Aliaksandr Valialkin
7ffa66d249
lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part
...
This results in more correct reporting of memory usage for in-memory parts
2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin
886ce94739
lib/mergeset: reduce the time needed for the slowest tests
2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin
10a17bfa16
lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}
2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin
0b8e7deabd
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
...
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin
d8d4d21d7a
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts
...
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin
b0b8f05fa4
lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS()
...
The issue has been introduced in 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin
06300fe9b8
lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf
...
Previously indexBlock.bhs pointed to the buffer, which could be changed over time.
This could result in incorrect time series search over time.
This is a follow-up for 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin
454060fd78
lib/mergeset: remove string allocation and copying when unmarshaling blockHeader
...
This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists
2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
1fb2be0cae
lib/{mergeset,storage}: simplify the code a bit after ae55ad8749
2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin
6855de311c
lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms
...
The panic has been introduced in 68f3a02589
2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin
68f3a02589
lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list
2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin
7a6e5f9224
lib/mergeset: mention in the error message the path to the part, which triggered the error
...
This should improve debuggability
2022-10-12 09:54:42 +03:00
Roman Khavronenko
fe71c73fe1
lib/mergeset: follow-up after a0e7432e42 ( #3145 )
...
* lib/mergeset: follow-up after a0e7432e42
2022-09-26 16:43:17 +03:00
Aliaksandr Valialkin
5b488a339d
lib/mergeset: atomically remove part dirs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll
...
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.
This is a follow-up for 02ca2342ab
2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
db049fed84
lib/mergeset: cleanup after de6dd1cd5a
...
Remove unused getInmemoryPart and putInmemoryPart functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin
962ed46583
lib/mergeset: optimize mergeInmemoryBlocks() function
...
Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs.
Just merge inmemoryBlock structs directly.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin
3bbe9054d3
lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge
...
Just read the current item directly from blockStreamReader.Block.Items
with the helper method - blockStreamReader.CurrItem()
2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin
547cb1edce
benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin
5f2b5bd173
lib/mergeset: add tests and benchmarks for commonPrefixLen function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 21:25:23 +03:00
zhenyuxie
14c6212a61
fix inmemoryBlock's Less method ( #2881 )
2022-07-18 12:00:45 +03:00
Dmytro Kozlov
5256af2291
lib/mergeset: fix linter error ( #2864 )
2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin
7cbcbea49d
lib/mergeset: optimize merge speed a bit
...
Use heap.Fix instead of heap.Pop + heap.Push when merging blocks
2022-07-12 12:52:36 +03:00
Loki's Wager
ca4730c00f
BugFix part_header.go ( #2763 )
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757
Co-authored-by: haotingyi <haotingyi@corp.netease.com>
2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin
fedfc9e686
lib/storage: stop background merge when storage enters read-only mode
...
This should prevent from `no space left on device` errors when VictoriaMetrics
under-estimates the additional disk space needed for background merge.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603
2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin
ed1b394a1a
app/vmstorage: expose vm_indexdb_items_added_total and vm_indexdb_items_added_size_bytes_total counters at /metrics page
...
These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index).
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471
2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin
edb139cfe4
lib/memory: export process_memory_limit_bytes metric, which shows the amounts of memory the current process has access to
...
This metric is equivalent to `vm_available_memory_bytes`, but it has better name,
since the metric is related to a process, not VictoriaMetrics itself.
Leave `vm_available_memory_bytes` for backwards compatibility.
2022-04-07 15:24:08 +03:00
Aliaksandr Valialkin
8ef9348801
lib/mergeset: use more rawItemsShard shards on multi-CPU systems
...
This should improve the scalability for registering of new time series on multi-CPU system
2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin
db00ddd23e
lib/mergeset: skip common prefixes when comparing inmemoryBlock items
...
This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData
if they have common prefix.
While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items.
This should improve performance for inmemoryBlock.MarshalSortedData during background merge.
2022-04-06 18:55:25 +03:00
Aliaksandr Valialkin
3f999b11f7
lib/mergeset: remove aux buffers from inmemoryPart
...
This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
ecf68da79e
lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock
...
This should improve performance when registering new time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
ecf4f7bf21
lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars
2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
f4e466955d
lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool
...
The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart,
so sync.Pool usage should reduce overall memory usage and improve performance
because of better locality of reference when marshaling inmemoryBlock to inmemoryPart.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin
b47f18f555
lib/{mergeset,storage}: tune compression levels for small blocks
...
This should reduce CPU usage spent on compression
2022-02-25 15:34:13 +02:00
Aliaksandr Valialkin
1967b9c211
lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort()
...
There is no need to sort the underlying data according to sorted items there.
This should reduce cpu usage when registering new time series in `indexdb`.
Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245
2022-02-24 11:19:44 +02:00
Aliaksandr Valialkin
986c12487a
lib/mergeset: typo fix after b6ed9afd6d
2022-02-21 19:00:04 +02:00
Roman Khavronenko
bd7837d524
lib: allow to configure cache size by type ( #2206 )
...
* lib: allow to configure cache size by type
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Apply suggestions from code review
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-02-21 13:55:51 +02:00
Aliaksandr Valialkin
895c9f4f11
lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin
a0a56d6c1c
lib/mergeset: tune sizes for indexdb/dataBlocks and indexdb/indexBlocks according to production workload
...
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742
2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin
d8d59ff760
lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
02b2bfcff3
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations
...
Follow-up for f4989edd96https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-02-01 11:20:20 +02:00
匠心零度
a1083d0531
optimized code ( #2103 )
...
* optimized code ,because only the first error,so no need var errors []error
* optimized code ,because only the first error,so no need var errors []error
Co-authored-by: lirenzuo <lirenzuo@shein.com>
2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin
6232eaa938
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions
...
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().
Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.
Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin
7ec0705b98
lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock
...
This should reduce the memory required for indexdb/dataBlocks cache.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin
9edf407144
lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin
6ae584b9b3
lib/{mergeset,storage}: properly limit cache sizes for indexdb
...
Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`,
since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded,
which could result to out of memory errors.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin
ab4be24397
app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches
...
The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query:
vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9
2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin
b885bd9b7d
lib/{mergeset,storage}: improve the detection of the needed free space for background merge
...
This should prevent from possible out of disk space crashes during big merges.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560
2021-08-25 10:01:09 +03:00
