VictoriaMetrics/lib/mergeset
Aliaksandr Valialkin 73f0a805e2
lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval
Previously the interval between item addition and its conversion to searchable in-memory part
could vary significantly because of too coarse per-second precision. Switch from fasttime.UnixTimestamp()
to time.Now().UnixMilli() for millisecond precision. It is OK to use time.Now() for tracking
the time when buffered items must be converted to searchable in-memory parts, since time.Now()
calls aren't located in hot paths.

Increase the flush interval for converting buffered samples to searchable in-memory parts
from one second to two seconds. This should reduce the number of blocks, which are needed
to be processed during high-frequency alerting queries. This, in turn, should reduce CPU usage.

While at it, hardcode the maximum size of rawRows shard to 8Mb, since this size gives the optimal
data ingestion pefromance according to load tests. This reduces memory usage and CPU usage on systems
with big amounts of RAM under high data ingestion rate.
2024-02-22 20:21:14 +02:00
..
block_header.go lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() 2022-11-16 21:23:35 +02:00
block_stream_reader.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_reader_test.go lib/mergeset: use deterministic random generator in tests 2023-01-23 19:43:49 -08:00
block_stream_writer.go lib/mergeset: properly record the firstItem in metaindexRow at blockStreamWriter.WriteBlock 2024-02-12 18:06:50 +02:00
encoding.go lib/mergeset: do not store commonPrefix in blockHeader if the block contains only a single item 2024-02-08 13:47:24 +02:00
encoding_test.go lib/mergeset: use deterministic random generator in tests 2023-01-23 19:43:49 -08:00
encoding_timing_test.go lib/mergeset: fix data race in BenchmarkInmemoryBlockMarshal 2023-01-23 19:43:18 -08:00
filenames.go lib/mergeset: consistently use OS-independent separator in file paths 2023-03-25 13:39:41 -07:00
inmemory_part.go lib/mergeset: verify that the index block for in-memory part doesnt exceed the 3*maxIndexBlockSize 2024-02-08 13:50:14 +02:00
merge.go lib/mergeset: make sure that the first and the last items are in the original range after prepareBlock() 2024-01-23 12:58:32 +02:00
merge_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
metaindex_row.go lib/mergeset: prevent from possible too big indexBlockSize panic 2024-02-08 12:54:10 +02:00
part.go lib/mergeset: typo fix: indexdb/indexBlock -> indexdb/indexBlocks 2024-02-08 13:53:18 +02:00
part_header.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
part_search.go lib/mergeset: remove inmemoryBlock pooling, since it wasn't effecitve 2024-01-26 21:34:57 +01:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
table.go lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval 2024-02-22 20:21:14 +02:00
table_search.go optimized code (#2103) 2022-01-28 14:15:41 +02:00
table_search_test.go lib/fs: add MustReadDir() function 2023-04-14 22:10:46 -07:00
table_search_timing_test.go lib/fs: add MustReadDir() function 2023-04-14 22:10:46 -07:00
table_test.go lib/mergeset: do not panic on too long items passed to Table.AddItems() 2024-02-12 19:32:18 +02:00