Aliaksandr Valialkin 73f0a805e2 lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval ... Previously the interval between item addition and its conversion to searchable in-memory part could vary significantly because of too coarse per-second precision. Switch from fasttime.UnixTimestamp() to time.Now().UnixMilli() for millisecond precision. It is OK to use time.Now() for tracking the time when buffered items must be converted to searchable in-memory parts, since time.Now() calls aren't located in hot paths. Increase the flush interval for converting buffered samples to searchable in-memory parts from one second to two seconds. This should reduce the number of blocks, which are needed to be processed during high-frequency alerting queries. This, in turn, should reduce CPU usage. While at it, hardcode the maximum size of rawRows shard to 8Mb, since this size gives the optimal data ingestion pefromance according to load tests. This reduces memory usage and CPU usage on systems with big amounts of RAM under high data ingestion rate.		2024-02-22 20:21:14 +02:00
..
block.go	lib/storage: move fixupTimestamps() call to Block.Init()	2024-02-06 22:43:48 +02:00
block_header.go	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block	2022-09-06 13:08:09 +03:00
block_header_test.go	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:47:59 +02:00
block_stream_merger.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
block_stream_reader.go	lib/storage: avoid superflouos copy of block header data	2024-02-22 20:21:14 +02:00
block_stream_reader_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
block_stream_reader_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_stream_writer.go	lib/storage: do not append headerData to bsw.indexData if its size exceeds maxBlockSize	2024-02-12 18:04:37 +02:00
block_stream_writer_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_test.go	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
dedup.go	docs: mention staleNaN handling during deduplication	2024-01-11 11:53:58 +01:00
dedup_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:16:57 -08:00
dedup_timing_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:16:57 -08:00
filenames.go	lib/storage: consistently use OS-independent separator in file paths	2023-03-25 14:33:58 -07:00
index_db.go	lib/storage: compress metricIDs, which match the given filters, before storing them in tagFiltersToMetricIDsCache	2024-01-23 16:09:55 +02:00
index_db_test.go	lib/storage: compress metricIDs, which match the given filters, before storing them in tagFiltersToMetricIDsCache	2024-01-23 16:09:55 +02:00
index_db_timing_test.go	Add flagutil.Duration to avoid conversion bugs (#4835)	2023-09-01 09:27:51 +02:00
inmemory_part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
inmemory_part_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
inmemory_part_timing_test.go	lib/storage: use deterministic random generator in tests	2023-01-23 20:10:32 -08:00
merge.go	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:12 -08:00
merge_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
merge_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
metaindex_row.go	all: subsitute ioutil.ReadAll with io.ReadAll	2022-08-22 00:16:37 +03:00
metaindex_row_test.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:12:56 +03:00
metric_name.go	lib/mergeset: do not panic on too long items passed to Table.AddItems()	2024-02-12 19:32:18 +02:00
metric_name_test.go	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:52:57 +03:00
part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
part_header.go	lib/storage: fix typo	2023-11-21 11:20:43 +01:00
part_search.go	lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid	2022-12-19 10:28:03 -08:00
part_search_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
part_search_timing_test.go	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
partition.go	lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval	2024-02-22 20:21:14 +02:00
partition_search.go	lib/storage: fixes finalDedup for backfilled data (#3737)	2023-02-01 09:54:21 -08:00
partition_search_test.go	lib/storage: do not keep rawRows buffer across flush() calls	2024-02-22 17:22:26 +02:00
partition_test.go	lib/storage: stop exposing vm_merge_need_free_disk_space metric	2023-09-25 16:52:39 +02:00
raw_block.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
raw_row.go	lib/storage: move fixupTimestamps() call to Block.Init()	2024-02-06 22:43:48 +02:00
search.go	app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery()	2024-01-21 23:45:31 +02:00
search_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
storage.go	app/vmstorage: expose vm_snapshots metric, which shows the current number of snapshots	2024-02-22 18:32:57 +02:00
storage_test.go	lib/storage: keep (date, metricID) entries only for the last two dates	2024-01-29 18:43:59 +01:00
storage_timing_test.go	lib/storage: replace OpenStorage() with MustOpenStorage()	2023-04-14 23:02:40 -07:00
table.go	lib/storage: do not keep rawRows buffer across flush() calls	2024-02-22 17:22:26 +02:00
table_search.go	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg	2022-10-24 01:31:04 +03:00
table_search_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
table_search_timing_test.go	lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval	2024-02-22 20:21:14 +02:00
table_test.go	flagutil: Make .Msecs private (#4906)	2023-09-03 10:33:37 +02:00
table_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
tag_filters.go	lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args	2023-11-11 12:30:08 +01:00
tag_filters_test.go	all: upgrade Go builder from Go1.21.7 to Go1.22.0	2024-02-12 21:59:51 +02:00
tag_filters_timing_test.go	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
time.go	app,lib: fix typos in comments (#3804)	2023-02-13 13:27:13 +01:00
time_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tsid.go	all: remove the remaining mentions of cluster version	2019-11-21 23:18:22 +02:00
tsid_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00