Aliaksandr Valialkin 9b4294e53e lib/storage: reduce the contention on dateMetricIDCache mutex when new time series are registered at high rate ... The dateMetricIDCache puts recently registered (date, metricID) entries into mutable cache protected by the mutex. The dateMetricIDCache.Has() checks for the entry in the mutable cache when it isn't found in the immutable cache. Access to the mutable cache is protected by the mutex. This means this access is slow on systems with many CPU cores. The mutabe cache was merged into immutable cache every 10 seconds in order to avoid slow access to mutable cache. This means that ingestion of new time series to VictoriaMetrics could result in significant slowdown for up to 10 seconds because of bottleneck at the mutex. Fix this by merging the mutable cache into immutable cache after len(cacheItems) / 2 cache hits under the mutex, e.g. when the entry is found in the mutable cache. This should automatically adjust intervals between merges depending on the addition rate for new time series (aka churn rate): - The interval will be much smaller than 10 seconds under high churn rate. This should reduce the mutex contention for mutable cache. - The interval will be bigger than 10 seconds under low churn rate. This should reduce the uneeded work on merging of mutable cache into immutable cache.		2024-01-22 18:40:32 +02:00
..
block.go	lib/storage: validate timestamps in the block only if they use encoding, which needs validation	2022-10-21 00:52:32 +03:00
block_header.go	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block	2022-09-06 13:08:09 +03:00
block_header_test.go	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:47:59 +02:00
block_stream_merger.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
block_stream_reader.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_stream_reader_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
block_stream_reader_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_stream_writer.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_stream_writer_timing_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
block_test.go	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
dedup.go	docs: mention staleNaN handling during deduplication	2024-01-11 11:53:58 +01:00
dedup_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:16:57 -08:00
dedup_timing_test.go	lib/storage: follow-up after 7c0ae3a86a	2022-12-08 18:16:57 -08:00
filenames.go	lib/storage: consistently use OS-independent separator in file paths	2023-03-25 14:33:58 -07:00
index_db.go	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
index_db_test.go	lib/storage: log fatal error inside searchMetricName() instead of propagating it to the caller	2023-09-22 11:41:06 +02:00
index_db_timing_test.go	Add flagutil.Duration to avoid conversion bugs (#4835)	2023-09-01 09:27:51 +02:00
inmemory_part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
inmemory_part_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
inmemory_part_timing_test.go	lib/storage: use deterministic random generator in tests	2023-01-23 20:10:32 -08:00
merge.go	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:12 -08:00
merge_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
merge_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
metaindex_row.go	all: subsitute ioutil.ReadAll with io.ReadAll	2022-08-22 00:16:37 +03:00
metaindex_row_test.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:12:56 +03:00
metric_name.go	lib/prompb: change type of Label.Name and Label.Value from []byte to string	2024-01-14 22:33:21 +02:00
metric_name_test.go	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:52:57 +03:00
part.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
part_header.go	lib/storage: fix typo	2023-11-21 11:20:43 +01:00
part_search.go	lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid	2022-12-19 10:28:03 -08:00
part_search_test.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
part_search_timing_test.go	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
partition.go	lib/storage/partition.go: remove misleading comment, which falsely states that inmemoryParts isn't visible to search	2024-01-21 04:49:35 +02:00
partition_search.go	lib/storage: fixes finalDedup for backfilled data (#3737)	2023-02-01 09:54:21 -08:00
partition_search_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
partition_test.go	lib/storage: stop exposing vm_merge_need_free_disk_space metric	2023-09-25 16:52:39 +02:00
raw_block.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
raw_row.go	lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart	2023-04-14 15:46:12 -07:00
search.go	app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery()	2024-01-21 23:45:31 +02:00
search_test.go	Makefile: update golangci-lint from v1.51.2 to v1.54.2	2023-09-01 10:16:42 +02:00
storage.go	lib/storage: reduce the contention on dateMetricIDCache mutex when new time series are registered at high rate	2024-01-22 18:40:32 +02:00
storage_test.go	lib/storage: follow-up after 188cfe3a85	2023-10-17 15:45:14 +02:00
storage_timing_test.go	lib/storage: replace OpenStorage() with MustOpenStorage()	2023-04-14 23:02:40 -07:00
table.go	lib/storage: do not create flock.lock files at partition directories, since it is created at the Storage level	2023-06-19 22:48:37 -07:00
table_search.go	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg	2022-10-24 01:31:04 +03:00
table_search_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
table_search_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
table_test.go	flagutil: Make .Msecs private (#4906)	2023-09-03 10:33:37 +02:00
table_timing_test.go	lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage()	2023-07-13 17:13:24 -07:00
tag_filters.go	lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args	2023-11-11 12:30:08 +01:00
tag_filters_test.go	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
tag_filters_timing_test.go	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
time.go	app,lib: fix typos in comments (#3804)	2023-02-13 13:27:13 +01:00
time_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tsid.go	all: remove the remaining mentions of cluster version	2019-11-21 23:18:22 +02:00
tsid_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00