Roman Khavronenko 5f33445f66 lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>		2022-06-07 15:02:55 +03:00
..
block.go	lib/storage: deduplicate samples more thoroughly	2021-12-15 16:00:30 +02:00
block_header.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
block_header_test.go	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:48:44 +02:00
block_stream_merger.go	lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used.	2022-01-20 20:37:24 +02:00
block_stream_reader.go	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations	2022-02-01 11:20:20 +02:00
block_stream_reader_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
block_stream_reader_timing_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
block_stream_writer.go	lib/storage: deduplicate samples more thoroughly	2021-12-15 16:00:30 +02:00
block_stream_writer_timing_test.go	lib/storage: deduplicate samples more thoroughly	2021-12-15 16:00:30 +02:00
block_test.go	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:37 +03:00
dedup.go	lib/storage: leave the last sample per each discrete interval during the deduplicaton	2022-05-02 21:59:31 +03:00
dedup_test.go	lib/storage: leave the last sample per each discrete interval during the deduplicaton	2022-05-02 21:59:31 +03:00
dedup_timing_test.go	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge()	2021-12-14 20:52:29 +02:00
index_db.go	lib/storage: make indexdb/tagFilters cache size configurable (#2667)	2022-06-01 14:57:39 +03:00
index_db_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
index_db_timing_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
inmemory_part.go	lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects	2021-07-06 16:33:25 +03:00
inmemory_part_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
inmemory_part_timing_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
merge.go	lib/storage: deduplicate samples more thoroughly	2021-12-15 16:00:30 +02:00
merge_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
merge_timing_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
metaindex_row.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
metaindex_row_test.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
metric_name.go	lib/storage: document why job-like and instance-like labels must be stored at mn.Tags[0] and mn.Tags[1]	2022-02-25 13:21:53 +02:00
metric_name_test.go	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
part.go	lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks	2022-01-21 12:46:05 +02:00
part_header.go	lib/promscrape: support prometheus-like duration in scrape configs (#2169)	2022-02-11 16:17:51 +02:00
part_header_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
part_search.go	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations	2022-02-01 11:20:20 +02:00
part_search_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
partition.go	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673)	2022-06-07 15:02:55 +03:00
partition_search.go	optimized code (#2103)	2022-01-28 12:10:47 +02:00
partition_search_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
partition_test.go	lib/{mergeset,storage}: improve the detection of the needed free space for background merge	2021-08-25 10:01:09 +03:00
raw_block.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
raw_row.go	lib/storage: deduplicate samples more thoroughly	2021-12-15 16:00:30 +02:00
search.go	all: add initial support for query tracing	2022-06-01 02:31:44 +03:00
search_test.go	all: add initial support for query tracing	2022-06-01 02:31:44 +03:00
storage.go	lib/storage: make indexdb/tagFilters cache size configurable (#2667)	2022-06-01 14:57:39 +03:00
storage_test.go	all: add initial support for query tracing	2022-06-01 02:31:44 +03:00
storage_timing_test.go	app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags	2021-05-20 15:31:57 +03:00
table.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
table_search.go	optimized code (#2103)	2022-01-28 12:10:47 +02:00
table_search_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
table_search_timing_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
table_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
table_timing_test.go	lib/storage: stop background merge when storage enters read-only mode	2022-06-01 14:22:12 +03:00
tag_filters.go	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293)	2022-03-26 12:57:27 +02:00
tag_filters_test.go	lib/storage: properly handle series selector matching multiple metric names plus a negative filter	2022-02-24 12:11:53 +02:00
tag_filters_timing_test.go	lib/storage: small code adjustements after d2960a20e0	2020-10-17 01:17:12 +03:00
time.go	all: add initial support for query tracing	2022-06-01 02:31:44 +03:00
time_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tsid.go	lib/storage: optimize TSID comparison	2019-09-26 14:20:02 +03:00
tsid_test.go	all: open-sourcing cluster version	2019-05-23 00:25:38 +03:00