Aliaksandr Valialkin 0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores ... VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit 56b6b893ce . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244		2021-06-17 13:51:42 +03:00
..
block.go	lib/storage: drop more samples outside the given retention during background merge	2020-10-31 20:44:47 +02:00
block_header.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
block_header_test.go	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:48:44 +02:00
block_stream_merger.go	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected	2020-07-30 20:02:22 +03:00
block_stream_reader.go	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps	2020-09-09 23:59:21 +03:00
block_stream_reader_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
block_stream_reader_timing_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
block_stream_writer.go	spelling fix: adjacent (#1115)	2021-03-09 09:19:16 +02:00
block_stream_writer_timing_test.go	lib/storage: skip deduplication when creating inmemory data blocks	2021-02-09 02:26:16 +02:00
block_test.go	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:37 +03:00
dedup.go	lib/storage/dedup.go: go fmt	2020-04-26 14:37:36 +03:00
dedup_test.go	all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9	2020-11-24 12:42:04 +02:00
dedup_timing_test.go	lib/storage: improve deduplication algorithm	2020-04-26 13:10:18 +03:00
index_db.go	lib/storage: move deletedMetricIDs set from indexDB to Storage	2021-06-15 15:07:54 +03:00
index_db_test.go	lib/storage: move deletedMetricIDs set from indexDB to Storage	2021-06-15 15:07:54 +03:00
index_db_timing_test.go	lib/storage: reset cache on disk during series deletion and during indexdb rotation	2021-06-11 12:54:36 +03:00
inmemory_part.go	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
inmemory_part_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
inmemory_part_timing_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
merge.go	lib/storage: do not update b.nextIdx if no samples are removed because of retention	2021-03-29 12:13:38 +03:00
merge_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
merge_timing_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
metaindex_row.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
metaindex_row_test.go	lib/storage: correctly use maxBlockSize in various checks	2020-09-24 18:13:15 +03:00
metric_name.go	lib/storage: remove possible data race when logging dropped labels	2021-05-20 11:54:06 +03:00
metric_name_test.go	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
part.go	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:43:37 +02:00
part_header.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
part_header_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
part_search.go	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:43:37 +02:00
part_search_test.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
partition.go	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores	2021-06-17 13:51:42 +03:00
partition_search.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
partition_search_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
partition_test.go	lib/{mergeset,storage}: allow merging smaller number of small parts	2021-02-21 21:43:37 +02:00
raw_block.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
raw_row.go	lib/storage: skip deduplication when creating inmemory data blocks	2021-02-09 02:26:16 +02:00
search.go	lib/storage: code clarification: remove caching the found metricName in searchMetricName	2021-04-13 10:20:35 +03:00
search_test.go	app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags	2021-05-20 15:31:57 +03:00
storage.go	lib/storage: move deletedMetricIDs set from indexDB to Storage	2021-06-15 15:07:54 +03:00
storage_test.go	lib/storage: reduce memory allocations when syncing dateMetricIDCache	2021-06-03 16:20:02 +03:00
storage_timing_test.go	app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags	2021-05-20 15:31:57 +03:00
table.go	lib/storage: sort tag filters by the number of loops they need for the execution	2021-02-18 12:52:29 +02:00
table_search.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
table_search_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
table_search_timing_test.go	lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard	2021-06-11 10:52:31 +03:00
table_test.go	app/vmstorage: support for -retentionPeriod smaller than one month	2020-10-20 14:42:46 +03:00
table_timing_test.go	all: properly handle CPU limits set on the host system/container	2020-12-08 21:07:03 +02:00
tag_filters.go	lib/storage: further tuning for time series search	2021-03-16 18:47:29 +02:00
tag_filters_test.go	lib/storage: use composite index for a query with a name filter and negative filters	2021-02-18 18:57:45 +02:00
tag_filters_timing_test.go	lib/storage: small code adjustements after d2960a20e0	2020-10-17 01:17:12 +03:00
time.go	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
time_test.go	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tsid.go	lib/storage: optimize TSID comparison	2019-09-26 14:20:02 +03:00
tsid_test.go	all: open-sourcing cluster version	2019-05-23 00:25:38 +03:00