github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib/storage
History
Aliaksandr Valialkin aa9b56a046 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
2021-06-17 13:52:08 +03:00
..
block.go lib/storage: drop more samples outside the given retention during background merge 2020-11-01 23:35:06 +02:00
block_header.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:12:56 +03:00
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:47:59 +02:00
block_stream_merger.go lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 2020-07-30 19:57:27 +03:00
block_stream_reader.go lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 2020-09-09 23:59:32 +03:00
block_stream_reader_test.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:05:11 +03:00
block_stream_reader_timing_test.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:05:11 +03:00
block_stream_writer.go spelling fix: adjacent (#1115) 2021-03-09 09:18:19 +02:00
block_stream_writer_timing_test.go lib/storage: skip deduplication when creating inmemory data blocks 2021-02-09 02:25:32 +02:00
block_test.go lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm 2020-09-29 13:10:22 +03:00
dedup.go happy fmt 2020-04-26 14:16:32 +03:00
dedup_test.go all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9 2020-11-24 12:42:22 +02:00
dedup_timing_test.go lib/storage: improve deduplication algorithm 2020-04-26 13:10:02 +03:00
index_db.go lib/storage: move deletedMetricIDs set from indexDB to Storage 2021-06-15 15:04:30 +03:00
index_db_test.go lib/storage: move deletedMetricIDs set from indexDB to Storage 2021-06-15 15:04:30 +03:00
index_db_timing_test.go lib/storage: reset cache on disk during series deletion and during indexdb rotation 2021-06-11 12:42:28 +03:00
inmemory_part.go lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:02:07 +03:00
inmemory_part_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
inmemory_part_timing_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
merge.go lib/storage: do not update b.nextIdx if no samples are removed because of retention 2021-03-29 12:00:21 +03:00
merge_test.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
merge_timing_test.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
metaindex_row.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:12:56 +03:00
metaindex_row_test.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:12:56 +03:00
metric_name.go lib/storage: remove possible data race when logging dropped labels 2021-05-20 02:47:22 +03:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:52:57 +03:00
part.go lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:18:59 +02:00
part_header.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:05:11 +03:00
part_header_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
part_search.go lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:18:59 +02:00
part_search_test.go Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" 2020-09-24 22:44:23 +03:00
partition.go lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 2021-06-17 13:52:08 +03:00
partition_search.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:05:11 +03:00
partition_search_test.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
partition_test.go lib/{mergeset,storage}: allow merging smaller number of small parts 2021-02-21 21:28:36 +02:00
raw_block.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
raw_row.go lib/storage: skip deduplication when creating inmemory data blocks 2021-02-09 02:25:32 +02:00
search.go lib/storage: code clarification: remove caching the found metricName in searchMetricName 2021-04-13 10:22:21 +03:00
search_test.go app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 14:15:19 +03:00
storage.go lib/storage: move deletedMetricIDs set from indexDB to Storage 2021-06-15 15:04:30 +03:00
storage_test.go lib/storage: reduce memory allocations when syncing dateMetricIDCache 2021-06-03 16:20:42 +03:00
storage_timing_test.go app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 14:15:19 +03:00
table.go lib/storage: sort tag filters by the number of loops they need for the execution 2021-02-18 12:47:38 +02:00
table_search.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
table_search_test.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
table_search_timing_test.go lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard 2021-06-11 10:57:23 +03:00
table_test.go app/vmstorage: support for -retentionPeriod smaller than one month 2020-10-20 14:31:44 +03:00
table_timing_test.go all: properly handle CPU limits set on the host system/container 2020-12-08 21:07:29 +02:00
tag_filters.go lib/storage: further tuning for time series search 2021-03-16 18:46:22 +02:00
tag_filters_test.go lib/storage: use composite index for a query with a name filter and negative filters 2021-02-18 18:57:23 +02:00
tag_filters_timing_test.go lib/storage: small code adjustements after d2960a20e0 2020-10-17 01:16:54 +03:00
time.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:05:11 +03:00
time_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tsid.go all: remove the remaining mentions of cluster version 2019-11-21 23:18:22 +02:00
tsid_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00