122 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	0074539441	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-26 01:11:17 +03:00
Aliaksandr Valialkin	c4265322f4	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin	8e3198ba29	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-24 18:04:48 +03:00
Aliaksandr Valialkin	8e998aa1a1	lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series) ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289	2022-10-24 16:40:20 +03:00
Aliaksandr Valialkin	6c7c0790a0	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-24 16:32:43 +03:00
Aliaksandr Valialkin	e2f0b76ebf	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg ... This makes code easier to read. This is a follow-up after d2d30581a0	2022-10-24 01:31:04 +03:00
Aliaksandr Valialkin	89a1108b1a	lib/storage: small code cleanups	2022-10-24 01:17:47 +03:00
Aliaksandr Valialkin	d2d30581a0	lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback ... This improves code readability a bit.	2022-10-23 16:10:04 +03:00
Aliaksandr Valialkin	07fe2c5361	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-21 15:03:12 +03:00
Aliaksandr Valialkin	4128ad71e2	lib/storage: move common code to newRawRowsBlock() function	2022-10-21 14:46:55 +03:00
Aliaksandr Valialkin	b5674164c6	lib/storage: simplify code a bit after 3f5959c053	2022-10-21 14:39:27 +03:00
Aliaksandr Valialkin	fd7c86ae25	lib/{mergeset,storage}: simplify the code a bit after ae55ad8749	2022-10-21 14:33:03 +03:00
Aliaksandr Valialkin	22e87b0088	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-21 01:11:40 +03:00
Aliaksandr Valialkin	3f5959c053	lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate ... This should improve background merge rate under high load a bit	2022-10-20 23:28:24 +03:00
Aliaksandr Valialkin	150e99d403	lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms ... The panic has been introduced in 68f3a02589 While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs This should improve scalability on systems with many CPU cores	2022-10-20 16:25:43 +03:00
Aliaksandr Valialkin	f105e2e8c3	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-10-18 20:55:52 +03:00
Aliaksandr Valialkin	fb50730ba7	lib/storage: double the number of rawRows shards on multi-core systems ... This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage	2022-10-17 18:19:51 +03:00
Aliaksandr Valialkin	ae55ad8749	lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list	2022-10-17 18:01:26 +03:00
Aliaksandr Valialkin	1c9f5b3580	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-09-14 17:54:06 +03:00
Aliaksandr Valialkin	68e32b0764	lib/storage: atomically remove parts inside partitions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	340ada871d	lib/storage: atomically remove partitions, which went outside the configured retention ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin	628e87e727	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-08-22 00:40:04 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:52:35 +03:00
Roman Khavronenko	d59d829cdb	lib/storage: bump max merge concurrency for small parts to 15 (#2997) ... * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:32:08 +03:00
Aliaksandr Valialkin	d2e293b5c9	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-08-15 01:43:51 +03:00
Roman Khavronenko	a0e7432e42	lib/storage: prevent excessive loops when storage is in RO (#2962) ... * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:00 +03:00
Aliaksandr Valialkin	cd91c29243	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-06-28 13:26:58 +03:00
Aliaksandr Valialkin	134751e43e	all: locate throttled loggers via logger.WithThrottler() only once and then use them ... This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 13:45:50 +03:00
Aliaksandr Valialkin	7fc62feddc	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-06-09 13:33:07 +03:00
Roman Khavronenko	1ee1e986da	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673) ... Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 14:55:09 +03:00
Aliaksandr Valialkin	0ea0168d98	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-06-04 01:13:48 +03:00
Aliaksandr Valialkin	ea06d2fd3c	lib/storage: stop background merge when storage enters read-only mode ... This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:36:45 +03:00
Aliaksandr Valialkin	ec01a188fd	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-04-07 15:25:52 +03:00
Aliaksandr Valialkin	57143e9435	lib/storage: increase the number of rawRowsShard shards on systems with more than 4 CPU cores ... This should improve data ingestion scalability on systems with many CPU cores	2022-04-06 19:49:20 +03:00
Aliaksandr Valialkin	50cf74ce4b	lib/storage: reuse sync.WaitGroup objects ... This reduces GC load by up to 10% according to memory profiling	2022-04-06 13:34:04 +03:00
Aliaksandr Valialkin	90a1502335	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-03-03 19:31:14 +02:00
Aliaksandr Valialkin	59877d9f32	lib/{mergeset,storage}: tune compression levels for small blocks ... This should reduce CPU usage spent on compression	2022-02-25 15:33:40 +02:00
Aliaksandr Valialkin	e8ff658b2e	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-01-23 13:24:13 +02:00
Aliaksandr Valialkin	145337792d	lib/{mergeset,storage}: properly limit cache sizes for indexdb ... Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`, since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded, which could result to out of memory errors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:37:17 +02:00
Aliaksandr Valialkin	3f5b1084eb	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2022-01-07 12:36:24 +02:00
Aliaksandr Valialkin	ce333f28d8	all: use logger.WithThrottler() where appropriate	2021-12-21 17:03:25 +02:00
Aliaksandr Valialkin	08219faf8d	Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files	2021-12-17 20:21:56 +02:00
Aliaksandr Valialkin	8a7f08ded3	lib/storage: properly update per-part min_dedup_interval file contents after merge ... Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value This is a follow-up for 4ff647137a	2021-12-17 20:13:24 +02:00
Aliaksandr Valialkin	288620ca40	lib/storage: initial support for multi-level downsampling ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36 Based on https://github.com/valyala/VictoriaMetrics/pull/203	2021-12-15 16:42:44 +02:00
Aliaksandr Valialkin	4ff647137a	lib/storage: deduplicate samples more thoroughly ... Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in higher disk space usage.	2021-12-15 15:59:58 +02:00
Aliaksandr Valialkin	7275ebf91a	app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches ... The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query: vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9	2021-12-02 10:30:43 +02:00
Aliaksandr Valialkin	2fb5a6ca78	lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges	2021-12-01 11:02:36 +02:00
Aliaksandr Valialkin	d666755159	lib/storage: take into account -storage.minFreeDiskSpaceBytes when performing big merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-11-30 12:56:35 +02:00
Aliaksandr Valialkin	ffc0ab1774	lib/{mergeset,storage}: improve the detection of the needed free space for background merge ... This should prevent from possible out of disk space crashes during big merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560	2021-08-25 09:35:44 +03:00
Aliaksandr Valialkin	c2deee9911	lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-27 12:04:50 +03:00