Aliaksandr Valialkin
e56d5e1918
lib/storage: follow-up after 7c0ae3a86a
...
- Update docs at https://docs.victoriametrics.com/#deduplication
- Optimize the deduplication loop a bit
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
2022-12-08 18:18:49 -08:00
Roman Khavronenko
909cd04c55
lib/storage: keep sample with the biggest value on timestamp conflict ( #3421 )
...
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.
```
benchstat old.txt new.txt
name old time/op new time/op delta
DeduplicateSamples/minScrapeInterval=1s-10 817ns ± 2% 832ns ± 3% ~ (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10 1.56µs ± 1% 2.12µs ± 0% +35.19% (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10 1.32µs ± 3% 1.65µs ± 2% +25.57% (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10 1.13µs ± 2% 1.50µs ± 1% +32.85% (p=0.000 n=10+10)
name old speed new speed delta
DeduplicateSamples/minScrapeInterval=1s-10 10.0GB/s ± 2% 9.9GB/s ± 3% ~ (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10 5.24GB/s ± 1% 3.87GB/s ± 0% -26.03% (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10 6.22GB/s ± 3% 4.96GB/s ± 2% -20.37% (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10 7.28GB/s ± 2% 5.48GB/s ± 1% -24.74% (p=0.000 n=10+10)
```
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:18:36 -08:00
Aliaksandr Valialkin
0a9992a9c6
lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown
2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin
7d5c64eb7a
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
...
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.
For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.
The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin
9ac1174493
lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function
2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin
a13d21513e
lib/mergeset: panic when too long item is passed to Table.AddItems()
2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin
dccd70ce10
lib/storage: remove duplicate logging for filepath on errors
2022-12-03 23:15:28 -08:00
Aliaksandr Valialkin
813e8402f6
lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args
2022-12-03 23:10:26 -08:00
Aliaksandr Valialkin
bb93494eac
lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers
...
This simplifies the code
2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin
106332cd9f
lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation
2022-12-03 23:03:30 -08:00
Aliaksandr Valialkin
ea55c16422
lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart
...
This allows packing in-memory blocks with different compression levels
depending on its contents. This may save memory usage.
2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin
7ffa66d249
lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part
...
This results in more correct reporting of memory usage for in-memory parts
2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin
10a17bfa16
lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}
2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin
233301a549
lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention
2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin
fd9d0a550b
lib/storage: remove logging redundant path values in a single error message
2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin
8e9822bc7f
lib/storage: speed up search for data block for the given tsids
...
Use binary search instead of linear scan for looking up the needed
data block inside index block.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425
2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin
d8303845ef
lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC
2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin
d8d4d21d7a
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts
...
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
2022-12-02 17:07:52 -08:00
匠心零度
d4808d5b84
lib/storage: remove extra error check ( #3396 )
2022-11-28 17:07:11 +01:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants ( #3348 )
...
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* app/vmselect: fix error message
* {app/vmstorage,app/vmselect}: fix error messages
* app/vmselect: change log level for error handling
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 10:32:45 -08:00
Aliaksandr Valialkin
d3035b1ca1
lib/storage: follow-up for 790768f20b
...
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin
be78950011
lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8
2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin
99e6a937a5
lib/storage: remove unused isFull field from hourMetricIDs struct
2022-11-07 13:15:59 +02:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists
2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search
...
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
2fc82b846e
lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg
...
This makes code easier to read.
This is a follow-up after d2d30581a0
2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin
d51f9b9284
lib/storage: small code cleanups
2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin
5ace1587e6
lib/storage: re-use newTestStorage() instead of manually initializing Storage mock
...
This is a follow-up for d2d30581a0
2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin
57ea7a3ee8
lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback
...
This improves code readability a bit.
2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin
63419d8e7c
lib/storage: small refactoring: move retentionDeadline to blockStreamMerger
...
This allows defining per-block retention in the future by updating the getRetentionDeadline function
2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin
31071347ca
lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop
2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin
5d0a91afd5
lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms
2022-10-23 12:48:43 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
...
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.
The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.
This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.
This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
fe5611d6e1
lib/storage: free up memory occupied by Storage.pendingHourEntries after a temporary spike in its memory usage
...
This reduces vmstorage memory usage by up to 20% in production workload
2022-10-21 14:59:14 +03:00
Aliaksandr Valialkin
32b6ce691b
lib/storage: move common code to newRawRowsBlock() function
2022-10-21 14:46:06 +03:00
Aliaksandr Valialkin
2f8861ed9c
lib/storage: simplify code a bit after 3f5959c053
2022-10-21 14:39:44 +03:00
Aliaksandr Valialkin
1fb2be0cae
lib/{mergeset,storage}: simplify the code a bit after ae55ad8749
2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
...
This reduces CPU usage when there is no sense in validating timestamps.
This is a follow-up for 5fa9525498https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
edf3b7be47
lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate
...
This should improve background merge rate under high load a bit
2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin
6855de311c
lib/{mergeset,storage}: avoid unaligned 64-bit atomic operation panic on 32-bit platforms
...
The panic has been introduced in 68f3a02589
2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin
6f69a88a5a
lib/storage: double the number of rawRows shards on multi-core systems
...
This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage
2022-10-17 18:19:28 +03:00
Aliaksandr Valialkin
68f3a02589
lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list
2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin
b96fe2e265
lib/storage: optimize matching speed for non-trivial regexp filters
...
Wrap re.Match into bytesutil.FastStringMatcher.
This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
fe52378f45
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic
2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin
6c9729d694
lib/storage: atomically remove parts inside partitions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 15:28:41 +03:00
Aliaksandr Valialkin
daa42e4f79
lib/storage: atomically remove partitions, which went outside the configured retention
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin
0a342f04b2
lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there
...
Previously the cache directory was removed. This could result in error when the cache directory
is mounted to a separate filesystem.
2022-09-13 13:32:05 +03:00
Aliaksandr Valialkin
ff7188b6a5
lib/storage: atomically remove snapshot directories
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 13:25:48 +03:00
Aliaksandr Valialkin
051e722112
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:07:49 +03:00
