github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib/storage
History
Nikolay 544fba6826
lib/storage: pre-create timeseries before indexDB rotation (#4652) 
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:20:21 -07:00
..
block.go lib/storage: validate timestamps in the block only if they use encoding, which needs validation 2022-10-21 00:52:32 +03:00
block_header.go lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block 2022-09-06 13:08:09 +03:00
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:47:59 +02:00
block_stream_merger.go lib/storage: skip blocks outside the configured retention during search 2022-10-24 02:52:44 +03:00
block_stream_reader.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_reader_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_writer.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_writer_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:10:32 -08:00
dedup.go app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:16:57 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:16:57 -08:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:33:58 -07:00
index_db.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:20:21 -07:00
index_db_test.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:20:21 -07:00
index_db_timing_test.go lib/storage: move series registration in caches from createAllIndexesForMetricName into a separate function - putSeriesToCache 2023-07-13 23:13:23 -07:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:10:32 -08:00
merge.go lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention 2022-12-03 22:14:12 -08:00
merge_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
merge_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:37 +03:00
metaindex_row_test.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:12:56 +03:00
metric_name.go app/vmselect/promql: add the ability to copy all the labels from one side of group_left()/group_right() operation 2023-07-17 19:07:39 -07:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:52:57 +03:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
part_header.go lib/storage: fix a bug, which prevents from reading pre-v1.90.0 parts 2023-04-14 22:33:08 -07:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:28:03 -08:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
part_search_timing_test.go lib/storage: skip missing tsids in the current block header by using binary search 2022-12-14 22:06:51 -08:00
partition.go docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix 2023-07-06 17:09:03 -07:00
partition_search.go lib/storage: fixes finalDedup for backfilled data (#3737) 2023-02-01 09:54:21 -08:00
partition_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
partition_test.go lib/storage: remove unused code (#4050) 2023-04-02 21:24:42 -07:00
raw_block.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
raw_row.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
search.go app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
search_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:02:40 -07:00
storage.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:20:21 -07:00
storage_test.go lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:20:21 -07:00
storage_timing_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:02:40 -07:00
table.go lib/storage: do not create flock.lock files at partition directories, since it is created at the Storage level 2023-06-19 22:48:37 -07:00
table_search.go lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg 2022-10-24 01:31:04 +03:00
table_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
table_search_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
table_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
table_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
tag_filters.go lib/storage: optimize matching speed for non-trivial regexp filters 2022-10-01 12:06:06 +03:00
tag_filters_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:12 +03:00
tag_filters_timing_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:12 +03:00
time.go app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
time_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tsid.go all: remove the remaining mentions of cluster version 2019-11-21 23:18:22 +02:00
tsid_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00