github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib/storage
History
Aliaksandr Valialkin 0ebfb91aba
lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) 
This reverts the following commits:
- e0e16a2d36
- 2ce02a7fe6

The reason for revert: the updated logic breaks assumptions made
when fixing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698 .
For example, if a time series stop receiving new samples during the first
day after the indexdb rotation, there are chances that the time series
won't be registered in the new indexdb. This is OK until the next indexdb
rotation, since the time series is registered in the previous indexdb,
so it can be found during queries. But the time series will become invisible
for search after the next indexdb rotation, while its data is still there.

There is also incompletely solved issue with the increased CPU and disk IO resource
usage just after the indexdb rotation. There was an attempt to fix it, but it didn't fix
it in full, while introducing the issue mentioned above. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

TODO: to find out the solution, which simultaneously solves the following issues:
- increased memory usage for setups high churn rate and long retention (e.g. what the reverted commit does)
- increased CPU and disk IO usage during indexdb rotation ( https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 )
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698
2023-05-18 11:28:54 -07:00
..
block.go lib/storage: validate timestamps in the block only if they use encoding, which needs validation 2022-10-21 00:54:37 +03:00
block_header.go lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block 2022-09-06 13:07:49 +03:00
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:48:44 +02:00
block_stream_merger.go lib/storage: skip blocks outside the configured retention during search 2022-10-24 02:56:13 +03:00
block_stream_reader.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_reader_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_writer.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_writer_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
dedup.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:34:36 -07:00
index_db.go lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) 2023-05-18 11:28:54 -07:00
index_db_test.go lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) 2023-05-18 11:28:54 -07:00
index_db_timing_test.go lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) 2023-05-18 11:28:54 -07:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
merge.go lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention 2022-12-03 22:14:20 -08:00
merge_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
merge_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:04 +03:00
metaindex_row_test.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:13:15 +03:00
metric_name.go lib/storage: log original labels set when label value is truncated (#3952) 2023-03-14 16:11:02 -07:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_header.go lib/storage: fix a bug, which prevents from reading pre-v1.90.0 parts 2023-04-14 22:33:29 -07:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:31:39 -08:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_search_timing_test.go lib/storage: skip missing tsids in the current block header by using binary search 2022-12-14 22:07:55 -08:00
partition.go lib/storage: reduce the unimportant logging during Storage start / stop 2023-05-16 15:32:35 -07:00
partition_search.go lib/storage: fixes finalDedup for backfilled data (#3737) 2023-02-01 09:57:02 -08:00
partition_search_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
partition_test.go lib/storage: remove unused code (#4050) 2023-04-02 21:23:24 -07:00
raw_block.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
raw_row.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
search.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
search_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
storage.go lib/storage: revert the migration from global to per-day index for (MetricName -> TSID) 2023-05-18 11:28:54 -07:00
storage_test.go lib/storage: follow-up after a50d63c376 (#4289) 2023-05-16 10:13:20 -07:00
storage_timing_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
table.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
table_search.go lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg 2022-10-24 01:32:56 +03:00
table_search_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
table_search_timing_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
table_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
table_timing_test.go lib/storage: replace OpenStorage() with MustOpenStorage() 2023-04-14 23:04:42 -07:00
tag_filters.go lib/storage: optimize matching speed for non-trivial regexp filters 2022-10-01 12:07:18 +03:00
tag_filters_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:43 +03:00
tag_filters_timing_test.go lib/regexutil: add Simplify() function for simplifying the regular expression 2022-08-26 11:57:43 +03:00
time.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
time_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tsid.go lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
tsid_test.go all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00