VictoriaMetrics/lib/storage
rtm0 4c31a6a1fc
lib/storage: properly handle maxMetrics limit at metricID search
`TL;DR` This PR improves the metric IDs search in IndexDB:

- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search

A unit test has been added to ensure that this refactoring does not
break anything.

---

Function calls before the fix:

```
idb.searchMetricIDs
    |__ is.searchMetricIDs
        |__ is.searchMetricIDsInternal
            |__ is.updateMetricIDsForTagFilters
                |__ is.tryUpdatingMetricIDsForDateRange
                |                       |
                |__ is.getMetricIDsForDateAndFilters
```

- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.

Problems with this solution:

1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.

Solution:

1.  Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in  `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-08-27 23:08:17 +02:00
..
block.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:48:44 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
block_stream_reader.go lib/storage: avoid superflouos copy of block header data 2024-02-23 01:11:31 +02:00
block_stream_reader_test.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:25:49 +02:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
block_stream_writer.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
block_stream_writer_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
block_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-26 09:44:40 +02:00
dedup.go docs: mention staleNaN handling during deduplication 2024-01-16 20:11:45 +02:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:18:49 -08:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:34:36 -07:00
index_db.go lib/storage: properly handle maxMetrics limit at metricID search 2024-08-27 23:08:17 +02:00
index_db_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:55:07 +02:00
index_db_timing_test.go lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency 2024-02-24 00:51:03 +02:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:12:32 -08:00
merge.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
merge_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
merge_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:04 +03:00
metaindex_row_test.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:13:15 +03:00
metric_name.go lib/storage: change default value for maxLabelValueLen to 1024 (#6313) 2024-05-22 21:55:21 +02:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_header.go lib/storage: add ability to use downsampling for the given series filter (#733) 2024-04-03 02:38:37 +03:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:31:39 -08:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:47:20 -07:00
part_search_timing_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-26 09:44:40 +02:00
partition.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:43:00 +02:00
partition_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
partition_search_test.go lib/storage: do not keep rawRows buffer across flush() calls 2024-02-23 01:06:09 +02:00
partition_test.go lib/storage: stop exposing vm_merge_need_free_disk_space metric 2023-09-25 17:00:14 +02:00
raw_block.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
raw_row.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:33:07 +02:00
search.go lib/storage: fix SearchQuery.Unmarshal() after 32193b6059 2024-05-14 01:39:01 +02:00
search_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:55:07 +02:00
storage.go lib/storage: properly register index records with RegisterMetricNames 2024-08-27 23:00:27 +02:00
storage_test.go lib/storage: properly handle maxMetrics limit at metricID search 2024-08-27 23:08:17 +02:00
storage_timing_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:55:07 +02:00
table.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:43:00 +02:00
table_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
table_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
table_search_timing_test.go lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency 2024-02-24 00:51:03 +02:00
table_test.go flagutil: Make .Msecs private (#4906) 2023-09-03 10:37:57 +02:00
table_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
tag_filters.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
tag_filters_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
tag_filters_timing_test.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-03 03:00:45 +03:00
time.go app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
time_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tsid.go lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
tsid_test.go all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00