mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
9fcfba3927
`TL;DR` This PR improves the metric IDs search in IndexDB:
- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search
A unit test has been added to ensure that this refactoring does not
break anything.
---
Function calls before the fix:
```
idb.searchMetricIDs
|__ is.searchMetricIDs
|__ is.searchMetricIDsInternal
|__ is.updateMetricIDsForTagFilters
|__ is.tryUpdatingMetricIDsForDateRange
| |
|__ is.getMetricIDsForDateAndFilters
```
- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.
Problems with this solution:
1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.
Solution:
1. Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.
---------
Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
|
||
|---|---|---|
| .. | ||
| block.go | ||
| block_header.go | ||
| block_header_test.go | ||
| block_stream_merger.go | ||
| block_stream_reader.go | ||
| block_stream_reader_test.go | ||
| block_stream_reader_timing_test.go | ||
| block_stream_writer.go | ||
| block_stream_writer_timing_test.go | ||
| block_test.go | ||
| dedup.go | ||
| dedup_test.go | ||
| dedup_timing_test.go | ||
| filenames.go | ||
| index_db.go | ||
| index_db_test.go | ||
| index_db_timing_test.go | ||
| inmemory_part.go | ||
| inmemory_part_test.go | ||
| inmemory_part_timing_test.go | ||
| merge.go | ||
| merge_test.go | ||
| merge_timing_test.go | ||
| metaindex_row.go | ||
| metaindex_row_test.go | ||
| metric_name.go | ||
| metric_name_test.go | ||
| part.go | ||
| part_header.go | ||
| part_search.go | ||
| part_search_test.go | ||
| part_search_timing_test.go | ||
| partition.go | ||
| partition_search.go | ||
| partition_search_test.go | ||
| partition_test.go | ||
| raw_block.go | ||
| raw_row.go | ||
| search.go | ||
| search_test.go | ||
| storage.go | ||
| storage_test.go | ||
| storage_timing_test.go | ||
| table.go | ||
| table_search.go | ||
| table_search_test.go | ||
| table_search_timing_test.go | ||
| table_test.go | ||
| table_timing_test.go | ||
| tag_filters.go | ||
| tag_filters_test.go | ||
| tag_filters_timing_test.go | ||
| time.go | ||
| time_test.go | ||
| tsid.go | ||
| tsid_test.go | ||