mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
9fcfba3927
`TL;DR` This PR improves the metric IDs search in IndexDB:
- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search
A unit test has been added to ensure that this refactoring does not
break anything.
---
Function calls before the fix:
```
idb.searchMetricIDs
|__ is.searchMetricIDs
|__ is.searchMetricIDsInternal
|__ is.updateMetricIDsForTagFilters
|__ is.tryUpdatingMetricIDsForDateRange
| |
|__ is.getMetricIDsForDateAndFilters
```
- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.
Problems with this solution:
1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.
Solution:
1. Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.
---------
Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
|
||
|---|---|---|
| .. | ||
| appmetrics | ||
| auth | ||
| awsapi | ||
| backup | ||
| blockcache | ||
| bloomfilter | ||
| bufferedwriter | ||
| buildinfo | ||
| bytesutil | ||
| cgroup | ||
| decimal | ||
| encoding | ||
| envflag | ||
| envtemplate | ||
| fastnum | ||
| fasttime | ||
| filestream | ||
| flagutil | ||
| formatutil | ||
| fs | ||
| htmlcomponents | ||
| httpserver | ||
| httputils | ||
| influxutils | ||
| ingestserver | ||
| leveledbytebufferpool | ||
| logger | ||
| logstorage | ||
| lrucache | ||
| memory | ||
| mergeset | ||
| metricsql | ||
| netutil | ||
| persistentqueue | ||
| procutil | ||
| promauth | ||
| prompb | ||
| prompbmarshal | ||
| promrelabel | ||
| promscrape | ||
| promutils | ||
| protoparser | ||
| proxy | ||
| pushmetrics | ||
| querytracer | ||
| ratelimiter | ||
| regexutil | ||
| slicesutil | ||
| snapshot | ||
| storage | ||
| streamaggr | ||
| stringsutil | ||
| syncwg | ||
| tenantmetrics | ||
| timerpool | ||
| timeutil | ||
| uint64set | ||
| workingsetcache | ||
| writeconcurrencylimiter | ||