github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
1115 commits 150 branches 748 tags 347 MiB
Commit graph

90 commits

Author SHA1 Message Date
Aliaksandr Valialkin
2f42b85e0e lib/storage: document that getnerateUniqueMetricID should return dense ids 2020-05-14 14:08:59 +03:00
Aliaksandr Valialkin
8c77cb436a lib/storage: typo fixes in error messages: or -> of 2020-05-12 12:12:33 +03:00
Aliaksandr Valialkin
d78ed50edd lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown 
Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet.
Log about this and delete the corresponding metricID, so it could be re-created next time.
 2020-04-28 12:01:32 +03:00
Aliaksandr Valialkin
13b4069c59 lib/storage: postpone label filters matching too many time series instead of giving up with error 
This should reduce the frequency of the following errors:

    cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters

    more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range
 2020-04-24 21:18:52 +03:00
Aliaksandr Valialkin
f9526809e5 app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
e9d9638627 lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search 
This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet.

This should fix the following error:

error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF
 2020-04-15 00:10:11 +03:00
Aliaksandr Valialkin
0ad7aaf535 lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init 2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin
ef714e01c1 lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet 2020-03-31 12:35:15 +03:00
Aliaksandr Valialkin
7e755b4bac lib/storage: optimize per-day inverted index search for tag filters matching big number of time series 
- Sort tag filters in the ascending number of matching time series
  in order to apply the most specific filters first.
- Fall back to metricName search for filters matching big number of time series
  (usually this are negative filters or regexp filters).
 2020-03-31 00:53:29 +03:00
Aliaksandr Valialkin
31a533656e lib/storage: remove obsolete code 2020-03-13 22:42:42 +02:00
Aliaksandr Valialkin
476c7fb109 lib/storage: reduce memory allocations when merging metricID sets 2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin
534da0a8c3 lib/storage: fall back to global inverted index if a filter match too many time series in per-day index 
Previously this resulted to error message. The query may succeed via search in global index.
 2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin
625f6ca761 lib/storage: optimize regexp filter search 2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin
4e22b521c2 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin
0f184affa7 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
b9e53490b9 lib/storage: move non-matching tag filters to the top at matchTagFilters 
This should reduce the amount of useless work needed for matching the next metricNames.
 2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin
33d9d63393 lib/storage: speed up time series search for queries with multiple filters 
Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.
 2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin
494ad0fdb3 lib/storage: remove inmemory index for recent hour, since it uses too much memory 
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.

Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
 2019-11-13 18:08:58 +02:00
Aliaksandr Valialkin
633dd81bb5 lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour 
This may be useful for saving RAM on high number of time series aka high cardinality
 2019-11-13 15:10:12 +02:00
Aliaksandr Valialkin
c48e39eea9 lib/storage: add tests for dateMetricIDCache 2019-11-11 13:21:05 +02:00
Aliaksandr Valialkin
9ea2bd822e lib/storage: implement per-day inverted index 2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin
dea2f3efed lib/storage: use specialized cache for (date, metricID) entries 
This improves ingestion performance.
 2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin
9a43902bd8 lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero 2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin
c16e17dede lib/storage: properly set time range when deleting time series 2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin
8126007c15 lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster 2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin
50773348d3 lib/storage: small code prettifying 2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
46e67bb78c lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate 2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin
0063c857f5 lib/storage: add inmemory inverted index for the last hour 
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
 2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin
1c777e0245 lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter 
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.

Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c567a4353a lib/storage: take into account the requested time range when caching TSIDs for the given tag filters 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c6564c5d26 lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting 2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin
e5b1fa0c38 lib/storage: separate the max inverted index scan loops per metric into fast and slow loops 
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
 2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
f93c4f2493 lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive 
This should improve performance for query filters over big number of time series.
 2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin
f48e97263c lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name 
The new value reduces the amount of wasted work during index scans over big number of time series.
 2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin
d2f688c550 lib/storage: try potentially faster tag filters at first, then apply slower tag filters 
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
 2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin
f5fbc3ffd7 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
6a22727676 lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups 2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
661b8ede5b lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order 2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin
7e410e1412 lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. 
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
 2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
95e3d648cb lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
4e3871ac1e lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966 lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00