Aliaksandr Valialkin
67e331ac62
lib/storage: optimize ingestion pefrormance for new time series
2020-05-15 12:12:19 +03:00
Aliaksandr Valialkin
f9526809e5
app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
0ad7aaf535
lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init
2020-04-01 17:40:27 +03:00
Aliaksandr Valialkin
494ad0fdb3
lib/storage: remove inmemory index for recent hour, since it uses too much memory
...
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.
Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
2019-11-13 18:08:58 +02:00
Mike Poindexter
955a592106
Add test for invalid caching of tsids ( #232 )
...
* Add test for invalid caching of tsids
* Clean up error handling
2019-11-12 15:52:46 +02:00
Aliaksandr Valialkin
9ea2bd822e
lib/storage: implement per-day inverted index
2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin
1c777e0245
lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter
...
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.
Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
6a22727676
lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups
2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
7e410e1412
lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering.
...
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
4e3871ac1e
lib/storage: add missing break in removeDuplicateMetricIDs
2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966
lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index
2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
67a2bcb98a
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
0b0153ba3d
lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
8c2158af24
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin
b7c4b0c6d2
lib/storage: fix matching against tag filter with empty name
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137
2019-07-30 15:15:21 +03:00
Aliaksandr Valialkin
ee23a143b9
lib/storage: make sure non-nil args are passed to openIndexDB
2019-06-25 20:10:08 +03:00
Aliaksandr Valialkin
51e2f3b48f
lib/storage: skip searching in extDB if it doesn't contain items for the given time range
...
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
2019-06-25 12:57:56 +03:00
Aliaksandr Valialkin
00e0760608
lib/storage: test GetSeriesCount
2019-06-10 12:40:33 +03:00
Aliaksandr Valialkin
d882afa905
lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)
2019-06-09 19:14:04 +03:00
Aliaksandr Valialkin
24578b4bb1
all: open-sourcing cluster version
2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
