github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
3746 commits 150 branches 748 tags 347 MiB
Commit graph

451 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a22f37599b lib/storage: tune tag filters search logic 
Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624

The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats.
This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters
for further queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin
a207be3ffb lib/storage: fix infinite loop introduced in aa9b56a046 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin
0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin
b133de1e37 lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin
ce10bdc82a lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin
eb335d2c29 lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard 2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin
d06c0e7a94 lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores 2021-06-11 10:49:02 +03:00
Aliaksandr Valialkin
1e4a64844d lib/storage: properly account the number of loops spent when matching for or suffixes 
This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin
fc2565b4ee lib/storage: reduce memory allocations when syncing dateMetricIDCache 2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin
10b2855949 lib/storage: fix spelling typo: borken->broken 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336
 2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin
1c16cbacf5 lib/storage: do not stop data ingestion on the first error in Storage.AddRows 
Continue data ingestion for the rest of blocks.
 2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin
2601844de3 lib/storage: limit the number of rows per each block in Storage.AddRows() 
This should reduce memory usage when ingesting big blocks or rows.
 2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin
95b735a883 lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows 
This should reduce memory usage a bit on data ingestion path
 2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin
402a8ca710 lib/storage: do not populate MetricID->MetricName cache during data ingestion 
This cache isn't needed during data ingestion, so there is no need in spending RAM on it.

This reduces RAM usage on data ingestion path by 30%
 2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin
0fc857d363 lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds 
Log these messages if the merge takes more than 30 seconds instead of 10 seconds.
 2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
165a9f9200 app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin
e228f479a5 lib/storage: remove possible data race when logging dropped labels 2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin
2839055513 lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss 2021-05-13 09:01:05 +03:00
Aliaksandr Valialkin
008ae25b3a lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate 
These functions are non-trivial, while their code has minimal differences.
It is better from maintainability PoV to merge these functions into a single function.
 2021-05-12 18:01:08 +03:00
Nikolay
be87be34a4 Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin
4e59cf4380 lib/storage: properly apply time range when matching an empty filter 
It must match all the time series on the given time range.
Previously it was matched to all the time series without the restriction on the given time range.
 2021-05-11 01:09:35 +03:00
Aliaksandr Valialkin
326cf83eb4 lib/storage: remove dead code after the commit 3ccf7ea20c 2021-05-08 20:15:59 +03:00
Aliaksandr Valialkin
4a5f45c77e app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
43c52ff77a lib/storage: use WARNING instead of INFO level for logging dropped labels 2021-05-03 13:57:28 +03:00
Nikolay
62d58324dd adds stalePartsRemover (#1261) 
for new created partitions
 2021-05-03 11:34:33 +03:00
Aliaksandr Valialkin
b43ba6d85f lib/storage: log dropped labels if the number of labels in a metric exceeds -maxLabelsPerTimeseries command-line flag value 
This should improve debuggability for this case.
 2021-05-01 09:29:56 +03:00
Aliaksandr Valialkin
e37e1b1e34 lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures 
The panic has been introduced in 56b6b893ce
 2021-04-27 16:42:19 +03:00
Aliaksandr Valialkin
2d1d60118d lib/mergeset: split rows ingestion among multiple shards 
This improves rows ingestion on systems with many CPU cores by reducing lock contention.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244

Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243
 2021-04-27 15:45:11 +03:00
Aliaksandr Valialkin
cba2d13456 lib/storage: typo fix in info message when deleting the part outside the configured retention 
Previously the message was displaying incorrect retention time
 2021-04-27 13:33:36 +03:00
Aliaksandr Valialkin
ab8008d6d7 lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
 2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin
72c41323fa lib/storage: code clarification: remove caching the found metricName in searchMetricName 2021-04-13 10:20:35 +03:00
Aliaksandr Valialkin
59ccc43e3a lib/storage: properly handle big time ranges passed to /api/v1/labels and /api/v1/label/<labelName>/values 
It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch
 2021-04-07 13:33:10 +03:00
Aliaksandr Valialkin
512addc608 app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage 
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
 2021-03-31 23:27:21 +03:00
Aliaksandr Valialkin
ae1c653d55 lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels 2021-03-31 21:22:40 +03:00
Aliaksandr Valialkin
940a547116 lib/storage: do not update b.nextIdx if no samples are removed because of retention 2021-03-29 12:13:38 +03:00
Aliaksandr Valialkin
9c2be144cf app/vmselect: log the metric which trigger rollup result cache reset 
This should help finding the source of stale metrics
 2021-03-25 21:32:28 +02:00
Aliaksandr Valialkin
f971fe86cd lib/storage: tune loopsCountPerMetricNameMatch according to production workload 2021-03-25 13:48:17 +02:00
Aliaksandr Valialkin
9947c65df3 lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock 
This should speed up Search.NextMetricBlock a bit
 2021-03-23 17:59:34 +02:00
Aliaksandr Valialkin
12ca0efc19 lib/storage: respect the deadline passed to Storage.SearchMetricNames 2021-03-22 23:03:00 +02:00
Aliaksandr Valialkin
40e47935e7 lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache 2021-03-22 23:02:59 +02:00
Aliaksandr Valialkin
1618b0ca6d lib/storage: tune loopsCountPerMetricNameMatch 2021-03-22 12:57:54 +02:00
Aliaksandr Valialkin
7503111feb lib/storage: small code simplification after 6cee5338b2 2021-03-18 15:22:39 +02:00
Aliaksandr Valialkin
4443254fb9 lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars 
The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137
 2021-03-18 14:57:39 +02:00
Aliaksandr Valialkin
b859fe7879 lib/storage: faster move heavy filters to the end of list 2021-03-17 15:11:56 +02:00
Aliaksandr Valialkin
41fe707bec lib/storage: limit loops count in order to reduce max CPU usage during filter search 2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
e2a0c8bd72 lib/storage: do not modify filterLoopsCount stats with loopsCount stats 
Such a modification can result in incorrect filter sorting later
 2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
727ded9d4e lib/storage: time series search optimization according to production workload profiling 
Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down
the function by multiple times when it finds big number of metricIDs (tens of millions).
 2021-03-16 20:08:43 +02:00
Aliaksandr Valialkin
f4a44d6c0d lib/storage: further tuning for time series search 2021-03-16 18:47:29 +02:00
Aliaksandr Valialkin
d074326970 app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate 2021-03-15 22:40:28 +02:00
Aliaksandr Valialkin
fc902734d9 lib/storage: further tuning for time series selector code 2021-03-15 20:32:37 +02:00
Aliaksandr Valialkin
1c26020080 lib/storage: tune per-day index search 2021-03-15 13:36:36 +02:00
Aliaksandr Valialkin
b2732575f7 lib/storage: further tune filters sorting logic 2021-03-12 00:51:35 +02:00
John Belmonte
edf39aa225 spelling fix: adjacent (#1115) 2021-03-09 09:19:16 +02:00
Aliaksandr Valialkin
c4a0bd5eac lib/storage: go fmt 2021-03-08 11:59:31 +02:00
Aliaksandr Valialkin
c76a904bb0 lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow 
The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload
 2021-03-07 21:17:48 +02:00
Aliaksandr Valialkin
c8dde1fd6b lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:48:44 +02:00
Aliaksandr Valialkin
06676c8feb lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache 2021-02-23 15:50:08 +02:00
faceair
b1409a7413 lib/storage: correct tagfilter match cost (#1079) 2021-02-22 21:54:37 +02:00
Aliaksandr Valialkin
587132555f lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
 2021-02-21 22:09:10 +02:00
Aliaksandr Valialkin
bd3bcdc43c lib/storage: do not re-calculate stats for heavy tag filters 
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
b8a5ee2e93 lib/{mergeset,storage}: allow merging smaller number of small parts 
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.

This partially reverts ebf8da3730
 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
34195218e1 lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains 2021-02-21 21:43:37 +02:00
Aliaksandr Valialkin
10ccb92e4d lib/storage: use composite index for a query with a name filter and negative filters 2021-02-18 18:57:45 +02:00
Aliaksandr Valialkin
418de71509 lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters 
Example: `node_cpu_seconds_total{mode!="idle"}`
 2021-02-18 18:33:05 +02:00
Aliaksandr Valialkin
628b8eb55e lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. 
This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters
 2021-02-18 14:01:18 +02:00
Aliaksandr Valialkin
fd41f070db lib/storage: sort tag filters by the number of loops they need for the execution 
This metric should work better than the filter execution duration, since it cannot be distorted
by concurrently running queries.
 2021-02-18 12:52:29 +02:00
Aliaksandr Valialkin
9566015a36 Revert "lib/mergeset: tune lifetime for entries inside block caches" 
This reverts commit 458c89324d.

Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.
 2021-02-17 20:42:15 +02:00
Aliaksandr Valialkin
b4c7d5992b lib/storage: move composite filters to the top during sorting 2021-02-17 20:26:32 +02:00
Aliaksandr Valialkin
2005d8212a lib/storage: return back filter arg to getMetricIDsForTagFilter function 
The filter arg has been removed in the commit c7ee2fabb8
because it was preventing from caching the number of matching time series per each tf.

Now the cache contains duration for tf execution, so the filter shouldn't break such caching.
 2021-02-17 19:33:15 +02:00
Aliaksandr Valialkin
83da939947 app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics 2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin
0c5bb2a397 lib/storage: revert ecf132933e, since negative filters require the same amount of work as non-negative filters 2021-02-17 18:56:13 +02:00
Aliaksandr Valialkin
bbc287ea6a lib/storage: tag filters sorting... 2021-02-17 18:56:12 +02:00
Aliaksandr Valialkin
f5e841c1e9 lib/storage: further tune tag filters sorting 2021-02-17 17:28:36 +02:00
Aliaksandr Valialkin
9d1f14d94c lib/storage: tune the logic for sorting tag filters according the their exeuction times 2021-02-17 15:02:19 +02:00
Aliaksandr Valialkin
1a19702d92 lib/storage: make sure that nobody uses partitions when closing the table 2021-02-17 15:02:18 +02:00
Aliaksandr Valialkin
1ab7a8dfd5 lib/storage: more tuning for tag filters sorting according the time they take 2021-02-16 21:27:27 +02:00
Aliaksandr Valialkin
35a23234ca lib/mergeset: tune lifetime for entries inside block caches 
This should reduce memory usage in general case without significant CPU usage increase
 2021-02-16 18:12:32 +02:00
Aliaksandr Valialkin
55952f8f2e lib/storage: tune sorting for tag filters 2021-02-16 13:07:42 +02:00
Aliaksandr Valialkin
3eae03a337 lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs 2021-02-15 16:39:52 +02:00
Aliaksandr Valialkin
46e98ed490 vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
93ff866e91 lib/storage: reduce the minimum supported retention for inverted index from one month to one day 2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
6f3bbf21b8 lib/storage: sort tag filters by actual execution time instead of by the number of matching time series 
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
 2021-02-15 00:19:46 +02:00
Aliaksandr Valialkin
9e3993c585 lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. 
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
 2021-02-14 23:39:19 +02:00
Aliaksandr Valialkin
4e645a5fd3 lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:43:07 +02:00
Aliaksandr Valialkin
eeb92eb7fc lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata 2021-02-10 17:55:51 +02:00
Aliaksandr Valialkin
08f21d8761 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin
b27288f1b0 lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 16:32:27 +02:00
Aliaksandr Valialkin
4262c2f7c2 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 16:32:26 +02:00
Aliaksandr Valialkin
681dfb7485 lib/storage: fix inconsistencies in error logs 2021-02-10 16:32:21 +02:00
Aliaksandr Valialkin
148422bcba lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
17d5a03f6e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:34 +02:00
Aliaksandr Valialkin
fa0ef143b1 lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin
5c9715a89a lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day 
This should help systems with multiple CPU cores
 2021-02-10 00:04:19 +02:00
Aliaksandr Valialkin
9ed7789fef optimize Storage.updatePerDateData() 2021-02-09 02:59:53 +02:00
Aliaksandr Valialkin
ea328b7391 lib/storage: skip deduplication when creating inmemory data blocks 
The deduplication will be performed later during merging such blocks.
 2021-02-09 02:26:16 +02:00
Aliaksandr Valialkin
7b7963a77f lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:49:59 +02:00
Aliaksandr Valialkin
e8ee9fa7fe app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin
2dbb12563b lib/storage: optimize data ingestion in the beginning of every hour 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-08 12:04:51 +02:00
Aliaksandr Valialkin
c6a7288109 lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion 
This should reduce possible CPU usage spikes at the beginning of every hour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-04 18:46:23 +02:00
Aliaksandr Valialkin
8249f13104 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:16:28 +02:00