github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
2276 commits 150 branches 748 tags 347 MiB
Commit graph

323 commits

Author SHA1 Message Date
Aliaksandr Valialkin
71c417427c lib/storage: sort tag filters by actual execution time instead of by the number of matching time series 
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
 2021-02-15 00:18:13 +02:00
Aliaksandr Valialkin
c727d2219b lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. 
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
 2021-02-14 23:38:14 +02:00
Aliaksandr Valialkin
0e26b7168a lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case 2021-02-10 22:41:04 +02:00
Aliaksandr Valialkin
b51c23dc5b lib/storage: parallelize tag filters execution a bit 
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
 2021-02-10 18:12:25 +02:00
Aliaksandr Valialkin
c7ee2fabb8 lib/storage: remove filter arg from getMetricIDsForDateTagFilter function 
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
 2021-02-10 18:12:20 +02:00
Aliaksandr Valialkin
57cac289e0 lib/storage: fix inconsistencies in error logs 2021-02-10 18:12:16 +02:00
Aliaksandr Valialkin
5d5f0b0627 lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata 2021-02-10 17:55:40 +02:00
Aliaksandr Valialkin
cdecf83ce5 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:08 +02:00
Aliaksandr Valialkin
553016ea99 lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
fcb7655d1e lib/storage: fix metric name match for composite filter 2021-02-10 01:27:45 +02:00
Aliaksandr Valialkin
c7dccebaef lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
6b4e6c229c lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day 
This should help systems with multiple CPU cores
 2021-02-10 00:01:13 +02:00
Aliaksandr Valialkin
d56390b925 optimize Storage.updatePerDateData() 2021-02-09 02:55:36 +02:00
Aliaksandr Valialkin
fda61e8e96 lib/storage: skip deduplication when creating inmemory data blocks 
The deduplication will be performed later during merging such blocks.
 2021-02-09 02:25:32 +02:00
Aliaksandr Valialkin
a4140de9e6 lib/mergeset: unconditionally cache indexdb blocks 
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
 2021-02-09 00:47:50 +02:00
Aliaksandr Valialkin
cb96a1865b app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:47:00 +02:00
Aliaksandr Valialkin
2242647a04 lib/storage: optimize data ingestion in the beginning of every hour 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-08 12:01:12 +02:00
Aliaksandr Valialkin
83d3e582ab lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion 
This should reduce possible CPU usage spikes at the beginning of every hour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
 2021-02-04 18:48:13 +02:00
Aliaksandr Valialkin
d16f22f3a1 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
a5a1b9bd66 lib/storage: fix a bug, which breaks searching by Graphite wildcard filters 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
6123aa3e75 sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
157c02622b app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
31ec79eaf6 lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here 
This is a follow-up commit after c8ea697db8
 2021-01-12 15:13:30 +02:00
Aliaksandr Valialkin
c8ea697db8 lib/storage: de-duplicate tags in MetricName.sortTags 
Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels
during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.
 2021-01-12 15:03:42 +02:00
Aliaksandr Valialkin
9dcb18e03d app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage 2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin
490c69c64e lib/storage: wait for pending transactions before closing and dropping the partition 
This deflakes `make test-full-386` test
 2020-12-25 11:45:53 +02:00
Aliaksandr Valialkin
cab7e936a3 lib/storage: physically remove stale parts 
Previously they were removed from partition struct, but the corresponding directories weren't removed.

This is a follow-up for 46dba00756
 2020-12-24 16:51:36 +02:00
Aliaksandr Valialkin
9e4ed5e591 lib/storage: do not remove parts outside the configured retention if they are currently merged 
These parts are automatically removed after the merge is complete.
 2020-12-24 08:51:28 +02:00
Aliaksandr Valialkin
46dba00756 lib/storage: remove stale parts as soon as they go outside the configured retention 
Previously such parts could remain undeleted for long durations until they are merged with other parts.
This should help for `-retentionPeriod` values smaller than one month.
 2020-12-22 19:54:31 +02:00
Aliaksandr Valialkin
d65c03c004 lib/storage: properly determine max rows for output part when merging small parts 2020-12-18 23:14:38 +02:00
Aliaksandr Valialkin
ebf8da3730 lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage 2020-12-18 20:01:08 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
357f886f97 Revert "lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test" 
This reverts commit f3e80eb70e493620e42a0cc22a62c9af75076c77, since it isn't needed for single-node version of VictoriaMetrics
 2020-11-29 01:27:25 +02:00
Aliaksandr Valialkin
ace969d595 lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test 2020-11-29 01:26:04 +02:00
Aliaksandr Valialkin
8a057e705a lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention 
This should simplify debugging when the source of the metric with unexpected timestamp must be found.
 2020-11-25 14:41:37 +02:00
Aliaksandr Valialkin
b65236530c lib/storage: typo fix in error message: allowd->allowed 2020-11-25 14:15:42 +02:00
Aliaksandr Valialkin
78d2715d04 all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9 2020-11-24 12:42:22 +02:00
Aliaksandr Valialkin
f765985947 lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin
266788be14 app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place 
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
 2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
0208d8c103 lib/storage: add a test for Storage.SearchMetricNames 2020-11-16 13:15:16 +02:00
Aliaksandr Valialkin
465923b181 app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
 2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
48d033a198 app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
 2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin
4aaee33860 lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page 2020-11-16 00:44:35 +02:00
immerrr again
51c529a2b6
app/vmstorage: add "/internal/force_flush" endpoint (#893) 2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin
368b69b4c4 app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange 2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin
1cb78ba1a0 lib/storage: remove data race when updating rowsDeleted 2020-11-05 01:12:21 +02:00
Aliaksandr Valialkin
b378cd6ed8 app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set 2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
caeb74f068 app/vmselect: reduce memory usage when query touches big number of time series 2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin
ae91a6883c lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively 
Previously such blocks were cleaned after they weren't accessed during 10 minutes.
Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.
 2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin
f3f62ab04e lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification 2020-11-03 15:32:02 +02:00