111 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	b5db69fe05	app/vmselect/netstorage: do not intern string representation of MetricName for time series received from vmstorage ... It has been appeared that this interning may lead to increased memory usage and increased CPU usage when vmselect performs queries, which select big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863	2023-03-12 00:52:35 -08:00
Oleksandr Redko	9fff48c3e3	app,lib: fix typos in comments (#3804)	2023-02-13 13:27:13 +01:00
Aliaksandr Valialkin	be8fba9b6a	app/vmselect/netstorage: tune the number of blocks per series which should be unpacked by a single goroutine instead of spinning up multiple goroutines ... This reduces overhead on time series data unpacking for typical cases, this reducing CPU usage at vmselect	2023-01-12 09:31:44 -08:00
Aliaksandr Valialkin	53d871d0b1	app/vmselect/netstorage: reduce tail latency during query processing ... Previously the selected time series were split evenly among available CPU cores for further processing - e.g unpacking the data and applying the given rollup function to the unpacked data. Some time series could be processed slower than others. This could result in uneven work distribution among available CPU cores, e.g. some CPU cores could complete their work sooner than others. This could slow down query execution. The new algorithm allows stealing time series to process from other CPU cores when all the local work is done. This should reduce the maximum time needed for query execution (aka tail latency). The new algorithm should also scale better on systems with many CPU cores, since every CPU processes locally assigned time series without inter-CPU communications. The inter-CPU communications are used only when all the local work is finished and the pending work from other CPUs needs to be stealed.	2023-01-10 13:43:14 -08:00
Aliaksandr Valialkin	e640ff72f1	app/vmselect/netstorage: reduce memory allocations when unpacking time series ... Unpack time series with less than 400K samples in the currently running goroutine. Previously a new goroutine was being started for unpacking the samples. This was requiring additional memory allocations.	2023-01-09 23:18:17 -08:00
Aliaksandr Valialkin	df2a494a7c	app/vmselect/netstorage: pre-allocate 4 block references per each time series during querying ... Usually the number of blocks returned per each time series during queries is around 4. So it is a good idea to pre-allocate 4 block references per time series in order to reduce the number of memory allocations.	2023-01-09 22:03:23 -08:00
Aliaksandr Valialkin	c5e0f527bc	app/vmselect/netstorage: cache canonical MetricName for time series returned from the storage ... This reduces memory allocations for repeated queries, which return (almost) the same set of time series.	2023-01-09 21:53:10 -08:00
Aliaksandr Valialkin	7afcca0c51	all: use metricsql.CompileRegexp instead of regexp.Compile for compiling regexps used in graphite queries ... This should speed up repeated queries, since metricsql.CompileRegexp returns regexps from the cache on subsequent calls for the same input regexp.	2023-01-09 21:43:08 -08:00
Aliaksandr Valialkin	c38a10e143	app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks()	2023-01-09 21:08:51 -08:00
Aliaksandr Valialkin	1f9d605988	app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333 This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 , but doesn't slow down the common case with merging replicated data blocks so significantly. Benchmark results: Before the change: BenchmarkMergeSortBlocks/replicationFactor-1-4 13968 85643 ns/op 956.53 MB/s 1700 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-2-4 10806 109171 ns/op 1500.77 MB/s 2191 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-3-4 8887 130623 ns/op 1881.45 MB/s 2660 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-4-4 7440 157348 ns/op 2082.52 MB/s 3174 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-5-4 6534 184473 ns/op 2220.38 MB/s 3612 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4 13419 85205 ns/op 961.44 MB/s 2213 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 579 1894900 ns/op 43.23 MB/s 46760 B/op 1 allocs/op After the change: BenchmarkMergeSortBlocks/replicationFactor-1-4 13832 85298 ns/op 960.40 MB/s 1716 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-2-4 8833 134222 ns/op 1220.66 MB/s 2675 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-3-4 6487 184830 ns/op 1329.65 MB/s 3636 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-4-4 4977 236318 ns/op 1386.61 MB/s 4733 B/op 1 allocs/op BenchmarkMergeSortBlocks/replicationFactor-5-4 4088 296734 ns/op 1380.36 MB/s 5761 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4 14083 84067 ns/op 974.47 MB/s 2110 B/op 1 allocs/op BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 536 2043534 ns/op 40.09 MB/s 50511 B/op 1 allocs/op	2023-01-09 13:01:48 -08:00
Aliaksandr Valialkin	cae0f37edd	app/vmselect/netstorage: remove superflouos map lookup at ProcessSearchQuery ... This should reduce CPU usage a bit during querying	2022-11-18 13:40:04 +02:00
Aliaksandr Valialkin	c53b7e66ef	app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints	2022-10-01 22:05:43 +03:00
Aliaksandr Valialkin	7478d423c5	app/vmselect/netstorage: cleanup after 92630c1ab4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896	2022-08-04 18:28:11 +03:00
Aliaksandr Valialkin	c2bd75926b	app/vmselect/netstorage: initialize tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths	2022-07-30 00:39:36 +03:00
Aliaksandr Valialkin	19a0b4679a	app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel ... This should reduce CPU usage needed for rand.Rand initialization	2022-07-30 00:30:37 +03:00
Aliaksandr Valialkin	92630c1ab4	app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system ... Reduce inter-CPU communications when processing the query over big number of time series. This should improve performance for queries over big number of time series on systems with many CPU cores. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896 Based on b596ac3745 Thanks to @zqyzyq for the idea.	2022-07-25 09:18:44 +03:00
Aliaksandr Valialkin	159c2e15e3	app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples	2022-07-12 12:31:38 +03:00
Aliaksandr Valialkin	cd09f583fe	app/vmselect/netstorage: add benchmarks for mergeSortBlocks ... This is a follow-up for 743ff84863	2022-07-11 12:54:48 +03:00
Aliaksandr Valialkin	743ff84863	app/vmselect/netstorage: optimize mergeSortBlocks function ... - Use binary search instead of linear scan when locating the run of smallest timestamps in blocks with intersected time ranges. This should improve performance when merging blocks with big number of samples - Skip samples with duplicate timestamps. This should increase query performance in cluster version of VictoriaMetrics with the enabled replication.	2022-07-09 00:34:42 +03:00
Aliaksandr Valialkin	77cbbacfdb	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics ... This reduces the number of args to vmselectapi calls	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	f435924ab3	lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	e1b8059086	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	a60e03b3a7	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() ... This improves the API consistency	2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin	a14188dd8e	app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload ... This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792	2022-06-28 20:18:13 +03:00
Aliaksandr Valialkin	a350d1e81c	lib/storage: return marshaled metric names from SearchMetricNames ... Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin	e578549b8a	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin	ba514284f1	lib/storage: add querytracer to more contexts ... querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	9a314106ca	app/vmselect/netstorage: remove Get prefix from netstorage functions ... This makes these function names more consistent with the server side	2022-06-27 00:45:05 +03:00
Aliaksandr Valialkin	ec7963208d	app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb ... This allows filling the seriesCountByFocusLabelValue list in the /api/v1/status/tsdb response with label values for the specified focusLabel, which contain the highest number of time series. TODO: add this to Cardinality explorer at VMUI - https://docs.victoriametrics.com/#cardinality-explorer	2022-06-14 18:36:54 +03:00
Aliaksandr Valialkin	374beb350e	app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them	2022-06-12 04:32:13 +03:00
Aliaksandr Valialkin	89b778902b	app/vmselect: add optional limit query arg to /api/v1/labels and /api/v1/label_values endpoints ... This arg allows limiting the number of sample values returned from these APIs	2022-06-10 09:50:33 +03:00
Aliaksandr Valialkin	2bcb960f17	all: improve query tracing coverage for indexdb search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:07:07 +03:00
Aliaksandr Valialkin	12ac255dae	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin	41958ed5dd	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin	6e364e19ef	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin	92070cbb67	lib/storage: return dedup interval in milliseconds from GetDedupInterval() ... This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:26:38 +02:00
Aliaksandr Valialkin	1d20a19c7d	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() ... This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:49:12 +02:00
Aliaksandr Valialkin	8ed95e82c6	app/vmselect/promql: follow-up after 57b3320478	2021-09-24 01:24:18 +03:00
Aliaksandr Valialkin	f4dead529f	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:09:21 +03:00
Aliaksandr Valialkin	a1911e1330	app/vmselect/netstorage: unpack time series data in mostly local big chunks ... This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples	2021-07-30 12:03:17 +03:00
Aliaksandr Valialkin	74ffaa45d9	app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17	2021-07-30 09:57:53 +03:00
Aliaksandr Valialkin	8ee8660ac4	app/vmselect: follow-up for 626073bca8 ... * Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries * Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes * Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md	2021-07-28 18:00:23 +03:00
Aliaksandr Valialkin	95aff47330	app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel	2021-07-26 15:47:30 +03:00
Aliaksandr Valialkin	ed10141ff8	app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores	2021-07-16 00:35:23 +03:00
Aliaksandr Valialkin	f4e81aef7e	app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series ... This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067	2021-07-15 16:03:28 +03:00
Aliaksandr Valialkin	e6ef97a5ee	app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems	2021-07-15 15:41:58 +03:00
Aliaksandr Valialkin	832651c6c2	app/vmselect: follow up after 8a0678678b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168	2021-05-12 17:18:30 +03:00
Nikolay	8a0678678b	Adds tsdb match filters (#1282) ... * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin	7962cf1af8	app/vmselect: prevent from possible incomplete query results after timed out query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2021-03-30 13:35:45 +03:00
Aliaksandr Valialkin	7bc728bf53	app/vmselect: add vm_index_search_duration_seconds histogram for monitoring the performance of index search	2021-03-17 01:17:41 +02:00