419 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	7dc632719d	app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045	2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin	4076277cf0	app/vmselect/promql: evaluate union() args in parallel in order to increase query performance ... Note that the parallel execution of `union()` args may take more memory and CPU time than the sequential execution if args contain heavy queries, which may load all the available CPU, disk and memory resources and vmselect and vmstorage levels.	2022-09-02 19:46:27 +03:00
Aliaksandr Valialkin	ad11b8d83d	app/vmselect/promql: follow-up after 2d71b4859c ... - Use getScalar() function for obtaining the expected scalar from phi arg - Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles - Improve the description of this bugfix in the docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026	2022-08-27 01:35:49 +03:00
Dmytro Kozlov	2d71b4859c	vmselect/promql: fix panic in histogram_quantiles function (#3029) ... * vmselect/promql: fix panic in histogram_quantiles function * Update docs/MetricsQL.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-08-27 01:33:56 +03:00
Dmytro Kozlov	463ea6897b	vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963) ... * vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries * vmselect/promql: cleanup * vmselect/promql: rename config flag * vmselect/promql: add tests * vmselect/promql: use test object instead of log * vmselect/promql: fix posible panic is subquery has more points. add description * vmselect/promql: update tests descriptions * vmselect/promql: update doInternal validation * vmselect/promql: fix linter * vmselect/promql: fix linter * vmselect/promql: update documentation and release notes * wip - Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries. Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries if it was smaller than the -search.maxPointsSubqueryPerTimeseries . - Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag . - Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits . - Update docs/CHANGELOG.md . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-24 15:25:18 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin	4ac79d29ad	app/vmselect: follow-up after 63e0f16062 ... * Explicitly store a pointer to UserReadableError in the error interface. Previously Go automatically converted the value to a pointer before storing in the error interface. * Add Unwrap() method to UserReadableError, so it can be used transparently with the other code, which calls errors.Is() and errors.As(). * Document the change in docs/CHANGELOG.md	2022-08-15 13:50:16 +03:00
Roman Khavronenko	63e0f16062	vmselect: introduce UserReadableError type of error (#2894) ... When read query fails, VM returns rich error message with all the details. While these details might be useful for debugging specific cases, they're usually too verbose for users. Introducing a new error type `UserReadableError` is supposed to allow to return to user only the most important parts of the error trace. This supposed to improve error readability in web interfaces such as VMUI or Grafana. The full error trace is still logged with the full context and can be found in vmselect logs. Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-15 13:38:47 +03:00
Dmytro Kozlov	a927814e7b	vmselect/promql: add tests for vmrangeBucketsToLE (#2907) ... * vmselect/promql: add tests for vmrangeBucketsToLE * vmselect/promql: cleanup * vmselect/promql: cleanup * vmselect/promql: fix panic tests want result * vmselect/promql: cleanup * vmselect/promql: update test name * vmselect/promql: fix linter error * vmselect/promql: refactor testcases * vmselect/promql: cleanup * vmselect/promql: remove unused reassign to workers, fix typo * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-26 20:42:41 +03:00
Aliaksandr Valialkin	c888e6b9be	app/vmselect/promql: reduce the diff for f148cffc8a ... This is a follow-up for c826f06366	2022-07-26 19:20:48 +03:00
Alan Liang	c826f06366	vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902) ... Co-authored-by: alanwzliang <alanwzliang@tencent.com>	2022-07-25 10:55:13 +03:00
Roman Khavronenko	9ccf695d57	vmselect: return correct error for second part of expression (#2893) ... Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-20 16:44:28 +02:00
Aliaksandr Valialkin	cc7d499bbd	app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied ... This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886	2022-07-19 14:27:48 +03:00
Aliaksandr Valialkin	b5da47bfaf	app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing	2022-07-18 14:24:54 +03:00
Aliaksandr Valialkin	0792c4ca90	app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results	2022-07-18 14:24:53 +03:00
Aliaksandr Valialkin	29e53b9f55	app/vmselect/promql: consistency update after 93fbd0c54b	2022-07-13 12:33:14 +03:00
Roman Khavronenko	93fbd0c54b	promql: return step as scrapeInterval when it can't be calculated (#2865) ... The change allows to specify default value for `getScrapeInterval` function when actual interval can't be calculated. Before the change, function were returning `maxSilenceInterval` (5m) in such cases, which may be not correct for instant queries processing. The specific scenario where using `maxSilenceInterval` caused issues is the following: 1. Series becomes stale; 2. Client (in this case vmalert) continues to request series every 15s; 3. Database returns empty results as expected; 4. But at some specific moment of time database returns datapoints from `now()-5m`, because lookback window was extended to `maxSilenceInterval`. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-13 12:27:38 +03:00
Aliaksandr Valialkin	c2197ad139	app/vmselect/promql: validate function name before evaluating its arguments ... This avoids unneeded evaluation of args for unknown functions	2022-07-12 19:48:26 +03:00
Roman Khavronenko	e1a41cfab5	metricsql: properly evaluate timezone_offset over time interval (#2842) ... * metricsql: properly evaluate `timezone_offset` over time interval https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-08 14:03:56 +03:00
Aliaksandr Valialkin	ae80cf76e0	app/vmselect: make fmt after f3ece83e67	2022-07-05 14:35:24 +03:00
Aliaksandr Valialkin	f3ece83e67	app/vmselect/promql: properly calculate histogram_quantile over unexpected le buckets ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819	2022-07-05 13:19:24 +03:00
Aliaksandr Valialkin	84e373e5c7	app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions	2022-06-30 22:39:38 +03:00
Aliaksandr Valialkin	3e2dd85f7d	all: readability improvements for query traces ... - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:20:33 +03:00
Aliaksandr Valialkin	a14188dd8e	app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload ... This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792	2022-06-28 20:18:13 +03:00
Aliaksandr Valialkin	a43f2d0bc5	app/vmselect/promql: show the number of scanned samples in the query trace	2022-06-28 19:26:17 +03:00
Aliaksandr Valialkin	e578549b8a	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin	a963b2a0aa	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	e6ed92529b	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package ... This package already has the same name, so there is no need in explicit name	2022-06-21 20:23:32 +03:00
Aliaksandr Valialkin	12ac255dae	lib/querytracer: make it easier to use by passing trace context message to New and NewChild ... The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin	41958ed5dd	all: add initial support for query tracing ... See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin	a7f18f8cb2	app/vmselect/promql: do not return values from label_value() if the original time series has no values at the selected timestamps	2022-05-09 17:57:39 +03:00
Aliaksandr Valialkin	cce1b6d7f9	app/vmselect/promql: add tlast_change_over_time(m[d]) function, which returns the timestamp for the last change of m on the given lookbehind window d	2022-04-27 10:59:03 +03:00
Aliaksandr Valialkin	25fe83577d	app/vmselect/promql: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries ... Previously `vector` time series could be unexpectedly returned from such queries	2022-04-21 15:34:36 +03:00
Aliaksandr Valialkin	d1a9fac894	app/vmselect/promql: fix comparison to nan ... The comparison to nan has been broken in d335cc886c Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2022-04-21 14:55:37 +03:00
Aliaksandr Valialkin	de892239a9	app/vmselect/promql: add drop_common_labels() function	2022-04-21 14:20:20 +03:00
Aliaksandr Valialkin	7a44ba1234	app/vmselect/promql: fix q default b where b may have empty time series	2022-04-21 11:42:42 +03:00
Aliaksandr Valialkin	d335cc886c	app/vmselect/promql: fix duplicate time series error on joins against time series filtered by values ... This should prevent from `duplicate time series` errors when executing the following query: kube_pod_container_resource_requests{resource="cpu"} * on (namespace,pod) group_left() (kube_pod_status_phase{phase=~"Pending|Running"}==1) where `kube_pod_status_phase{phase=~"Pending|Running"}==1` filters out diplicate time series	2022-04-20 22:18:44 +03:00
Aliaksandr Valialkin	ed97908ca9	app/vmselect/promql: rename removeNaNs() to more clear removeEmptySeries()	2022-04-20 19:53:46 +03:00
Aliaksandr Valialkin	61c7f6beae	app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2406	2022-04-11 12:34:43 +03:00
Aliaksandr Valialkin	6e364e19ef	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:29:49 +02:00
Aliaksandr Valialkin	0d47c23a03	app/vmselect/promql: reduce the maximum number of label values, which can be propagated from one side of the binary operation to another side of the binary operation from 10K to 1K ... There are user reports that 10K unique values in a single label filter may lead to performance and memory usage issues	2022-02-24 04:05:18 +02:00
Aliaksandr Valialkin	62b46007c5	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes ... This should reduce memory usage under high time series churn rate	2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin	b1f94f7f0e	app/vmselect/promql: return at most one time series from absent_over_time() in the same way as Prometheus does ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2130	2022-02-12 15:45:09 +02:00
Aliaksandr Valialkin	96b7de6736	app/vmselect/promql: clarify comments on why the right side of if and and operators are executed at first	2022-02-03 00:26:14 +02:00
Aliaksandr Valialkin	4b850c2a59	app/vmselect/promql: do not push down filters, which enumerate more than 10k unique values ... Such filters may slow down time series search, so just skip them. This is a follow-up for e7f1ceeb84 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1827	2022-02-02 23:40:02 +02:00
Aliaksandr Valialkin	2016a2c899	app/vmselect/promql: properly handle foo or bar queries ... Such queries may miss `bar` results after the commit e7f1ceeb84 because common label filters from `foo` could be mistakenly applied to `bar`.	2022-02-01 17:40:51 +02:00
Aliaksandr Valialkin	4bdd10ab90	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations ... Follow-up for f4989edd96 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 00:18:42 +02:00
Aliaksandr Valialkin	a8d22e1223	app/vmselect/promql: check for binary operation in case-insensitive manner when deciding which side of the operation to perform the first ... PromQL and MetricsQL operators are case-insensitive	2022-01-31 22:06:15 +02:00
Aliaksandr Valialkin	e7f1ceeb84	app/vmselect/promql: optimize queries, which join on _info metrics. ... Automatically add common filters from one side of binary operation to the other side before sending the query to storage subsystem. See https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/ and https://www.robustperception.io/exposing-the-software-version-to-prometheus	2022-01-31 19:32:36 +02:00
Aliaksandr Valialkin	f4989edd96	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions ... Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:24:44 +02:00