Aliaksandr Valialkin
fc81ea38d4
app/vmselect/promql: add outliersk(N, m)
aggregate function for anomaly detection across groups of similar time series
2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin
3845420a8f
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
f3d9a5b0ec
app/vmselect/promql: suppress "SA4006: this value of dstValues
is never used" error in golangci-lint
2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin
18a0caee43
app/vmselect/promql: fix any(..)
calculations - return all the data points instead of the first one
2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin
81b8811cf4
app/vmselect/promql: remove -search.maxPointsPerTimeseries
command-line flag
...
Limit the estimated time series count after aggregation with grouping by the number of source time series.
2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin
408ade27a9
app/vmselect/promql: add any(x) by (y)
aggregate function, which returns any time series from q
for each group y
2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin
21c2982ac8
app/vmselect/promql: support for sum(x) by (y) limit N
syntax in order to limit the number of output time series after aggregation
2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin
f341c6fcc4
Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation
command-line flag for tuning the probability of OOMs or false-positive not enough memory
errors"
...
This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c.
Reason for revert: this commit has been removed from single-node version
2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin
9ed4951ec8
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin
cd1145e5f4
app/vmselect: add -search.estimatedSeriesCountAfterAggregation
command-line flag for tuning the probability of OOMs or false-positive not enough memory
errors
2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin
b59f1f1504
app/vmselect: add -search.minStalenessInterval
command-line flag for removing gaps on graphs built from time series with irregular duration between samples
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin
603d4c9217
app/vmselect: merge -search.maxLookback
and -search.maxStalenessInterval
flags, since it has been appeared they have identical purpose :(
...
Leave both flags for backwards compatibility reasons.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin
a873b553cf
app/vmselect: handle timestamp(metric offset X)
the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin
a012f6fe70
app/vmselect/promql: keep metric name after applying first_over_time
and last_over_time
functions
2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin
58cb7fc476
app/vmselect: adjust label_map()
handling for corner cases
...
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series
All the unmatched labels are kept unchanged.
2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin
0e7a71a245
app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN)
function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin
50555d89d3
app/vmselect: add -search.maxStalenessInterval
for tuning Prometheus data model closer to Influx-style data model
2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin
0d893eff36
Makefile: add build and test rules with enabled race detector. These rules have -race
suffix
...
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
6ca1e58d98
app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick
2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin
b58e3fc8a9
app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-23 18:06:26 +02:00
Yaroslav
c69d4b01f0
fix rollupOpen(), rollupHigh(), rollupLow() functions ( #328 )
2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin
f22aefdb16
app/vmselect/promql: log when rollupResult cache is cleared
2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin
d5c2a0ce64
app/vmselect: add -search.cacheTimestampOffset
command-line flag
...
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.
This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin
846d7fa7e9
app/vmselect: add sort_by_label(q, label)
and sort_by_label_desc(q, label)
functions
...
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
e6d9ea3094
app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8
2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin
4a1de7fee9
app/vmselect/promql: properly adjust time range for data to select
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin
8e77b54846
app/vmselect: unconditionally offset -step
to rollup_candlestick
. This makes results more consistent
2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin
ce38b176bc
app/vmselect/promql: automatically apply offset -step
to rollup_candlestick
function in order to obtain the expected OHLC results
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin
4f7116d1ee
app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin
45bc6c62f2
app/vmselect/promql: adjust and
and unless
binary operator handling to be consistent with Prometheus
2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin
170c1c3a4e
app/vmselect/promql: add keep_next_value(q)
for filling gaps with the next non-empty value
2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
b28c9a3944
app/vmselect/promql: return expected results from increase()
over the beginning of time series, which start from big value
...
Examples for such counters: OS-level counters for network or cpu stats.
2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin
3e304890a6
app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin
ea53a21b02
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
e127173984
app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors
2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
40e564eb9c
app/vmselect/promql: add range_over_time(m[d])
function for calculating value range for m
over d
2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871
app/vmselect/promql: add label_match(q, label, regexp)
and label_mismatch(q, label, regexp)
functions for filtering out time series with labels matching the given regexp
2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
ce0b602405
app/vmselect/promql: fix panic on sum(aggr_over_time(...))
with incorrect number of args
2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin
bcd3f0c5bd
app/vmselect/promql: add hoeffding_bound_upper(phi, m[d])
and hoeffding_bound_lower(phi, m[d])
functions
...
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin
fc01b11ddc
app/vmselect/promql: return continuous values for min_over_time
and max_over_time
when step
is smaller than scrape_interval
2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin
16fb128bbc
app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time
, max_over_time
, rollup_first
and rollup_last
functions
...
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
2020-01-11 00:26:11 +02:00
Aliaksandr Valialkin
adc36d00b7
app/vmselect/promql: properly handle aggr(aggr_over_time(...))
2020-01-10 21:57:11 +02:00
Aliaksandr Valialkin
87a106702b
app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])
function
...
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin
c314d9a219
app/vmselect/promql: add tmin_over_time(m[d])
and tmax_over_time(m[d])
functions
...
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin
7c6df1e51d
app/vmselect/promql: skip rate
calculation for the first point on time series
2020-01-08 14:42:44 +02:00
Aliaksandr Valialkin
76707b2ab9
app/vmselect/promql: fix calculations for histogram_share
2020-01-04 14:44:15 +02:00
Aliaksandr Valialkin
accad01b3e
app/vmselect/promql: add missing MetricName into netstorage.Result in tests
2020-01-04 12:53:14 +02:00
Aliaksandr Valialkin
6f29d37cb5
app/vmselect/promql: add histogram_share(le, buckets)
function
2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin
2290503140
app/vmselect/promql: add absent_over_time(m[d])
func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin
67f94bbe12
app/vmselect/promql: add histogram_over_time(m[d])
rollup function
2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin
9a1f6848ca
app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time
...
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
2020-01-03 20:44:54 +02:00
Aliaksandr Valialkin
3d0c7b095a
app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate
for the first point on the time series
...
This should provide better estimation for `rate` in the beginning of time series.
2020-01-03 19:03:32 +02:00
Aliaksandr Valialkin
6ea7f23446
app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something)
from 100 to 1K, since 100 may result in OOM for high number of time series
2020-01-03 01:02:30 +02:00
Aliaksandr Valialkin
e0abf45d45
app/vmselect/promql: add share_le_over_time
and share_gt_over_time
functions for SLI and SLO calculations
2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin
eb1a66c577
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin
453d71d082
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:09:09 +02:00
Mike Poindexter
009d1559db
Split Extended PromQL parsing to a separate library
2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin
ff18101d30
app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range
...
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:
BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
2019-12-24 22:45:49 +02:00
Aliaksandr Valialkin
e24ee43109
app/vmselect/promql: adjust calculations for rate
and increase
for the first value
...
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
2019-12-24 19:41:03 +02:00
Aliaksandr Valialkin
9a2554691c
app/vmselect/promql: properly calculate rate
on the first data point
...
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
2019-12-24 15:55:15 +02:00
Aliaksandr Valialkin
b238997a84
all: rename Extended PromQL
to PromQL extensions
2019-12-12 19:29:59 +02:00
Aliaksandr Valialkin
c25b97829f
app/vmselect/promql: return lower
and upper
bounds for the estimated percentile from histogram_quantile
if third arg is passed
...
Updates https://github.com/prometheus/prometheus/issues/5706
2019-12-11 14:00:18 +02:00
Aliaksandr Valialkin
f79b61e2a1
app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query
like Prometheus does
2019-12-11 00:57:54 +02:00
Aliaksandr Valialkin
5d2ff573aa
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
d39bba3547
app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median}
aggregate functions for returning the exact k time series on the given time range
...
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
2019-12-05 19:27:45 +02:00
Aliaksandr Valialkin
819bb36852
app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals
...
This should improve scrape interval estimation for tiem series with gaps.
2019-12-02 13:43:04 +02:00
Aliaksandr Valialkin
1e2019b1b6
app/vmselect/promql: fix corner case for increase
over time series with gaps
...
In this case `increase` could return invalid high value for the first point after the gap.
2019-11-30 01:34:18 +02:00
Aliaksandr Valialkin
28cc4c09b5
app/vmselect/promql: remove zero timeseries from prometheus_buckets
output
2019-11-25 19:10:13 +02:00
Aliaksandr Valialkin
f7da9b2db2
app/vmselect/promql: allow nested parens
2019-11-25 16:13:33 +02:00
Aliaksandr Valialkin
d2619d6dce
vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1
2019-11-25 15:22:50 +02:00
Aliaksandr Valialkin
f46fb6c740
app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph
...
This should reduce the amounts memory allocations
2019-11-25 14:24:30 +02:00
Aliaksandr Valialkin
0f184affa7
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
dbd07041ae
app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0
2019-11-25 13:44:08 +02:00
Aliaksandr Valialkin
8bb254d960
app/vmselect/promql: add histogram
aggregate function, which is useful for building heatmaps from multiple time series
2019-11-24 00:04:15 +02:00
Aliaksandr Valialkin
414259f47b
app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets
2019-11-23 14:19:19 +02:00
Aliaksandr Valialkin
193d553f6d
app/vmselect/promql: properly handle histogram_quantile(0, ...)
with zero buckets
2019-11-23 14:02:25 +02:00
Aliaksandr Valialkin
4d76977745
app/vmselect/promql: transparently apply prometheus_buckets
in histogram_quantile
2019-11-23 11:49:16 +02:00
Aliaksandr Valialkin
5f6f03c692
app/vmselect/promql: add prometheus_buckets
function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics
to Prometheus-compatible buckets
2019-11-23 00:21:56 +02:00
Aliaksandr Valialkin
77bb66a5be
app/vmselect/promql: properly calculate integrate(q[d])
2019-11-13 21:11:03 +02:00
Aliaksandr Valialkin
c33640664a
app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
2019-11-13 20:26:07 +02:00
Aliaksandr Valialkin
33abbec6b4
app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions
...
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.
Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.
This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
2019-11-08 19:37:43 +02:00
Aliaksandr Valialkin
4a8251feff
app/vmselect/promql: add lag(q[d])
function, which returns the lag between the current timestamp and the timstamp for the last data point in q
2019-11-01 12:21:43 +02:00
Aliaksandr Valialkin
5b01b7fb01
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
99786c2864
app/vmselect/prometheus: add -search.maxLookback
command-line flag for overriding dynamic calculations for max lookback interval
...
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.
The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:37:17 +03:00
Aliaksandr Valialkin
a5302a6651
app/vmselect/promql: take into account the previous point when calculating max_over_time
and min_over_time
...
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
2019-10-08 12:30:16 +03:00
Aliaksandr Valialkin
ba460f62e6
app/vmselect/promql: do not generate timestamps for NaN values in timestamp
function according to Prometheus logic
2019-09-27 18:55:16 +03:00
Aliaksandr Valialkin
bd1cf053f6
app/vmselect/promql: add increases_over_time
and decreases_over_time
functions
...
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:51 +03:00
Aliaksandr Valialkin
ee4585db33
app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step]))
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-13 21:42:11 +03:00
Aliaksandr Valialkin
828e5f6d26
app/vmselect/promql: binary operation fixes according to Prometheus behaviour
...
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
2019-09-13 17:43:09 +03:00
Aliaksandr Valialkin
d0953e9f02
app/vmselect/promql: ignore grouping by destination label in count_values
, since such a grouping is performed automatically
2019-09-04 19:59:02 +03:00
Aliaksandr Valialkin
f78ffe565f
app/vmselect/promql: do not return artificial points beyond the last point in time series
2019-09-04 16:34:29 +03:00
Aliaksandr Valialkin
b08f085082
app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does
2019-09-03 20:43:29 +03:00
Aliaksandr Valialkin
604a4312f9
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
e9db22a551
app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
2019-08-22 14:39:24 +03:00
Aliaksandr Valialkin
90a4b00b10
app/vmselect/promql: fix panic on -search.disableCache
...
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
2019-08-21 17:12:01 +03:00
Aliaksandr Valialkin
491b1762c8
app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue
2019-08-21 11:29:41 +03:00
Aliaksandr Valialkin
99331606e1
app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries
...
This should reduce memory allocations for big number of timeseries
2019-08-20 23:01:57 +03:00
Aliaksandr Valialkin
1101765adb
app/vmselect/promql: add label_value(q, label_name)
func, which returns numeric value labels with name label_name
in q
2019-08-20 00:28:44 +03:00
Aliaksandr Valialkin
940349ccb9
app/vmselect/promql: independently track offset hints for tStart and tEnd
...
This should improve performance if timeseries starts or ends on the selected time range
2019-08-19 13:40:24 +03:00
Aliaksandr Valialkin
6ae4b4190f
app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do
...
This should improve the performance of queries over big number of time series
with big number of output points.
2019-08-19 13:03:38 +03:00
Aliaksandr Valialkin
005aabd305
app/vmselect/promql: add scrape_interval(q[d])
function, which would return scrape interval for q
over d
2019-08-18 21:08:15 +03:00
Aliaksandr Valialkin
218cb4623a
app/vmselect/promql: hande comparisons with NaN
similar to Prometheus
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-08-18 00:25:58 +03:00
Aliaksandr Valialkin
dcce92c63c
app/vmselect/promql: add lifetime(q[d])
function, which returns the lifetime of q
over d
in seconds.
...
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
2019-08-16 11:59:51 +03:00
Aliaksandr Valialkin
0cb66a8f95
app/vmselect/promql: fix corner-case calculation for ideriv
2019-08-16 11:59:50 +03:00
Aliaksandr Valialkin
1b5b9ced27
app/vmselect/promql: properly handle corner cases for rollup functions
2019-08-15 23:31:28 +03:00
Aliaksandr Valialkin
b8bbe92de1
app/vmselect/promql: store compressed results in the cache
...
This should increase rollup results cache capacity.
2019-08-14 02:32:16 +03:00
Aliaksandr Valialkin
8c2158af24
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin
880b1d80b1
app/vmselect: optimize /api/v1/series
by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
000c154641
app/vmselect/promql: tune automatic window adjustement
...
Increase the windows adjustement for small scrape intervals,
since they usually have higher jitter.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
2019-08-04 19:34:11 +03:00
Aliaksandr Valialkin
1d4ddadbb1
app/vmselect/promql: further increase the allowed jitter for scrape interval
...
Real-world production data shows higher jitter than 1/8 of scrape interval.
This may results in gaps on the graph. So increase the allowed jitter to 1/4
of scrape interval in order to reduce the probability of gaps on the graphs
over time series with high jitter for scrape_interval.
2019-08-02 20:16:41 +03:00
Aliaksandr Valialkin
ade7bc30db
app/vmselect/promql: tolerate higher jitter in scrape interval
...
Allow jitter for up to 1/8 instead of 1/16 for the scrape interval.
This should imrpove graphs when `step` is smaller than the `scrape_interval`.
2019-08-01 23:25:53 +03:00
Aliaksandr Valialkin
c994fbf500
app/vmselect/promql: add vm_slow_queries_total
metric for counting slow queries
...
The query is slow if its execution time exceeds `-search.logSlowQueryDuration`
2019-07-31 03:36:45 +03:00
Aliaksandr Valialkin
071a122119
app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken
2019-07-31 01:18:11 +03:00
Aliaksandr Valialkin
b9a16b93e7
app/vmselect/promql: allow adjusting window for default rollup function
...
Default rollup function is `last_over_time`. It must support adjusting
the provided window in order to prevent from gaps on the graph
for window values smaller than scrape interval.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
2019-07-31 00:45:58 +03:00
Aliaksandr Valialkin
c901a6472f
app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
2019-07-30 22:05:55 +03:00
Aliaksandr Valialkin
aac482517f
app/vmselect/promql: return NaN from count()
over zero time series
...
This aligns `count` behavior with Prometheus.
2019-07-25 22:02:34 +03:00
Aliaksandr Valialkin
0e52357f35
app/vmselect/promql: properly calculate incremental aggregations grouped by __name__
...
Previously the following query may fail on multiple distinct metric names match:
sum(count_over_time{__name__!=''}) by (__name__)
2019-07-25 21:53:26 +03:00
Aliaksandr Valialkin
092c9b39a8
app/vmselect/promql: remove empty time series after applying filters like q > 0
...
This should reduce CPU and RAM usage for queries over high number of time series.
2019-07-12 19:59:49 +03:00
Aliaksandr Valialkin
6875fb411a
app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores
...
This may reduce response times for aggregation over big number of time series
with small step between output data points.
2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin
cbab86fd9d
app/vmselect/promql: reduce RAM usage for aggregates over big number of time series
...
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
2019-07-10 13:03:36 +03:00
Aliaksandr Valialkin
ba8195c58e
all: consistency renaming: bytesSize -> sizeBytes
2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
df6f17b82c
app/vmselect/promql: mention -search.logSlowQueryDuration
flag value in the slow query log message
2019-07-10 00:43:01 +03:00
Aliaksandr Valialkin
73ae889244
app/vmselect/promql: extract rmoeveGroupTags
function for removing unneeded tags from MetricName according to the given modifierExpr
2019-07-09 23:20:58 +03:00
Aliaksandr Valialkin
603b34edbd
app/vmselect/promql: properly preserve metric name after applying functions in any case from transformFuncsKeepMetricGroup
2019-07-09 23:10:49 +03:00
Aliaksandr Valialkin
bba07d05fe
app/vmselect/promql: remove empty timeseries left after topk
call
2019-07-04 19:43:07 +03:00
Aliaksandr Valialkin
858746fa6c
app/vmselect/promql: gracefully handle duplicate timestamps in irate
and rollup_rate
funcs
...
Previously such timestamps result in `+Inf` results. Now the previous timestamp is used
for the calculations.
2019-07-03 12:41:30 +03:00
Aliaksandr Valialkin
a3abed80ff
app/vmselect: do not return empty time series in /api/v1/query
result
2019-07-01 17:16:26 +03:00
Aliaksandr Valialkin
c3c60bee45
app/vmselect: add -search.denyPartialResponse
flag for disabling partial responses if some of vmstorage nodes are unavailable
...
Also accept `deny_partial_response` query arg in Prometheus API handlers. If it is set to true,
then return error if some of vmstorage nodes are unavailable.
2019-06-30 01:27:07 +03:00
Aliaksandr Valialkin
72a3050c41
app/vmselect/promql: consistency renaming: candlestick -> rollup_candlestick
2019-06-29 03:13:25 +03:00
Aliaksandr Valialkin
96342f1422
app/vmselect: add candlestick(m[d])
func for returning open
, close
, low
and high
rollups on the given time range d
...
This function is frequently used in financial apps. See https://en.wikipedia.org/wiki/Candlestick_chart
2019-06-27 18:46:54 +03:00
Aliaksandr Valialkin
4f54bcf90b
app/vmselect/promql: suppress error when template func is used inside modifier list. Just leave it as is
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/78
2019-06-25 20:43:57 +03:00
Aliaksandr Valialkin
d0bf4393a9
app/vmselect/promql: increase default value for -search.maxPointsPerTimeSeries from 10k to 30k
...
This may be required for subqueries with small steps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/77
2019-06-24 22:53:25 +03:00
Aliaksandr Valialkin
334cf253c7
app/vmselect/promql: adjust value returned by linearRegression to the end of time range like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
2019-06-24 22:46:03 +03:00
Aliaksandr Valialkin
14cd628948
app/vmselect/promql: add sum2
and sum2_over_time
, geomean
and geomean_over_time
funcs.
...
These functions may be useful for statistic calculations.
2019-06-24 16:45:00 +03:00
Aliaksandr Valialkin
0eac538fc8
app/vmselect/promql: adjust the provided window only for range functions with dt in denominator
...
This should fix range function calculations such as `changes(m[d])` where `d` is smaller
than the scrape interval.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
2019-06-23 19:27:25 +03:00
Aliaksandr Valialkin
ec57e59154
app/vmselect/promql: use deriv_fast
instead of deriv
in ttf
, since deriv
calculations have been changed recently
2019-06-23 15:54:12 +03:00
Aliaksandr Valialkin
516062b162
app/vmselect/promql: adjust ttf
calculation, so deriv(freev)
for freev
=m[d]
could be properly calculated
2019-06-23 14:31:36 +03:00
Aliaksandr Valialkin
a4e040f5ef
app/vmselect/promql: typo fixes in comments
2019-06-21 23:22:54 +03:00
Aliaksandr Valialkin
c05d443791
app/vmselect/promql: add deriv_fast
function for calculating fast derivative
...
`deriv_fast` calculates derivative based on the first and the last point on the interval
instead of calculating linear regression based on all the data points on the interval.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
2019-06-21 23:05:48 +03:00
Aliaksandr Valialkin
98eafdbd58
app/vmselect/promql: use linear regression in deriv
func like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
2019-06-21 22:54:34 +03:00
Aliaksandr Valialkin
f334908c22
app/vmselect/promql: ajdust data model to the model used in Prometheus
...
Do not take into account data points on the range `[timestamp .. timestamp+step)`
when calculating value on the given `timestamp`.
Use only data points from the past when performing these calculations like Prometheus does.
This should reduce discrepancies between results returned by VictoriaMetrics
and results returned by Prometheus.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
2019-06-21 21:55:25 +03:00
Aliaksandr Valialkin
837e349b7d
app/vmselect/promql: do not strip __name__
form time series after binary comparison operation
...
Example:
foo > 10
Would leave `foo` name for all the matching time series on the left.
2019-06-21 13:08:02 +03:00
Aliaksandr Valialkin
e84b7641ef
app/vmselect/promql: remove unused func keepLastValue
; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:35:19 +03:00
Aliaksandr Valialkin
db042bf6d6
app/vmselect/promql: typo fix; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:33:52 +03:00
Aliaksandr Valialkin
947bc16f8c
app/vmselect/promql: use dynamic limit on memory for concurrent queries
2019-06-12 23:18:23 +03:00
Aliaksandr Valialkin
8567e3463d
app/vmselect/promql: merge non-overlapping duplicate time series in group_left
and group_right
joins
2019-06-12 20:33:01 +03:00
Aliaksandr Valialkin
88005237f4
app/vmselect/promql: swap binary operation with modifier in the error message for improved readability
2019-06-12 17:14:33 +03:00
Aliaksandr Valialkin
a71381ad2a
app/vmselect/promql: list a sample of duplicate time series in the error message for group_left
or group_right
...
This should improve troubleshooting for complex queries involving `group_left` and `group_right` modifiers.
2019-06-12 16:57:34 +03:00
Aliaksandr Valialkin
eea7da8e0c
app/vmselect/promql: prevent from count_values
explosion of timeseries, which could result in OOM
2019-06-11 01:03:18 +03:00
Aliaksandr Valialkin
e87a602209
app/vmselect/promql: skip superflouos timestamps copying in count_values
2019-06-11 00:44:09 +03:00
Aliaksandr Valialkin
ec84febc1c
app/vmselect/promql: remove superflouos timeseries copy in histogram_quantile
func
2019-06-11 00:39:35 +03:00
Aliaksandr Valialkin
1fab34fb5c
app/vmselect/promql: remove superflouos timeseries copy in union
func
2019-06-11 00:35:09 +03:00
Aliaksandr Valialkin
a6f368499d
app/vmselect/promql: skip NaN values in count_values
func
2019-06-10 22:42:41 +03:00
Aliaksandr Valialkin
913f888d0c
app/vmselect/promql: properly handle {__name__ op "string"}
queries
...
This has been broken in 7294ef333ad26f4f6578b783e97649e58b1f8945 .
2019-06-07 02:02:09 +03:00
Aliaksandr Valialkin
8f4790625d
app/vmselect/promql: return the correct time series from quantile
...
Previously arbitrary time series could be returned from `quantile`
depending on sort order for the last data point in the selected range.
Fix this by returning the calculated time series.
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/55
2019-06-06 17:33:53 +03:00
Aliaksandr Valialkin
2ff0d595b0
app/vmselect/promql: add -search.disableCache
flag for disabling response caching
...
This may be useful for data back-filling, when the response caching
could interfere badly with newly added data points with timestamps
in the past.
2019-06-04 17:30:41 +03:00
Aliaksandr Valialkin
53242105fb
app/vmselect/promql: allow escaping identifiers with \
and \xXX
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/42
2019-05-31 17:35:54 +03:00
Aliaksandr Valialkin
a2c71f18a3
app/vmselect/promql: misspeling fix
2019-05-25 21:53:48 +03:00
Aliaksandr Valialkin
bdf696ef18
all: fix misspellings
2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
121a920a18
Makefile: add -s flag to go fmt
in make fmt
command
2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin
2ff996e276
app/vmselect: log slow queries if their execution time exceeds -search.logSlowQueryDuration
2019-05-24 16:14:46 +03:00
Aliaksandr Valialkin
8e3eb5b39d
app/vmselect/promql: add alias(q, name)
function that sets the given name to all the time series in q
2019-05-24 02:42:10 +03:00
Aliaksandr Valialkin
bb048937bc
app/vmselect/promql: add label_transform(q, label, regexp, replacement)
function for replacing all the occurences of regexp with replacement in the given label for q
2019-05-23 16:26:07 +03:00
Aliaksandr Valialkin
24578b4bb1
all: open-sourcing cluster version
2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00