github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
3649 commits 150 branches 748 tags 347 MiB
Commit graph

581 commits

Author SHA1 Message Date
Aliaksandr Valialkin
95688cbfc5 all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin
82973f8ae7 Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" 
This reverts commit bab6a15ae0.

Reason for revert: the `fetchData` arg is used in cluster branch.
Leaving this arg in master branch makes smaller the diff with cluster branch.
 2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin
bab6a15ae0 lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock 
This arg became unused after 23bdc1f107
 2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin
23bdc1f107 app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls 2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin
c584aece38 app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
 2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin
27500d7d4c app/vmselect/prometheus: code cleanup after 3ba507000c 2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin
3ba507000c app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin
bed25e3c24 app/vmselect/netstorage: properly pre-allocate space for sbs 2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin
09b0f7c202 app/vmselect/netstorage: release search resources on timeout errors 
Previously these resources weren't released, which could lead to resource leaks.
 2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin
3b1e3a03e0 app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin
00b5145c69 app/vmselect/searchutils: fixed tests after 2eb72e09ab 2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin
2eb72e09ab app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-21 21:24:40 +03:00
Aliaksandr Valialkin
98d1cd0971 app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API 2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin
9f79bcf64a app/vmselect: improve description for -search.maxQueryDuration 2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin
1fec47a289 app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series 2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin
8c3d7c1a59 app/vmselect: typo fix in -search.maxStalenessInterval description 2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin
0e533d1a9c app/vmselect/promql: support composite durations like Prometheus 2.21 does 
The following durations are supported now: `1h5m35s` or `1s543ms`

See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
 2020-09-11 23:39:13 +03:00
Aliaksandr Valialkin
a2f647d142 app/vmselect/prometheus: typo fix in the description for -search.latencyOffset command-line flag 2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin
f0005c3007 app/vmselect: move Deadline from netstorage to searchutils 
This removes dependency on netstorage from searchutils.
 2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin
2114179e19 app/vmselect: substitute inf values at smooth_exponential with the previous values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin
204ec415b4 app/vmselect: skip infinite values when calculating smooth_exponential 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin
8a8b5a73d3 app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric 2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin
f6bc608e86 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
62919eaf7e app/vmselect/promql: go fmt 2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin
e6da63dffe app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin
8e85b56737 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin
c0343a661b app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3 2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin
f4e7e5fb90 app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions 
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
 2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin
e706e59d49 app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does 2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin
147c35ebd4 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin
7d89fafe1a app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3) 2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin
e969ef2639 app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin
8adba82c02 app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores 
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
 2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin
f3d33e23c9 app/vmstorage: improve error logging when the request times out 2020-08-10 13:23:26 +03:00
Aliaksandr Valialkin
716754fae6 app/vmselect/promql: properly handle -n^m like Prometheus does 
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
 2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin
bb61a4769b app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin
ac45082216 app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin
e5202a4eae app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus 
Rollup functions:

  - avg_over_time
  - min_over_time
  - max_over_time
  - quantile_over_time

This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin
68e4f40a72 app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does 
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin
bc8381613d app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names 2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
3f85c06b65 app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block 
This should improve query performance on systems with big number of CPU cores (16 and more)
 2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin
d20c2156e4 app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans 2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin
5d0c37bec0 app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration 2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin
e6eee2bebf app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...) 2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin
509d12643b app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin
f0c678c41b app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
 2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin
328b52e5ff app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window 
Just divide the data point value by the window duration in this case.
 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
700737c181 app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now 
See e00cfc854d for the approach used now.
 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
2f735f112d app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
79c30cf4cb app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query 2020-07-28 15:13:08 +03:00
Aliaksandr Valialkin
0da202023b app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp 
This aligns `group()` behaviour to Prometheus
 2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin
894dcb7c1c app/vmselect/promql: improve further the accuracy of buckets_limit() function 
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
 2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin
edb1eca6f1 app/vmselect/promql: avoid dropping inf bucket in buckets_limit 
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
 2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin
97b6f5d223 app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets 2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin
bb161497cf app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function 
Now it properly merges the bucket with the previous bucket after deletion.
 2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin
994fa2f3bf app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k 
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
 2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin
e151c5c644 app/vmselect: fix tests for rate_over_sum 2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin
3107c633e3 app/vmselect/promql: typo fix after 3e557c9861 2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin
3e557c9861 app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration 
Something like `sum_over_time(m[d]) / d`, but more accurate.
 2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin
b1f6843bd0 app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time 
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
 2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin
039c9d2441 lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
461481fbdf app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery 2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin
e79de9774b app/vmselect: typo fix after 34563916f7 2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin
34563916f7 app/vmselect: reduce memory usage when querying big number of time series with long labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
 2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin
4f526cc816 app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
 2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin
dfb113f175 app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries 
This should slightly reduce load on GC when processing queries that touch big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.
 2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin
caa2952aa6 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin
e00cfc854d app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin
b9c8f6bf34 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin
5a4675c528 app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range 
This matches Prometheus behaviour.

This should fix https://github.com/jacksontj/promxy/issues/329
 2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin
ecb1b2564a app/vmselect/promql: add mode() aggregate function 2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin
b35cb293f5 lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
1c641037e8 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin
6b5ad535ae app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin
8949d65ad1 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin
aa5d88055d app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin
df01836818 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin
e4e1cd1de2 app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage 2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin
25f80d320b app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time 
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
 2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin
55d83e777d app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9 2020-07-13 21:40:38 +03:00
faceair
1033dc7e2a
fix empty response template (#617) 2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin
9c350bc20d app/vmselect/promql: add missing tests for ifnot binary operation 2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin
256fd9a87e app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation 2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin
2d9b3ad5b3 app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function 2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin
742da690f4 app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575
 2020-07-08 18:55:38 +03:00
DexterZhang
99f54e44ff
feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598) 2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin
acf828a759 app/vmselect/prometheus: small fixes on top of 8bb762124a 2020-07-05 18:17:06 +03:00
faceair
8bb762124a
fix adjust last points avoid influence earlier value (#606) 2020-07-05 17:56:54 +03:00
Aliaksandr Valialkin
5d3db3ff7c app/vmselect: add interpolate function for filling gaps with linearly interpolated values 
See https://stackoverflow.com/q/62565021/274937 for details
 2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin
84a37098ed app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 
VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention
if `-denyQueriesOutsideRetention` command-line flag is set.
 2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin
7c2c8b2981 all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode 2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
05d754d7bb app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series 
This should reduce GC pressure when processing time series with big number of rows
 2020-06-24 19:38:39 +03:00
nicbaz
774f7ca1c1
vmselect: fix label_replace when mismatch (#579) 
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".

Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.

Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
 2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin
7209d58fbd app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series 
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
 2020-06-23 22:46:15 +03:00
Aliaksandr Valialkin
7532dbcdf5 app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin
88e1b7d144 app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
 2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin
dbd8beccfa app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions 
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
 2020-05-21 12:07:48 +03:00
Aliaksandr Valialkin
70ea4e28a7 app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3 2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin
538fdfe133 app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries 2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin
f52769f6ee app/vmselect/promql: fix outilersk calculations 2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin
a441cdd1d9 app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series 2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin
0afd48d2ee lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
7882a0dbbf app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint 2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin
faf92a0965 app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one 2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin
cc311e20fe app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y 2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin
574289c3fb app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation 2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin
57407cca83 app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag 
Limit the estimated time series count after aggregation with grouping by the number of source time series.
 2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin
4e4f57b121 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
17d96e4503 app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs vs false-positive not enough memory errors 2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin
b4afe562c1 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
fcf57f9883 app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks 
This should minimize the number of disk seeks when reading data from temporary file.
 2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin
a596aec82c app/vmselect: fix description for -search.resetCacheAuthKey 2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin
364db13c9c app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
9ebc937685 app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin
fe57d46687 app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( 
Leave both flags for backwards compatibility reasons.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin
f41e6a7bd9 app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query 2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin
071fdf5518 lib/logger: add WARN level for logging expected errors such as invalid user queries 2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin
6f7f64f757 app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
 2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin
f9b24d4899 app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions 2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin
318326c309 lib/storage: properly handle {label=~"foo|"} filters as Prometheus does 
Such filters must match all the time series with `label="foo"` plus all the time series without `label`

Previously only time series with `label="foo"` were matched.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
0c485f14d1 app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries 2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin
2ebf7d86ff app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds() 2020-03-29 21:50:28 +03:00
Aliaksandr Valialkin
3c7c71a49c app/vmselect: adjust label_map() handling for corner cases 
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series

All the unmatched labels are kept unchanged.
 2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin
fdc2a9d1d7 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin
8023ad7dbd app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model 2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin
e17702fada app/vmselect: add optional max_rows_per_line query arg to /api/v1/export 
This arg allows limiting the number of data points that may be exported on a single line.
 2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin
c5f894b361 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin
7d178a40bd app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter 
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin
5701b2f7bb app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin
18af31a4c2 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
ae51300973 app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin
e65ec88779 app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-23 18:03:57 +02:00
Yaroslav
a6d0645539
fix rollupOpen(), rollupHigh(), rollupLow() functions (#328) 2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin
2f55cabaa4 app/vmselect/promql: log when rollupResult cache is cleared 2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin
ec9bf39b5b app/vmselect: add -search.cacheTimestampOffset command-line flag 
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.

This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin
539139391c app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 13:58:05 +02:00
Aliaksandr Valialkin
7b1c7051a3 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin
a6c6a2debc app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8 2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin
78b62dee87 app/vmselect/promql: properly adjust time range for data to select 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin
366693b9f1 app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent 2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin
525101339e app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin
ada6a3da8d app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin
36a1ac8360 app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded 
This will prevent from excess CPU usage for timed out queries.
 2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin
834051e5b2 app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana 
See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata

VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.
 2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin
42864bb52f all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin
90cf356ea1 app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus 2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin
c3d86eef96 all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
d70ba7eb37 lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor 
This should improve performance when reading many small blocks.
 2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin
a462355b2f app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin
adbbc4fa1a app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value 
Examples for such counters: OS-level counters for network or cpu stats.
 2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin
2392a359e1 app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
 2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin
680080887d all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
9773022e50 app/vmselect: mention the original query and time range in error messages 
This should simplify debugging invalid or heavy queries.
 2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin
efc7ad88ec app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin
ec9651e266 app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data 2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin
582dd01f42 app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin
36973ee975 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin
cdcacaea6d app/vmselect/netstorage: make fmt 2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin
7327adbc86 app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file 
This should reduce memory usage on systems with more than 8GB RAM.
 2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin
d0d258b314 app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs 
A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value.
This will give more chances for executing insert path.
 2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin
d88725f133 app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects 2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin
0d7505b00e all: mention command-line flags used for limiting the incoming request size in error messages 
This should improve error logs usability.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
 2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin
2839f4688a app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args 2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin
8b14572f70 app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin
8eaced8cae app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval 2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin
46bfdbe6cf app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions 
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
 2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin
b829fe5e39 app/vmselect/promql: properly handle aggr(aggr_over_time(...)) 2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin
164278151f app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function 
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
 2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin
c4632faa9d app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions 
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
 2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin
d5aeda0e1a app/vmselect/promql: skip rate calculation for the first point on time series 2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin
d410c78c7e app/vmselect/promql: fix calculations for histogram_share 2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin
d64699bb9f app/vmselect/promql: add missing MetricName into netstorage.Result in tests 2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin
f409f2d050 app/vmselect/promql: add histogram_share(le, buckets) function 2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin
b1ded7cf9a app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 
See https://github.com/prometheus/prometheus/issues/2882
 2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0 app/vmselect/promql: add histogram_over_time(m[d]) rollup function 2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
3e09d38f29 app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time 
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
 2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
a774120460 app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series 
This should provide better estimation for `rate` in the beginning of time series.
 2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin
b5645ccbdf app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series 2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin
cb3a342882 app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations 2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
6f67e0b56b lib/metricsq: add ExpandWithExprs 2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d Rename lib/promql to lib/metricsql and apply small fixes 2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43 Split Extended PromQL parsing to a separate library 2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
d880325cf6 app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range 
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:

    BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
 2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin
9e19949c6b app/vmselect/promql: adjust calculations for rate and increase for the first value 
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
 2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin
0455c03cb9 app/vmselect/promql: properly calculate rate on the first data point 
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
 2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin
5b75984aa9 app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs 2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin
bea5a8700a app/vmselect: add -search.maxExportDuration command-line flag for limiting /api/v1/export duration 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275
 2019-12-20 11:35:22 +02:00
Aliaksandr Valialkin
96ff8d9adb app/vmselect: add ability to pass match[], start and end to /api/v1/labels 
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.

See https://github.com/prometheus/prometheus/issues/6178 for more details.
 2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin
cea5a14853 all: rename Extended PromQL to PromQL extensions 2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin
73b2a3d4b7 app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed 
Updates https://github.com/prometheus/prometheus/issues/5706
 2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin
caa4eb72d9 app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does 2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin
3c076544bf app/vmselect/promql: allow negative offsets 
Updates https://github.com/prometheus/prometheus/issues/6282
 2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin
c09472dfd9 app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range 
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
 2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin
face3d57bf app/vmselect: add placeholders for /api/v1/rules and /api/v1/alerts 2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin
20812008a7 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
9476b73527 app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals 
This should improve scrape interval estimation for tiem series with gaps.
 2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin
75ff524a4e app/vmselect/promql: fix corner case for increase over time series with gaps 
In this case `increase` could return invalid high value for the first point after the gap.
 2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin
4c859d980c app/vmselect/prometheus: consistently apply nocache arg to /api/v1/query the same way ast to /api/v1/query_range 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
 2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin
110235f789 app/vmselect/prometheus: fix content-type for /api/v1/export responses 
The correct Content-Type should be `application/stream+json` instead of `application/json`
Thanks to Joshua Ryder for pointing to this.
 2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin
205233d9a7 app/vmselect/promql: remove zero timeseries from prometheus_buckets output 2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin
3f99f39e9b app/vmselect/prometheus: reduce default value for -search.latencyOffset from 60s to 30s 
30 seconds should be enough for almost all the cases
 2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin
e91cb34c0e app/vmselect/promql: allow nested parens 2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin
826dfd63a5 vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1 2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin
0401969d78 app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph 
This should reduce the amounts memory allocations
 2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin
da98703748 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
c28876172f app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0 2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin
50ae1879c6 app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series 2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin
8582b50360 app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets 2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin
19dfe52254 app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets 2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin
4bb88843cf app/vmselect: add vm_per_query_{rows,series}_processed_count histograms 2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
7753c8c0a1 app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile 2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin
c4287b3c86 app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets 2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin
1f3fd2c910 app/vmselect: adjust end arg instead of adjusting start arg if start > end 
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
 2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin
7a4635f853 all: remove the remaining mentions of cluster version 2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
0094bc4fc9 app/vmselect/prometheus: properly adjust too big time time on /api/v1/query 
Too big `time` must be adjusted to `now()-queryOffset`.
 2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin
3f1637fae8 app/vmselect/promql: properly calculate integrate(q[d]) 2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin
3fd32e331a app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
 2019-11-13 20:24:26 +02:00
Oleg Kovalov
b4f44befa3 fix misspelled words (#229) 2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
63b05c0b9f app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions 
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.

Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.

This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
 2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
2f58f37f07 app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q 2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin
b0295dbf2e app/vmselect: add -search.latencyOffset flag for tuning the time after data collection when data points become visible in query results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
 2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin
97ce4e03a5 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
c66a691593 app/vmselect/prometheus: add -search.maxLookback command-line flag for overriding dynamic calculations for max lookback interval 
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.

The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
 2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
956fdd89d3 app/vmselect/promql: take into account the previous point when calculating max_over_time and min_over_time 
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
 2019-10-08 12:30:05 +03:00
Aliaksandr Valialkin
4e9e1ca0f7 app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially 
This became the case after b7ee2e7af2 .
 2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin
c1d3705be0 app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock 
This allows re-using the destination buffer for marshaling in the outer loop.
 2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin
b7ee2e7af2 app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series 2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin
67d44b0845 app/vmselect/promql: do not generate timestamps for NaN values in timestamp function according to Prometheus logic 2019-09-27 18:54:43 +03:00
Aliaksandr Valialkin
26dc21cf64 app/vmselect/promql: add increases_over_time and decreases_over_time functions 
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
 2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin
d4218d27e6 app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step])) 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
 2019-09-13 21:41:04 +03:00
Aliaksandr Valialkin
4d636c244d app/vmselect/promql: binary operation fixes according to Prometheus behaviour 
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
 2019-09-13 17:42:52 +03:00
Aliaksandr Valialkin
6954e126fc app/vmselect/promql: ignore grouping by destination label in count_values, since such a grouping is performed automatically 2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin
305e7bc981 app/vmselect/promql: do not return artificial points beyond the last point in time series 2019-09-04 16:35:34 +03:00
Aliaksandr Valialkin
9721d06c6a app/vmselect/prometheus: do not adjust start and end args in /api/v1/query_range if nocache=1 arg is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171
 2019-09-04 13:10:09 +03:00
Aliaksandr Valialkin
db4560ca31 app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does 2019-09-03 20:42:54 +03:00
Aliaksandr Valialkin
1575a560f0 app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks 
The maximum inmemory file size now depends on `-memory.allowedPercent`.
This should improve performance and reduce the number of filesystem calls
on machines with big amounts of RAM when performing heavy queries
over big number of samples and time series.
 2019-09-03 13:32:09 +03:00
Aliaksandr Valialkin
9196c085a7 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
603a79b357 app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries 2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin
ac004665b5 all: return 503 http error if service is temporarily unavailable 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
 2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin
1272e407b2 app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
 2019-08-22 14:39:46 +03:00
Aliaksandr Valialkin
f4a38f7fb1 app/vmselect/promql: fix panic on -search.disableCache 
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
 2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin
ab740afd0d app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue 2019-08-21 11:29:24 +03:00
Aliaksandr Valialkin
7b5168adfb app/vmselect/promql: remove NaNs from /api/v1/query_range output like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
 2019-08-20 23:01:41 +03:00
Aliaksandr Valialkin
a0d480fbf3 app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries 
This should reduce memory allocations for big number of timeseries
 2019-08-20 23:01:39 +03:00
Aliaksandr Valialkin
d3fd113a80 app/vmselect/promql: add label_value(q, label_name) func, which returns numeric value labels with name label_name in q 2019-08-20 00:28:34 +03:00
Aliaksandr Valialkin
dd86e6130c app/vmselect/promql: independently track offset hints for tStart and tEnd 
This should improve performance if timeseries starts or ends on the selected time range
 2019-08-19 13:40:14 +03:00
Aliaksandr Valialkin
6a27657d73 app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do 
This should improve the performance of queries over big number of time series
with big number of output points.
 2019-08-19 13:03:29 +03:00
Aliaksandr Valialkin
e74fb23189 app/vmselect/promql: add scrape_interval(q[d]) function, which would return scrape interval for q over d 2019-08-18 21:08:26 +03:00
Aliaksandr Valialkin
582fdc059a app/vmselect/promql: hande comparisons with NaN similar to Prometheus 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
 2019-08-18 00:25:50 +03:00
Aliaksandr Valialkin
1c108fc494 app/vmselect/promql: add lifetime(q[d]) function, which returns the lifetime of q over d in seconds. 
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
 2019-08-16 11:59:32 +03:00
Aliaksandr Valialkin
d6b5ed6d39 app/vmselect/promql: fix corner-case calculation for ideriv 2019-08-16 11:59:28 +03:00
Aliaksandr Valialkin
639b14e8ab app/vmselect/promql: properly handle corner cases for rollup functions 2019-08-15 23:29:59 +03:00
Aliaksandr Valialkin
9ecb994671 app/vmselect/promql: store compressed results in the cache 
This should increase rollup results cache capacity.
 2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin
09fc6e22e5 all: use workingsetcache instead of fastcache 
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.

The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
 2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin
5c83f8e203 app: add vm_concurrent_ metrics for visibility in concurrency limiters for vminsert and vmselect 2019-08-05 18:30:57 +03:00
Aliaksandr Valialkin
99e048c9df app/vmselect: allow passing match[], start and time to /api/v1/label/<label_name>/values 
`/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)`
call in Grafana templating.
 2019-08-04 23:09:21 +03:00
Aliaksandr Valialkin
47e4b50112 app/vmselect: optimize /api/v1/series by skipping storage data 
Fetch and process only time series metainfo.
 2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin
241170dc05 app/vmselect/prometheus: prevent from fetching and scanning all the data on /api/v1/searies call by default 2019-08-04 19:42:36 +03:00
Aliaksandr Valialkin
1c69f4eadc app/vmselect/promql: tune automatic window adjustement 
Increase the windows adjustement for small scrape intervals,
since they usually have higher jitter.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
 2019-08-04 19:34:05 +03:00
Aliaksandr Valialkin
8d93b15b86 app/vmselect/promql: further increase the allowed jitter for scrape interval 
Real-world production data shows higher jitter than 1/8 of scrape interval.
This may results in gaps on the graph. So increase the allowed jitter to 1/4
of scrape interval in order to reduce the probability of gaps on the graphs
over time series with high jitter for scrape_interval.
 2019-08-02 20:10:23 +03:00
Aliaksandr Valialkin
f090b2e917 app/vmselect/promql: tolerate higher jitter in scrape interval 
Allow jitter for up to 1/8 instead of 1/16 for the scrape interval.
This should imrpove graphs when `step` is smaller than the `scrape_interval`.
 2019-08-01 23:26:00 +03:00
Aliaksandr Valialkin
3a38b23fa3 app/vmselect/promql: add vm_slow_queries_total metric for counting slow queries 
The query is slow if its execution time exceeds `-search.logSlowQueryDuration`
 2019-07-31 03:36:37 +03:00
Aliaksandr Valialkin
8bd6f1f6df app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken 2019-07-31 01:18:07 +03:00
Aliaksandr Valialkin
4aaa5c2efc app/vmselect/promql: allow adjusting window for default rollup function 
Default rollup function is `last_over_time`. It must support adjusting
the provided window in order to prevent from gaps on the graph
for window values smaller than scrape interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134
 2019-07-31 00:45:54 +03:00
Aliaksandr Valialkin
10f5a26bec app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
 2019-07-30 22:05:10 +03:00
Aliaksandr Valialkin
afcc7fb167 app/vmselect/netstorage: improve error message when reading data blocks from storage 
Mention the block number in the error. This should simplify troubleshooting in this code.
 2019-07-28 12:12:35 +03:00
Aliaksandr Valialkin
17096b5750 app/vmselect/promql: return NaN from count() over zero time series 
This aligns `count` behavior with Prometheus.
 2019-07-25 22:02:30 +03:00
Aliaksandr Valialkin
66efa5745f app/vmselect/promql: properly calculate incremental aggregations grouped by __name__ 
Previously the following query may fail on multiple distinct metric names match:

    sum(count_over_time{__name__!=''}) by (__name__)
 2019-07-25 21:53:20 +03:00
Aliaksandr Valialkin
6a943a6a58 app/vmselect/promql: remove empty time series after applying filters like q > 0 
This should reduce CPU and RAM usage for queries over high number of time series.
 2019-07-12 19:59:27 +03:00
Aliaksandr Valialkin
ab88890523 app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores 
This may reduce response times for aggregation over big number of time series
with small step between output data points.
 2019-07-12 15:52:22 +03:00
Aliaksandr Valialkin
62a7353479 app/vmselect/prometheus: set start arg in /api/v1/series to the minimum allowed time by default as Prometheus does 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
 2019-07-11 17:10:14 +03:00
Aliaksandr Valialkin
54bd21eb4a app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage 2019-07-11 17:07:20 +03:00
Aliaksandr Valialkin
a336bb4e22 app/vmselect/promql: reduce RAM usage for aggregates over big number of time series 
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
 2019-07-10 13:04:39 +03:00
Aliaksandr Valialkin
1fe6d784d8 all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin
55fe36149c app/vmselect/promql: mention -search.logSlowQueryDuration flag value in the slow query log message 2019-07-10 00:41:24 +03:00
Aliaksandr Valialkin
9203170eb2 app/vmselect/promql: extract rmoeveGroupTags function for removing unneeded tags from MetricName according to the given modifierExpr 2019-07-09 23:20:48 +03:00
Aliaksandr Valialkin
2db685c19c app/vmselect/promql: properly preserve metric name after applying functions in any case from transformFuncsKeepMetricGroup 2019-07-09 23:10:35 +03:00
Aliaksandr Valialkin
40a6c0d672 app/vmselect/prometheus: typo fix 2019-07-07 23:34:23 +03:00
Aliaksandr Valialkin
1371024747 app/vmselect/prometheus: handle minTime and maxTime values that may be set by Promxy or Prometheus client 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/88
 2019-07-07 21:53:48 +03:00
Aliaksandr Valialkin
4dbd642c86 app/vmselect/promql: remove empty timeseries left after topk call 2019-07-04 19:42:39 +03:00
Aliaksandr Valialkin
9a4b2b8315 app/vmselect/prometheus: update adjustLastPoints function 
- Do not overwrite last points by the previous NaNs, since this may result in empty time series.
- Overwrite the last 2 points instead of 3. This should be enough in most cases.
 2019-07-04 09:14:18 +03:00
Aliaksandr Valialkin
e06866005d app/vmselect/promql: gracefully handle duplicate timestamps in irate and rollup_rate funcs 
Previously such timestamps result in `+Inf` results. Now the previous timestamp is used
for the calculations.
 2019-07-03 12:39:55 +03:00
Aliaksandr Valialkin
b9166a60ff app/vmselect: do not return empty time series in /api/v1/query result 2019-07-01 17:16:34 +03:00
Aliaksandr Valialkin
74c0fb04f3 app/vmselect/promql: consistency renaming: candlestick -> rollup_candlestick 2019-06-29 03:13:02 +03:00
Aliaksandr Valialkin
55bcf60ea6 app/vmselect: fix 32bit arm build 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/83
 2019-06-27 19:36:17 +03:00
Aliaksandr Valialkin
796b010139 app/vmselect: add candlestick(m[d]) func for returning open, close, low and high rollups on the given time range d 
This function is frequently used in financial apps. See https://en.wikipedia.org/wiki/Candlestick_chart
 2019-06-27 18:46:13 +03:00
Aliaksandr Valialkin
844ce4731e app/vmselect/promql: suppress error when template func is used inside modifier list. Just leave it as is 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/78
 2019-06-25 20:43:22 +03:00
Aliaksandr Valialkin
feacfffe89 app/vmselect/promql: increase default value for -search.maxPointsPerTimeSeries from 10k to 30k 
This may be required for subqueries with small steps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/77
 2019-06-24 22:53:18 +03:00
Aliaksandr Valialkin
4bb738ddd9 app/vmselect/promql: adjust value returned by linearRegression to the end of time range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
 2019-06-24 22:45:58 +03:00
Aliaksandr Valialkin
90e72c2a42 app/vmselect/promql: add sum2 and sum2_over_time, geomean and geomean_over_time funcs. 
These functions may be useful for statistic calculations.
 2019-06-24 16:44:44 +03:00
Aliaksandr Valialkin
61926bae01 app/vmselect/promql: adjust the provided window only for range functions with dt in denominator 
This should fix range function calculations such as `changes(m[d])` where `d` is smaller
than the scrape interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
 2019-06-23 19:27:31 +03:00
Aliaksandr Valialkin
ee13256f74 app/vmselect/promql: use deriv_fast instead of deriv in ttf, since deriv calculations have been changed recently 2019-06-23 15:54:18 +03:00
Aliaksandr Valialkin
3b3b2f1e6e app/vmselect/promql: adjust ttf calculation, so deriv(freev) for freev=m[d] could be properly calculated 2019-06-23 14:31:19 +03:00
Aliaksandr Valialkin
146c6e1f72 app/vmselect/promql: typo fixes in comments 2019-06-21 23:22:59 +03:00
Aliaksandr Valialkin
d261fa2885 app/vmselect/promql: add deriv_fast function for calculating fast derivative 
`deriv_fast` calculates derivative based on the first and the last point on the interval
instead of calculating linear regression based on all the data points on the interval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
 2019-06-21 23:05:39 +03:00
Aliaksandr Valialkin
5b47c00910 app/vmselect/promql: use linear regression in deriv func like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73
 2019-06-21 22:59:46 +03:00
Aliaksandr Valialkin
9e1119dab8 app/vmselect/promql: ajdust data model to the model used in Prometheus 
Do not take into account data points on the range `[timestamp .. timestamp+step)`
when calculating value on the given `timestamp`.
Use only data points from the past when performing these calculations like Prometheus does.

This should reduce discrepancies between results returned by VictoriaMetrics
and results returned by Prometheus.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71
 2019-06-21 21:54:48 +03:00
Aliaksandr Valialkin
47a3228108 app/vmselect/promql: do not strip __name__ form time series after binary comparison operation 
Example:

  foo > 10

Would leave `foo` name for all the matching time series on the left.
 2019-06-21 13:09:38 +03:00