Yury Molodov
6d1d558c4f
vmui: fix graph reset ( #1788 )
...
* feat: add query history
* fix: change detect keyUp for nav query history
* feat: set default query history
* feat: change graph legend
* update dependencies
* update codemirror version
* fix: correct update period time after zoom/pan
* fix: optimize data processing for the graph
* fix: eliminate memory leaks related to mouse events
* fix: correct display of straight line
* Merge branch 'master' into vmui-fix-reset-graph
* app/vmselect: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-11-05 20:56:57 +02:00
Aliaksandr Valialkin
2178335618
app/vmselect: make vmui-update
2021-11-04 12:13:12 +02:00
Aliaksandr Valialkin
27044b84d2
app/vmselect/promql: add limit_offset(limit, offset, q) function, which can be used for paging over big number of time series
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1778
2021-11-03 16:02:27 +02:00
Aliaksandr Valialkin
43a58bd618
app/vmselect/promql: add label_graphite_group() function for extracting groups from Graphite metric names
2021-11-03 13:19:08 +02:00
Yury Molodov
47ee3744f2
vmui: correct migration material-ui ( #1758 )
...
* migration material-ui
* fix: rollback popover
* app/vmselect: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-27 20:41:26 +03:00
Aliaksandr Valialkin
1952ab99aa
all: fix build issues and tests for Apple M1
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653
2021-10-27 15:06:34 +03:00
Yury Molodov
956cf83e7b
vmui: update dependencies ( #1754 )
...
* update dependencies
* update codemirror version
* app/vmui: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-26 19:31:20 +03:00
Aliaksandr Valialkin
df8f967040
app/vmselect/promql: reduce the precision from 15 significant digits to 13 significant digits when comparing float64 results in tests
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1738
2021-10-24 13:31:14 +03:00
Yury Molodov
2b266cb87e
vmui: query history ( #1732 )
...
* feat: add query history
* fix: change detect keyUp for nav query history
* feat: set default query history
* app/vmselect/vmui: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-10-22 12:21:22 +03:00
Yury Molodov
a3e09a57c2
vmui: features ( #1711 )
...
* feat: initial uPlot graph
* feat: add zoom/pan for graph
* fix: add zoom by ctrl/mac
* fix: remove unused code
* feat: add toggle cache for fetch
* feat: add fix y-axis limits
* fix: stop point events while panning
* fix: change getting cursor position when scaling
* feat: add cursor tooltip to graph
* fix: uninstall chart.js
* fix: change link for create an issue
* fix: set default cache value to true
* app/vmalert: follow-up after 0e2486df565416e18007
2021-10-18 15:16:57 +03:00
Aliaksandr Valialkin
da97e58979
app/vmselect/promql: randomize the static selection of time series returned from limitk()
...
Sort series by a hash calculated from the series labels. This should guarantee "random" selection of the returned time series.
Previously the selection could be biased, since time series were sorted alphabetically by label names and label values.
2021-10-16 21:16:49 +03:00
Aliaksandr Valialkin
cae174b11c
app/vmselect/promql: typo fix in comment: didsn't -> didn't
2021-10-16 13:00:34 +03:00
Aliaksandr Valialkin
a5001b9c20
app/vmselect/promql: add atan2 binary operator, which is going to be added in Prometheus 2.31
...
See https://github.com/prometheus/prometheus/pull/9248
2021-10-11 21:15:53 +03:00
Aliaksandr Valialkin
81c6720392
app/vmselect/promql: add missing trigonometric functions, which are going to be added in Prometheus 2.31
...
See https://github.com/prometheus/prometheus/issues/9233
2021-10-11 21:01:33 +03:00
Aliaksandr Valialkin
92b92d4d2c
app/vmselect/promql: consistently return the same set of time series from limitk() function
...
This is the expected behaviour by most users.
2021-10-08 19:53:52 +03:00
Aliaksandr Valialkin
0ff8fcac6a
app/vmui: follow-up after 7bfb44113e
...
* Run `vmui-update`
* Document the changes in README.md and CHANGELOG.md
2021-10-08 15:09:29 +03:00
Aliaksandr Valialkin
2748255c8b
app/vmselect/promql: substitute rollupFuncsCannotAdjustWindow with rollupFuncsCanAdjustWindow
...
The list of functions, which can adjust lookbehind window is more limited than the rest of functions,
so it is better from maintainability and readability PoV using the allowlist instead of blocklist.
2021-10-07 13:18:42 +03:00
Aliaksandr Valialkin
c45210a6f9
app/vmselect/promql: return back the behaviour for deriv() function when the lookbehind window doesnt contain enough points
...
It is expected that the `deriv(m[d])` returns non-empty value if the lookbehind window `d`
contains less than 2 samples in the same way as `rate()` does.
This is a follow-up after 3e084be06b
2021-10-07 12:52:27 +03:00
Roman Khavronenko
3e084be06b
app/vmselect: make predict_linear and deriv compatible with Prometheus ( #1681 )
...
Previously, `predict_linear` returned slightly different results comparing
to Prometheus. The change makes linear regression algorithm compatible
with Prometheus.
`deriv` was excluded from the list of functions which can adjust the time
window for the same reasons.
2021-10-07 12:50:49 +03:00
Aliaksandr Valialkin
0e3de5a0cc
app/vmselect/promql: add topk_last and bottomk_last functions
2021-09-30 13:22:52 +03:00
Roman Khavronenko
a31407006c
app/vmselect: fix binary comparison func ( #1667 )
...
The fix makes the binary comparison func to check for NaNs
before executing the actual comparison. This prevents VM
to return values for non-existing samples for expressions
which contain bool comparisons. Please see added test
for example.
2021-09-30 12:24:17 +03:00
Roman Khavronenko
344490d89b
app/vmselect: fix testRowsEqual func NaN checks ( #1666 )
...
It appeared, that `testRowsEqual` NaN comparison was incorrect.
The fix caused some tests to fail. Please see the change and
tests updated.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-30 12:08:47 +03:00
Yury Molodov
893af0a92c
vmui: fixed bug with time range (time zone) ( #1661 )
...
* fix: set date in query string in utc format
* app/vmselect: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-29 01:00:44 +03:00
Roman Khavronenko
5dc84bf210
app/vmselect: disable time-window adjustment for min/max_over_time funcs ( #1658 )
...
Adjustment results into discrepancy between Prometheus and VM on time windows
smaller than scrape interval.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:43:21 +03:00
Roman Khavronenko
de810031bf
app/vmselect: always return zero for stddev func if there is only one value ( #1659 )
...
The fix will always return zero if received set of items consists of one
element only, which also means no deviation.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:38:55 +03:00
Roman Khavronenko
dd536b475c
app/vmselect: return NaN instead of 0 for empty value sets ( #1660 )
...
The change affects `count/stddev/stdvar_over_time` funcs and makes
them to return NaN instead of zero when there is no datapoints
in a time window.
This is needed for improving compatibility with Prometheus.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:37:04 +03:00
Roman Khavronenko
03cd93bf1a
app/vmselect: rm quantile_over_time fast-path optimisations ( #1662 )
...
The removed fast path optimisations weren't consistent with
`quantile` function behavior and results into discrepancy.
Specifically, results didn't match in cases when:
* 0 < phi > 1;
* values contain only one element.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-29 00:35:14 +03:00
Yury Molodov
a64155d91e
vmui: use Chart.js as default engine for graph ( #1634 )
...
* feat: add Plotly as default engine for graph
* fix: remove unused components
* feat: use Chart.js as default engine graph
* fix: correct styles for loader
* feat: add zoom/pan for chart
* feat: add height for chart
* fix: remove unused code
* fix: remove empty units from duration
* fix: change debounce for pan to 500ms
* fix: add utility for plugins register globally
* fix: optimize render graph
* feat: add buffer data for zoom
* fix: add limits for zoom in/out
* fix: change update data while zooming
* app/vmselect: `make vmui-update`
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-27 22:26:14 +03:00
Aliaksandr Valialkin
2efe0acfc9
app/vmselect/promql: add rollup_scrape_interval(m[d]) function
...
It calculates the min, max and avg scrape intervals for m over the given lookbehind window d
2021-09-27 19:21:24 +03:00
Aliaksandr Valialkin
c4c77aa2dd
app/vmselect/promql: follow-up after 526dd93b32
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
2021-09-27 18:55:39 +03:00
Roman Khavronenko
526dd93b32
app/vmselect: quantile func compatiblity with Prometheus ( #1646 )
...
* app/vmselect: `quantile` func compatiblity with Prometheus
The `quantile` func was previously calculated by https://github.com/valyala/histogram
package. The result of such calculation was always the closest real value to
requested quantile. While in Prometheus implementation interpolation is used.
Such difference may result into discrepancy in output between Prometheus and
VictoriaMetrics.
This commit adds a Prometheus-like `quantile` function. It also used by other
functions which depend on it, such as `quantiles`, `quantile_over_time`, `median` etc.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* app/vmselect: `quantile` review fixes
* quantile functions were split into multiple to provide
different API for already sorted data;
* float64sPool is used for reducing allocations. Items in pool may have
different sizes, but defining a new pool was complicates due to name collisions;
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2021-09-27 18:02:41 +03:00
Aliaksandr Valialkin
8ed95e82c6
app/vmselect/promql: follow-up after 57b3320478
2021-09-24 01:24:18 +03:00
Roman Khavronenko
57b3320478
app/vmselect: make sorting for query result similar to Prometheus ( #1647 )
...
* app/vmselect: make sorting for query result similar to Prometheus
Updated sorting allows to get the order of series in result similar or equal
to what Prometheus returns.
The change is needed for compatibility reasons.
* Update app/vmselect/promql/exec_test.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-24 01:03:12 +03:00
Aliaksandr Valialkin
e564411a62
app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus
...
Previously VictoriaMetrics could return unexpected result of the right-hand side operand
had multiple time series with the given set of labels mentioned in `on(labels)`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643
2021-09-24 00:46:25 +03:00
Aliaksandr Valialkin
a22aa0608b
app/vmselect: fix accessing /graphite/* endpoints
2021-09-21 13:56:35 +03:00
Aliaksandr Valialkin
9a3d0c43b5
app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once
2021-09-17 23:35:10 +03:00
Aliaksandr Valialkin
2951dd0a57
app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function
...
This function calculates multiple quantiles over the given buckets at once
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
2021-09-17 13:32:39 +03:00
Aliaksandr Valialkin
5a44be0e52
app/vmselect/promql: optimize quantiles() calculation
...
Calculate quantiles in one go instead of calculating each quantile individually
2021-09-17 12:33:42 +03:00
f41gh7
9375b60c5f
adds stub for functions api
2021-09-16 13:49:52 +03:00
Aliaksandr Valialkin
e60dfc96ff
app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL
2021-09-16 13:33:53 +03:00
Aliaksandr Valialkin
eca75cc650
app/vmselect/prometheus: make more clear log messages for errors during sending data to remote clients
2021-09-16 12:56:58 +03:00
Aliaksandr Valialkin
2f86d4cf38
app/vmui: follow-up after 777ff75874
...
The commit contains the following changes:
- Show vmui when requesting /graph page in order to be compatible with Prometheus datasource in Grafana.
- Properly encode query args at vmui url.
- Set the number of points on the graph to the number of horizontal pixels divided by 2. Previously it was hardcoded to 30.
- Do not save server url to persistent storage at browser, since it should be always obtained from the url.
- Run `make vmui-update` for updating vmui embedded into VictoriaMetrics.
2021-09-15 17:40:48 +03:00
Aliaksandr Valialkin
d1a16e0891
app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function
2021-09-13 13:48:06 +03:00
Aliaksandr Valialkin
f4dead529f
lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"}
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
2021-09-09 21:09:21 +03:00
Aliaksandr Valialkin
ea943911bc
app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600
2021-09-09 19:21:18 +03:00
Aliaksandr Valialkin
5ea689d61b
app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
2021-08-27 18:37:20 +03:00
Aliaksandr Valialkin
bec18e4fe9
app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570
2021-08-27 17:15:31 +03:00
Aliaksandr Valialkin
6c5760db9c
app/vmselect/promql: make fmt after 0078486ea7
2021-08-23 23:06:00 +03:00
Aliaksandr Valialkin
0078486ea7
app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus
...
See https://github.com/prometheus/prometheus/pull/8457 for details.
2021-08-23 11:45:51 +03:00
Aliaksandr Valialkin
4f3a5742eb
app/vmselect/prometheus: do not extend [d] to the detected interval between samples for first_over_time(m[d])
...
This is for the sake of consistency with similar change for the last_over_time(m[d]) at a724229b5d
2021-08-21 19:56:14 +03:00
Roman Khavronenko
b5d6a0e499
vmselect: update vm_request_duration_seconds value when request fails ( #1537 )
...
Before, metric `vm_request_duration_seconds` was update only on successful
attempts which could be misleading. For example, timeout errors on netstorage
request may be not accounted in the metric and won't be visible on dashboards.
Using `defer` statement to update the metric after query arguments validation
may improve the situation.
2021-08-19 13:58:54 +03:00
Aliaksandr Valialkin
a724229b5d
app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval
...
Since most users do not expect the overriding of explicitly set `[d]`.
2021-08-19 10:31:48 +03:00
Aliaksandr Valialkin
cdc372bb98
app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries
...
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
2021-08-18 13:59:02 +03:00
Aliaksandr Valialkin
db1e62495b
app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541
2021-08-17 13:21:21 +03:00
Aliaksandr Valialkin
bd14b0887e
app/vmselect/promql: move common condition to dropStaleNaNs in order to improve code maintainability
2021-08-17 11:01:16 +03:00
Aliaksandr Valialkin
113f0a8a07
app/vmselect/promql: drop staleness marks before calling rollupConfig.Do
...
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2021-08-15 13:21:10 +03:00
Aliaksandr Valialkin
25997a70f1
Revert "app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions"
...
This reverts commit 94dfcb6747a3b29a11d14e71bea21a2312bb6346.
It is better to remove staleness marks (decimal.StaleNaN) before calling rollupConfig.Do, e.g. in preFunc
2021-08-15 13:19:16 +03:00
Aliaksandr Valialkin
73d7b568da
app/vmselect/promql: properly handle Prometheus staleness marks in removeCounterResets functions
...
Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2021-08-14 12:45:57 +03:00
Aliaksandr Valialkin
4401464c22
all: add support for Prometheus staleness markers
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin
95dd5a48bb
app/vmselect: make vmui-update after the commit 4ae14df864a7e327955f44941295a286175423b3
2021-08-11 13:41:41 +03:00
Aliaksandr Valialkin
e92fde7945
app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0
...
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
2021-08-03 16:11:49 +03:00
Nikolay
7bbff7fb86
adds /rules and /alerts api for grafana ( #1504 )
...
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-08-02 17:28:09 +03:00
Aliaksandr Valialkin
a1911e1330
app/vmselect/netstorage: unpack time series data in mostly local big chunks
...
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
2021-07-30 12:03:17 +03:00
Aliaksandr Valialkin
74ffaa45d9
app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17
2021-07-30 09:57:53 +03:00
Aliaksandr Valialkin
192dfbfd90
app/vmselect: follow-up for ed95bc9531
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493
2021-07-29 09:53:28 +03:00
arnoldyahad
00af4ff5a4
Add case prometheus/rules for grafana 8 ( #1502 )
2021-07-29 09:53:27 +03:00
Aliaksandr Valialkin
8ee8660ac4
app/vmselect: follow-up for 626073bca8
...
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
2021-07-28 18:00:23 +03:00
Aliaksandr Valialkin
8b7917cd81
all: add go:build lines for Go1.17
...
See https://tip.golang.org/doc/go1.17#gofmt for more details
2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin
95aff47330
app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel
2021-07-26 15:47:30 +03:00
Aliaksandr Valialkin
fcaf152480
app/vmselect: make vmui-update after a91d41f12a
2021-07-26 10:31:11 +03:00
Aliaksandr Valialkin
05672ffc32
app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series
...
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
2021-07-16 01:44:58 +03:00
Aliaksandr Valialkin
ed10141ff8
app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores
2021-07-16 00:35:23 +03:00
Aliaksandr Valialkin
ca75432e66
app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries
...
This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries
or queries that take too much cpu time.
2021-07-15 16:44:28 +03:00
Aliaksandr Valialkin
f4e81aef7e
app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series
...
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
2021-07-15 16:03:28 +03:00
Aliaksandr Valialkin
e6ef97a5ee
app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems
2021-07-15 15:41:58 +03:00
Aliaksandr Valialkin
a925d5a3e1
app/vmselect/promql: duration handling improvements in MetricsQL queries
...
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
2021-07-12 17:16:41 +03:00
Aliaksandr Valialkin
8c764e88f0
app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-09 17:15:23 +03:00
Aliaksandr Valialkin
c5f0b454f0
app/vmselect: follow-up after aa11ef6d3b
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
2021-07-07 17:43:35 +03:00
tony
e9e35a7d6a
add vmui for vmselect component ( #1431 )
...
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-07-07 17:33:02 +03:00
Aliaksandr Valialkin
ed944313b0
app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms
2021-07-07 13:25:21 +03:00
Aliaksandr Valialkin
766edbc421
lib/httpserver: print full requestURI in httpserver.Errorf
...
This should simplify debugging.
2021-07-07 13:09:40 +03:00
Aliaksandr Valialkin
cdfae0117a
app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect
2021-06-23 14:19:00 +03:00
Aliaksandr Valialkin
83a4db813e
app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration
2021-06-18 19:04:42 +03:00
Roman Khavronenko
fb4f758715
promql: fix increase_pure calculation for cases with stale series ( #1381 )
...
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
2021-06-15 17:37:19 +03:00
John Belmonte
67b17cdd68
spelling fix: synonym ( #1363 )
2021-06-10 08:32:52 +03:00
Aliaksandr Valialkin
d0dca62026
app/vmselect/promql: typo fix in the comment
2021-06-09 18:34:31 +03:00
Aliaksandr Valialkin
0842bb9294
app/vmselect/promql: add timezone_offset(tz) function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
2021-05-20 11:53:09 +03:00
Aliaksandr Valialkin
832651c6c2
app/vmselect: follow up after 8a0678678b
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168
2021-05-12 17:18:30 +03:00
Nikolay
8a0678678b
Adds tsdb match filters ( #1282 )
...
* init work on filters
* init propose for status filters
* fixes tsdb status
adds test
* fix bug
* removes checks from test
2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin
6bc52fe41a
all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com
2021-04-20 20:16:17 +03:00
Aliaksandr Valialkin
544821b719
app/vmselect/promql: fix tests after d3fa0ccabd
2021-04-08 00:18:01 +03:00
Aliaksandr Valialkin
d3fa0ccabd
app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-08 00:09:40 +03:00
Aliaksandr Valialkin
cb12a8f0a8
app/vmselect: return data:null instead of data:[] from /api/v1/query_exemplars, since Grafana throws an error otherwise
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-07 23:34:06 +03:00
Aliaksandr Valialkin
1177dca3da
app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-07 14:16:08 +03:00
Aliaksandr Valialkin
7d23598b33
app/vmselect: return dumb response on /api/v1/query_exemplars request
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-05 23:25:08 +03:00
Aliaksandr Valialkin
7a0b964e8d
app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label)
2021-04-03 22:05:06 +03:00
Aliaksandr Valialkin
3055ab0115
app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value")
2021-04-02 23:55:54 +03:00
Aliaksandr Valialkin
c79e4a2f90
app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users
...
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
2021-04-02 15:02:08 +03:00
Aliaksandr Valialkin
e7fdea5953
app/vmselect: add -search.maxStatusRequestDuration command-line flag for limiting the duration of requests to /api/v1/status/* and /api/v1/series/count
2021-03-30 21:41:35 +03:00
Aliaksandr Valialkin
7962cf1af8
app/vmselect: prevent from possible incomplete query results after timed out query
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2021-03-30 13:35:45 +03:00
Aliaksandr Valialkin
aa81039b42
app/vmselect: log the metric which trigger rollup result cache reset
...
This should help finding the source of stale metrics
2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
0b2a66db30
app/vmselect/promql: do not merge time series during requests to /api/v1/query
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141
2021-03-25 13:56:07 +02:00
Aliaksandr Valialkin
331a6a2015
app/vmselect/graphite: accept and enforce extra_label in all the Graphite APIs
2021-03-23 15:29:16 +02:00
Aliaksandr Valialkin
b521d1d4f2
app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon
2021-03-23 14:16:29 +02:00
Aliaksandr Valialkin
71b72304ae
app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag
2021-03-22 16:45:34 +02:00
Aliaksandr Valialkin
cef010d5f7
app/vmselect/promql: increment key prefix for faster reset for rollup result cache
2021-03-22 11:59:07 +02:00
Aliaksandr Valialkin
b0c956a178
app/vmselect/graphite: follow-up after 529d7be26b
2021-03-18 16:30:20 +02:00
Nikolay
529d7be26b
changes metricsFind api ( #1137 )
...
it should be able mitigate crash if label value contains *,[ or { symbols
2021-03-18 16:12:02 +02:00
Aliaksandr Valialkin
b1713e3fcd
app/vmselect/promql: typo fix after 9666834045
2021-03-17 15:12:11 +02:00
Aliaksandr Valialkin
9666834045
app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function
...
This should improve accuracy for the returned buckets
2021-03-17 14:31:41 +02:00
Aliaksandr Valialkin
7bc728bf53
app/vmselect: add vm_index_search_duration_seconds histogram for monitoring the performance of index search
2021-03-17 01:17:41 +02:00
Aliaksandr Valialkin
0ce557951f
app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores
2021-03-16 21:51:31 +02:00
Aliaksandr Valialkin
f0a4157f89
app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time()
2021-03-16 12:24:21 +02:00
Aliaksandr Valialkin
85a95bf60c
all: various fixes in command-line flag descriptions
2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin
923cdb0552
app/vmselect/promql: reduce overhead on scrape interval estimation
...
It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.
2021-03-15 20:31:33 +02:00
Aliaksandr Valialkin
82aab87446
app/vmselect/promql: fix tests after 2dae0a2c47
2021-03-15 20:18:59 +02:00
Aliaksandr Valialkin
2dae0a2c47
app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point
2021-03-15 12:36:33 +02:00
Aliaksandr Valialkin
2096c6e464
app/vmselect/prometheus: typo fix after 7c002023d7
2021-03-12 12:19:36 +02:00
Aliaksandr Valialkin
7c002023d7
app/vmselect/prometheus: do not include datapoints with timestamps matching t-d when returning results from /api/v1/query?query=m[d]&time=t as Prometheus does
2021-03-12 12:16:50 +02:00
Aliaksandr Valialkin
fe8b12fbad
app/vmselect/promql: follow up for 433fff0006
2021-03-09 12:48:44 +02:00
Nikolay
433fff0006
duplicate timeseries fix for prometheus_buckets function ( #1119 )
...
* try fix for prometheus_buckets
* merge possible end of the bucket collision
2021-03-09 12:26:23 +02:00
Aliaksandr Valialkin
a14053ffa0
app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL
2021-03-04 14:12:07 +02:00
Aliaksandr Valialkin
2b53add6b2
app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration
...
This should help debugging slow queries.
2021-02-28 19:40:19 +02:00
Nikolay
186c078fac
adds enforced tag filters into cache key ( #1095 )
2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin
d86e9b49c4
app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge
...
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
2021-02-26 22:56:36 +02:00
Aliaksandr Valialkin
98854e5f2b
app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release
...
See https://twitter.com/roidelapluie/status/1363428376162295811
The `last_over_time(m[d])` function already exists in MetricsQL.
2021-02-24 17:24:56 +02:00
Aliaksandr Valialkin
f4135b0d14
app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does
2021-02-24 00:42:22 +02:00
Aliaksandr Valialkin
fa03e0d210
app/vmselect/promql: add increase_pure() function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
2021-02-22 19:14:15 +02:00
Aliaksandr Valialkin
9c70c1f21f
app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics
2021-02-18 22:07:29 +02:00
Aliaksandr Valialkin
c4756f94da
app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations
2021-02-16 16:08:53 +02:00
Aliaksandr Valialkin
5a401225c7
app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key
...
This should prevent from a memory allocation and a string copy.
2021-02-16 15:43:10 +02:00
Aliaksandr Valialkin
4e39bf148c
vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0
...
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
acdb401585
app/vmselect/prometheus: treat match query arg in the same way as match[] query arg
2021-02-11 15:02:21 +02:00
Aliaksandr Valialkin
2d33230793
app/vmselect/promql: properly make copies of EvalConfig
2021-02-11 12:41:15 +02:00
Aliaksandr Valialkin
7a3a9421f3
app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2
...
This should prevent from data races if the underlying functions modify EvalConfig contents.
2021-02-10 23:05:55 +02:00
Aliaksandr Valialkin
04faea8b45
app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel
...
This should reduce query execution times.
2021-02-10 22:59:39 +02:00
Aliaksandr Valialkin
553016ea99
lib/storage: disable composite index usage when querying old data
2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
c7dccebaef
lib/storage: optimize search by label filters matching big number of time series
2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
803a00102a
app/{vminsert,vmselect}: accept requests to paths with /graphite and /prometheus prefixes
...
This should improve compatibility with path prefixes from VictoriaMetrics cluster.
See https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#url-format
2021-02-04 20:01:18 +02:00
Aliaksandr Valialkin
4d43ab0875
app/vmselect: typo fix when stripping url path prefixes
2021-02-04 19:29:25 +02:00
Aliaksandr Valialkin
8629fd8a72
app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax
2021-02-03 20:36:01 +02:00
Aliaksandr Valialkin
d16f22f3a1
app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards
...
Example: foo{bar{x,yz},a[b-c],*de}
2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
157c02622b
app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax
2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
b43b498fd8
app/vmselect: add ability to pass extra_label=<label>=<value> query arg to Prometheus Querying API
...
This enforced `{label="value"}` label filter to the query.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021
2021-02-01 18:04:17 +02:00
Nikolay
195341a7cf
Graphite vmalert wip ( #112 )
...
* init implementation for graphite alerts
* adds graphite support for vmalert
* small fix
* changes vmalert graphite api with type
* updates tests
* small fix
* fixes graphite parse
* Fixes graphite from time
2021-02-01 15:05:32 +02:00
Aliaksandr Valialkin
c164a8d231
app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag
...
This should reduce incorrect usage and assumptions for this flag.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020
2021-01-22 13:00:10 +02:00
Aliaksandr Valialkin
8749c2dd92
app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time
2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin
011c5da785
app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()
2021-01-18 17:30:26 +02:00
Aliaksandr Valialkin
1051d8aa2d
app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin
acf1a2c72b
app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
2021-01-13 10:59:42 +02:00
Aliaksandr Valialkin
1837f2f7d3
app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d
2021-01-12 16:12:12 +02:00
Aliaksandr Valialkin
c86286ec1d
app/vmselect/promql: do not ajdust offset value provided in the query
...
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin
4b7105a65b
app/vmselect: sync query stats handling with cluster version
2020-12-27 13:00:29 +02:00
Aliaksandr Valialkin
df0309eae0
app/vmselect/promql: simplify defer call for querystats.RegisterQuery
2020-12-27 12:06:04 +02:00
Aliaksandr Valialkin
ad4e6a9283
app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K
...
This should reduce memory usage in constrained environments
2020-12-25 17:40:47 +02:00
Aliaksandr Valialkin
59183f66d0
app/vmselect: refactor /api/v1/stats/top_queries
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
2020-12-25 16:44:29 +02:00
Nikolay
86630350bf
Adds query stats handler ( #945 )
...
* Adds query stat handler,
for query and query_range api, victoriametrics tracks query execution time,
stats are expored at /api/v1/status/queries endpoint with topN param
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
* fixed query stats bugs
* improves queryStats tracker
* improves query stat
* small fix
* fix tests
* added more tests
* fixes 386 tests
* naming fixes
* adds drop for outdated records
2020-12-25 16:42:05 +02:00
Aliaksandr Valialkin
88ac4dfc07
app/vmselect: properly parse negative combined offsets such as -1h2m3s
...
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin
11fa458e39
app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100
2020-12-15 13:40:46 +02:00
Aliaksandr Valialkin
5ebfc275e6
app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested
2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin
ba8ac08739
app/vmselect/graphite: properly handle wildcards and charsets inside curly braces
...
For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952
2020-12-11 17:24:12 +02:00
Aliaksandr Valialkin
4146fc4668
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
1906f841c9
app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect
2020-12-08 15:16:17 +02:00
Aliaksandr Valialkin
007dbf273d
app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942
2020-12-07 01:08:52 +02:00
Aliaksandr Valialkin
e2e8ef86d9
app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N
2020-12-05 12:30:46 +02:00
Aliaksandr Valialkin
66379cc69f
app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936
2020-12-03 21:47:36 +02:00
Aliaksandr Valialkin
44a34a0f5f
app/vmselect/promql: make fmt
2020-12-02 21:33:35 +02:00
Aliaksandr Valialkin
1982505c2b
app/vmselect/promql: return nan from minute(m) when m equals to nan
...
This aligns VictoriaMetrics behaviour with Prometheus behaviour.
The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
2020-12-02 20:16:58 +02:00
Aliaksandr Valialkin
9d87496b50
app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window.
...
This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.
2020-12-02 13:12:50 +02:00
Aliaksandr Valialkin
91a4c279cc
app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true
...
This aligns MetricsQL behavior to Prometheus' one.
The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
2020-12-02 12:09:34 +02:00
Aliaksandr Valialkin
700bda8e2e
app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does
2020-12-02 00:28:26 +02:00
Aliaksandr Valialkin
efdc3c71af
app/vmselect/searchutils: return elapsed time in Deadline.String() output
...
This should improve debuggability for error messages containing Deadline.String() output
2020-12-01 00:15:18 +02:00
Aliaksandr Valialkin
a724dde90a
app/vmselect: protect /tags/delSeries with -deleteAuthKey in the same way as /api/v1/admin/tsdb/delete_series
2020-11-23 15:35:59 +02:00
Aliaksandr Valialkin
f0c207fae2
app/vmselect: add /tags/delSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb
2020-11-23 15:27:21 +02:00
Aliaksandr Valialkin
d3794eb994
app/{vminsert,vmselect}: move /tags/tagSeries and /tags/tagMultiSeries api from vminsert to vmselect
...
This is needed for consistency, since all the `/tags*` api handlers are located in vmselect.
2020-11-23 12:33:19 +02:00
Aliaksandr Valialkin
f765985947
lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt
...
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin
2859a452d4
app/vmselect: add remoteAddr to slow query log in order to improve debuggability
...
This will simplify identifying the client that sends slow queries to VictoriaMetrics.
2020-11-18 20:38:32 +02:00
Aliaksandr Valialkin
266788be14
app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place
...
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
cf18df367d
app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name
2020-11-16 15:52:16 +02:00
Aliaksandr Valialkin
2c67232565
app/vmselect/graphite: add /tags/autoComplete/values handler from Graphite Tags API
2020-11-16 15:29:35 +02:00
Aliaksandr Valialkin
86f99c6b55
app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support
2020-11-16 14:50:05 +02:00
Aliaksandr Valialkin
3c1434118e
app/vmselect/prometheus: return __name__ label if match[] query to /api/v1/labels matches at least a single time series
2020-11-16 13:54:34 +02:00
Aliaksandr Valialkin
27a417bcd3
app/vmselect/prometheus: improve performance for /api/v1/labels and /api/v1/label/<labelName>/values on time ranges exceeding one day when match[] query arg is set
2020-11-16 13:51:59 +02:00
Aliaksandr Valialkin
6fa806f1ca
app/vmselect/prometheus: fix deadlock in /api/v1/series on a time range exceeding one day
2020-11-16 13:30:47 +02:00
Aliaksandr Valialkin
465923b181
app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
414cd39659
app/vmselect/graphite: apply filter then limit
2020-11-16 04:09:14 +02:00
Aliaksandr Valialkin
d100341394
app/vmselect/graphite: add /tags/<tag_name> handler for Graphite Tags API
2020-11-16 03:42:25 +02:00
Aliaksandr Valialkin
6251762787
app/vmselect/graphite: add /tags handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 03:00:01 +02:00
Aliaksandr Valialkin
d9d01f976b
app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points
...
Do not take into account spare data point value if the next point will is located too far from the current point.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
2020-11-13 15:23:44 +02:00
Aliaksandr Valialkin
1f19c167a4
app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta()
...
This should remove unexpected spikes at the end of gaps.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
2020-11-13 14:59:24 +02:00
Aliaksandr Valialkin
47a038401b
all: consistently return text-based HTTP responses with charset=utf-8
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
2020-11-13 10:35:41 +02:00
Aliaksandr Valialkin
348edd92fe
app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters
2020-11-11 12:39:07 +02:00
Aliaksandr Valialkin
47390d8947
app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2020-11-10 14:51:44 +02:00
Aliaksandr Valialkin
ba4a2c8bca
app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum
2020-11-10 01:18:08 +02:00
Aliaksandr Valialkin
dd6bfa50e9
app/vmselect/promql: code cleanup after 43823addea
2020-11-06 01:30:50 +02:00
n4mine
43823addea
app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result ( #883 )
2020-11-06 01:29:24 +02:00
Aliaksandr Valialkin
368b69b4c4
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin
b378cd6ed8
app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set
2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
caeb74f068
app/vmselect: reduce memory usage when query touches big number of time series
2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin
5e75c389e6
app/vmselect/promql: allow dropping trailing sample only for default_rollup function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
2020-11-02 02:10:59 +02:00
Aliaksandr Valialkin
cb71af216a
app/vmselect/promql: go fmt
2020-11-02 00:15:29 +02:00
Aliaksandr Valialkin
daacbc7e34
app/vmselect/promql: do not drop trailing datapoints for instant queries
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-11-02 00:12:37 +02:00
Aliaksandr Valialkin
c539494b36
app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
28353e48ca
app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series
...
Now trailing data points are additionally dropped for time series with a single raw sample
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-17 10:44:34 +03:00
Aliaksandr Valialkin
a2b9476897
app/vmselect/promql: return a single time series at max from absent() function like Prometheus does
2020-10-13 15:56:04 +03:00
Aliaksandr Valialkin
9aa3b65766
app/vmselect/promql: improve time series staleness detection
...
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-13 12:19:57 +03:00
Aliaksandr Valialkin
d8af290947
app/vmselect/promql: fix mode_over_time calculations
...
Previously `mode_over_time` could return garbage due to improper shuffling of input data points.
2020-10-13 11:58:25 +03:00
Aliaksandr Valialkin
1e27420243
app/vmselect/prometheus: fix golangci-lint warning
2020-10-13 09:36:11 +03:00
Aliaksandr Valialkin
4f16a964e3
app/vmselect: add ability to export data in CSV format via /api/v1/export/csv
2020-10-12 20:08:17 +03:00
Aliaksandr Valialkin
762c967855
app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning
...
Functions are:
* keep_last_value
* keep_next_value
* interpolate
* running_min
* running_max
* running_avg
* range_min
* range_max
* range_avg
* range_first
* range_last
* range_quantile
* smooth_exponential
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:47:06 +03:00
Aliaksandr Valialkin
45f7cdc532
Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus"
...
This reverts commit ac45082216https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:40:34 +03:00
Aliaksandr Valialkin
a94825b169
Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus"
...
This reverts commit bb61a4769bhttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:38:27 +03:00
Aliaksandr Valialkin
f7d28bddbf
Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus"
...
This reverts commit e5202a4eaehttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:35:18 +03:00
Aliaksandr Valialkin
e9f2e2cbc9
app/vmselect/promql: add missing label filters to binary operands before query execution
...
This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization
See also https://github.com/cortexproject/cortex/issues/3253
2020-10-07 21:15:09 +03:00
Aliaksandr Valialkin
59c26feefa
app/vmselect/promql: fill gaps on graphs for range_* and running_* functions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806
2020-10-02 13:59:45 +03:00
Aliaksandr Valialkin
f7636b0342
app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output
2020-09-29 18:03:26 +03:00
Aliaksandr Valialkin
96ee276e6e
app/vmselect/prometheus: check for errors returned from bufferedwriter.Write
...
This makes `make errcheck` happy
2020-09-29 11:37:01 +03:00
Aliaksandr Valialkin
6fdfc67620
app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.*
...
In this case only node must be returned with stripped dot in the end of id as carbonapi does
2020-09-29 11:01:59 +03:00
Aliaksandr Valialkin
1b3efccb24
app/vmselect: stop /api/v1/export/* execution if client disconnects
2020-09-27 23:53:13 +03:00
Aliaksandr Valialkin
95688cbfc5
all: add native format for data export and import
...
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format ) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format ) handler.
2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin
82973f8ae7
Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock"
...
This reverts commit bab6a15ae0
2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin
bab6a15ae0
lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock
...
This arg became unused after 23bdc1f107
2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin
23bdc1f107
app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls
2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin
c584aece38
app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin
27500d7d4c
app/vmselect/prometheus: code cleanup after 3ba507000c
2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin
3ba507000c
app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin
bed25e3c24
app/vmselect/netstorage: properly pre-allocate space for sbs
2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin
09b0f7c202
app/vmselect/netstorage: release search resources on timeout errors
...
Previously these resources weren't released, which could lead to resource leaks.
2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin
3b1e3a03e0
app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin
00b5145c69
app/vmselect/searchutils: fixed tests after 2eb72e09ab
2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin
2eb72e09ab
app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-21 21:24:40 +03:00
Aliaksandr Valialkin
98d1cd0971
app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API
2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin
9f79bcf64a
app/vmselect: improve description for -search.maxQueryDuration
2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin
1fec47a289
app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series
2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin
8c3d7c1a59
app/vmselect: typo fix in -search.maxStalenessInterval description
2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin
0e533d1a9c
app/vmselect/promql: support composite durations like Prometheus 2.21 does
...
The following durations are supported now: `1h5m35s` or `1s543ms`
See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
2020-09-11 23:39:13 +03:00
Aliaksandr Valialkin
a2f647d142
app/vmselect/prometheus: typo fix in the description for -search.latencyOffset command-line flag
2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin
f0005c3007
app/vmselect: move Deadline from netstorage to searchutils
...
This removes dependency on netstorage from searchutils.
2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin
2114179e19
app/vmselect: substitute inf values at smooth_exponential with the previous values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin
204ec415b4
app/vmselect: skip infinite values when calculating smooth_exponential
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin
8a8b5a73d3
app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric
2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin
f6bc608e86
app/vmselect: initial implementation of Graphite Metrics API
...
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
62919eaf7e
app/vmselect/promql: go fmt
2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin
e6da63dffe
app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin
8e85b56737
app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin
c0343a661b
app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3
2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin
f4e7e5fb90
app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions
...
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin
e706e59d49
app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does
2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin
147c35ebd4
all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin
7d89fafe1a
app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)
2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin
e969ef2639
app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin
8adba82c02
app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores
...
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin
f3d33e23c9
app/vmstorage: improve error logging when the request times out
2020-08-10 13:23:26 +03:00
Aliaksandr Valialkin
716754fae6
app/vmselect/promql: properly handle -n^m like Prometheus does
...
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin
bb61a4769b
app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin
ac45082216
app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin
e5202a4eae
app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus
...
Rollup functions:
- avg_over_time
- min_over_time
- max_over_time
- quantile_over_time
This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin
68e4f40a72
app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does
...
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin
bc8381613d
app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names
2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
3f85c06b65
app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block
...
This should improve query performance on systems with big number of CPU cores (16 and more)
2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin
d20c2156e4
app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans
2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin
5d0c37bec0
app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration
2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin
e6eee2bebf
app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...)
2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin
509d12643b
app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin
f0c678c41b
app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin
328b52e5ff
app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window
...
Just divide the data point value by the window duration in this case.
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
700737c181
app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now
...
See e00cfc854d
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
2f735f112d
app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty
2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
79c30cf4cb
app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query
2020-07-28 15:13:08 +03:00
Aliaksandr Valialkin
0da202023b
app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp
...
This aligns `group()` behaviour to Prometheus
2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin
894dcb7c1c
app/vmselect/promql: improve further the accuracy of buckets_limit() function
...
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin
edb1eca6f1
app/vmselect/promql: avoid dropping inf bucket in buckets_limit
...
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin
97b6f5d223
app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets
2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin
bb161497cf
app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function
...
Now it properly merges the bucket with the previous bucket after deletion.
2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin
994fa2f3bf
app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k
...
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin
e151c5c644
app/vmselect: fix tests for rate_over_sum
2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin
3107c633e3
app/vmselect/promql: typo fix after 3e557c9861
2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin
3e557c9861
app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration
...
Something like `sum_over_time(m[d]) / d`, but more accurate.
2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin
b1f6843bd0
app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time
...
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin
039c9d2441
lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index
...
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823
lib/storage: add more fine-grained pace limiting for search
2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
461481fbdf
app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery
2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin
e79de9774b
app/vmselect: typo fix after 34563916f7
2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin
34563916f7
app/vmselect: reduce memory usage when querying big number of time series with long labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin
4f526cc816
app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin
dfb113f175
app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries
...
This should slightly reduce load on GC when processing queries that touch big number of time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.
2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin
caa2952aa6
app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query
2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin
e00cfc854d
app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value
...
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin
b9c8f6bf34
app/vmselect: log the total available memory for concurrent requests on not enough memory errors
...
This should simplify root cause analysis
2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin
5a4675c528
app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range
...
This matches Prometheus behaviour.
This should fix https://github.com/jacksontj/promxy/issues/329
2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin
ecb1b2564a
app/vmselect/promql: add mode() aggregate function
2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin
b35cb293f5
lib/httpserver: log remote address in error message from httpserver.Errorf
...
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
1c641037e8
app/vmselect/promql: add mode_over_time(m[d]) function
...
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin
6b5ad535ae
app/vmselect/promql: optimize group(rollup(m)) calculations
2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin
8949d65ad1
app/vmselect/promql: check that any() doesn't touch metric name
2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin
aa5d88055d
app/vmselect/promql: add group() aggregate function to MetricsQL
...
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin
df01836818
app/vmselect/promql: keep all labels for time series from any() call
2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin
e4e1cd1de2
app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage
2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin
25f80d320b
app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time
...
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin
55d83e777d
app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9
2020-07-13 21:40:38 +03:00
faceair
1033dc7e2a
fix empty response template ( #617 )
2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin
9c350bc20d
app/vmselect/promql: add missing tests for ifnot binary operation
2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin
256fd9a87e
app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation
2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin
2d9b3ad5b3
app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function
2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin
742da690f4
app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575
2020-07-08 18:55:38 +03:00
DexterZhang
99f54e44ff
feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch ( #598 )
2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin
acf828a759
app/vmselect/prometheus: small fixes on top of 8bb762124a
2020-07-05 18:17:06 +03:00
faceair
8bb762124a
fix adjust last points avoid influence earlier value ( #606 )
2020-07-05 17:56:54 +03:00
Aliaksandr Valialkin
5d3db3ff7c
app/vmselect: add interpolate function for filling gaps with linearly interpolated values
...
See https://stackoverflow.com/q/62565021/274937 for details
2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin
84a37098ed
app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention
...
VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention
if `-denyQueriesOutsideRetention` command-line flag is set.
2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin
7c2c8b2981
all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode
2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
05d754d7bb
app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series
...
This should reduce GC pressure when processing time series with big number of rows
2020-06-24 19:38:39 +03:00
nicbaz
774f7ca1c1
vmselect: fix label_replace when mismatch ( #579 )
...
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".
Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.
Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin
7209d58fbd
app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series
...
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
2020-06-23 22:46:15 +03:00
Aliaksandr Valialkin
7532dbcdf5
app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin
88e1b7d144
app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin
dbd8beccfa
app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions
...
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
2020-05-21 12:07:48 +03:00
Aliaksandr Valialkin
70ea4e28a7
app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3
2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin
538fdfe133
app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries
2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin
f52769f6ee
app/vmselect/promql: fix outilersk calculations
2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin
a441cdd1d9
app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series
2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin
0afd48d2ee
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
7882a0dbbf
app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint
2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin
faf92a0965
app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one
2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin
cc311e20fe
app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y
2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin
574289c3fb
app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation
2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin
57407cca83
app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag
...
Limit the estimated time series count after aggregation with grouping by the number of source time series.
2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin
4e4f57b121
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
17d96e4503
app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs vs false-positive not enough memory errors
2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin
b4afe562c1
lib/storage: postpone reading data from blocks during search
...
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.
This improves single-node VM performance on heavy queries by up to 2x.
2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
fcf57f9883
app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks
...
This should minimize the number of disk seeks when reading data from temporary file.
2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin
a596aec82c
app/vmselect: fix description for -search.resetCacheAuthKey
2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin
364db13c9c
app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
9ebc937685
app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin
fe57d46687
app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :(
...
Leave both flags for backwards compatibility reasons.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin
f41e6a7bd9
app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query
2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin
071fdf5518
lib/logger: add WARN level for logging expected errors such as invalid user queries
2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin
6f7f64f757
app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin
f9b24d4899
app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions
2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin
318326c309
lib/storage: properly handle {label=~"foo|"} filters as Prometheus does
...
Such filters must match all the time series with `label="foo"` plus all the time series without `label`
Previously only time series with `label="foo"` were matched.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
0c485f14d1
app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries
2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin
2ebf7d86ff
app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()
2020-03-29 21:50:28 +03:00
Aliaksandr Valialkin
3c7c71a49c
app/vmselect: adjust label_map() handling for corner cases
...
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series
All the unmatched labels are kept unchanged.
2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin
fdc2a9d1d7
app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin
8023ad7dbd
app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model
2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin
e17702fada
app/vmselect: add optional max_rows_per_line query arg to /api/v1/export
...
This arg allows limiting the number of data points that may be exported on a single line.
2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin
c5f894b361
Makefile: add build and test rules with enabled race detector. These rules have -race suffix
...
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin
7d178a40bd
app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter
...
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin
5701b2f7bb
app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin
18af31a4c2
all: properly split vm_deduplicated_samples_total among cluster components
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
ae51300973
app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin
e65ec88779
app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-23 18:03:57 +02:00
Yaroslav
a6d0645539
fix rollupOpen(), rollupHigh(), rollupLow() functions ( #328 )
2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin
2f55cabaa4
app/vmselect/promql: log when rollupResult cache is cleared
2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin
ec9bf39b5b
app/vmselect: add -search.cacheTimestampOffset command-line flag
...
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.
This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin
539139391c
app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 13:58:05 +02:00
Aliaksandr Valialkin
7b1c7051a3
app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions
...
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin
a6c6a2debc
app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8
2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin
78b62dee87
app/vmselect/promql: properly adjust time range for data to select
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin
366693b9f1
app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent
2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin
525101339e
app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin
ada6a3da8d
app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin
36a1ac8360
app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded
...
This will prevent from excess CPU usage for timed out queries.
2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin
834051e5b2
app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata
VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.
2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin
42864bb52f
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin
90cf356ea1
app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus
2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin
c3d86eef96
all: add -dedup.minScrapeInterval command-line flag for data de-duplication
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
d70ba7eb37
lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor
...
This should improve performance when reading many small blocks.
2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin
a462355b2f
app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value
2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin
adbbc4fa1a
app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value
...
Examples for such counters: OS-level counters for network or cpu stats.
2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin
2392a359e1
app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin
680080887d
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
9773022e50
app/vmselect: mention the original query and time range in error messages
...
This should simplify debugging invalid or heavy queries.
2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin
efc7ad88ec
app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors
2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin
ec9651e266
app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data
2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin
582dd01f42
app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d
2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin
36973ee975
app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp
2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin
cdcacaea6d
app/vmselect/netstorage: make fmt
2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin
7327adbc86
app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file
...
This should reduce memory usage on systems with more than 8GB RAM.
2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin
d0d258b314
app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs
...
A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value.
This will give more chances for executing insert path.
2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin
d88725f133
app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects
2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin
0d7505b00e
all: mention command-line flags used for limiting the incoming request size in error messages
...
This should improve error logs usability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin
2839f4688a
app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args
2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin
8b14572f70
app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions
...
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin
8eaced8cae
app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval
2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin
46bfdbe6cf
app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions
...
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin
b829fe5e39
app/vmselect/promql: properly handle aggr(aggr_over_time(...))
2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin
164278151f
app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function
...
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin
c4632faa9d
app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions
...
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin
d5aeda0e1a
app/vmselect/promql: skip rate calculation for the first point on time series
2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin
d410c78c7e
app/vmselect/promql: fix calculations for histogram_share
2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin
d64699bb9f
app/vmselect/promql: add missing MetricName into netstorage.Result in tests
2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin
f409f2d050
app/vmselect/promql: add histogram_share(le, buckets) function
2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin
b1ded7cf9a
app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0
app/vmselect/promql: add histogram_over_time(m[d]) rollup function
2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
3e09d38f29
app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time
...
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
a774120460
app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series
...
This should provide better estimation for `rate` in the beginning of time series.
2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin
b5645ccbdf
app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series
2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin
cb3a342882
app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations
2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
6f67e0b56b
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43
Split Extended PromQL parsing to a separate library
2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
d880325cf6
app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range
...
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:
BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin
9e19949c6b
app/vmselect/promql: adjust calculations for rate and increase for the first value
...
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin
0455c03cb9
app/vmselect/promql: properly calculate rate on the first data point
...
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin
5b75984aa9
app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs
2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin
bea5a8700a
app/vmselect: add -search.maxExportDuration command-line flag for limiting /api/v1/export duration
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275
2019-12-20 11:35:22 +02:00
Aliaksandr Valialkin
96ff8d9adb
app/vmselect: add ability to pass match[], start and end to /api/v1/labels
...
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.
See https://github.com/prometheus/prometheus/issues/6178 for more details.
2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin
cea5a14853
all: rename Extended PromQL to PromQL extensions
2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin
73b2a3d4b7
app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed
...
Updates https://github.com/prometheus/prometheus/issues/5706
2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin
caa4eb72d9
app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does
2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin
3c076544bf
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin
c09472dfd9
app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range
...
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin
face3d57bf
app/vmselect: add placeholders for /api/v1/rules and /api/v1/alerts
2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin
20812008a7
lib/storage: remove metricID with missing metricID->metricName entry
...
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.
Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
9476b73527
app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals
...
This should improve scrape interval estimation for tiem series with gaps.
2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin
75ff524a4e
app/vmselect/promql: fix corner case for increase over time series with gaps
...
In this case `increase` could return invalid high value for the first point after the gap.
2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin
4c859d980c
app/vmselect/prometheus: consistently apply nocache arg to /api/v1/query the same way ast to /api/v1/query_range
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin
110235f789
app/vmselect/prometheus: fix content-type for /api/v1/export responses
...
The correct Content-Type should be `application/stream+json` instead of `application/json`
Thanks to Joshua Ryder for pointing to this.
2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin
205233d9a7
app/vmselect/promql: remove zero timeseries from prometheus_buckets output
2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin
3f99f39e9b
app/vmselect/prometheus: reduce default value for -search.latencyOffset from 60s to 30s
...
30 seconds should be enough for almost all the cases
2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin
e91cb34c0e
app/vmselect/promql: allow nested parens
2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin
826dfd63a5
vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1
2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin
0401969d78
app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph
...
This should reduce the amounts memory allocations
2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin
da98703748
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
c28876172f
app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0
2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin
50ae1879c6
app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series
2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin
8582b50360
app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets
2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin
19dfe52254
app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets
2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin
4bb88843cf
app/vmselect: add vm_per_query_{rows,series}_processed_count histograms
2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
7753c8c0a1
app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile
2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin
c4287b3c86
app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets
2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin
1f3fd2c910
app/vmselect: adjust end arg instead of adjusting start arg if start > end
...
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin
7a4635f853
all: remove the remaining mentions of cluster version
2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
0094bc4fc9
app/vmselect/prometheus: properly adjust too big time time on /api/v1/query
...
Too big `time` must be adjusted to `now()-queryOffset`.
2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin
3f1637fae8
app/vmselect/promql: properly calculate integrate(q[d])
2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin
3fd32e331a
app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
2019-11-13 20:24:26 +02:00
Oleg Kovalov
b4f44befa3
fix misspelled words ( #229 )
2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
63b05c0b9f
app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions
...
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.
Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.
This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
2f58f37f07
app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q
2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin
b0295dbf2e
app/vmselect: add -search.latencyOffset flag for tuning the time after data collection when data points become visible in query results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin
97ce4e03a5
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
c66a691593
app/vmselect/prometheus: add -search.maxLookback command-line flag for overriding dynamic calculations for max lookback interval
...
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.
The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
956fdd89d3
app/vmselect/promql: take into account the previous point when calculating max_over_time and min_over_time
...
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
2019-10-08 12:30:05 +03:00
Aliaksandr Valialkin
4e9e1ca0f7
app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially
...
This became the case after b7ee2e7af2
2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin
c1d3705be0
app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock
...
This allows re-using the destination buffer for marshaling in the outer loop.
2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin
b7ee2e7af2
app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series
2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin
67d44b0845
app/vmselect/promql: do not generate timestamps for NaN values in timestamp function according to Prometheus logic
2019-09-27 18:54:43 +03:00
Aliaksandr Valialkin
26dc21cf64
app/vmselect/promql: add increases_over_time and decreases_over_time functions
...
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:44 +03:00
