Commit graph

767 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a10647e0bf
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:13:47 +03:00
Aliaksandr Valialkin
3bb3893b2d
app/vmselect/vmui: make vmui-update after 080562030d 2022-10-18 10:50:35 +03:00
Yury Molodov
ff6151fa49
vmui: limit number of plotted series (#3229)
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:13:47 +03:00
Aliaksandr Valialkin
e6fd33044f
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:03:08 +03:00
Siqi Liu
930f1ee153
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225)
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
2022-10-13 11:57:16 +03:00
Aliaksandr Valialkin
b8da90b893
app/vmselect/promql: properly handle zero and negative values for -search.maxMemoryPerQuery
This is a follow-up for 04a05f161c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-12 09:25:17 +03:00
Aliaksandr Valialkin
04a05f161c
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-10 21:45:13 +03:00
Aliaksandr Valialkin
8e1ccecd97
docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag
This is a follow-up for 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-09 13:56:59 +03:00
Aliaksandr Valialkin
5138eaeea0
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-08 01:08:05 +03:00
Yury Molodov
27ed4b853e
vmui: auto-update chart after query field removed (#3210)
* feat: run query after query field removed

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-07 23:02:41 +03:00
Aliaksandr Valialkin
8322760647
app/vmselect/vmui: make vmui-update after a54987f671 2022-10-07 03:31:17 +03:00
Aliaksandr Valialkin
f452c84579
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:22:24 +03:00
Aliaksandr Valialkin
40e899fd67
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:21 +03:00
Aliaksandr Valialkin
d9282027e6
app: follow-up after ec04fcac93
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:52:02 +03:00
Dmytro Kozlov
ec04fcac93
Properly parse json when export import metric (#3180)
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API

* app/vmselect: update convert function

* app/vmselect: export null if `math.IsNaN(v)`

* app/vmselect: get float from json

* lib/protoparser: add test

* docs: add change log

* lib/protoparser: make export import api compatible
2022-10-06 13:54:20 +03:00
Aliaksandr Valialkin
c53b7e66ef
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:05:43 +03:00
Aliaksandr Valialkin
49311ae977
app/vmselect/prometheus: improve scalability of /federate endpoint on systems with many CPU cores
Minimize usage of global lock inside bufferedwriter.Write() when processing `/federate` data
on systems with many CPU cores
2022-10-01 20:13:24 +03:00
Aliaksandr Valialkin
fb1cc3cc94
app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores
Use sync.Map instead of a global mutex there. This should lift scalability limits
on systems with many CPU cores.
2022-10-01 20:00:03 +03:00
Aliaksandr Valialkin
fcc7ab71b3
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-01 19:47:37 +03:00
Aliaksandr Valialkin
725dfb0ed6
lib/httpserver: use 302 redirects instead of 301 redirects
Incorrect 301 redirects can be cached by user agents such as web browsers.
This can complicate recovery procedure after the incorrect redirect is fixed,
e.g. web browser cache must be reset.

The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:53:35 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:20:34 +03:00
Roman Khavronenko
166d444159
vmselect/rollup: rm workaround for slow-changing counters (#3163)
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 15:33:25 +03:00
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:41:09 +03:00
Aliaksandr Valialkin
99bc18774c
app/vmselect/vmui: make vmui-update after 1304824201 2022-09-14 13:33:06 +03:00
Yury Molodov
1b41169415
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:55:11 +03:00
Yury Molodov
9541ef2e9e
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:43:37 +03:00
Dmytro Kozlov
4415c71a2b
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049)
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
2022-09-06 13:25:59 +03:00
Aliaksandr Valialkin
ae31b2363f
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:55:48 +03:00
匠心零度
50e2524bc2
api prometheus/api/v1/label/../values time not specified, (#3052)
modify default start values
2022-09-05 11:52:19 +03:00
Aliaksandr Valialkin
7dc632719d
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin
4076277cf0
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 19:46:27 +03:00
匠心零度
6d81584d2a
reduce unnecessary vmstorage query (#3031)
* reduce unnecessary vmstorage query

* reduce unnecessary vmstorage query

* rollback limit logic /api/v1/label/*
2022-08-30 12:36:54 +03:00
Aliaksandr Valialkin
ad11b8d83d
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:35:49 +03:00
Dmytro Kozlov
2d71b4859c
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:33:56 +03:00
Dmytro Kozlov
463ea6897b
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:25:18 +03:00
Aliaksandr Valialkin
343241680b
all: remove the remaining bits of io/ioutil
The io/ioutil package is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics requires at least Go1.18, so it is time to remove the io/ioutil from source code

This is a follow-up for 02ca2342ab
2022-08-22 00:20:58 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
4ac79d29ad
app/vmselect: follow-up after 63e0f16062
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
2022-08-15 13:50:16 +03:00
Roman Khavronenko
63e0f16062
vmselect: introduce UserReadableError type of error (#2894)
When read query fails, VM returns rich error message with
all the details. While these details might be useful
for debugging specific cases, they're usually too verbose
for users.
Introducing a new error type `UserReadableError` is supposed
to allow to return to user only the most important parts
of the error trace. This supposed to improve error readability
in web interfaces such as VMUI or Grafana.

The full error trace is still logged with the full context
and can be found in vmselect logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-15 13:38:47 +03:00
Yury Molodov
1513866d51
vmui: shortcut keys legend (#2971)
* feat: add shortcut modal

* feat: add shortcut descriptions

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-15 11:54:47 +03:00
Yury Molodov
3cb013aeb8
vmui: graph action on moush hold and move (#2915)
* fix: change event for graph panning

* fix: change detect key

* feat: add zoom in with mouse selection

* - document the change
- run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-07 23:55:09 +03:00
Aliaksandr Valialkin
7478d423c5
app/vmselect/netstorage: cleanup after 92630c1ab4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896
2022-08-04 18:28:11 +03:00
Aliaksandr Valialkin
6b0550c023
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin
c2bd75926b
app/vmselect/netstorage: initialize tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths 2022-07-30 00:39:36 +03:00
Aliaksandr Valialkin
19a0b4679a
app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel
This should reduce CPU usage needed for rand.Rand initialization
2022-07-30 00:30:37 +03:00
Dmytro Kozlov
a927814e7b
vmselect/promql: add tests for vmrangeBucketsToLE (#2907)
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-26 20:42:41 +03:00
Aliaksandr Valialkin
c888e6b9be
app/vmselect/promql: reduce the diff for f148cffc8a
This is a follow-up for c826f06366
2022-07-26 19:20:48 +03:00
Alan Liang
c826f06366
vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902)
Co-authored-by: alanwzliang <alanwzliang@tencent.com>
2022-07-25 10:55:13 +03:00
Aliaksandr Valialkin
92630c1ab4
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
2022-07-25 09:18:44 +03:00