Commit graph

744 commits

Author SHA1 Message Date
Dmytro Kozlov
b75f1854c5
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:41:09 +03:00
Aliaksandr Valialkin
99bc18774c
app/vmselect/vmui: make vmui-update after 1304824201 2022-09-14 13:33:06 +03:00
Yury Molodov
1b41169415
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:55:11 +03:00
Yury Molodov
9541ef2e9e
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:43:37 +03:00
Dmytro Kozlov
4415c71a2b
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049)
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
2022-09-06 13:25:59 +03:00
Aliaksandr Valialkin
ae31b2363f
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:55:48 +03:00
匠心零度
50e2524bc2
api prometheus/api/v1/label/../values time not specified, (#3052)
modify default start values
2022-09-05 11:52:19 +03:00
Aliaksandr Valialkin
7dc632719d
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:18:04 +03:00
Aliaksandr Valialkin
4076277cf0
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 19:46:27 +03:00
匠心零度
6d81584d2a
reduce unnecessary vmstorage query (#3031)
* reduce unnecessary vmstorage query

* reduce unnecessary vmstorage query

* rollback limit logic /api/v1/label/*
2022-08-30 12:36:54 +03:00
Aliaksandr Valialkin
ad11b8d83d
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:35:49 +03:00
Dmytro Kozlov
2d71b4859c
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:33:56 +03:00
Dmytro Kozlov
463ea6897b
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:25:18 +03:00
Aliaksandr Valialkin
343241680b
all: remove the remaining bits of io/ioutil
The io/ioutil package is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics requires at least Go1.18, so it is time to remove the io/ioutil from source code

This is a follow-up for 02ca2342ab
2022-08-22 00:20:58 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
4ac79d29ad
app/vmselect: follow-up after 63e0f16062
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
2022-08-15 13:50:16 +03:00
Roman Khavronenko
63e0f16062
vmselect: introduce UserReadableError type of error (#2894)
When read query fails, VM returns rich error message with
all the details. While these details might be useful
for debugging specific cases, they're usually too verbose
for users.
Introducing a new error type `UserReadableError` is supposed
to allow to return to user only the most important parts
of the error trace. This supposed to improve error readability
in web interfaces such as VMUI or Grafana.

The full error trace is still logged with the full context
and can be found in vmselect logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-15 13:38:47 +03:00
Yury Molodov
1513866d51
vmui: shortcut keys legend (#2971)
* feat: add shortcut modal

* feat: add shortcut descriptions

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-15 11:54:47 +03:00
Yury Molodov
3cb013aeb8
vmui: graph action on moush hold and move (#2915)
* fix: change event for graph panning

* fix: change detect key

* feat: add zoom in with mouse selection

* - document the change
- run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-07 23:55:09 +03:00
Aliaksandr Valialkin
7478d423c5
app/vmselect/netstorage: cleanup after 92630c1ab4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896
2022-08-04 18:28:11 +03:00
Aliaksandr Valialkin
6b0550c023
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin
c2bd75926b
app/vmselect/netstorage: initialize tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths 2022-07-30 00:39:36 +03:00
Aliaksandr Valialkin
19a0b4679a
app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel
This should reduce CPU usage needed for rand.Rand initialization
2022-07-30 00:30:37 +03:00
Dmytro Kozlov
a927814e7b
vmselect/promql: add tests for vmrangeBucketsToLE (#2907)
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-26 20:42:41 +03:00
Aliaksandr Valialkin
c888e6b9be
app/vmselect/promql: reduce the diff for f148cffc8a
This is a follow-up for c826f06366
2022-07-26 19:20:48 +03:00
Alan Liang
c826f06366
vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902)
Co-authored-by: alanwzliang <alanwzliang@tencent.com>
2022-07-25 10:55:13 +03:00
Aliaksandr Valialkin
92630c1ab4
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
2022-07-25 09:18:44 +03:00
Aliaksandr Valialkin
79d967d35a
app/vmselect/vmui: make vmui-update after edecd2493c 2022-07-21 20:59:52 +03:00
Roman Khavronenko
9ccf695d57
vmselect: return correct error for second part of expression (#2893)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-20 16:44:28 +02:00
Aliaksandr Valialkin
cc7d499bbd
app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied
This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886
2022-07-19 14:27:48 +03:00
Aliaksandr Valialkin
b5da47bfaf
app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing 2022-07-18 14:24:54 +03:00
Aliaksandr Valialkin
0792c4ca90
app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results 2022-07-18 14:24:53 +03:00
Yury Molodov
17c33132df
vmui: optimize table view (#2867)
* feat: optimize table view

* fix: add column display setting

* app/vmselect: `make vmui-update`

Also document the change at docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-14 00:15:43 +03:00
Aliaksandr Valialkin
29e53b9f55
app/vmselect/promql: consistency update after 93fbd0c54b 2022-07-13 12:33:14 +03:00
Roman Khavronenko
93fbd0c54b
promql: return step as scrapeInterval when it can't be calculated (#2865)
The change allows to specify default value for `getScrapeInterval`
function when actual interval can't be calculated.

Before the change, function were returning `maxSilenceInterval` (5m)
in such cases, which may be not correct for instant queries processing.
The specific scenario where using `maxSilenceInterval` caused issues
is the following:
1. Series becomes stale;
2. Client (in this case vmalert) continues to request series every 15s;
3. Database returns empty results as expected;
4. But at some specific moment of time database returns datapoints from `now()-5m`,
because lookback window was extended to `maxSilenceInterval`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-13 12:27:38 +03:00
Aliaksandr Valialkin
c2197ad139
app/vmselect/promql: validate function name before evaluating its arguments
This avoids unneeded evaluation of args for unknown functions
2022-07-12 19:48:26 +03:00
Aliaksandr Valialkin
159c2e15e3
app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples 2022-07-12 12:31:38 +03:00
Aliaksandr Valialkin
8429d4af5a
app/vmselect/netstorage: add mergeSortBlocks benchmark for the worstcase 2022-07-12 12:31:36 +03:00
Aliaksandr Valialkin
cad471037a
app/vmselect/prometheus: follow-up after 3efe33b917 2022-07-11 20:35:28 +03:00
Dmytro Kozlov
3efe33b917
vmselect/prometeus: Add limit param to api/v1/series api endpoint (#2851)
* issue-2841: Add limit param to api/v1/series api endpoint

* issue-2841: add change log

* issue-2841: update logic

* issue-2841: simplify logic

* issue-2841: simplify logic, add information to documentation
2022-07-11 20:18:30 +03:00
Aliaksandr Valialkin
ce68e76d62
app/vmselect: follow-up after 8667307d73 2022-07-11 20:14:34 +03:00
Roman Khavronenko
8667307d73
vmselect: cover special cases for vmalert's routing in single-node version (#2845)
* vmselect: cover special cases for vmalert's routing in single-node version

* remove trailing `/` from requests
* redirect to vmalert's home page when `/vmalert` is requested.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmselect/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-11 19:52:22 +03:00
Aliaksandr Valialkin
5c8eee26bf
all: make fmt via the upcoming Go1.19 2022-07-11 19:22:15 +03:00
Aliaksandr Valialkin
cd09f583fe
app/vmselect/netstorage: add benchmarks for mergeSortBlocks
This is a follow-up for 743ff84863
2022-07-11 12:54:48 +03:00
Aliaksandr Valialkin
743ff84863
app/vmselect/netstorage: optimize mergeSortBlocks function
- Use binary search instead of linear scan when locating the run of smallest timestamps
  in blocks with intersected time ranges. This should improve performance
  when merging blocks with big number of samples

- Skip samples with duplicate timestamps. This should increase query performance
  in cluster version of VictoriaMetrics with the enabled replication.
2022-07-09 00:34:42 +03:00
Roman Khavronenko
e1a41cfab5
metricsql: properly evaluate timezone_offset over time interval (#2842)
* metricsql: properly evaluate `timezone_offset` over time interval

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-08 14:03:56 +03:00
Aliaksandr Valialkin
2e9ae40d56
app/vmselect/vmui: follow-up after 0bf6841140
* Document the bugfix in docs/CHANGELOG.md
* Run `make vmui-update` for updating static js files for vmui, which are included into vmselect
2022-07-08 13:14:17 +03:00
Aliaksandr Valialkin
a6bd442ef9
app/vmui: tune visual presentation of trace view 2022-07-06 14:07:12 +03:00
Aliaksandr Valialkin
bd5b20445e
app/vmselect: add ability to query vmselect from another vmselect 2022-07-06 13:30:12 +03:00
Aliaksandr Valialkin
17de8a41c2
all: follow-up after ed89106274 2022-07-06 12:44:46 +03:00