github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
6464 commits 119 branches 748 tags 355 MiB
Commit graph

1038 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a521135b7b
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2022-12-10 12:47:47 -08:00
Aliaksandr Valialkin
72ea387c86
app/vmselect: properly serve static css for /select/.../metric-relabel-debug page 2022-12-10 03:04:43 -08:00
Aliaksandr Valialkin
97b41e727c
lib/promscrape: implement target-level and metric-level relabel debugging 
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
 2022-12-10 02:25:56 -08:00
Roman Khavronenko
909cd04c55
lib/storage: keep sample with the biggest value on timestamp conflict (#3421) 
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-12-08 18:18:36 -08:00
Aliaksandr Valialkin
353aad9da9
app/vmselect/searchutils: do not print flag name responsible for query timeout if the timeout isn't reached 
This should make the log message more clear
 2022-12-08 13:08:02 -08:00
Aliaksandr Valialkin
a257c5fe53
app/vmselect: typo fixes in code comments 2022-12-06 20:59:17 -08:00
Aliaksandr Valialkin
6423c0b901
app/vmselect/vmui: make vmui-update after 7645d9ae00 2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin
c902500efc
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:24 -08:00
Aliaksandr Valialkin
3a25a4b1de
app/{vminsert,vmselect}: speed up TestInitStopNodes() 2022-12-03 23:53:14 -08:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348) 
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* app/vmselect: fix error message

* {app/vmstorage,app/vmselect}: fix error messages

* app/vmselect: change log level for error handling

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-11-25 10:32:45 -08:00
Aliaksandr Valialkin
693d68f307
app/vmselect/vmui: make vmui-update after 37cda9abd0 2022-11-25 07:34:03 -08:00
Aliaksandr Valialkin
2f7152deb3
app/vmselect/vmui: make vmui-update after eb772aa50e 2022-11-24 17:37:33 -08:00
Aliaksandr Valialkin
587f90b4cb
app/vmselect/vmui: make vmui-update after 7dc2349913 2022-11-22 15:36:19 +02:00
Aliaksandr Valialkin
2f9df62795
app/vmselect/vmui: make vmui-update after 7d1b3e7e14 2022-11-22 00:35:05 +02:00
Aliaksandr Valialkin
ad548abd6a
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range 
This may be useful for analyzing correlation between time series with different value ranges
 2022-11-21 23:25:35 +02:00
Aliaksandr Valialkin
2d361a834d
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
 2022-11-21 16:48:10 +02:00
Aliaksandr Valialkin
b0fefe562a
app/vmselect/promql: optimize e1 op e2 when e1 returns an empty result 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
 2022-11-21 16:09:49 +02:00
Aliaksandr Valialkin
353b137ff0
app/vmselect/netstorage: typo fix after 61736e4a1d 2022-11-19 00:53:54 +02:00
Aliaksandr Valialkin
97eafbe4a7
app/vmselect: clarify that it isnt recommended setting -replicationFactor at vmselect nodes even if the replication is enabled at vminsert nodes 2022-11-18 14:04:12 +02:00
Aliaksandr Valialkin
61736e4a1d
app/vmselect/netstorage: remove superflouos map lookup at ProcessSearchQuery 
This should reduce CPU usage a bit during querying
 2022-11-18 13:49:59 +02:00
Aliaksandr Valialkin
eb784ff399
app/vmselect/netstorage: emit more useful information in query traces when some of vmstorage nodes return errors or if there is no need to wait for their responses 2022-11-18 13:01:42 +02:00
Yury Molodov
60aaf4b2c2
vmui: add trace analyzer (#3310) 
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add page with trace analyzer

* fix: change detect trace data

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* fix: change descriptions on trace page

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* feat: add base components

* feat: add reset styles

* docs: add description about trace analyzer

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

* feat: add paste json for trace analyzer

* vmui: update trace analyzer docs

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-11-17 22:23:05 +02:00
Aliaksandr Valialkin
7969ca6c80
app/vmselect/vmui: make vmui-update after 51bfd1ab80 2022-11-17 18:57:44 +02:00
Aliaksandr Valialkin
f67c2a2f8f
app/vmselect/vmui: make vmui-update after bc8a782f74 2022-11-17 01:21:56 +02:00
Aliaksandr Valialkin
0cc59b9f95
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:02 +02:00
Aliaksandr Valialkin
facf7efa3a
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:39:17 +02:00
Aliaksandr Valialkin
58e7963828
app/vmselect/vmui: make vmui-update after 7130af7fd2 2022-11-09 16:43:09 +02:00
Aliaksandr Valialkin
fe8d40f12c
app/{vminsert,vmselect}: test initialization with different number of storage nodes 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
 2022-11-09 11:48:39 +02:00
Zakhar Bessarab
fd36c9b3b1
docs/{vminsert,vmselect}: clarify flag description for clusternativeListenAddr (#3314) 
docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`
 2022-11-04 09:49:53 +01:00
Zakhar Bessarab
7afa67dba0
{app/vmselect,app/vminsert}: ensure -storageNodes flag has no empty values (#3291) 2022-11-01 14:54:55 +02:00
Aliaksandr Valialkin
a02a5f646f
app/vmselect/vmui: make vmui-update after 54e1865d17 2022-10-28 14:52:16 +03:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes 
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
 2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
976bbe3677
app/{vminsert,vmselect}: limit the access to storageNodes to getStorageNodesBucket and setStorageNodesBucket functions 
This makes the code more maintainable and earier to test.
 2022-10-28 11:41:55 +03:00
Aliaksandr Valialkin
46450995ae
app/vmselect/vmui: make vmui-update after eae6063450 2022-10-26 02:51:09 +03:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop 
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
 2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
596251eb87
app/vmselect/vmui: make vmui-update after 274e235bf7 2022-10-24 21:29:57 +03:00
Aliaksandr Valialkin
c92aef39b5
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:14:07 +03:00
Aliaksandr Valialkin
574b9f4f78
app/vmselect/vmui: make vmui-update after 080562030d 2022-10-18 10:51:05 +03:00
Yury Molodov
a50f2e141c
vmui: limit number of plotted series (#3229) 
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-10-13 12:14:53 +03:00
Aliaksandr Valialkin
9bf8d64605
app/vmselect/promql: follow-up for 930f1ee153 
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
 2022-10-13 12:06:24 +03:00
Siqi Liu
c0fc716426
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225) 
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
 2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
a0f3247b14
app/vmselect/promql: properly handle zero and negative values for -search.maxMemoryPerQuery 
This is a follow-up for 04a05f161c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin
098c9bda27
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set 
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin
d8057fca0b
docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag 
This is a follow-up for 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-09 13:58:18 +03:00
Aliaksandr Valialkin
938ff7bba6
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
 2022-10-08 01:14:01 +03:00
Yury Molodov
9e4e5abac6
vmui: auto-update chart after query field removed (#3210) 
* feat: run query after query field removed

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-10-07 23:08:22 +03:00
Aliaksandr Valialkin
98bd843f3c
app/{vminsert,vmselect,vmstorage}: add package-{vminsert,vmselect,vmstorage}-{goarch} Makefile rules for building arch-specific docker images 2022-10-07 20:17:01 +03:00
Aliaksandr Valialkin
58ee1b9edb
app/vmselect/vmui: make vmui-update after a54987f671 2022-10-07 03:31:00 +03:00
Aliaksandr Valialkin
b05ff477c8
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:23:17 +03:00
Aliaksandr Valialkin
baa11a778d
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:38 +03:00
Aliaksandr Valialkin
958c1f291c
app: follow-up after ec04fcac93 
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
 2022-10-06 14:54:15 +03:00
Dmytro Kozlov
4064db27a8
Properly parse json when export import metric (#3180) 
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API

* app/vmselect: update convert function

* app/vmselect: export null if `math.IsNaN(v)`

* app/vmselect: get float from json

* lib/protoparser: add test

* docs: add change log

* lib/protoparser: make export import api compatible
 2022-10-06 14:54:14 +03:00
Aliaksandr Valialkin
43bdd96a6e
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:16:07 +03:00
Aliaksandr Valialkin
5497997b72
app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores 
Use sync.Map instead of a global mutex there. This should lift scalability limits
on systems with many CPU cores.
 2022-10-01 20:14:14 +03:00
Aliaksandr Valialkin
972fcfcd17
app/vmselect/prometheus: improve scalability of /federate endpoint on systems with many CPU cores 
Minimize usage of global lock inside bufferedwriter.Write() when processing `/federate` data
on systems with many CPU cores
 2022-10-01 20:14:13 +03:00
Aliaksandr Valialkin
ae6c7edf04
app/vmselect: do not export NaN values for stale metrics at /federate endpoint 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
 2022-10-01 19:48:21 +03:00
Aliaksandr Valialkin
6f9ce3f6d6
lib/flagutil: rename Array to ArrayString 
This makes the ArrayString more consistent with other Array* types.

While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
 2022-10-01 18:28:19 +03:00
Aliaksandr Valialkin
93e84a1c57
lib/httpserver: use 302 redirects instead of 301 redirects 
Incorrect 301 redirects can be cached by user agents such as web browsers.
This can complicate recovery procedure after the incorrect redirect is fixed,
e.g. web browser cache must be reset.

The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
 2022-10-01 16:56:43 +03:00
Aliaksandr Valialkin
98175493d7
app/vmselect/promql: remove empty series before applying aggregate function 
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
 2022-09-30 08:44:06 +03:00
Roman Khavronenko
273472dc20
app/vmselect: ignore empty series for limit_offset (#3178) 
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-09-30 08:44:05 +03:00
Roman Khavronenko
4acbf6e7d4
vmselect/rollup: rm workaround for slow-changing counters (#3163) 
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-09-26 15:39:49 +03:00
Dmytro Kozlov
36ea8537d9
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982) 
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-14 17:42:07 +03:00
Aliaksandr Valialkin
b6bdbb4787
app/vmselect/vmui: make vmui-update after 1304824201 2022-09-14 13:32:43 +03:00
Yury Molodov
465894d720
vmui: fix data processing (#3092) 
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-12 10:56:39 +03:00
Yury Molodov
c8de98e03f
vmui: add lists of top queries (#3065) 
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-08 21:44:43 +03:00
Dmytro Kozlov
ec273eafef
app/{vmselect, vminsert}: fails with error when user defines equal addresses in the -storageNodes flag (#3082) 2022-09-08 21:17:58 +03:00
Aliaksandr Valialkin
f0eea5b02d
app/vmselect/netstorage: fix a typo, which leads to incorrect query results in VictoriaMetrics cluster 
The typo has been introduced in the commit 1a254ea20c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067
 2022-09-08 13:46:40 +03:00
Dmytro Kozlov
96ecec877d
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049) 
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
 2022-09-06 14:48:20 +03:00
Aliaksandr Valialkin
b7f3569522
app/vmselect/prometheus: follow-up after 50e2524bc2 
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
 2022-09-05 11:57:08 +03:00
匠心零度
65d1124dc4
api prometheus/api/v1/label/../values time not specified, (#3052) 
modify default start values
 2022-09-05 11:51:32 +03:00
Aliaksandr Valialkin
c6b74148cf
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
 2022-09-02 23:19:05 +03:00
Aliaksandr Valialkin
9cca3a0a1b
app/vmselect/netstorage: fix potential panic under high load 
The panic may trigger during data blocks' processing received
from vmstorage nodes when some of vmstorage nodes return an error
or when `-replicationFactor` is set to values higher than 2 at `vmselect`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058
 2022-09-02 21:36:15 +03:00
Aliaksandr Valialkin
024e2f18da
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance 
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
 2022-09-02 21:01:04 +03:00
匠心零度
80067fd03a
reduce unnecessary vmstorage query (#3031) 
* reduce unnecessary vmstorage query

* reduce unnecessary vmstorage query

* rollback limit logic /api/v1/label/*
 2022-08-30 12:34:23 +03:00
Aliaksandr Valialkin
8aaaf221cc
app/vmselect/promql: follow-up after 2d71b4859c 
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
 2022-08-27 01:38:17 +03:00
Dmytro Kozlov
2cf29cc4d1
vmselect/promql: fix panic in histogram_quantiles function (#3029) 
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
 2022-08-27 01:34:54 +03:00
Dmytro Kozlov
d32a6359b0
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963) 
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-08-24 15:27:41 +03:00
Aliaksandr Valialkin
08b8467e97
app/vmselect/netstorage: make golangci-lint happy by naming the unused padding field as _ 2022-08-22 00:32:37 +03:00
Aliaksandr Valialkin
9ddd2699fd
all: remove the remaining bits of io/ioutil 
The io/ioutil package is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics requires at least Go1.18, so it is time to remove the io/ioutil from source code

This is a follow-up for 02ca2342ab
 2022-08-22 00:22:41 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
87e0d69bf4
app/vmselect/netstorage: fix a bug introduced in 1a254ea20c 
The bug results in `duplicate output time series` error
because the same time series is added two times into the orderedMetricNames list
inside the tmpBlocksFileWrapper.Finalize().

While at it, properly release all the tmpBlocksFile structs on tbf.Finalize() error.
Previously only the remaining tbf entries were released. This could result in resource leak.
 2022-08-17 14:07:51 +03:00
Aliaksandr Valialkin
7d7cf2b6fd
app/vmselect: follow-up after 63e0f16062 
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
 2022-08-15 13:53:19 +03:00
Roman Khavronenko
8a26ec435d
vmselect: introduce UserReadableError type of error (#2894) 
When read query fails, VM returns rich error message with
all the details. While these details might be useful
for debugging specific cases, they're usually too verbose
for users.
Introducing a new error type `UserReadableError` is supposed
to allow to return to user only the most important parts
of the error trace. This supposed to improve error readability
in web interfaces such as VMUI or Grafana.

The full error trace is still logged with the full context
and can be found in vmselect logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-08-15 13:53:18 +03:00
Yury Molodov
dc52b283a3
vmui: shortcut keys legend (#2971) 
* feat: add shortcut modal

* feat: add shortcut descriptions

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-08-15 12:04:30 +03:00
Aliaksandr Valialkin
1a254ea20c
app/vmselect/netstorage: remove common contention points related to inter-CPU communcations 
This should improve vmselect performance scalability on systems with many CPU cores.

The following tasks were done:

- Use separate temporary files for storing the data read from each vmstorage node.
  This may result in the following potential issues:
  - Up to N times higher memory usage for performing each query where N is the number
    of vmstorage nodes known to vmselect.
    This issue shouldn't increase chances of out of memory errors in most cases,
    since per-query memory overhead is quite low comparing to the overall vmselect memory usage.
  - Up to N times higher number of open temporary files where N is the number
    of vmstorage nodes known to vmselect.
    This issue should be fixed by increasing the limit on the number of open files.

- Use separate counters per each vmstorage node for various stats calculation
  when reading the data from vmstorage nodes.
 2022-08-11 23:22:56 +03:00
Aliaksandr Valialkin
ec3df0b913
app/vmselect/netstorage: improve scalability of blocks processing on systems with multiple CPU cores 
Previously a single syncwg.WaitGroup was used for tracking the lifetime of processBlock callbacks
across all the per-vmstorage goroutines. This could be slow on systems with many CPU cores
because of inter-CPU synchronization overhead.

Use a separate per-vmstorage sync.WaitGroup instead in order to reduce inter-CPU synchronization overhead.
This should imrpove performance for heavy queries over big number of blocks on multi-CPU systems.
 2022-08-11 21:37:24 +03:00
Aliaksandr Valialkin
1996e36cf0
app/vmselect/netstorage: prevent from calling processBlocks callback after the exit from ProcessBlocks function 
This should prevent from panic at multi-level vmselect
when the top-level vmselect is configured with -replicationFactor > 1
 2022-08-08 13:32:44 +03:00
Aliaksandr Valialkin
2635211bf4
app/vmselect/netstorage: properly detect and log timeout errors when querying vmstorage from vmselect 
This change is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2937

Thanks to @isodude for the initial pull request.
 2022-08-08 00:21:05 +03:00
Yury Molodov
a2ecf311b8
vmui: graph action on moush hold and move (#2915) 
* fix: change event for graph panning

* fix: change detect key

* feat: add zoom in with mouse selection

* - document the change
- run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-08-07 23:56:08 +03:00
Aliaksandr Valialkin
21f588e9cf
app/vmselect/promql: fix TestVmrangeBucketsToLE test after 8516670582 
Remove 'AccountID=0, ProjectID=0, ' prefix for cluster version
 2022-08-07 00:15:10 +03:00
Aliaksandr Valialkin
43185353bc
app/vmselect/netstorage: cleanup after 92630c1ab4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896
 2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin
a1e49606ed
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
 2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin
c81d2b4c18
app/vmselect/netstorage: initializes tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths 2022-07-30 00:39:14 +03:00
Aliaksandr Valialkin
5ddae2e293
app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel 
This should reduce CPU usage needed for rand.Rand initialization
 2022-07-30 00:31:00 +03:00
Dmytro Kozlov
ab645b5fab
vmselect/promql: add tests for vmrangeBucketsToLE (#2907) 
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-26 20:43:43 +03:00
Aliaksandr Valialkin
8a83c59956
app/vmselect/promql: reduce the diff for f148cffc8a 
This is a follow-up for c826f06366
 2022-07-26 19:21:16 +03:00
Alan Liang
f97de473f4
vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902) 
Co-authored-by: alanwzliang <alanwzliang@tencent.com>
 2022-07-25 10:49:48 +03:00
Aliaksandr Valialkin
764fc04756
app/vmalert: fix after da10962d4c 2022-07-25 09:48:41 +03:00
Roman Khavronenko
da10962d4c
vmselect: cover special cases for vmalert's routing in single-node version (#2845) 
* vmselect: cover special cases for vmalert's routing in single-node version

* remove trailing `/` from requests
* redirect to vmalert's home page when `/vmalert` is requested.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmselect/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-25 09:43:44 +03:00
Aliaksandr Valialkin
3d4c312ba2
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system 
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
 2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin
c0c9f30870
lib/pushmetrics: properly handle errors when initializing pushmetrics 2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin
59bb42be06
app/vmselect/vmui: make vmui-update after edecd2493c 2022-07-21 21:00:36 +03:00
Roman Khavronenko
04c4f8bafd
vmselect: return correct error for second part of expression (#2893) 
Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-07-21 20:40:47 +03:00
Aliaksandr Valialkin
fe68bb3ba7
all: follow-up after 46f803fa7a 
Add -pushmetrics.* command-line flags to all the VictoriaMetrics apps
 2022-07-21 20:18:25 +03:00
Aliaksandr Valialkin
f992f96a88
app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied 
This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886
 2022-07-19 14:29:41 +03:00
Aliaksandr Valialkin
d920b0afec
app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing 2022-07-18 14:24:23 +03:00
Aliaksandr Valialkin
f933d5f142
app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results 2022-07-18 14:24:22 +03:00
Aliaksandr Valialkin
70b9925bf7
app: fix make publish-* after ed93330e66 
Add missing `-linux` substring to built binary names for copying into Docker images
 2022-07-14 11:01:34 +03:00
Yury Molodov
1ca20caa4b
vmui: optimize table view (#2867) 
* feat: optimize table view

* fix: add column display setting

* app/vmselect: `make vmui-update`

Also document the change at docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-14 00:16:57 +03:00
Aliaksandr Valialkin
da6c85a2f6
all: follow-up for d99ba3481b 2022-07-13 17:17:08 +03:00
Dmytro Kozlov
4e4def9df8
Rename release packages (#2810) 
* makefile: add os to each release file

* makefile: update vmutils arm64

* makefile: update victoria-metrics release process

* makefile: update publish with os

* makefile: update publish with os

* makefile: change tar library

* update release logic

* copy all releases

* sort command by GOOS

* rollback commands

* rollback OSARCH

* fix commands

* cleanup

* fix windows build

* sort build by GOOS, update README.md
 2022-07-13 17:11:01 +03:00
Aliaksandr Valialkin
919939ee9f
app/vmselect/promql: consistency update after 93fbd0c54b 2022-07-13 12:34:28 +03:00
Roman Khavronenko
214e01499c
promql: return step as scrapeInterval when it can't be calculated (#2865) 
The change allows to specify default value for `getScrapeInterval`
function when actual interval can't be calculated.

Before the change, function were returning `maxSilenceInterval` (5m)
in such cases, which may be not correct for instant queries processing.
The specific scenario where using `maxSilenceInterval` caused issues
is the following:
1. Series becomes stale;
2. Client (in this case vmalert) continues to request series every 15s;
3. Database returns empty results as expected;
4. But at some specific moment of time database returns datapoints from `now()-5m`,
because lookback window was extended to `maxSilenceInterval`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin
5867708b3d
app/vmselect/promql: validate function name before evaluating its arguments 
This avoids unneeded evaluation of args for unknown functions
 2022-07-12 19:49:54 +03:00
Aliaksandr Valialkin
fbb403b5c0
app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples 2022-07-12 12:30:24 +03:00
Aliaksandr Valialkin
aee08117e9
app/vmselect/netstorage: add mergeSortBlocks benchmark for the worstcase 2022-07-12 12:26:27 +03:00
Dmytro Kozlov
6734d77d97
vmselect/prometeus: Add limit param to api/v1/series api endpoint (#2851) 
* issue-2841: Add limit param to api/v1/series api endpoint

* issue-2841: add change log

* issue-2841: update logic

* issue-2841: simplify logic

* issue-2841: simplify logic, add information to documentation
 2022-07-11 20:37:20 +03:00
Aliaksandr Valialkin
ed2126aec3
app/vmselect: follow-up after 6dda254e01 2022-07-11 19:48:18 +03:00
Roman Khavronenko
6dda254e01
vmselect: cover special cases for vmalert's routing in cluster version (#2844) 
* vmselect: cover special cases for vmalert's routing

* remove trailing `/` from requests
* redirect to vmalert's home page when `/vmalert` is requested.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmselect: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmselect/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-11 19:33:33 +03:00
Aliaksandr Valialkin
eab8ebbe11
all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin
c0af52228a
app/vmselect/netstorage: add benchmarks for mergeSortBlocks 
This is a follow-up for 743ff84863
 2022-07-11 12:53:46 +03:00
Aliaksandr Valialkin
d442ee4610
app/vmselect/netstorage: optimize mergeSortBlocks function 
- Use binary search instead of linear scan when locating the run of smallest timestamps
  in blocks with intersected time ranges. This should improve performance
  when merging blocks with big number of samples

- Skip samples with duplicate timestamps. This should increase query performance
  in cluster version of VictoriaMetrics with the enabled replication.
 2022-07-09 00:35:38 +03:00
Roman Khavronenko
78da074ad3
metricsql: properly evaluate timezone_offset over time interval (#2842) 
* metricsql: properly evaluate `timezone_offset` over time interval

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-08 14:04:40 +03:00
Aliaksandr Valialkin
decbf84e55
app/vmselect/vmui: follow-up after 0bf6841140 
* Document the bugfix in docs/CHANGELOG.md
* Run `make vmui-update` for updating static js files for vmui, which are included into vmselect
 2022-07-08 13:22:57 +03:00
Aliaksandr Valialkin
6bb4bcd30c
app/vmui: tune visual presentation of trace view 2022-07-06 14:16:24 +03:00
Aliaksandr Valialkin
eee384c9c9
app/vmselect: add -clusternative.tls* options for mTLS setup in multi-level clusters 2022-07-06 13:48:21 +03:00
Aliaksandr Valialkin
195dccf678
app/vmselect: add ability to query vmselect from another vmselect 2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin
714958deab
all: follow-up after ed89106274 2022-07-06 12:48:52 +03:00
Aliaksandr Valialkin
cdd89d9cc2
app/vmselect: properly generate response for /api/v1/series 
The response has been broken in 7d5d33fd71
 2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin
270e555f47
lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call 2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin
f4df43f7cc
app/vmselect/netstorage: remove unused auth.Token arg 2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin
78eeca6f0d
lib/vmselectapi: rename deleteMetrics to more correct deleteSeries 2022-07-06 12:46:21 +03:00
Roman Khavronenko
f51bc07d97
vmselect: allow proxying requests to vmalert (#2833) 
The change allows to proxy requests with prefix `/vmalert`
to the vmalert component if `-vmalert.proxyURL` is set.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825
and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2831

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-07-06 10:47:51 +02:00
Aliaksandr Valialkin
10c63f5853
app/vmselect: make fmt after f3ece83e67 2022-07-05 14:56:53 +03:00
Aliaksandr Valialkin
8516670582
app/vmselect/promql: properly calculate histogram_quantile over unexpected le buckets 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819
 2022-07-05 13:20:41 +03:00
Aliaksandr Valialkin
fa08220d27
app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787
 2022-06-30 22:40:26 +03:00
Aliaksandr Valialkin
4fb0f15322
all: readability improvements for query traces 
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
 2022-06-30 18:19:43 +03:00
Dmytro Kozlov
67753cc6f0
vmui: update render logic for nested component (#2795) 
* vmui: update render logic for nested component, avoid rerender, remove local storage usage for tracing flag

* docs/url-examples.md: fix various documentation issues there

* docs: add Troubleshooting doc

This doc contains troubleshooting guides for typical problems with VictoriaMetrics.

* docs/Troubleshooting.md: add troubleshooting guide for cluster instability

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-30 15:48:22 +03:00
Aliaksandr Valialkin
daefb64f38
app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload 
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792
 2022-06-28 20:18:31 +03:00
Aliaksandr Valialkin
9adff0b686
app/vmselect/promql: show the number of scanned samples in the query trace 2022-06-28 19:32:03 +03:00
Aliaksandr Valialkin
c21f5a508f
app/vmselect/prometheus: reduce the default value for -search.maxSeries from 100k to 30k 
Production experience shows that 100k is too big for /api/v1/series .
It leads to increased CPU usage when Grafana queries /api/v1/series over VictoriaMetrics
with big number of time series during auto-completion and when modifying template variables.
 2022-06-28 18:22:24 +03:00
Aliaksandr Valialkin
7d5d33fd71
lib/storage: return marshaled metric names from SearchMetricNames 
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.

While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.

Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
 2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin
399d4c36ae
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day 2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin
a667d339be
app/vmselect/netstorage/netstorage.go: group metrics in order to improve readability a bit 2022-06-27 14:00:24 +03:00
Aliaksandr Valialkin
6386f117c8
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds 2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin
08de733924
app/vmselect/netstorage: assume the response is full if up to -replicationFactor-1 vmstorage nodes are unavailable 
This is a follow-up for ee5c502446

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767
 2022-06-27 12:21:26 +03:00
Aliaksandr Valialkin
bc9d704ef4
app/vmselect/netstorage: remove Get prefix from netstorage functions 
This makes these function names more consistent with the server side
 2022-06-27 00:37:49 +03:00
hagen1778
e40d015e9a vmselect: make vm_partial_results_total consistent 
Metrics `vm_partial_results_total` and `vm_requests_total` serving
the similar purpose, but contain inconsistent set of labels.
This change updates `vm_partial_results_total` labels to be consistent
with `vm_requests_total`.

The change breaks backward compatibility with assumption that
`vm_partial_results_total` wasn't widely used, since it is
not documented and absent in the alerts and dashboards.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-06-24 13:50:26 +02:00
Dmytro Kozlov
f28cbcc7b5
vmui: added query tracing (#2748) 
* vmui: added query tracing

* vmui: updated ui

* vmui: update tracing logic, fix bugs, disable tracing by default

* vmui: use empty message as props

* vmui: fixed ui, added delete for each tacing data, show query in header

* vmui: added timelines

* vmui: speedup render

* vmui: use memo for sorting

* vmui: use Trace model, remove unused functions, simplify part of code

* vmui: update recursive logic

* vmui: fix set query to header

* vmui: code cleanup, remove unused code

* vmui: remove unused type, rename component

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-23 23:00:45 +03:00
Nikolay
ee5c502446
app/vmselect: fixes partial response with replicationFactor (#2777) 
* app/vmselect: fixes partial response with replicationFactor
Allow partial response if it meets replicationFactor configured at vmselect
https://t.me/VictoriaMetrics_ru1/38490

* docs/CHANGELOG.md: document this change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-23 20:17:24 +03:00
Aliaksandr Valialkin
7429dfbe9e
app/vmselect: add -search.setLookbackToStep command-line flag for making the gap filling algorithm similar to InfluxDB data model 
This option should override `-search.maxStalenessInterval` for most cases when users migrate from InfluxDB to VictoriaMetrics
 2022-06-22 14:20:02 +03:00
Aliaksandr Valialkin
4f64da874d
app/vmselect: typo fix in the exported metric name: vm_http_request_total -> vm_http_requests_total 2022-06-22 13:18:03 +03:00
Aliaksandr Valialkin
dceca7e864
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This is a follow-up for fe2269b999
 2022-06-21 20:27:30 +03:00
Aliaksandr Valialkin
fe2269b999
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This package already has the same name, so there is no need in explicit name
 2022-06-21 20:24:28 +03:00
Aliaksandr Valialkin
b28c6febf9
app/{vminsert,vmselect}: add -vmstorageDialTimeout command-line flag for tuning the maximum time needed for establishing connections to vmstorage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2022-06-20 15:17:34 +03:00
Dmytro Kozlov
5bc13e2fe8
vmui: added focusLabel, enable cardinality app configuratior (#2736) 
* vmui: added focusLabel, enable app configuratior

* vmui: set focusLabel if {labelName!=""}

* wip

* docs/CHANGELOG.md: mention about focusLabel feature in cardinality explorer

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2730

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-17 13:04:09 +03:00
Roman Khavronenko
cbe39bbb8d
vmselect: limit end param max value by 2d in future (#2729) 
* vmselect: limit `end` param max value by 2d in future

The change is applied only to service handlers like `/labels` or `/series`
and limits the `end` param by max value <= now() + 2 days. The same limit
is applied for the ingested data, so no reason to allow to request data
in future far than that.

The change is also needed for corner cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2669
where too high `end` value triggers inefficient global index search.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs/CHANGELOG.md: document the bugfix

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-16 20:51:40 +03:00
Aliaksandr Valialkin
da1d1e83df
app/{vmselect,vmstorage}: properly pass seriesCountByLabelName and seriesCountByFocusLabelValue entries from vmstorage to vmselect 2022-06-16 10:44:29 +03:00
Aliaksandr Valialkin
ee9954082f
app/vmselect/netstorage: properly aggregate seriesCountByLabelName and seriesCountByFocusLabelValue obtained from multiple vmselect nodes at /api/v1/status/tsdb 2022-06-15 16:48:40 +03:00
Aliaksandr Valialkin
45fa9d798d
app/vmselect: accept focusLabel query arg at /api/v1/status/tsdb 2022-06-14 18:39:00 +03:00
Aliaksandr Valialkin
fb77843639
lib/storage: show top labels with the highest number of series in cardinality explorer 2022-06-14 16:34:13 +03:00
Dmytro Kozlov
ed9b4c6f6d
vmui: refactor Cardinality panel (#2726) 
* vmui: refactor Cardinality panel

* vmui: change width of the search panel

* vmui: code cleanup

* vmui: code cleanup

* vmui: fixed vulnerability (npm audit fix)

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-14 14:40:44 +03:00
Yury Molodov
65107a4585
vmui: enhancements (#2638) (#2717) 
* feat: make datepicker to be set to last 30 min by default

* fix: correct spinner while loading data

* feat: change legend style

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-13 09:49:58 +03:00
Aliaksandr Valialkin
61e03f172b
app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them 2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin
4a94cd81ce
app/vmselect: add optional limit query arg to /api/v1/labels and /api/v1/label_values endpoints 
This arg allows limiting the number of sample values returned from these APIs
 2022-06-10 10:24:07 +03:00
Aliaksandr Valialkin
1335698ba7
app/vmselect/prometheus: extract common code for obtaining common query args into getCommonParams() function 2022-06-09 20:33:36 +03:00
Aliaksandr Valialkin
cb39eada77
all: improve query tracing coverage for indexdb search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
 2022-06-09 20:04:02 +03:00
Aliaksandr Valialkin
ad44eadd94
app/vmselect/graphite: remove additional redundant Request.ParseForm() calls after 38c785b851 2022-06-09 13:22:23 +03:00
Aliaksandr Valialkin
7fe60dad10
app/vmselect: remove redundant calls to Request.ParseForm() 
Request.ParseForm() is implicitly called by the first call to Request.FormValue()
 2022-06-09 13:09:52 +03:00
Aliaksandr Valialkin
a9ea3fee38
lib/querytracer: make it easier to use by passing trace context message to New and NewChild 
The context message can be extended by calling Donef.
If there is no need to extend the message, then just call Done.
 2022-06-08 21:16:12 +03:00
Aliaksandr Valialkin
2b343d8bd0
app: properly collect and merge /api/v1/status/tsdb info from vmstorage nodes 
The collection has been broken in f2754c3e90

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233
 2022-06-08 19:26:09 +03:00
Dmytro Kozlov
f2754c3e90
Cardinality explorer (#2625) 
* Cardinality explorer

* vmui, vmselect: updated field name, added description to spinner

* make vmui-update

* updated const name, make vmui-update

* lib/storage: changes calculation for totalSeries values

* added static files

* wip

* wip

* wip

* wip

* docs/CHANGELOG.md: document cardinality explorer feature

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233

Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-08 18:54:27 +03:00
Aliaksandr Valialkin
68b6ddfb14
all: follow-up after 8edb390e21 
- Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb
- Add /service-discovery page for API compatibility with Prometheus
- Properly load bootstrap.min.css from /prometheus/targets
- Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there
 2022-06-07 01:05:53 +03:00
Dmytro Kozlov
ce8aade80e
lib/promscrape: adds service discovery visualization for /targets page(#2675) 
* lib/promscrape: updated template

* lib/promscrape: fixed click on unhealthy and all btns

* app/vmselect: jquery scripts into static folder

Co-authored-by: f41gh7 <nik@victoriametrics.com>
 2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin
c92bc5394f
app/vmselect/netstorage: properly read trace from vmstorage when it returns error message to vmselect 2022-06-01 14:35:00 +03:00
Aliaksandr Valialkin
afced37c0b
all: add initial support for query tracing 
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
 2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin
f0ab5ca61f
app/vmselect/vmui: make vmui-update after 492a615a88 2022-05-26 09:43:30 +03:00
Yury Molodov
b9ef9ddbfd
vmui: import dashboards (#2642) 
* fix: switch dashboards import to fetch

* make vmui-update
 2022-05-26 09:43:29 +03:00
Aliaksandr Valialkin
4409b9f376
app/vmselect/vmui: make vmui-update 2022-05-20 14:56:25 +03:00
Yury Molodov
d13437afd0
vmui: setup predefined dashboards without build (#2541) 
vmui: support predefined dashboards in json format

See https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmselect/vmui/dashboards
 2022-05-20 13:46:19 +03:00
Aliaksandr Valialkin
68bfa05437
app/vmselect/promql: do not return values from label_value() if the original time series has no values at the selected timestamps 2022-05-09 17:58:39 +03:00
Aliaksandr Valialkin
8babb4aebc
app/vmselect/vmui: make vmui-update after 450d879eaa 2022-05-05 22:11:41 +03:00
Aliaksandr Valialkin
3fee369997
app/vmselect: follow-up after 8639e79d38 2022-05-04 23:41:00 +03:00
Nikolay
48041305a2
app/vmselect: adds proxy for rules API (#2516) 
* app/vmselect: adds proxy for rules and alerts API
It allows to visualization for rules at grafana
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739

* Update app/vmselect/main.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
 2022-05-04 10:12:15 +03:00
Dima Lazerka
b5fce4190c
Code cleanup (#343) 
* Small code cleanup: remove Request from params

* Extract common params to all export handlers

* Renamed ExportParams -> exportParams

* wip

Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-03 15:53:58 +03:00
Aliaksandr Valialkin
4fa3cd701c
docs/Cluster-VictoriaMetrics.md: typo fix: serparated -> separated 2022-05-03 15:20:11 +03:00
Aliaksandr Valialkin
75bd7f1539
app/vmui: fix up/down arrow keys on multi-line query after a186434b50 2022-05-03 10:51:08 +03:00
Aliaksandr Valialkin
361b08c30e
lib/storage: leave the last sample per each discrete interval during the deduplicaton 
This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness
 2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin
7ca32c21c8
app/vmui: execute query by pressing enter in the same way as Prometheus does 
Multi-line query can be entered via `shift-enter` in the query input field
 2022-05-02 20:24:43 +03:00
Yury Molodov
6292185abf
vmui: support node v.18 (#2529) 
* fix: add support vmui node18

* fix: remove @mui/styles (legacy styling solution)

* Update app/vmui/Dockerfile-build

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-02 20:24:42 +03:00
Nikolay
022b2875ad
app/vmselect: adds API /api/v1/status/buildinfo (#2515) 
* app/vmselect: adds API /api/v1/status/buildinfo
it should fix an compability error with grafana 8.5 prometheus datasource
https://github.com/grafana/grafana/pull/46771

* Update main.go
 2022-04-29 13:02:43 +03:00
Aliaksandr Valialkin
90f9ed260e
app/vmselect/vmui: make vmui-update after da04e9d1de 2022-04-29 13:02:43 +03:00
Dima Lazerka
e6ee235707
Export "null" in jsonl instead of NaN (#2518) 
* Export "null" in jsonl instead of NaN

The NaN appeared because of staleness markers that were added for compatibility. I think it's better to use json `null`, implemented here.

Also maybe it also makes sense to add a flag like `?skip-staleness-markers=true` to `/export`, to skip nulls at all?

* Update app/vmselect/prometheus/export.qtpl

* app/vmselect/prometheus/export.qtpl.go: `make quicktemplate-gen`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-04-29 12:51:03 +03:00
Yury Molodov
dac203320a
vmui: expression alias (#2495) 
* feat: add alias for queries

* docs: update docs for predefined dashboards

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-04-27 10:59:51 +03:00
Aliaksandr Valialkin
6eb1580158
app/vmselect/promql: add tlast_change_over_time(m[d]) function, which returns the timestamp for the last change of m on the given lookbehind window d 2022-04-27 10:58:40 +03:00
Yury Molodov
eae6f68be2
vmui: add support relative time (#2504) 
* feat: add support relative time

* app/vmselect: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-04-26 15:46:47 +03:00
Aliaksandr Valialkin
dac24aa342
app/vmselect/promql: properly handle scalar default vector, scalar if vector and scalar ifnot vector queries 
Previously `vector` time series could be unexpectedly returned from such queries
 2022-04-21 15:34:14 +03:00
Aliaksandr Valialkin
bc54ae9608
app/vmselect/promql: fix comparison to nan 
The comparison to nan has been broken in d335cc886c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
 2022-04-21 14:56:41 +03:00
Aliaksandr Valialkin
a5cfe5d13e
app/vmselect/promql: add drop_common_labels() function 2022-04-21 14:20:36 +03:00