Commit graph

2082 commits

Author SHA1 Message Date
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
56a9ea3753
docs/vmalert.md: mention latency_offset query arg, which has been added in 86dae56bd0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:20:37 -08:00
Aliaksandr Valialkin
285841bcce
app/vmselect/prometheus: follow-up after 86dae56bd0
Return error if the provided latency_offset query arg cannot be parsed.
This should simplify debugging in production.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:13:20 -08:00
Roman Khavronenko
86dae56bd0
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 16:54:57 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
5d30080555
lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes 2022-12-14 17:52:32 -08:00
Aliaksandr Valialkin
38341802c2
app/vmselect: add /expand-with-exprs page 2022-12-14 16:18:55 -08:00
Aliaksandr Valialkin
231569f89b
docs/vmagent.md: small formatting fix 2022-12-14 14:25:35 -08:00
Aliaksandr Valialkin
ac5948b3f3
app/vmselect/vmui: make vmui-update 2022-12-14 12:01:48 -08:00
Aliaksandr Valialkin
7a61bafe59
docs/vmagent.md: clarify that relabeling is actually a debugging at relabel debug section 2022-12-13 15:46:21 -08:00
Yury Molodov
d3418bafc0
fix: prevent run query when selecting autocomplete option (#3480) 2022-12-13 09:30:05 -08:00
Aliaksandr Valialkin
33bff00890
app/vmselect/vmui: make vmui-update 2022-12-12 17:46:30 -08:00
Yury Molodov
cca3bc756b
vmui: minor enhancements (#3471)
* update package-lock.json

* fix: correct handle click by "action" on cardinality page

* fix: correct styles for icons width

* feat: add layout with copyright

* feat: add website and issue to footer
2022-12-12 17:44:13 -08:00
Aliaksandr Valialkin
d1af6046c7
lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead
Blocked small merges may result into big number of small parts, which, in turn,
may result in increased CPU and memory usage during queries, since queries need to inspect
all the existing small parts.

The issue has been introduced in 8189770c50

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-12 17:00:50 -08:00
Zakhar Bessarab
ebcb4ab617
{app/{vmbackup/vmrestore}: update path example to use Azure terminology for consistency (#3475) 2022-12-12 22:17:22 +03:00
Roman Khavronenko
8286f9608f
vmalert: support $for or .For template variables (#3474)
support `$for` or `.For` template variables  in alert's annotations.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-12 22:16:10 +03:00
Aliaksandr Valialkin
88597f187b
app/{vmagent,vminsert}/datadog: make the host label optional in DataDog data ingestion protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-10 23:32:31 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
b01607e3fb
docs: clarify that single-node VictoriaMetrics also provides functionality for relabel debugging
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2022-12-10 20:49:52 -08:00
Aliaksandr Valialkin
a30ae502ef
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2022-12-10 12:44:45 -08:00
Aliaksandr Valialkin
3e7276639e
app/{vminsert,vmselect}: move the handler for /metric-relabel-debug from vminsert to vmselect to be consistent with the cluster version 2022-12-10 02:45:40 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
6f0179405a
app/vmalert: properly handle nil req passed to requestToCurl()
This fixes a panic in the TestAlertingRule_Exec_Negative test.
The panic has been introduced in the commit b97bd01605
2022-12-10 02:04:17 -08:00
Aliaksandr Valialkin
ca59d3de59
app/vmalert: do not show system links at http://vmalert:8880/ page when it is requested via proxy
The system links are absolute, e.g. they start from `/`, so there are high chances
they won't work as expected when requested via proxy such as vmselect with -vmalert.proxyURL
command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3424
2022-12-09 11:46:28 -08:00
Roman Khavronenko
b97bd01605
vmalert: do not hold pointer to http.Request (#3467)
http.Request was used as a part of state struct
for generating the curl command when viewing the rule's
state changes.
It appears, that holding a referencing is far more expensive
than generating the curl command immediately.
On the test with 40k rules, this change reduces memory
and CPU usage by 50%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-09 18:13:29 +03:00
Roman Khavronenko
7c0ae3a86a
lib/storage: keep sample with the biggest value on timestamp conflict (#3421)
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:06:11 -08:00
Aliaksandr Valialkin
eeacbaf0b6
all: update Go builder from v1.19.3 to v1.19.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved
2022-12-08 16:41:24 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
f9730676d9
app/vmselect/searchutils: do not print flag name responsible for query timeout if the timeout isn't reached
This should make the log message more clear
2022-12-08 13:07:33 -08:00
Aliaksandr Valialkin
3f82e3fa36
docs: follow-up after e1bf2a85d0559d112908ce81597f3261d3a085c0
- Document the change at docs/CHANGELOG.md
- Run `make docs-sync` for copying app/vmgateway/README.md to docs/vmgateway.md
  in order to propagate docs' changes to https://docs.victoriametrics.com/vmgateway.html
2022-12-06 21:05:22 -08:00
Aliaksandr Valialkin
758e8a15fd
app/vmselect: typo fixes in code comments 2022-12-06 20:58:16 -08:00
Roman Khavronenko
3dec847c93
vmalert: correctly return error for RW failures (#3452)
* vmalert: correctly return error for RW failures

By mistake, in 0989649ad0 the error
for remote write failures weren't return to user.
This change fixes it.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-06 15:36:46 +01:00
Aliaksandr Valialkin
e2e341da9f
app/vmselect/vmui: make vmui-update after 7645d9ae00 2022-12-05 23:07:08 -08:00
Pedro Gonçalves
1e0666abb4
Datadog - Add device as a tag if it's present as a field in the series object (#3431)
* Datadog - Add device as a tag if it's present as a field in the series object

* address PR comments
2022-12-05 23:06:03 -08:00
Yury Molodov
7645d9ae00
feat: add toggle query display by Ctrl (#3449) 2022-12-05 22:45:15 -08:00
Yury Molodov
01a9b36a95
vmui: timezone select (#3414)
* feat: add timezone selection

* vmui: provide feature timezone select

* fix: correct timezone with relative time

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:44:31 -08:00
Yury Molodov
fd43b5bad0
vmui: fix multi-line query (#3448)
* fix: remove prevent nav by up/down keys for multi-line query

* fix: add query params encode in URL
2022-12-05 21:56:54 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Zakhar Bessarab
86c31f2955
app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450) 2022-12-05 17:18:09 -08:00
Aliaksandr Valialkin
8189770c50
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:16:14 -08:00
Yury Molodov
461158a437
fix: add word-break for tooltip (#3437) 2022-12-05 08:50:34 +01:00
Roman Khavronenko
91a8afa172
vmalert: reduce allocations for Prometheus resp parse (#3435)
Method `metrics()` now pre-allocates slices for labels
and results from query responses. This reduces the number 
of allocations on the hot path for instant requests.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-05 08:34:54 +01:00
Aliaksandr Valialkin
14660d4df5
all: typo fix: the the -> the 2022-12-03 21:53:01 -08:00
Roman Khavronenko
bcba5d2a78
vmalert: fix replay step param (#3428)
The recent change in modifying default value
of `datasource.queryStep` flag resulted in situation
where replay mode was always running queries with
step=`datasource.queryStep`. When it should always
use rule's evaluation interval.

The fix is related not to replay mode only, but
for all Range requests. Now step param is set
individually for each mode.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 13:57:53 +01:00
Zakhar Bessarab
30fea30685
app/vmalert: add remoteWrite.sendTimeout command-line flag to configure timeout for sending data to remoteWrite.url (#3423)
* app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url`

* vmalert: remove WriteTimeout from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the WriteTimeout.

* vmalert: remove DisablePathAppend from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the DisablePathAppend.

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:57:19 +01:00
Roman Khavronenko
8cc4f7eac6
vmalert: properly pass headers during the restore procedure (#3420)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:27:39 +01:00
Aliaksandr Valialkin
3cdff3de23
docs/vmagent.md: update after 959f06d175 2022-11-29 21:33:49 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
c7ce4979ec
all: follow-up after 05cf8a6ecc 2022-11-29 21:03:59 -08:00
Aliaksandr Valialkin
4822406b64
app/vmalert: substitute -datasource.disablePathAppend with -remoteRead.disablePathAppend in the description for -datasource.url command-line flag
This is a follow-up for 959f06d175
2022-11-29 20:36:41 -08:00