Commit graph

1085 commits

Author SHA1 Message Date
Roman Khavronenko
e4e9dfb785
vmagent: respect -usePromCompatibleNaming if no relabeling is set (#3511)
* vmagent: respect `-usePromCompatibleNaming` if no relabeling is set

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: upd test

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-20 14:32:45 -08:00
Yury Molodov
731d189fa9
fix: change the logic for hide query (#3514) 2022-12-20 14:29:46 -08:00
Aliaksandr Valialkin
4e55b67a44
lib/storage: clear the err if it is set to io.EOF when searching for the TSID by metricID
This is expected error after when recently added indexdb data isn't available for search yet
or wasn't flushed to disk after unclean shutdown of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515
2022-12-20 14:05:29 -08:00
Aliaksandr Valialkin
1d7b5cb83c
docs/CHANGELOG.md: document the change at 547f07463b29c09c62c9af35eac9cee6764b3286
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612
2022-12-20 10:22:07 -08:00
Aliaksandr Valialkin
6bf46c7bf5
docs/CHANGELOG.md: formatting fix 2022-12-20 01:06:34 -08:00
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
680925f872
docs/CHANGELOG.md: add a warning for releases between v1.83.0 and v1.85.1 that it is recommended upgrading to v1.85.2 because of the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
2022-12-19 13:36:49 -08:00
Aliaksandr Valialkin
d2ad184377
docs/CHANGELOG.md: consistently use YYYY-MM-DD format for release dates
The previously used DD-MM-YYYY format could be confused with the MM-DD-YYYY format.
The YYYY-MM-DD format reduces this confusion.
2022-12-19 13:33:05 -08:00
Aliaksandr Valialkin
6530344a8f
docs/CHANGELOG.md: cut v1.85.2 2022-12-19 13:09:29 -08:00
Aliaksandr Valialkin
0bf3ae9559
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname>
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
2022-12-19 13:08:11 -08:00
Aliaksandr Valialkin
6c98b56935
lib/storage: search for TSIDs for the given metricIDs in the previous indexdb if they aren't found in the current indexdb
The issue triggers after the indexdb rotation for time series, which stop receiving new samples.
This results in missing data for such time series in query responses.

This commit should address the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502

The issue has been introduced in 2dd93449d8
2022-12-19 12:03:09 -08:00
Aliaksandr Valialkin
4cb83f0f4a
lib/logger: follow-up for 72f8fce107
- Document the change at docs/CHANELOG.md
- Log fatal errors if the -loggerJSONFields contains unexpected values
- Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348
2022-12-16 17:42:07 -08:00
Aliaksandr Valialkin
65f8fc527f
lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page 2022-12-16 17:02:41 -08:00
Roman Khavronenko
86dae56bd0
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 16:54:57 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
38341802c2
app/vmselect: add /expand-with-exprs page 2022-12-14 16:18:55 -08:00
Aliaksandr Valialkin
0c54ff20eb
docs/CHANGELOG.md: fix the link to the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 2022-12-14 14:17:17 -08:00
Aliaksandr Valialkin
76445bfd98
docs/CHANGELOG.md: add release date for v1.85.1 2022-12-14 14:12:07 -08:00
Aliaksandr Valialkin
ea44c39377
docs/CHANGELOG.md: cut v1.85.1 2022-12-14 11:57:49 -08:00
Aliaksandr Valialkin
b89d862aa5
docs/CHANGELOG.md: document the bugfix at a50120a212 2022-12-13 09:36:24 -08:00
Aliaksandr Valialkin
d1af6046c7
lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead
Blocked small merges may result into big number of small parts, which, in turn,
may result in increased CPU and memory usage during queries, since queries need to inspect
all the existing small parts.

The issue has been introduced in 8189770c50

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-12 17:00:50 -08:00
Aliaksandr Valialkin
3b18931050
lib/bytesutil: cache results for all the input strings, which were passed during the last 5 minutes from FastStringMatcher.Match(), FastStringTransformer.Transform() and InternString()
Previously only up to 100K results were cached.
This could result in sub-optimal performance when more than 100K unique strings were actually used.
For example, when the relabeling rule was applied to a million of unique Graphite metric names
like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466

This commit should reduce the long-term CPU usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466
after all the unique Graphite metrics are registered in the FastStringMatcher.Transform() cache.

It is expected that the number of unique strings, which are passed to FastStringMatcher.Match(),
FastStringTransformer.Transform() and to InternString() during the last 5 minutes,
is limited, so the function results fit memory. Otherwise OOM crash can occur.
This should be the case for typical production workloads.
2022-12-12 14:41:13 -08:00
Roman Khavronenko
8286f9608f
vmalert: support $for or .For template variables (#3474)
support `$for` or `.For` template variables  in alert's annotations.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-12 22:16:10 +03:00
Aliaksandr Valialkin
7ae744fce6
lib/protoparser/datadog: do not re-use previously parsed field values if they are missing in the currently parsed message
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-11 13:09:25 -08:00
Aliaksandr Valialkin
9f642d10ff
docs/CHANGELOG.md: cut v1.85.0 2022-12-11 02:01:11 -08:00
Aliaksandr Valialkin
38f8e8adc3
docs/CHANGELOG.md: document changes at v1.79.6 2022-12-11 01:51:36 -08:00
Aliaksandr Valialkin
d141cb28a6
docs/CHANGELOG.md: document 461158a437
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3427
2022-12-10 23:42:16 -08:00
Aliaksandr Valialkin
88597f187b
app/{vmagent,vminsert}/datadog: make the host label optional in DataDog data ingestion protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-10 23:32:31 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
b7aec1be4d
docs/CHANGELOG.md: add a link to the issue related to reduced CPU and memory usage at vmalert
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3464

The related commit - b97bd01605
2022-12-10 22:21:19 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
765ee5f7ba
docs/CHANGELOG.md: document b97bd01605 2022-12-09 11:49:29 -08:00
Aliaksandr Valialkin
2406c0dcfd
docs/CHANGELOG.md: document the bugfix at 05b42601c3
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
2022-12-08 18:35:28 -08:00
Aliaksandr Valialkin
5b9e6b9d24
lib/storage: follow-up after 7c0ae3a86a
- Update docs at https://docs.victoriametrics.com/#deduplication
- Optimize the deduplication loop a bit

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
2022-12-08 18:16:57 -08:00
Roman Khavronenko
7c0ae3a86a
lib/storage: keep sample with the biggest value on timestamp conflict (#3421)
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:06:11 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
189217a069
docs/CHANGELOG.md: document the addition of file-based discovery of vmstorage nodes 2022-12-07 12:04:57 -08:00
Aliaksandr Valialkin
49ebc48809
lib/querytracer: put the version of VictoriaMetrics in the first message of query trace
This should simplify further debugging, since the first thing to start the debugging by query trace
is to know the version of VictoriaMetrics, which produced this trace.
2022-12-07 09:46:39 -08:00
Aliaksandr Valialkin
3f82e3fa36
docs: follow-up after e1bf2a85d0559d112908ce81597f3261d3a085c0
- Document the change at docs/CHANGELOG.md
- Run `make docs-sync` for copying app/vmgateway/README.md to docs/vmgateway.md
  in order to propagate docs' changes to https://docs.victoriametrics.com/vmgateway.html
2022-12-06 21:05:22 -08:00
Aliaksandr Valialkin
35a3170d97
docs: document the addition of -storageNode.discoveryInterval command-line flag in VictoriaMetrics cluster enterprise
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3417
2022-12-06 19:57:06 -08:00
Roman Khavronenko
3dec847c93
vmalert: correctly return error for RW failures (#3452)
* vmalert: correctly return error for RW failures

By mistake, in 0989649ad0 the error
for remote write failures weren't return to user.
This change fixes it.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-06 15:36:46 +01:00
Aliaksandr Valialkin
a40c50f4fe
docs/CHANGELOG.md: document 1e0666abb4 2022-12-05 23:10:17 -08:00
Aliaksandr Valialkin
caa1c43166
docs: follow-up for 7645d9ae00
- Document the change at docs/CHANGELOG.md
- Document the feature at https://docs.victoriametrics.com/#vmui

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3446
2022-12-05 22:50:41 -08:00
Yury Molodov
01a9b36a95
vmui: timezone select (#3414)
* feat: add timezone selection

* vmui: provide feature timezone select

* fix: correct timezone with relative time

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:44:31 -08:00
Aliaksandr Valialkin
718d1d90b6
docs/CHANGELOG.md: document fd43b5bad0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445
2022-12-05 22:01:42 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Zakhar Bessarab
86c31f2955
app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450) 2022-12-05 17:18:09 -08:00
Aliaksandr Valialkin
f3e84b4dea
{dashboards,alerts}: subtitute {type="indexdb"} with {type=~"indexdb.*"} inside queries after 8189770c50
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 16:00:22 -08:00
Aliaksandr Valialkin
8189770c50
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:16:14 -08:00
Aliaksandr Valialkin
ddc3d6b5c3
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:17:31 -08:00