Commit graph

5505 commits

Author SHA1 Message Date
Yurii Kravets
272f00dbb6
docs: Update How to send data from DataDog agent (#3168)
docs: Update How to send data from DataDog agent
2022-10-03 11:45:05 +02:00
Aliaksandr Valialkin
c53b7e66ef
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:05:43 +03:00
Aliaksandr Valialkin
49311ae977
app/vmselect/prometheus: improve scalability of /federate endpoint on systems with many CPU cores
Minimize usage of global lock inside bufferedwriter.Write() when processing `/federate` data
on systems with many CPU cores
2022-10-01 20:13:24 +03:00
Aliaksandr Valialkin
fb1cc3cc94
app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores
Use sync.Map instead of a global mutex there. This should lift scalability limits
on systems with many CPU cores.
2022-10-01 20:00:03 +03:00
Aliaksandr Valialkin
fcc7ab71b3
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-01 19:47:37 +03:00
Aliaksandr Valialkin
7812761ab4
docs/Cluster-VictoriaMetrics.md: add a security note for multitenant support via labels 2022-10-01 18:57:28 +03:00
Aliaksandr Valialkin
d7327d2f02
docs/vmagent.md: fix incorrect url for multitenant writes to VictoriaMetrics cluster 2022-10-01 18:51:37 +03:00
Aliaksandr Valialkin
0dc93cca7f
app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
2022-10-01 18:40:59 +03:00
Aliaksandr Valialkin
c1fa9828b3
lib/flagutil: rename Array to ArrayString
This makes the ArrayString more consistent with other Array* types.

While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
2022-10-01 18:26:36 +03:00
Aliaksandr Valialkin
366f04001b
vendor: make vendor-update 2022-10-01 17:20:11 +03:00
Zakhar Bessarab
87c77727e4
vmbackup: update AWS SDK to v2 (#3174)
* lib/backup/s3remote: update AWS SDK to v2

* Update lib/backup/s3remote/s3.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

* lib/backup/s3remote: refactor error handling

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-01 17:12:07 +03:00
Aliaksandr Valialkin
725dfb0ed6
lib/httpserver: use 302 redirects instead of 301 redirects
Incorrect 301 redirects can be cached by user agents such as web browsers.
This can complicate recovery procedure after the incorrect redirect is fixed,
e.g. web browser cache must be reset.

The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:53:35 +03:00
Aliaksandr Valialkin
a296994fed
app/vmauth: do not remove trailing slash from the proxied path
This should fix the issue with opening VMUI at /vmui/ page.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:52:30 +03:00
Aliaksandr Valialkin
4998402004
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin
3a98ef2f5f
lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests 2022-10-01 16:05:51 +03:00
Aliaksandr Valialkin
f86070169d
lib/promscrape/discovery/azure: remove unneeded conversion to string 2022-10-01 16:04:37 +03:00
Aliaksandr Valialkin
973ce4b561
app/vmagent: accept requests with /prometheus and /influx path prefixes in the same way as VictoriaMetrics does
This allows using vmagent as a drop-in replacement for VictoriaMetrics
for push protocols.
2022-10-01 14:10:48 +03:00
Aliaksandr Valialkin
63a94b1d54
docs/vmagent.md: formatting fixes 2022-10-01 13:36:24 +03:00
Aliaksandr Valialkin
5b83e6e14e
app/vmagent/README.md: apply typo fix from 1fda517af9 2022-10-01 12:31:06 +03:00
lionelee
1fda517af9
docs/vmagent.md: fix typo (#3186) 2022-10-01 12:30:01 +03:00
Aliaksandr Valialkin
db16759c68
lib/storage: optimize matching speed for non-trivial regexp filters
Wrap re.Match into bytesutil.FastStringMatcher.

This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:06:06 +03:00
Aliaksandr Valialkin
9e8fbef27e
docs/CHANGELOG.md: clarify the description of the improvement in relabeling performance 2022-10-01 11:54:48 +03:00
Aliaksandr Valialkin
e8a64f6e7a
lib/promrelabel: remove redundant memory allocations by using interned strings 2022-10-01 11:50:21 +03:00
Aliaksandr Valialkin
73dc17ef64
lib/promrelabel: add a benchmark for realistic Kubernetes relabeling
The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes

This benchmark has been copied from d521933053/model/relabel/relabel_test.go (L505)

See also https://github.com/prometheus/prometheus/pull/11147
2022-10-01 10:38:22 +03:00
Aliaksandr Valialkin
c54e14cdec
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:48:32 +03:00
Aliaksandr Valialkin
4d27fa41c8
Makefile: run errcheck for all the app/... subdirs 2022-09-30 18:35:53 +03:00
Aliaksandr Valialkin
d0b7172316
app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints
These labels are substituted by documented vm_account_id and vm_project_id labels.

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels

This is a follow up for 505d359b39
2022-09-30 18:35:53 +03:00
Nikolay
33f40f4a5f
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 18:35:53 +03:00
Roman Khavronenko
740bb2cc00
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 17:10:17 +02:00
Aliaksandr Valialkin
171dd14aa3
lib/promrelabel: go fmt 2022-09-30 12:28:55 +03:00
Aliaksandr Valialkin
a18d6d5ccc
lib/promrelabel: optimize action: replace for non-trivial regex values
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.

Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:25:05 +03:00
Aliaksandr Valialkin
146021a076
lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481 2022-09-30 10:49:18 +03:00
Aliaksandr Valialkin
899d2c40fb
lib/promrelabel: optimize action: labelmap for non-trivial regexs 2022-09-30 10:43:31 +03:00
Aliaksandr Valialkin
17289ff481
lib/regexutil: cache MatchString results for unoptimized regexps
This increases relabeling performance by 3x for unoptimized regexs
2022-09-30 10:41:29 +03:00
Dmytro Kozlov
e220bc3cd5
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:55:12 +03:00
Aliaksandr Valialkin
b70f815dc4
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:39:54 +03:00
Roman Khavronenko
b64b9b9fec
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:20:34 +03:00
Aliaksandr Valialkin
fda60b3d4d
lib/promrelabel: properly parse regex with escaped $ at the end
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131

Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin
bf2f14a3a6
docs/CHANGELOG.md: document 39f559d22b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013
2022-09-30 07:48:25 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil
lib/bytesutil is more appropriate place for InternString() function
2022-09-30 07:44:35 +03:00
Nikolay
f61b8cec69
lib/awsapi: fixes sign encoding (#3183)
* lib/awsapi: fixes sign encoding

previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171

* Update lib/awsapi/sign.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:43:44 +03:00
Roman Khavronenko
39f559d22b
vmalert: allow using extra labels in annotations (#3181)
According to Ruler specification, only labels returned within time series
should be available for use in annotations.

For long time, vmalert didn't respect this rule. And in PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2403
this was fixed for the sake of compatibility. However, this resulted
into users confusion, as they expected all configured and extra labels
to be available - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013

This fix allows to use extra labels in Annotations. But in the case of conflicts
the original labels (extracted from time series) are preferred.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-29 18:22:50 +02:00
Aliaksandr Valialkin
6a32a64073
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed 2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin
92b3622253
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
This is a follow-up for 7c2474dac7

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin
ef435f8cc4
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
Optimize this function by using results cache for input strings.
Use this function all over the code.

This is a follow-up for fcffdba9dc

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:40:59 +03:00
nemobis
6a818dbddf
docs: fix typo in Cluster-VictoriaMetrics.md (#3172) 2022-09-28 09:18:29 +02:00
panguicai
fbc85e654c
docs: fix typo for vmalert docs (#3173)
Signed-off-by: panguicai008 <1121906548@qq.com>
2022-09-28 09:15:05 +02:00
Roman Khavronenko
4ad3b36630
docs: update img for key concepts (#3176)
Make dots bigger for range_query, so it will be easier
to read the graph.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-27 21:22:59 +02:00
Aliaksandr Valialkin
6411bbcce7
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
This should simplify further code maintenance and refactoring

This is a follow-up after 6ab1cede62
2022-09-26 17:58:01 +03:00
Dmytro Kozlov
6ab1cede62
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:35:45 +03:00