Commit graph

4970 commits

Author SHA1 Message Date
lionelee
f9843442f2
docs/vmagent.md: fix typo (#3186) 2022-10-01 12:31:41 +03:00
Aliaksandr Valialkin
b96fe2e265
lib/storage: optimize matching speed for non-trivial regexp filters
Wrap re.Match into bytesutil.FastStringMatcher.

This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
72d8653233
docs/CHANGELOG.md: clarify the description of the improvement in relabeling performance 2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
969ae90941
lib/promrelabel: remove redundant memory allocations by using interned strings 2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
d8d455856c
lib/promrelabel: add a benchmark for realistic Kubernetes relabeling
The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes

This benchmark has been copied from d521933053/model/relabel/relabel_test.go (L505)

See also https://github.com/prometheus/prometheus/pull/11147
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
c628f5b6eb
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:49:08 +03:00
Aliaksandr Valialkin
ae2f669e73
Makefile: run errcheck for all the app/... subdirs 2022-09-30 18:40:06 +03:00
Roman Khavronenko
408d7043a1
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 18:38:11 +03:00
Aliaksandr Valialkin
c8000c029a
app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints
These labels are substituted by documented vm_account_id and vm_project_id labels.

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels

This is a follow up for 505d359b39
2022-09-30 18:06:00 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin
39ba55dbb3
lib/promrelabel: go fmt 2022-09-30 12:28:27 +03:00
Aliaksandr Valialkin
9fc2817f41
lib/promrelabel: optimize action: replace for non-trivial regex values
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.

Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
f38c9db74d
lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481 2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
fa46c28c5f
lib/promrelabel: optimize action: labelmap for non-trivial regexs 2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin
b4bb1477fe
lib/regexutil: cache MatchString results for unoptimized regexps
This increases relabeling performance by 3x for unoptimized regexs
2022-09-30 12:28:25 +03:00
Dmytro Kozlov
d0a9fff70c
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:56:43 +03:00
Aliaksandr Valialkin
98175493d7
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:44:06 +03:00
Roman Khavronenko
273472dc20
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:44:05 +03:00
Aliaksandr Valialkin
f1eebc0a99
lib/promrelabel: properly parse regex with escaped $ at the end
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131

Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:20:57 +03:00
Aliaksandr Valialkin
33590c5fb9
docs/CHANGELOG.md: document 39f559d22b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013
2022-09-30 07:49:54 +03:00
Nikolay
909709346e
lib/awsapi: fixes sign encoding (#3183)
* lib/awsapi: fixes sign encoding

previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171

* Update lib/awsapi/sign.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:49:18 +03:00
Roman Khavronenko
a2ded58600
vmalert: allow using extra labels in annotations (#3181)
According to Ruler specification, only labels returned within time series
should be available for use in annotations.

For long time, vmalert didn't respect this rule. And in PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2403
this was fixed for the sake of compatibility. However, this resulted
into users confusion, as they expected all configured and extra labels
to be available - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013

This fix allows to use extra labels in Annotations. But in the case of conflicts
the original labels (extracted from time series) are preferred.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 07:48:59 +03:00
Aliaksandr Valialkin
c0aa10bd73
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil
lib/bytesutil is more appropriate place for InternString() function
2022-09-30 07:34:59 +03:00
nemobis
6723bdf867
docs: fix typo in Cluster-VictoriaMetrics.md (#3172) 2022-09-28 10:43:17 +03:00
panguicai
156e7035c7
docs: fix typo for vmalert docs (#3173)
Signed-off-by: panguicai008 <1121906548@qq.com>
2022-09-28 10:42:13 +03:00
Roman Khavronenko
3f78608da2
docs: update img for key concepts (#3176)
Make dots bigger for range_query, so it will be easier
to read the graph.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-28 10:41:35 +03:00
Aliaksandr Valialkin
4afa25fb38
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed 2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
9c6c691471
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
This is a follow-up for 7c2474dac7

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin
7f0b95b50a
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
Optimize this function by using results cache for input strings.
Use this function all over the code.

This is a follow-up for fcffdba9dc

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin
41882222d3
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
This should simplify further code maintenance and refactoring

This is a follow-up after 6ab1cede62
2022-09-26 17:59:07 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Denys Holius
3f38a88f71
docs: managed_victoriametrics/quickstart.md fix image link (#3165) 2022-09-26 17:38:41 +03:00
Roman Khavronenko
b8c4c36b89
docs: clarify numeric label values for label_value (#3129)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3114
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 16:49:45 +03:00
Max Golionko
322e5006d2
added security policy (#3140)
* added security policy

* Update SECURITY.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:47:38 +03:00
Denys Holius
ebfccada32
Adds packer build for server with VM Single node in vultr.com marketplace (#3142)
* adds packer build for server with VM Single node in vultr.com marketplace

* fix missed varibale
2022-09-26 16:45:29 +03:00
Roman Khavronenko
fe71c73fe1
lib/mergeset: follow-up after a0e7432e42 (#3145)
* lib/mergeset: follow-up after a0e7432e42

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:43:17 +03:00
Aliaksandr Valialkin
d91bf89651
docs/CHANGELOG.md: document f022296d96
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147
2022-09-26 16:32:54 +03:00
Zakhar Bessarab
6c65ee18d9
vmbackup: configure retries for GCS remote FS (#3156) 2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin
6611417732
vendor: make vendor-update 2022-09-26 15:47:35 +03:00
Aliaksandr Valialkin
89e1060101
docs/CHANGELOG.md: document 166d444159 2022-09-26 15:39:51 +03:00
Roman Khavronenko
4acbf6e7d4
vmselect/rollup: rm workaround for slow-changing counters (#3163)
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 15:39:49 +03:00
Aliaksandr Valialkin
2b98f2bc1a
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102

See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin
938d7a932c
docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics sanitizes metric names in DataDog data by default
This is a follow-up for 7c2474dac7
2022-09-26 14:03:26 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin
6a6cf9c590
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:13:31 +03:00
Aliaksandr Valialkin
594a4ab345
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-09-26 08:49:47 +03:00
Aliaksandr Valialkin
a6a869c365
docs/vmalert.md: follow-up after 7748a9d629
- Consistently use single dash in front of command-line flags instead of double dashes.
- Add a warning that too small -search.latencyOffset may lead to incomplete query results.
2022-09-26 08:49:47 +03:00
Aliaksandr Valialkin
2fe38a3243
docs: follow-up for 03d54ac890
- Typo fix: 'adn' -> 'and'
- Remove duplicate link to https://victoriametrics.com/blog/victoriametrics-monitoring/
  from docs/Articles.md . The initial link has been already added in 27254096b2
2022-09-26 08:49:47 +03:00
Roman Khavronenko
0efc20d7b8
dashboards: replace Index size panel with Active series (#3157)
Panel `Index size` showed itself impractical for users. So
replacing it with `Active series` panel.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/776#issuecomment-1255823734
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:48:25 +03:00
Roman Khavronenko
cffceba0f5
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:40:17 +03:00