Commit graph

4738 commits

Author SHA1 Message Date
Aliaksandr Valialkin
3282829e30
go.mod: increase the minimum supported Go version from 1.17 to 1.19 2022-11-10 14:03:39 +02:00
Aliaksandr Valialkin
6b54ad8cea
vendor: make vendor-update 2022-11-10 13:55:47 +02:00
Aliaksandr Valialkin
7ac3f19ee1
docs/CHANGELOG.md: cut v1.79.5 2022-11-10 13:52:40 +02:00
Dmytro Kozlov
ac03ad8b6c
vmui: fix vmui vulnerability (#3336)
* vmui: fix vmui vulnerability

* vmui: code cleanup
2022-11-10 13:33:00 +02:00
dependabot[bot]
66f0015cd5
build(deps): bump loader-utils in /app/vmui/packages/vmui (#3328)
Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/webpack/loader-utils/releases)
- [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.3/CHANGELOG.md)
- [Commits](https://github.com/webpack/loader-utils/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: loader-utils
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 16:50:20 +02:00
Aliaksandr Valialkin
b931b9151a
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 16:45:53 +02:00
Roman Khavronenko
2ddea5be36
protoparser/opentsdb: allow lines without tags (#3303)
According to http://opentsdb.net/docs/build/html/api_telnet/put.html
"At least one tag pair must be present".
However, in VictoriaMetrics datamodel tags aren't required.
This could be confusing for users. Allowing accept lines without
tags seems to do no harm.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 16:45:00 +02:00
Aliaksandr Valialkin
1fbf94a1c4
lib/storage: follow-up for 790768f20b
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:51:25 +02:00
Aliaksandr Valialkin
3d7e03cffa
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-11-07 14:44:59 +02:00
Siqi Liu
3195aa44fd
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225)
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
2022-11-07 14:43:40 +02:00
Aliaksandr Valialkin
ae64be4f2c
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-11-07 14:40:24 +02:00
Roman Khavronenko
c32d3695e7
vmalert: lower severity level for RW retries (#3237)
The message about dropped data still remains at `error` level.
The change supposed to make log message more clear about how
serious it is.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-07 14:33:37 +02:00
Aliaksandr Valialkin
b5ec4379d0
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-11-07 14:29:39 +02:00
Aliaksandr Valialkin
3ddab2d789
deployment/docker: update Go builder from v1.19.2 to v1.19.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved
2022-11-07 14:26:04 +02:00
Aliaksandr Valialkin
8b08761c56
docs/CHANGELOG.md: typo fix 2022-10-07 03:12:24 +03:00
Aliaksandr Valialkin
82d6610426
docs/CHANGELOG.md: add a note about LTS release for v1.79.4 2022-10-07 02:49:04 +03:00
Aliaksandr Valialkin
b4f4eaf710
docs/CHANGELOG.md: cut v1.79.4 2022-10-07 01:27:23 +03:00
Aliaksandr Valialkin
7d4101931c
deployment/docker: update Go builder from v1.19.0 to v1.19.2
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.19.1+label%3ACherryPickApproved
2022-10-07 01:25:19 +03:00
Aliaksandr Valialkin
70a579b725
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:26:57 +03:00
Aliaksandr Valialkin
f6211309c5
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:48 +03:00
Aliaksandr Valialkin
f088162c8a
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-10-06 16:12:54 +03:00
Roman Khavronenko
7115b8610a
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 16:11:00 +03:00
Roman Khavronenko
4fc3495bed
lib/mergeset: follow-up after a0e7432e42 (#3145)
* lib/mergeset: follow-up after a0e7432e42

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-06 16:05:46 +03:00
Roman Khavronenko
00e55947e4
vmalert: always re-evaluate Annotations (#3119)
* vmalert: always re-evaluate Annotations

Previously, Annotations were evaluated only:
1. On alert creating.
2. On alert's value change.

This is premature optimization. It was assumed that since annotations
could contain only text with alert's labels or value - there is no need
in spending resources to re-compile Annotations.

Later, template function `query` was added, which can execute
arbitrary queries and return different results on every evaluation.
So if it was used in annotations, it would be executed only on init
or value change.

Another case when optimization caused an issue - annotations hot reload.
In this case, annotations of the active alert won't change even if Rule's
annotations were changed.

This fix enables Annotations re-evaluation on each iteration to resolve
issues above. It would have some impact on performance, but it is unlikely
it will be noticeable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add tp Changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 16:03:00 +03:00
Yury Molodov
9b4ebf95b8
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-06 15:56:57 +03:00
Roman Khavronenko
f68333a8ce
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 15:49:34 +03:00
Aliaksandr Valialkin
7543bdfd54
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-10-06 15:48:46 +03:00
Aliaksandr Valialkin
2db5ec5509
docs/CHANGELOG.md: document 166d444159 2022-10-06 15:46:19 +03:00
Roman Khavronenko
5839112cda
vmselect/rollup: rm workaround for slow-changing counters (#3163)
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 15:45:31 +03:00
Aliaksandr Valialkin
dca89c7d2f
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-10-06 15:44:12 +03:00
Nikolay
64c0133b88
lib/awsapi: fixes sign encoding (#3183)
* lib/awsapi: fixes sign encoding

previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171

* Update lib/awsapi/sign.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-06 15:31:22 +03:00
Aliaksandr Valialkin
dbf0ef5b38
app/vmauth: do not remove trailing slash from the proxied path
This should fix the issue with opening VMUI at /vmui/ page.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-06 15:28:51 +03:00
Aliaksandr Valialkin
4ef2d46b8b
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-06 15:18:33 +03:00
Aliaksandr Valialkin
dd46521676
docs/Release-Guide.md: sync with the master branch after the commit b9231c715a 2022-09-08 21:03:53 +03:00
Aliaksandr Valialkin
a037180167
Makefile: remove github-create-release and github-upload-assets commands from publish-release
This is a follow-up for b9231c715a
2022-09-08 21:03:12 +03:00
Aliaksandr Valialkin
32cbc0f497
docs/CHANGELOG.md: document e7119de7f7 2022-09-08 21:00:10 +03:00
Dmytro Kozlov
e7119de7f7
vmagent: expose metric vmagent_remotewrite_queues (#2871) (#3087)
* vmagent: expose metric `vmagent_remotewrite_queues` (#2871)

The new metric `vmagent_remotewrite_queues` exports a static value of
number of configured remote write queus. This metric is useful to
calculate total saturation per each configured URL with given number
of queues. See corresponding changes to vmagent alerts and dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update dashboards/vmagent.json

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 20:53:51 +03:00
Max Golionko
d54cf15478
simplify release process (#3012)
* simplify release process

* address comments

* address comments

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-31 02:31:13 +03:00
Aliaksandr Valialkin
56d7f3e37b
docs/CHANGELOG.md: cut v1.79.3 2022-08-30 13:23:02 +03:00
Aliaksandr Valialkin
cd422a5435
app/vmselect/promql: typo fix after 992f36702f 2022-08-30 12:04:06 +03:00
Aliaksandr Valialkin
c35b63cd0c
docs/CHANGELOG.md: clarify the change at 28441711e6 2022-08-30 12:03:03 +03:00
Roman Khavronenko
3e2b434bad
vmalert: follow-up after 28441711e6 (#2972)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-30 11:58:28 +03:00
Matthew Blewitt
240acdf3b7
vmalert: mark some url flags as sensitive (#2965)
Other components, such as `vmagent`, mark these flags as sensitive and
hide them from the `/metrics` endpoint by default. This commit adds
similar handling to the `vmalert` component, hiding them by default, to
prevent logging of secrets inappropriately.

Showing of these values is controlled by an additional flag.

Follow up to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2947
2022-08-30 11:55:55 +03:00
Aliaksandr Valialkin
a9c5766ebc
docs/CHANGELOG.md: document the 044d51b668 2022-08-30 11:28:03 +03:00
Denys Holius
0aa41430dd
deployment/docker/Makefile: bump version of Alpine linux to latest 3.16.2 to fix CVE-2022-37434 (#3035)
see https://alpinelinux.org/posts/Alpine-3.13.12-3.14.8-3.15.6-3.16.2-released.html
2022-08-30 11:27:10 +03:00
Aliaksandr Valialkin
992f36702f
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-30 11:16:29 +03:00
Dmytro Kozlov
86d85591a4
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-30 11:16:13 +03:00
Aliaksandr Valialkin
e8b1131f97
docs/CHANGELOG.md: document d59d829cdb
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673
2022-08-30 11:09:20 +03:00
Roman Khavronenko
2838ee93c6
lib/storage: bump max merge concurrency for small parts to 15 (#2997)
* lib/storage: bump max merge concurrency for small parts to 15

The change is based on the feedback from users on github.
Thier examples show, that limit of 8 sometimes become a
bottleneck. Users report that without limit concurrency
can climb up to 15-20 merges at once.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update lib/storage/partition.go

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-30 11:02:35 +03:00
Roman Khavronenko
72342939d6
lib/storage: fix the search for empty label name (#2991)
* lib/storage: fix the search for empty label name

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-30 11:00:48 +03:00