Commit graph

8558 commits

Author SHA1 Message Date
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
Zhu Jiekun
9c3dabccac
docs: [vmagent] Add CHANGELOG for Statsd support in v1.102.0-rc1 (#6494)
### Describe Your Changes
Add CHANGELOG for Statsd support in v1.102.0-rc1
- CHANGELOG is missing for #5053.
- It would be better to include it so that users can be aware of when
this feature is released, rather than attempting to use it with lower
version VM components. See discussion
[here](https://victoriametrics.slack.com/archives/CGZF1H6L9/p1718462859065049?thread_ts=1718451117.668789&cid=CGZF1H6L9).

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
Fred Navruzov
73fbce30bd
docs/vmanomaly - add config example for min_dev arg (#6490)
### Describe Your Changes

Added config example for `min_dev_from_expected` arg; also, small
styling fixes and alignments

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-17 12:11:50 +02:00
hagen1778
43c53271bf
docs: update wording for 6e395048d3
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit da4fbf61a4)
2024-06-14 16:31:42 +02:00
Hui Wang
5ecf439078
app/vmselect: fix the way of counting raw samples in single query (#6464)
The limit is specified with command-line flag
`-search.maxSamplesPerQuery`.
Previously, samples might be over-counted and query can't be fixed by
reducing time range.
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851

(cherry picked from commit 6e395048d3)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 16:24:31 +02:00
Andrii Chubatiuk
8ca1813bd2
lib/flagutil: use month limit for duration flag for parsed duration assessment (#6486)
use maxMonths limit for parsed duration flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6330

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit faf67aa8b5)
2024-06-14 15:21:32 +02:00
jackyin
f69495cd5f
app/vmalert: fix VMAlert oauth2 error (#6478)
Properly set ClientSecret param for notifier.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6471

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5223981fed)
2024-06-14 15:21:30 +02:00
hagen1778
64df1383e7
app/vmalert: properly configure authentication with S3 when -s3.configFilePath is specified.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-14 14:22:35 +02:00
Andrii Chubatiuk
abc233a902
lib/backup/s3remote: fixed credsFilePath flag (#6488)
properly use credsFilePath flag value

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e678a9aa51)
2024-06-14 14:14:58 +02:00
Zakhar Bessarab
a91a2b97ed
docs/guides/otel-guide: update sentence to make it less obscure (#6483)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit f11261dc44)
2024-06-14 14:14:56 +02:00
Andrii Chubatiuk
779436bd9c
app/vmalert: fixed path prefixes for system routes (#6435)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit eea361defb)
2024-06-14 14:14:54 +02:00
Artem Navoiev
6c587eefbf
docs: add more redirects
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit d1baa41b6a)
2024-06-14 13:26:47 +02:00
Fred Navruzov
455d761de9
docs/vmanomaly: fix-typos-in-v1.13-preset (#6484)
### Describe Your Changes

small fix of typos in v1.13 presets (vmanomaly docs)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1cf28598c6)
2024-06-14 13:26:44 +02:00
Roman Khavronenko
5df50e5645
lib/streamaggr: prevent rate_sum and rate_avg from producing NaNs (#6482)
### Describe Your Changes

* check if `lastValue` was seen at least twice with different
timestamps. Otherwise, the difference between last timestamp and
previous timestamp could be `0` and will result into `NaN` calculation
* check if there items left in lastValue map after staleness cleanup.
Otherwise, `rate_avg` could have produce `NaN` result.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 51d19485bb)
2024-06-14 13:26:42 +02:00
Github Actions
5683610bf8
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@bc562ef (#6481)
(cherry picked from commit 2d575cd27a)
2024-06-14 13:26:40 +02:00
Fred Navruzov
aa85b772bb
docs/vmanomaly: update preset guide and cross-links (#6480)
### Describe Your Changes

Updates and cross-references for `preset` mode of `vmanomaly` [in
docs](https://docs.victoriametrics.com/anomaly-detection/presets/)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2184c0c730)
2024-06-14 13:26:37 +02:00
Daria Karavaieva
ae39b02496
docs/vmanomaly: preset guide (#6151)
- added instructions of how to run presets
- description of Node Exporter indicators covered

(cherry picked from commit c32b8d39dd)
2024-06-14 13:26:29 +02:00
Aliaksandr Valialkin
b455d356fc
docs/CONTRIBUTING.md: add checks about backwards compatibility and historical conventions 2024-06-13 17:08:54 +02:00
Aliaksandr Valialkin
2bbf62b6f6
lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes
Previously byte slices up to 2^20 bytes (e.g. 1Mb) were cached because of a typo in the commit c14dafce43 .

This could result in increased memory usage when vmagent scrapes many regular targets, which expose
relatively small number of metrics (e.g. up to a few thousand per target) and a few large targets such as kube-state-metrics,
which expose more than 10 thousand metrics. This is common case for Kubernetes monitoring.

While at it, remove pools for very small byte slices, since they are rarely used during scraping.
2024-06-13 17:02:05 +02:00
Aliaksandr Valialkin
7af2b7905f
docs/CONTRIBUTING.md: various typo fixes 2024-06-13 17:02:05 +02:00
Aliaksandr Valialkin
faf07fbc67
lib/bytesutil: optimize internStringMap cleanup
- Make it in a separate goroutine, so it doesn't slow down regular intern() calls.

- Do not lock internStringMap.mutableLock during the cleanup routine, since now
  it is called from a single goroutine and reads only the readonly part of the internStringMap.
  This should prevent from locking regular intern() calls for new strings during cleanups.

- Add jitter to the cleanup interval in order to prevent from synchornous increase in resource usage
  during cleanups.

- Run the cleanup twice per -internStringCacheExpireDuration . This should save 30% CPU time spent
  on cleanup comparing to the previous code, which was running the cleanup 3 times per -internStringCacheExpireDuration .
2024-06-13 15:09:42 +02:00
Zakhar Bessarab
ac16d1dc1b
lib/promscrape: increase default value for promscrape.maxDroppedTargets to 10_000 (#6459)
### Describe Your Changes
This limit can be increased since after
4513893ead
tracking of dropped targets uses much less memory per entry.

See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6381#issuecomment-2156708228

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit 34071ac660)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-13 09:28:16 +02:00
LHHDZ
41e4135371
app/vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix (#6401)
This change fixes the following panic:
```
2024-06-04T11:16:52.899Z        warn    app/vmauth/auth_config.go:353   cannot discover backend SRV records for http://srv+localhost:8080: lookup localhost on 10.100.10.4:53: server misbehaving; use it literally
panic: runtime error: integer divide by zero

goroutine 9 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper.func1()
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/lib/httpserver/httpserver.go:291 +0x58
panic({0x103115100?, 0x10338d700?})
        /Users/lhhdz/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/panic.go:770 +0x124
main.getLeastLoadedBackendURL({0x0?, 0x22?, 0x1400014757b?}, 0x1400013c120?)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:473 +0x210
main.(*URLPrefix).getBackendURL(0x140000aa080)
        /Users/lhhdz/wd/projects/go/VictoriaMetrics/app/vmauth/auth_config.go:312 +0xb8
```

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-06-12 11:47:44 +02:00
Aliaksandr Valialkin
27efb9499b
docs/VictoriaLogs/logsql-examples.md: added an example on how to search for a phrase with whitespace 2024-06-11 19:13:47 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress 2024-06-11 17:51:01 +02:00
Aliaksandr Valialkin
9bd16790c0
lib/streamaggr: prevent from data race inside dedupAggrShard when samplesBuf can be updated in pushSamples() while their values are read in the flush() loop without das.mu lock
This issue has been introduced in the commit 253c0cffbe
2024-06-11 17:31:38 +02:00
Aliaksandr Valialkin
81a3387bb7
.golangci.yml: properly specify functions to exclude for return values check after the upgrade to v1.59.1 at 239a7b6e6f
(cherry picked from commit 4ad1f8ee3a)
2024-06-11 17:05:17 +02:00
Aliaksandr Valialkin
b74dbdbc82
Makefile: update golangci-lint from v1.57.1 to v1.59.1
See https://github.com/golangci/golangci-lint/releases/tag/v1.59.1

Thanks to ccoVeille for the suggestion at https://www.reddit.com/r/golang/comments/1dcl4ne/log_viewer_in_go/l82vsfd/

(cherry picked from commit 239a7b6e6f)
2024-06-11 17:05:12 +02:00
Fred Navruzov
ea90d91a30
docs/vmanomaly - fix 404 img links & remove outdated models (#6460)
### Describe Your Changes

Fix 404 relative img links for v1.13.0 update of vmanomaly docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 9a355c3512)
2024-06-11 17:05:10 +02:00
Fred Navruzov
0219d34b21
docs/vmanomaly - release 1.13.0 preparation (#6436)
### Describe Your Changes

[vmanomaly docs](https://docs.victoriametrics.com/anomaly-detection/)
update for changes, introduced in v1.13.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1feb5d04d7)
2024-06-11 17:05:07 +02:00
Nikolay
ae76794a19
follow-up docs update after 77f22fdb8d (#6454)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 33d07e915f)
2024-06-11 17:05:05 +02:00
James Rhoat
f4b52b8137
updating operator dashboard chart to be titled working instead of wokring (#6455)
### Describe Your Changes

Corrected spelling mistake in the operator json to be "working" instead
of "wokring"

### Checklist

The following checks are **mandatory**:

- [ x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit fbd4b8e1ab)
2024-06-11 17:05:03 +02:00
Github Actions
c96b334b3e
Automatic update operator docs from VictoriaMetrics/operator@310a860 (#6449)
(cherry picked from commit 020d3f3d88)
2024-06-11 17:05:01 +02:00
Lapo Luchini
dd311aa76f
Small fix in MetricsQL.md (#6452)
### Describe Your Changes

value→valid

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Lapo Luchini <lapo@lapo.it>
(cherry picked from commit 14f586238f)
2024-06-11 17:04:59 +02:00
Nikolay
66fbea70a5
follow-up after 77f22fdb8d (#6458)
* fixes linter error
* simplify code a bit
* fixes bug with incorrectly set configSuccess metric. It was not set to
1 in case of config rollback

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-06-11 12:08:00 +02:00
noodles2hg
77f22fdb8d
[cluster/vminsert]:add reload -relabelConfig on the request to /-/reload (#3923)
When I use vminsert's `relabelConfig`, I found that now there is no
reloaded api. However, `vminsert` under `VM-Single` has it. So, I hope
to add it to the `cluster/vminster`.

---------

Signed-off-by: z-anshun <1179798460@qq.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-06-10 19:36:41 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin
7e24bf99de
lib/streamaggr: return back string interning to dedupAggr after 78953723200f15ffc417064d1912bdbb7551505c
It should reduce memory allocation rate during stream deduplication
2024-06-10 18:06:25 +02:00
Aliaksandr Valialkin
6470eac7dc
lib/bytesutil: reduce the number of memory allocations per each interned string in bytesutil.InternString() from 5 to 1
This should reduce GC overhead when tens of millions of strings are interned (for example, during stream deduplication
of millions of active time series).
2024-06-10 18:06:24 +02:00
Roman Khavronenko
8c8d84e30a
lib/protoparser/opentelemetry/firehose: escape requestID before returning it to user (#6451)
All user input should be sanitized before rendering. This should prevent
possible attacks. See
https://github.com/VictoriaMetrics/VictoriaMetrics/security/code-scanning/203

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 18:06:24 +02:00
Aliaksandr Valialkin
883c0e6221
lib/streamaggr: reduce memory allocations by using dedupAggrSample buffer per each dedupAggrShard 2024-06-10 16:39:26 +02:00
Aliaksandr Valialkin
422225bfa5
lib/streamaggr: reduce the number of duplicates per each sample in BenchmarkDedupAggr from 100 to 2
This is closer to typical production setups when deduplication is used for de-duplicating of 2 samples per series.
2024-06-10 16:39:26 +02:00
Yury Molodov
2300e30ff3
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:39:25 +02:00
Aliaksandr Valialkin
d269a95da3
lib/streamaggr: use strings.Clone() instead of bytesutil.InternString() for creating series key in dedupAggr
Our internal testing shows that this reduces GC overhead when deduplicating tens of millions of active series.
2024-06-10 16:08:47 +02:00
Aliaksandr Valialkin
9ed9e766e8
lib/streamaggr: improve performance for dedupAggr.sizeBytes() and dedupAggr.itemsCount()
These functions are called every time `/metrics` page is scraped, so it would be great
if they could be sped up for the cases when dedupAggr tracks tens of millions of active time series.
2024-06-10 16:00:05 +02:00
Aliaksandr Valialkin
387c22da49
lib/streamaggr: remove flushState arg at dedupAggr.flush(), since it is always set to true in production 2024-06-10 16:00:05 +02:00
hagen1778
fdf0a936f0
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8d95522529)
2024-06-10 14:05:58 +02:00
Nikolay
0ce7f38e1c
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d44058bcd6)
2024-06-10 12:41:51 +02:00
Andrii Chubatiuk
1b49413d4c
removed exclusion of recording label in alert expression, which is utilized in alert annotations (#6441)
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6439

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit ff458af25e)
2024-06-10 12:41:48 +02:00
Dmytro Kozlov
a4bdc14bc5
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 8b46bb0c41)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:41:44 +02:00