Commit graph

8735 commits

Author SHA1 Message Date
James Rhoat
fbd4b8e1ab
updating operator dashboard chart to be titled working instead of wokring (#6455)
### Describe Your Changes

Corrected spelling mistake in the operator json to be "working" instead
of "wokring"

### Checklist

The following checks are **mandatory**:

- [ x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-11 12:39:17 +04:00
Github Actions
020d3f3d88
Automatic update operator docs from VictoriaMetrics/operator@310a860 (#6449) 2024-06-10 22:17:32 +02:00
Lapo Luchini
14f586238f
Small fix in MetricsQL.md (#6452)
### Describe Your Changes

value→valid

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Lapo Luchini <lapo@lapo.it>
2024-06-10 22:17:07 +02:00
Aliaksandr Valialkin
0521e58a09
lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
Aliaksandr Valialkin
bf2d299420
lib/streamaggr: return back string interning to dedupAggr after 78953723200f15ffc417064d1912bdbb7551505c
It should reduce memory allocation rate during stream deduplication
2024-06-10 18:05:42 +02:00
Aliaksandr Valialkin
6a0a36aa93
lib/bytesutil: reduce the number of memory allocations per each interned string in bytesutil.InternString() from 5 to 1
This should reduce GC overhead when tens of millions of strings are interned (for example, during stream deduplication
of millions of active time series).
2024-06-10 18:05:41 +02:00
Roman Khavronenko
cd1aca217c
lib/protoparser/opentelemetry/firehose: escape requestID before returning it to user (#6451)
All user input should be sanitized before rendering. This should prevent
possible attacks. See
https://github.com/VictoriaMetrics/VictoriaMetrics/security/code-scanning/203

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 16:55:59 +02:00
Aliaksandr Valialkin
253c0cffbe
lib/streamaggr: reduce memory allocations by using dedupAggrSample buffer per each dedupAggrShard 2024-06-10 16:38:42 +02:00
Aliaksandr Valialkin
a1e8003754
lib/streamaggr: reduce the number of duplicates per each sample in BenchmarkDedupAggr from 100 to 2
This is closer to typical production setups when deduplication is used for de-duplicating of 2 samples per series.
2024-06-10 16:38:41 +02:00
Yury Molodov
84088e5a2d
vmui/logs: add markdown support (#6292)
Add support for markdown format and emoji for the `_msg` field in the
"Group" view.
Add markdown rendering toggle. Disabled by default. Value is stored in
`localStorage`.
2024-06-10 16:38:13 +02:00
Aliaksandr Valialkin
0b7c47a40c
lib/streamaggr: use strings.Clone() instead of bytesutil.InternString() for creating series key in dedupAggr
Our internal testing shows that this reduces GC overhead when deduplicating tens of millions of active series.
2024-06-10 16:08:34 +02:00
Aliaksandr Valialkin
e8bb4359bb
lib/streamaggr: improve performance for dedupAggr.sizeBytes() and dedupAggr.itemsCount()
These functions are called every time `/metrics` page is scraped, so it would be great
if they could be sped up for the cases when dedupAggr tracks tens of millions of active time series.
2024-06-10 15:59:37 +02:00
Aliaksandr Valialkin
f45d02a243
lib/streamaggr: remove flushState arg at dedupAggr.flush(), since it is always set to true in production 2024-06-10 15:59:33 +02:00
hagen1778
8d95522529
vmctl: rm --vm-disable-progress-bar flag
It is better to remove deprecated flag completely, so vmctl will
fail if this flag is used and user can immediately fix the issue.

Before, flag was ignored and it is worse then fail fast.

follow-up after 8b46bb0c41 (diff-2bfab3db5cc1baf4c6d3ff6b19901926e3bdf4411ec685dac973e5fcff1c723b)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 14:02:46 +02:00
Nikolay
d44058bcd6
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:36:37 +02:00
Andrii Chubatiuk
ff458af25e
removed exclusion of recording label in alert expression, which is utilized in alert annotations (#6441)
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6439

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-10 12:26:03 +02:00
Dmytro Kozlov
8b46bb0c41
vmctl: disable progress bar for prometheus snapshot migrations (#6385)
* deprecate `--vm-disable-progress-bar` in favour of `--disable-progress-bar`
* new `--disable-progress-bar` consistently disables usage of progress bar
for all migration modes.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:20:52 +02:00
Hui Wang
61dce6f2a1
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329, 
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:09:47 +02:00
Andrii Chubatiuk
2da45a8368
vmagent: updated dashboard and alert for stream aggregation (#6427)
### Describe Your Changes

Added streaming aggregation section to vmagent dashboards
Added alert for streaming aggregation and deduplication flush timeouts
Removed deprecated compose versions from compose files


Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 11:49:00 +02:00
Artem Navoiev
318e9e9de0
docs victorialogs: add alias for quickstart for VL to be consistent with other quickstarts
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-06-10 09:36:26 +02:00
Artem Navoiev
8f254232bb
add alias for vmalert-tools
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-06-09 14:30:45 +02:00
Aliaksandr Valialkin
e492bf0ae9
docs/CHANGELOG.md: document v1.93.15 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.15
2024-06-07 23:43:07 +02:00
Aliaksandr Valialkin
a9448f372b
docs/LTS-releases.md: update v1.97.4 LTS release to v1.97.5
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:15:17 +02:00
Aliaksandr Valialkin
914b9068a8
docs/CHANGELOG.md: add changelog for v1.97.5 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.5
2024-06-07 20:03:10 +02:00
Aliaksandr Valialkin
09c7accac9
docs/CHANGELOG.md: cut v1.202.0-rc1 release 2024-06-07 16:53:42 +02:00
Aliaksandr Valialkin
f8f9a58163
docs: run make docs-sync after c740a8042e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362
2024-06-07 16:45:16 +02:00
Aliaksandr Valialkin
b12864f905
vendor: run make vendor-update 2024-06-07 16:40:42 +02:00
Aliaksandr Valialkin
e8ab8944e6
app/vmselect/vmui: run make vmui-update after c236e3c03c1bf8ca00292b800a839fcb300e7e51 and 04744c274c269f6b6efb45f68df11abe0fb0ce25 2024-06-07 16:39:19 +02:00
Aliaksandr Valialkin
36be090cd5
lib/streamaggr: follow-up for 7cb894a777
- Use bytesutil.InternString() instead of strings.Clone() for inputKey and outputKey in aggregatorpushSamples().
  This should reduce string allocation rate, since strings can be re-used between aggrState flushes.
- Reduce memory allocations at dedupAggrShard by storing dedupAggrSample by value in the active series map.
- Remove duplicate call to bytesutil.InternBytes() at Deduplicator, since it is already called inside dedupAggr.pushSamples().
- Add missing string interning at rateAggrState.pushSamples().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402
2024-06-07 16:27:26 +02:00
Roman Khavronenko
7cb894a777
lib/streamaggr: reduce number of inuse objects (#6402)
The main change is getting rid of interning of sample key. It was
discovered that for cases with many unique time series aggregated by
vmagent interned keys could grow up to hundreds of millions of objects.
This has negative impact on the following aspects:
1. It slows down garbage collection cycles, as GC has to scan all inuse
objects periodically. The higher is the number of inuse objects, the
longer it takes/the more CPU it takes.
2. It slows down the hot path of samples aggregation where each key
needs to be looked up in the map first.

The change makes code more fragile, but suppose to provide performance
optimization for heavy-loaded vmagents with stream aggregation enabled.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-06-07 15:45:52 +02:00
Roman Khavronenko
5f46f8a11d
lib/promrelabel: speedup label match by __name__ (#6432)
The change adds a fastpath for `equalValue` comparisons against
`__name__` label by avoiding calls to `toCanonicalLabelName` func. This
speedups matches by metric name like `'foo'`. See bench stats below:
```
benchcmp old.txt new.txt

benchmark                                           old ns/op     new ns/op     delta
BenchmarkIfExpression/equal_label:_last-10          35.6          35.1          -1.18%
BenchmarkIfExpression/equal_label:_middle-10        18.3          17.3          -5.41%
BenchmarkIfExpression/equal_label:_first-10         1.20          1.24          +2.74%
BenchmarkIfExpression/equal___name__:_last-10       10.1          4.96          -50.75%
BenchmarkIfExpression/equal___name__:_middle-10     5.79          3.16          -45.41%
BenchmarkIfExpression/equal___name__:_first-10      1.17          1.05          -9.76%
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-07 15:44:48 +02:00
Andrii Chubatiuk
185fac03b3
lib/streamaggr: metrics to track dropped, nan samples and samples lag (#6358)
### Describe Your Changes

Added streamaggr metrics to:
 - `vm_streamaggr_samples_lag_seconds` - samples lag
- `vm_streamaggr_ignored_samples_total{reason="nan"}` - ignored NaN
samples
- `vm_streamaggr_ignored_samples_total{reason="too_old"}` - ignored old
samples
2024-06-06 14:06:11 +02:00
Aliaksandr Valialkin
55d8379ae6
lib/logstorage: work-in-progress 2024-06-06 12:27:05 +02:00
Aliaksandr Valialkin
d54d5a17de
docs/VictoriaLogs/CHANGELOG.md: document the a68c2c0f17 2024-06-06 12:21:17 +02:00
Aliaksandr Valialkin
21fafd550c
app/vlselect/vmui: run make vmui-logs-update after a68c2c0f17
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6419
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6408
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6405
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6406
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6407
2024-06-06 12:18:08 +02:00
Yury Molodov
a68c2c0f17
vmui/logs: improve log display for group view (#6419)
### Describe Your Changes

1) Set the default limit to `50`. 
    #6408
2) Configure the default search to cover the `last 5 minutes` and
include all messages (`*`).
     #6405
3) In the header, display only streams and group by stream.
     #6406
4) Add log processing, without the fields `msg`, `time`, and `stream`.
5) When clicking on logs, display a list of all fields.
     #6407

<img width="400" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/666dcaa3-20fb-4828-b77b-1d849dd9a8ed">

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-06 12:14:06 +02:00
Dima Lazerka
c57c16925d
vmui: Improve DownloadConfig button interaction with VMAnomaly (#6397)
Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
2024-06-06 11:07:59 +02:00
Github Actions
ff513b40c0
Automatic update operator docs from VictoriaMetrics/operator@32f7674 (#6414) 2024-06-06 10:20:11 +08:00
Aliaksandr Valialkin
1d86ea49a0
docs/VictoriaLogs/LogsQL.md: typo fix: replace from _json with from _msg where appropriate 2024-06-05 10:15:46 +02:00
Aliaksandr Valialkin
c2cd47da07
docs/VictoriaLogs/LogsQL.md: substitute TAG with APP-NAME[PROCID] in rfc3164 syslog message format at unpack_syslog pipe docs 2024-06-05 10:13:41 +02:00
Aliaksandr Valialkin
80a7c65ab7
lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:07:49 +02:00
Aliaksandr Valialkin
868c4c0253
docs/VictoriaLogs/LogsQL.md: typo fix: substitute "'" with "" in front of John` 2024-06-05 09:51:12 +02:00
Aliaksandr Valialkin
b9c0eb3dde
docs/VictoriaLogs/LogsQL.md: typo fix: its -> it 2024-06-05 09:45:55 +02:00
Zhu Jiekun
215f36a402
docs: [deployment] update CHANGELOG.md to include go version change 1.22.4 (#6412)
### Describe Your Changes
Update CHANGELOG to include go version change introduced in
43cf221681

Also see:
- https://go.dev/doc/devel/release
-
https://github.com/golang/go/issues?q=milestone%3AGo1.22.4+label%3ACherryPickApproved

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-05 11:17:36 +04:00
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
hagen1778
f497d3c222
docs: add technical articles by @jiekun
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-04 15:52:33 +02:00
pludov
3ddae77c63
lib/fs: support NFS implementations that return EEXIST instead of ENOTEMPTY (#6398)
### Describe Your Changes

Fix for issue #6396: according to rmdir manpage, ENOTEMPTY and EEXIST
should be treated equally

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6396

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Ludovic Pollet <ludovic.pollet@exfo.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-04 15:17:38 +02:00
Aliaksandr Valialkin
cbf606979d
docs/VictoriaLogs/LogsQL.md: mention unpack_syslog pipe in the Transformations section 2024-06-04 03:17:07 +02:00
Aliaksandr Valialkin
96c29ab403
lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc 2024-06-04 02:29:10 +02:00
Aliaksandr Valialkin
68c07b0599
docs/VictoriaLogs/Roadmap.md: remove LogsQL transformations from roadmap, since they are already implemented 2024-06-04 02:21:00 +02:00