Commit graph

5873 commits

Author SHA1 Message Date
nemobis
65ff04bc09
docs: Fix typo in changelog for v113
Fix a typo `scrapped` for `scraped`.
2025-04-08 16:14:50 +02:00
Zakhar Bessarab
e2715f94af
docs/guides/vmgateway-grafana-oidc: update guide for recent versions of components
- update grafana & keycloak to latest versions
- update UI images with the latest screenshots
- update wording to reflect UI changes
2025-04-08 16:13:58 +02:00
Max Kotliar
3f5bf4bd03
vmagent/remotewrite: set content encoding header based on actual body
Improve remote write handling in vmagent by setting the
`Content-Encoding` header based on the actual request body, rather than
relying on configuration.

- Detects Zstd compression via the Zstd magic number.
- Falls back to Snappy if Zstd is not detected.
- Persistent queue may now contain mixed-encoding content.
- Add basic vmagent integration tests

Follow up on
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5344 and
12cd32fd75.

Extracted from
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301
2025-04-08 16:12:06 +02:00
f41gh7
038419663b
docs: release follow-up
* mention lts release changes
* update vm apps versions at docs and deployment examples

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-04-07 12:59:53 +02:00
f41gh7
123f373537
CHANGELOG.md: cut v1.115.0 release 2025-04-04 14:30:16 +02:00
f41gh7
57121c828f
make docs-update-version 2025-04-04 14:23:08 +02:00
Andrii Chubatiuk
f9d8c86b0a
lib/streamaggr: fix panic in rate output
This commit properly reset aggregator state. Previously, it was not checked for `nil` and it lead to the panic on access.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8634
2025-04-04 14:14:52 +02:00
hansemschnokeloch
b733fc5b83
docs/vlogs: fix typo in README 2025-04-04 14:10:59 +02:00
Zakhar Bessarab
f2eaad62dc
docs/changelog: correct entry location after 298f862f
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-04 12:24:16 +04:00
Zakhar Bessarab
298f862fc0
deps: downgrade AWS dependencies
Pin AWS libraries to version before 2025-01-15 (see
https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15).

This version enabled request and response checksum verification by
default which breaks compatibility with non-AWS S3-compatible storage
providers.

See: https://github.com/victoriaMetrics/victoriaMetrics/issues/8622

Supersedes https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8630

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-03 18:05:07 +04:00
Zakhar Bessarab
aff1580a1d
app/vmauth: return non-OK response for timeouts and request cancellation
Currently, requests failing due to network timeout would receive "200
OK" while producing a warning log message about the timeout. This
behaviour is confusing and might produce unexpected issues as it is not
possible to retry errors properly.

Change this to return "502 Bad Gateway" response so that error can be
handled by the client.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8621

Config for testing:
```
unauthorized_user:
  url_prefix: "http://example.com:9800"
```

Before the change:
```
*   Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
/* NOTE: 30 seconds timeout passes */
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 08:54:05 GMT
Date: Tue, 01 Apr 2025 08:54:05 GMT
<

* Connection  to host 127.0.0.1 left intact
```

After:
```
*   Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 09:13:57 GMT
Date: Tue, 01 Apr 2025 09:13:57 GMT
< Content-Length: 109
Content-Length: 109
<

* Connection  to host 127.0.0.1 left intact
```

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-03 13:44:51 +04:00
hagen1778
d4c0a42c1b
docs: improve wording for recent vmalert changes
follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8522

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3cbc3eb19f)
2025-04-03 09:47:34 +01:00
Emre Yazıcı
a9736a5bfb
app/vmalert: show partial responses in debug logs ()
### Describe Your Changes

Log when the data response from vmselect is partial during
rule(recording, alertingrule) evaluations.

vmselect returns `isPartial: true` in case data is not fully fetched
from scattered vmstorages. At the time of rule evals, it may be drifting
apart from real values due to missing points. This is an important event
that should be logged to inform users to see how often that happens as
it may lead to false positive alerts.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: emreya <emre.yazici@adyen.com>
Signed-off-by: emreya <e.yazici1990@gmail.com>
Signed-off-by: Emre Yazici <e.yazici1990@gmail.com>
(cherry picked from commit 56f60e8be9)
2025-04-03 09:47:34 +01:00
Artem Fetishev
2e4beeefb1
Update series count docs ()
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-04-03 10:37:35 +02:00
Aliaksandr Valialkin
ca3910748f
deployment: update Go builder from Go1.24.1 to Go1.24.2
See https://github.com/golang/go/issues?q=milestone%3AGo1.24.2+label%3ACherryPickApproved
2025-04-02 18:01:21 +02:00
Aliaksandr Valialkin
4b2276608b
docs/victoriametrics/vmagent.md: mention that increasing scrape_interval can reduce CPU usage 2025-04-02 12:41:57 +02:00
Aliaksandr Valialkin
b352470ae1
docs/victoriametrics/vmagent.md: mention that -promscrape.disableKeepAlive option can reduce RAM usage when scraping thousands of targets 2025-04-01 23:22:13 +02:00
Aliaksandr Valialkin
6d5973dcb0
docs/victoriametrics/vmagent.md: change GOGC from 50 to 100 in the example of optimized config for vmagent
This is a follow-up after bf024d3dce,
2025-04-01 21:36:04 +02:00
Aliaksandr Valialkin
74f17bb67e
docs/victoriametrics/vmagent.md: remove the recommendation to set GOGC to 50 at vmagent in order to reduce CPU usage
The default GOGC is set to 50 at vmagent after bf024d3dce,
so this recommendation makes no sense. Leave the recommendation to increase GOGC to 100.
2025-04-01 21:14:33 +02:00
Aliaksandr Valialkin
34d35869fa
docs/victoriametrics/vmagent.md: add Performance optimizations chapter
Enumerate the most commonly used options for reducing CPU usage and RAM usage
for vmagent, which scrapes thousands of targets.

See https://docs.victoriametrics.com/vmagent/#performance-optimizations
2025-04-01 18:35:43 +02:00
Phuong Le
346db8a606
vmui: fix auto-suggestion doesn't work inside functions ()
Fixes 

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-31 16:22:25 +02:00
hagen1778
b277a62e94
docs: mention pull request checklist in doc guides
Checklist is a more practical list of actions than a full Contributing doc.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-31 15:48:10 +02:00
Roman Khavronenko
66e7b908ec
dashboards: drop all dashboards tags except victoriametrics or victorialogs tags for consistency ()
Having `victoriametrics` or `victorialogs` tags should be enough for
filtering dashboards related to VictoriaMetrics components.

Related ticket
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8618

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-31 14:27:05 +02:00
Roman Khavronenko
a2ba37be68
lib/promscrape: support filtering targets via scrapePool GET param in /api/v1/targets API ()
This improves compatibility with Prometheus `/api/v1/targets` API.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5343

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-31 14:26:25 +02:00
Hui Wang
b9c777a578
vmgateway: properly set the Host header when routing requests to `-… ()
* vmgateway: properly set the `Host` header when routing requests to `-write.url` and `-read.url`

* Update docs/victoriametrics/changelog/CHANGELOG.md

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-28 12:27:19 +01:00
Hui Wang
d8fe739aba
vmalert: properly attach tenant labels vm_account_id and `vm_projec… ()
* vmalert: properly attach tenant labels `vm_account_id` and `vm_project_id` to alerting rules when enabling `-clusterMode`

Previously, these labels were lost in alert messages to Alertmanager. Bug was introduced in [v1.112.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.112.0).
2025-03-28 12:26:55 +01:00
Andrii Chubatiuk
0e142e4e11
docs: update /VictoriaLogs path in hugo to /victorialogs ()
### Describe Your Changes

replace /VictoriaLogs aliases to /victorialogs as generated directories
are anyway renamed to victorialogs before deployment

need to merge this PR immediately after
https://github.com/VictoriaMetrics/vmdocs/pull/116

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 11:29:36 +01:00
hagen1778
ef16681dbf
dashboards: rm ETA panel from single and cluster dashboards
The panel was producing wrong predictions as it is almost impossible,
without making too expensive queries, to make a precise predictions.

More details on reasoning why it is better to remove it than fix it
is here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8492.

This change also removes ETA panels from alerting rules annotations.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-27 10:36:58 +01:00
Denys Holius
c1663f2175
docs/Articles.md: add a link to https://helgeklein.com/blog/victoriametrics-long-term-storage-of-home-assistant-data/ ()
### Describe Your Changes

This PR will add a link to
https://helgeklein.com/blog/victoriametrics-long-term-storage-of-home-assistant-data/

### Checklist

The following checks are **mandatory**:

- [ x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 10:20:20 +01:00
hagen1778
2bce56b348
docs: add cmd-line flag example to vlogs quickstart
Add example of using command-line flags when running vlogs
docker image or binary.
This might be helpful for users - see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8599

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-27 10:15:39 +01:00
Andrii Chubatiuk
63222a512e
docs: support new docs site version updates ()
### Describe Your Changes

- add smaller search weights for changelog content
- remove replace `<details>` tag with collapse shortcode

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 10:12:13 +01:00
Andrii Chubatiuk
f152021521
docs: move victorialogs, guides, anomaly-detection and victoriametrics-cloud to separate folders ()
related PR https://github.com/VictoriaMetrics/vmdocs/pull/115

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 10:05:14 +01:00
hagen1778
35319a414b
docs: update quickstart guide
* add more detailed instructions to docker sections
* fix typos
* re-word for simplicity

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-27 09:58:45 +01:00
hagen1778
e09c3f7938
docs: update guide for capacity planning
* update wording
* rm extra verbosity
* typo fixes
* add extra info about cluster sizing

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-26 15:55:57 +01:00
hagen1778
28c9f617c2
docs: update guide for migrating from influx
* update wording
* rm extra verbosity
* typo fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-26 14:47:04 +01:00
Zakhar Bessarab
099b2fdba7
docs/changelog: restore tip
Restore tip after e9508465.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-24 20:17:21 +04:00
Zakhar Bessarab
9a3b5114db
docs/changelog: document bugfix
Bugfix was introduced at 682e8d8af5 and was included as a part of package update here b1fab92d, but lacks a changelog change.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-24 19:50:27 +04:00
Zakhar Bessarab
34e1e18bcc
{docs,deployment}: post-release update
- update references to the latest version
- port LTS changelog
- revert changes from 47201ace96 and overwrite by an actual latest enterprise release version instead of latest LTS release

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-24 17:38:05 +04:00
Aliaksandr Valialkin
47201ace96
docs/enterprise.md: refer the real latest enterprise release in the examples
This is the follow-up for 44b0466281
2025-03-22 13:13:06 +01:00
Zakhar Bessarab
e950846534
docs/CHANGELOG.md: cut v1.114.0
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-21 16:42:46 +04:00
Zhu Jiekun
ca9bb48f3f
doc: [guide] update k8s vmsingle vmcluster helm and operator guide
This commit updates some wording for the k8s guide here:
- [Kubernetes monitoring via VictoriaMetrics
Single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single)
- [Kubernetes monitoring with VictoriaMetrics
Cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster)
- [Getting started with VM Operator,
](https://docs.victoriametrics.com/guides/getting-started-with-vm-operator)
2025-03-21 11:07:55 +01:00
Fred Navruzov
8239c119ca
docs/vmanomaly: release v1.21.0 + HC/HA docs ()
### Describe Your Changes

PR updates docs to release v1.21.0, in particular, adjust docs and its
structure to High Availability (HA) and horizontal scalability (HS)
capabilities.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-20 18:21:50 +02:00
Max Kotliar
c05ffa906d
lib/promscrape: improve streamParse performance
Previously, performance of stream.Parse could be limited by mutex.Lock on callback function. It used shared writeContext. With complicated relabeling rules and any slowness at pushData function, it could significantly decrease parsed rows processing performance.

 This commit removes locks and makes parsed rows processing lock-free in the same manner as `stream.Parse` processing implemented at push ingestion processing.

 Implementation details:
- Removing global lock around stream.Parse callback.
- Using atomic operations for counters
- Creating write contexts per callback instead of sharing
- Improving series limit checking with sync.Once
- Optimizing labels hash calculation with buffer pooling
- Adding comprehensive tests for concurrency correctness

 Benchmark performance:
```
# before
BenchmarkScrapeWorkScrapeInternalStreamBigData-10             13          81973945 ns/op          37.68 MB/s    18947868 B/op        197 allocs/op

# after
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape
cpu: Apple M1 Pro
BenchmarkScrapeWorkScrapeInternalStreamBigData-10             74          15761331 ns/op         195.98 MB/s    15487399 B/op        148 allocs/op
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape       1.806s
```

Related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8159
---------
Signed-off-by: Maksim Kotlyar <kotlyar.maksim@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-03-20 16:49:24 +01:00
Zakhar Bessarab
30625e83b6
app/vmbackupmanager: properly set vm_backup_last_run_failed metric
Previously, `getBackupsList` was appending `latest` backup in all cases without checking if it actually exists.
This lead to `vm_backup_last_run_failed` metric being set to `1` since folder did not contain successful completion marker.

This commit adds a check to handle a case when remote storage does not contain any backups yet.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8490

---
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-20 15:50:13 +01:00
Roman Khavronenko
4c9c2501f3
docs: fix a few typos 2025-03-20 15:37:16 +01:00
Zakhar Bessarab
3447cb0bd3
lib/backup/s3remote: add retries for "IncompleteBody" errors
These errors could be caused by intermittent network issues, especially
in case of using proxies when accessing S3 storage. Previously, such
error would abort backup/restore process and require manual intervention
to ensure backups consistency.

This commit adds automatic retries to handle this to improve backups
reliability and resilience to network issues.
2025-03-20 12:43:18 +01:00
f41gh7
985f7faad8
app/vmgateway: remove vmalert dependency
1. remove "vmalert" word from vmgateway doc and exposed metrics;
2. remove unrelated flags like -datasource.roundDigits, remoteRead.disablePathAppend, -datasource.disableStepParam
2025-03-20 12:39:24 +01:00
Zakhar Bessarab
3155542168
app/vmbackupmanager: properly close run channel when stopping
vmbackupmanager uses `runC` channel for inter-goroutine communication between `scheduler` and `execute` goroutines.

 Previously, `runC` wasn't closed during graceful shutdown. And vmbackupmanager process couldn't gracefully stop. It could only be killed with-in configured timeout.

This commit properly closes `runC` by `scheduler` when stopping vmbackupmanager in order to avoid shutdown delay.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
2025-03-20 12:39:01 +01:00
f41gh7
756c93c66a
app/vmbackupmanager: prevent backups being scheduled one after another
Previously, backup was first scheduled at 00:00:00 and `getSleepDuration` was immediately executed to get the sleep duration for the next backup. Since it was returning `1 * time.Second` the next backup was attempted and failed to be scheduled.

Update logic to wait for full backup interval in this case so that there will be no attempt to schedule an unneeded backup.

 It also adds the following changes:
* fix error log entry reference to type of policy
* add a message about retention completion similar to existing message for backups to make it more consistent

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8499

This is a follow-up for fb6d2e92e3a1cf412d1f7dee64a4852941a8aa1b
2025-03-20 12:38:50 +01:00
Andrii Chubatiuk
511517f491
lib/streamaggr: fix threshold update, when deduplication and windows are enabled ()
### Describe Your Changes

during initial flush with deduplication and windows enabled lower
timestamps threshold is set to an upper bound of the next deduplication
interval, which leads to ignoring all samples on subsequent intervals

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-20 09:54:35 +01:00
Yury Molodov
db66ab1852
vmui: show hidden common labels in group name ()
### Describe Your Changes

Changes:
1. When `hide common labels` is enabled, they will now be displayed in
the group name.
2. Legend settings toggles have been moved below the graph for better
accessibility.


![image](https://github.com/user-attachments/assets/fc8c7f4c-c155-4056-8862-301ad375d7ae)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-20 09:41:15 +01:00
Yury Molodov
31f662a0f7
vmui/logs: implement nanosecond log sorting ()
### Describe Your Changes

This PR adds nanosecond precision to log sorting, ensuring accurate
ordering of entries with sub-millisecond differences.

Related issue:  

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-20 09:30:57 +01:00
hagen1778
7f69553230
docs: fix typos in release versions
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-20 09:23:34 +01:00
hagen1778
90d547dec0
docs: mark LTS releases explicitly in changelog
The mark is important for readers to understand the type of release.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-20 09:12:31 +01:00
Hui Wang
4587d092fd
vmgateway: fix vmgateway_ratelimit_refresh_duration_seconds
* vmgateway: fix `vmgateway_ratelimit_refresh_duration_seconds`

* revert reload ep change

---------

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-20 09:41:42 +04:00
Hui Wang
bd11e00a59
app/vmalert: properly register group and rules metrics
Commit 9ca74d1fff introduced an issue with metrics registration. Due to metrics.Summary type always registered at the global state of metrics package, vmalert had increased memory and CPU usage after multiple configuration reloads.

 This commit addresses this issue and properly registers metrics.Summary metric. Now metrics for group and rules must be explicitly registered before group.Start with group.Init method. It simplifies metrics usage an ensures that all needed metrics were registered and group is ready to start.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
2025-03-19 13:57:34 +01:00
Hui Wang
b97cacad45
app/vmalert: fix possible data race on group checksum
1. fix possible data race on group checksum when reload is called
concurrently. Before, it didn't affect much but might update the group
one more time.
2. remove the unnecessary g.mu.RLock() and compute group.id at newGroup creation. Changes to group.ID()
indicate that type and interval have changed, and the group is new.

Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8540
2025-03-19 12:58:51 +01:00
Aliaksandr Valialkin
acdd6faecf
docs/VictoriaLogs/querying/README.md: fix the docs for /select/logsql/stream_field_values when the limit arg is set
It returns up to `limit` values for the given log stream field with the biggest number of hits.
2025-03-19 12:41:27 +01:00
Aliaksandr Valialkin
ea4534d154
lib/logstorage: support for {field in (*)} and {field not_in (*)} syntax in LogsQL
This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238
to be consistent with `in(*)` feature, which has been added in the commit 84d5771b41
2025-03-19 12:09:55 +01:00
Hui Wang
dec237b7d6
app/vmalert: fix memory leak with -notifier.blackhole
Previous commit 9ca74d1fff added a regression for notifier's metrics exposed by vmalert. vmalert returned new notifier instances for the blackhole notifier type. And it registered new metrics each get notifiers function was called. It registered duplicate metrics and lead to OOM crash.

 This commit properly init blachole notifier instances and add metrics for it only once, during application start.

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
2025-03-19 10:43:57 +01:00
Nikolay
27d7d0b25c
lib/promscrape: properly send staleness markers
Previously, vmagent may incorrectly store partial scrape response
in case of scrapping error. It may happen if `sw.ReadData` call fetched
some chunked response and store it at buffer. And later context deadline
exceed error happened.
 As a result, at the next scrape iteration this partial response could
 be forwarded to the `sw.sendStaleSeries(lastScrape...)` function call
 and lead to `Prometheus line` parsing error.

 This commit properly set response body to the empty value in case of
scrapping error. It prevents storing partial scrape response body. And
it no longer sends partial staleness markers to the remote storage.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8528
2025-03-19 10:37:54 +01:00
Roman Khavronenko
3511e2e6af
dashboards: add Memory allocations rate to ResourceUsage tab ()
This panel should have help us to identify
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501 during
release checks. While we already track the GC pressure, its value is
relative and change wasn't noticeable for the workloads that we
observed.

The absolute values of allocations rate could have helped to see the
anomaly.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-17 16:42:53 +01:00
Yury Molodov
44a54e4590
vmui/logs: fix endless group expansion loop bug ()
### Describe Your Changes

Fix endless group expansion loop.
Related issue: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-17 16:39:41 +01:00
hagen1778
d4560ee015
docs: add update note for renamed metric vm_mmapped_files
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-17 16:34:36 +01:00
Guillem Jover
76d205feae
spelling and grammar fixes via codespell ()
### Describe Your Changes

Fix many spelling errors and some grammar, including misspellings in
filenames. 

The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards. 
So it seems to have low impact on users.

The change also deprecates `cspell` as it is much heavier and less usable. 
---------

Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
2025-03-17 16:32:10 +01:00
Jose Gómez-Sellés
d852e5e0b4
docs/cloud: add FAQ for VM Cloud ()
### Describe Your Changes

This PR adds the dedicated FAQ for VictoriaMetrics Cloud

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-17 15:54:52 +01:00
Fred Navruzov
ee3ed8ab86
docs/vmanomaly: update to patch release v1.20.1 ()
### Describe Your Changes

Doc updates to a patch release v1.20.1, fixing a bug in
`PeriodicScheduler` that may affect some of the customers' deployments

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-16 21:34:19 +02:00
Aliaksandr Valialkin
c005cb89fc
deployment: update VictoriaLogs Docker image tag from v1.16.0-victorialogs to v1.17.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.17.0-victorialogs
2025-03-16 01:21:59 +01:00
Aliaksandr Valialkin
771233ebcd
docs/VictoriaLogs/CHANGELOG.md: cut v1.17.0-victorialogs 2025-03-16 01:15:54 +01:00
Aliaksandr Valialkin
39082103a6
app/vlinsert/opentelemetry: follow-up for a884949aba
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8511
2025-03-16 01:09:07 +01:00
Devops
a884949aba
fix:Fixed an issue where and were incorrectly displayed ()
### Describe Your Changes

Fixed an issue where and were incorrectly displayed when sent from
OpenTelemetry Collector to Victoria Logs

Fixes 
2025-03-16 00:33:52 +01:00
Aliaksandr Valialkin
d2f4698e3f
docs/VictoriaLogs/querying/README.md: mention that /select/logsql/query endpoint may return arbitary number of logs matching the given query filter, and this is OK
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8507
and https://github.com/VictoriaMetrics/victorialogs-datasource/issues/261
2025-03-16 00:01:52 +01:00
Aliaksandr Valialkin
23c4e4cdb2
app/vlinsert: send 204 No Content response code at /insert/loki/api/v1/push endpoint
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8505
2025-03-15 23:34:49 +01:00
Roman Khavronenko
3d9f2e3937
lib/bytesutil: don't drop ByteBuffer.B when its capacity is bigger th… ()
…an 64KB at Reset

This commit reverts
b58e2ab214
as it has negative impacts when ByteBuffer is used for workloads that
always exceed 64KiB size. This significantly slows down affected
components because:
* buffers aren't beign reused;
* growing new buffers to >64KiB is very slow.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-15 01:38:36 +01:00
Aliaksandr Valialkin
2c7dd2b991
lib/logstorage: support for {label in (v1,...,vN)} and {label not_in (v1, ..., vN)} syntax 2025-03-15 01:35:13 +01:00
Aliaksandr Valialkin
0451a1c9e0
app/vlinsert: follow-up for 37ed1842ab
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
  Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
  when Content-Encoding header is missing.

- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
  This eliminates the churn in tests for these functions. This also fixes broken
  benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
  the whole request body on the first iteration and do nothing on subsequent iterations.

- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
  versions of VictoriaMetrics and VictoriaLogs.

- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
  at VictoriaLogs and VictoriaMetrics.

- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
  the passed reader arg in order to properly deal with it.

- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
  This allows removing repeated code across request-based protocol parsers without streaming mode.

- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
  Previously this wasn't the case for all the supported protocols.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
2025-03-15 00:03:03 +01:00
Aliaksandr Valialkin
c60b4175bb
app/vlinsert: add an ability to ignore log fields starting with the given prefixes
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
2025-03-15 00:03:02 +01:00
hagen1778
972f14d540
docs: add vmsingle to affected components
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 19:12:05 +01:00
hagen1778
f62b690599
changelog: mention in update notes
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 19:04:00 +01:00
Roman Khavronenko
dc1f7ef0d0
app/vmselect/promql: optimize binary operator or for common cases ()
The optimization touches 2 things:
1. Reduces amount of allocations when comparing canonical metric names
between left and right parts of expressions.
2. Adds fast path for cases when right part of expression returns
scalar: `series_selector or on() vector(1)`, which is a typical
expression.

```
benchcmp old.txt new.txt
benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark                                             old ns/op     new ns/op     delta
BenchmarkBinaryOpOr/tss:1_or_tss:1-14                 291           272           -6.56%
BenchmarkBinaryOpOr/tss:1_or_tss:1000-14              44590         28592         -35.88%
BenchmarkBinaryOpOr/tss:1000_or_tss:1-14              103124        39563         -61.64%
BenchmarkBinaryOpOr/tss:1000_or_tss:1000-14           20386150      1859335       -90.88%
BenchmarkBinaryOpOr/tss:1000_or_on()_vector(0)-14     91382         36805         -59.72%
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8382

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 12:07:22 +01:00
hagen1778
8b0129f29b
docs: re-organize order of items in vmagent docs
* tie relevant functionality together
* change hierarchy of related options to visually group it

No breaking changes to links.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 12:02:23 +01:00
Zhu Jiekun
9548b7e442
docs: revert doc change for on-disk persistence and move new content to another section ()
### Describe Your Changes

revert doc change in
815bad3687
and move new content to another section.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 11:50:17 +01:00
Roman Khavronenko
85f1bd172b
docs: re-organize docs ()
* move related sections clother to each other
* group related sections within the same section

The intention of the change is to tie related documentation together.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 11:47:00 +01:00
Aliaksandr Valialkin
8c079602c1
lib/logstorage: optimize handling long constant fields
Long constant fields cannot be stored in columnsHeader as a const column,
because their size exceeds maxConstColumnValueSize, so they are stored as regular values.
This commit optimizes storing such fields by storing only a single value
across the field values in a block instead of storing multiple values.
This should improve data ingestion performance a bit. This also should improve query
performance when the query accesses such fields because of better cache locality.

Also improve persisting of constant string lengths by storing them only once.
2025-03-14 03:14:01 +01:00
f41gh7
5301af33c0
app/vmselect: properly cancel multitenant query request
Previously, vmselect didn't stop multitenant query execution if it
receives error from vmstorage. Such as limit error or any other. It
continued to execute queries until it did it for all tenants. It leads
to the potential waste of resources.
 In addition, callback error was incorrectly reference and can be updated by
subsequent callback call.

This commit returns error earlier, cancels sub-sequent requests for
tenants and properly return storageNode request error.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8461
2025-03-14 00:52:01 +01:00
Andrii Chubatiuk
37ed1842ab
lib/protoparser: support zstd in all logs http ingestion, datadog and otel metrics protocols ()
This commit introduces common readers for multiple compression encoding algorithms. 

Currently, supported encodings are:
* zstd
* gzip
* deflat
* snappy

 It adds new common reader to the all VictoriaLogs ingestion protocols.
And updates opentelemetry metrics parsing for VictoriaMetrics components.

Also, it ports zstd stream parses from cluster branch.

Related issues:
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300

---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-14 00:39:52 +01:00
Zhu Jiekun
815bad3687
app/vmagent: prevent dropping persistent queue if -remoteWrite.showURL changed
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
 Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
 Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.

 This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
 It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision. 

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.


### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-13 23:50:01 +01:00
Andrii Chubatiuk
6e34ea62c7
lib/awsapi: add EKS Pod Identity auth method
AWS introduced a new secure way for Kubernetes Pod authorization at AWS API.
The feature is called Pod Identity.
 It adds the following env variables to the Pod:
* AWS_CONTAINER_CREDENTIALS_FULL_URI -  endpoint URI served by the EKS Pod Identity Agent running on the worker node.
* AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE - projected JWT token that is used to exchange for IAM credentials.

See related blog post https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-a-new-way-for-applications-on-eks-to-obtain-iam-credentials/

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5780
2025-03-13 23:39:19 +01:00
Zakhar Bessarab
7dfdee5709
lib/httputils: always set up TLS config
Previously, TLS config was only created for URLs with `https` scheme.
This could lead to unexpected errors when original URL was redirecting
to `https` one as TLS config is not applied.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8494
2025-03-13 23:27:49 +01:00
Aliaksandr Valialkin
dfc22950bd
deployment: update VictoriaLogs Docker image from v1.15.0-victorialogs to v1.16.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.15.0-victorialogs
2025-03-12 23:48:26 +01:00
Aliaksandr Valialkin
4b52f7973d
docs/VictoriaLogs/CHANGELOG.md: cut v1.16.0-victorialogs 2025-03-12 23:41:16 +01:00
Aliaksandr Valialkin
b1fab92d1f
vendor: run make vendor-update 2025-03-12 22:40:42 +01:00
Aliaksandr Valialkin
19165c436f
app/vlinsert/loki: automatically parse JSON-encoded log fields from the plaintext log message
Loki doesn't support well high-cardinality log fields (e.g. fields with big number of unique values).
That's why Promtail, Grafana Agent and Grafana Alloy encode such fields into a JSON and push them
as a plaintext log message to the remote storage. This isn't an efficient way to store high-cardinality
log fields in VictoriaLogs, since it is optimized for storing and querying such fields when they are stored
distinctly as a regular log fields according to VictoriaLogs data model ( https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ).

This commit enables automatic parsing of JSON-encoded log fields at plaintext log message received over Loki protocol
and storing them as a separate log fields. This should improve data compression ratio and reduce disk space usage.
This should also improve query performance when the parsed log fields are used in queries for filtering and aggregation.
The old behaviour can be restored by passing -loki.disableMessageParsing command-line flag to VictoriaLogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8486
2025-03-12 21:23:28 +01:00
Artem Fetishev
ee66d601b4
lib/storage: fix active timeseries collection when per-day index is disabled ()
Fix metric that shows number of active time series when per-day index is disabled. Previously, once per-day index was disabled, the active time series metric would stop being populated and the `Active time series` chart would show 0.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8411.
2025-03-12 17:12:09 +01:00
Aliaksandr Valialkin
cb240eda70
app/vlinsert: follow-up for 67f8fa66ed
- Properly handle negative timestamps (e.g. timestamps before 1970-01-01)

- Optimize parsing floating-point timestamps by eliminating the memory allocation
  needed for returning an error from strconv.ParseInt. Instead, check whether the string contains a dot,
  and then parse it as a floating-point number.

- Add tests for ParseUnixTimestamp function.

- Make the code easier to understand and maintain by removing unneeded generic function toNano().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8472
2025-03-12 16:28:40 +01:00
Aliaksandr Valialkin
02bef62a66
docs/VictoriaLogs/CHANGELOG.md: move the description of the fix for the proper OpenTelemetry attributes conversion into JSON into the correct place
The bugfix isn't released yet, so move it from v1.15.0-victorialogs release to the tip.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8387

This is a follow-up for 26fba57cfa
2025-03-12 15:57:10 +01:00
Aliaksandr Valialkin
4d44c3e154
lib/logstorage: properly parse floating-point numbers with leading zeroes in fractional part
Parsing for floating-point numbers with leading zeroes such as 1.023, 1.00234 has been broken
in the commit ae5e28524e .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8464
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-03-12 15:25:58 +01:00
Emre Yazıcı
cfd2c6e5e7
app/vmalert: add vmalert_alerts_send_duration_seconds metric ()
### Describe Your Changes

Add `vmalert_alerts_send_latency_seconds` metric for
alertmanager.notifier.

To measure the time for alertmanager calls to send alerts per notifier.
This is needed to see the latency for each notifier from vmalert calls.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: emreya <e.yazici1990@gmail.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-03-12 14:30:54 +01:00
alicja-karasiewicz
d47d329ce7
feat: make topN limit configurable from CLI
Implement changes mentioned in 

Allow the administrator to specify the limit of returned TSDB series in
`/api/v1/status/tsdb` by making a TopN limit configurable from CLI.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: alicja-karasiewicz <alicja.karasiewicz@allegro.com>
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-12 11:33:30 +04:00
Nikolay
11436d5f00
docs: update metric names stats description ()
* add version since feature is available
* add cluster endpoint paths

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-03-12 11:01:24 +04:00
Evgeny
486b9e1c64
lib/promscrape: use original job name as scrapePool value in targets api ()
### Fix scrapePool name

If in the scrape file, I do some magic and manipulate the job name then
Prometheus will show scrapePool as the original job name in the targets
API, but vmagent will set it to the final value which is wrong.
example
```
job: consul-targets
...

- source_labels: [ __meta_consul_service ]
      regex: (\w+)[_-]exporter
      target_label: job
      replacement: $1
```

curl to prom API will show
`"scrapePool": "consul-targets",`
vmagent:
`""scrapePool": "node",`

before changes:
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
pgbackrest
postgres
```
after changes
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
consul-targets
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-11 13:11:35 +01:00
hagen1778
6a1c70115a
docs: order releases in changelog by their version
Ordering changes by release versions enhances the searchability
of the documentation. For example, tracking which release got
the bugfix becomes easier if releases are already sorted.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-11 10:11:58 +01:00
Alexander Marshalov
9007a4803c
vmcloud docs: information about new APIs in Cloud Public API: cloud providers, regions, tiers, deployments and access tokens. ()
2025-03-10 20:33:42 +01:00
Jose Gómez-Sellés
0873d1d8ab
docs/cloud: add account management section ()
### Describe Your Changes

This PR updates the documentation by removing old assets and adding the
user management chapter, divided in different sections.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-10 20:28:55 +01:00
Aliaksandr Valialkin
edecc433ff
deployment: update Go builder from Go1.24.0 to Go1.24.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.24.1+label%3ACherryPickApproved
2025-03-10 18:59:59 +01:00
Artem Fetishev
77b08cdbb0
docs: update vm apps versions to the v1.113.0 release
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-10 15:17:14 +01:00
Artem Fetishev
44b0466281
docs/changelog: mention LTS releases
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-10 14:45:55 +01:00
Naveen
179c530095
docs: update vlogs README.md ()
### Describe Your Changes

Fixed the typo in the documentation. Updated `Ir provides` to `It
provides`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-10 13:52:43 +01:00
Andrii Chubatiuk
c174a046e2
lib/streamaggr: fixed streamaggr panic ()
### Describe Your Changes

fixes 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-10 13:50:55 +01:00
Andrii Chubatiuk
67f8fa66ed
app/vlinsert: support floats for elasticseach timestamps ()
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-10 13:48:48 +01:00
Artem Fetishev
a7d0a75f4c
docs/CHANGELOG.md: cut v1.113.0
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-07 14:57:47 +01:00
Artem Fetishev
ecc46a4f42
make docs-update-version 2025-03-07 14:51:56 +01:00
hagen1778
b131c3bc22
docs: add available release mark to vmalert chaining groups
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-07 14:04:46 +01:00
f41gh7
2b8b9b8536
lib/storage: reject downsampling rules with zero interval configuration
Using zero interval for downsampling rules is not useful and caused a panic when performing validation of intervals.

Reject such rules during parsing in order to highlight incorrect usage and prevent panics.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8454
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-07 13:27:41 +01:00
hagen1778
021c2552dd
docs: change #tip changes order to reflect importance
Put more important features first in the list.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-07 10:49:27 +01:00
hagen1778
f7e1c430bb
docs: restore accidentally dropped changelog line
Line about `$__interval` was accidentally dropped in
b85b28d30a (diff-6564e3f60c3a7942189fe87a0c8f02e0f9841a71f914d64cd5487eb8b23ad66a)

The order was changed intentionally, so this commit could be cherry-picked
to cluster branch.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-07 09:59:15 +01:00
Hui Wang
e8e2ef54a0
vmalert: allow chaining groups with eval_offset ()
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/860,
see
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/change-evaloffset-behavior/docs/vmalert.md#chaining-groups

Also related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8154
2025-03-07 09:45:16 +01:00
Zakhar Bessarab
fd7b016c5b
docs/victoria-logs/data-ingestion/promtail: fix typo ()
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-07 11:03:04 +04:00
Nikolay
b85b28d30a
lib/storage: add tracker for time series metric names statistics
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.

This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.

  New API added to the select component:

* /api/v1/status/metric_names_stats - which returns a JSON
object
    with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
    state of the tracker and reset tsid/cache.

   New metrics were added for this feature:

  * vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
  * vm_cache_size{type="storage/metricNamesUsageTracker"}
  * vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}

  Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-06 22:06:50 +01:00
Zakhar Bessarab
7dfaef9088
app/vmselect/promql: fix panic with using @ with series which is not present at the start of the query ()
### Describe Your Changes

Previously, "selector @ another_selector" assumed that
"another_selector" metric is supposed to exist since "start" used in the
query.

If the query was evaluated in the following case (timestamps):
- start - 2, end - 10
- "another_selector" 5,6,7,8,9,10
- "selector" The resulting "at" timestamp would be taken from NaN (as
`int64(NaN * 1000)`), causing a panic or invalid behavior later.

Note that type cast of `NaN` to int64 is also platform-dependent, so
value of `int64(math.NaN() * 1000)` can produce `0` or max int64 on
different platforms and versions of Go.

This commit changes this and checks for the first non-NaN value. This
makes it easier to use for users as series are not always aligned and
returning an error in this case would disallow using this for some time
ranges.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8444

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-06 16:42:19 +01:00
Dmytro Kozlov
75601c2d9a
vendore: bump metricsql ot v0.84.1 ()
### Describe Your Changes

Updated MetricsQL dependency to v0.84.1

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8435

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-06 15:13:18 +01:00
hagen1778
bcbe5e80b3
docs/changelog: fix metric name in changelog for vlogs request duration
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-06 11:17:58 +01:00
f41gh7
f7d5b11f00
app/vmgateway properly handle trailing slash when applying rate limiter
Previously, the trailing slash was removed and caused an incorrect redirect path when visiting VMUI.

 This commit leaves it as is. Also it applies minor refactoring to url formatting.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8439
2025-03-05 18:43:02 +01:00
Andrii Chubatiuk
26fba57cfa
lib/protoparser/opentelemetry: properly marshal nested attributes into JSON
Previously, opentelemetry attribute parsed added extra field names according to 
golang JSON parser spec for structs:

```
struct AnyValue{
 StringValue string
}
```
 Was serialized into:
```
{"StringValue": "some-string"}
```
 While opentelemetry-collector serializes it as
```
"some-string"
```

 This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
2025-03-05 16:35:07 +01:00
hagen1778
e3e5733b77
docs/changelog: fix formatting of update notes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-05 15:57:21 +01:00
Max Kotliar
e11f5eda1c
docs/vmalert: fix available from version ()
### Describe Your Changes

Fix available from version hint. The feature was introduced in
[v1.91.0](https://docs.victoriametrics.com/changelog/changelog_2023/#v1910).

I noticed that the sentence uses both `{{% available_from "v1.91.0" %}}`
and a manual reference like `starting from
[v1.91](https://docs.victoriametrics.com/changelog/#v1910)`. Does {{%
available_from %}} fully supersede the manual changelog reference, and
should the later be removed? Or should\could both be used together?

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-05 14:40:52 +01:00
Andrii Chubatiuk
bbdb650f2f
docs: remove VictoriaMetrics prefix from anomaly detection menu items titles ()
### Describe Your Changes

Removed VictoriaMetrics prefix from anomaly detection menu items

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-05 14:38:20 +01:00
Jose Gómez-Sellés
400101c674
Add features and guides to VMCloud docs ()
### Describe Your Changes

This PR adds the remaining subsections for the get started part. Some
content is taken from the product page.
Ideally, the get-started page would have some cards instead of raw
links. We should explore that.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-05 01:52:45 -08:00
Zhu Jiekun
08baa8139a
app/vmgateway: properly ratelimit for ingestion path
Commit cd39df1 introduced regression, which caused any write path related limits to be ignored.

This commit fixes match typo and adds check to prevent such kind of regression in future.
2025-03-04 18:42:43 +01:00
Aliaksandr Valialkin
97e99e1fc1
docs/VictoriaLogs/README.md: mention about JSONBench benchmark results in the benchmarks section
See https://docs.victoriametrics.com/victorialogs/#benchmarks
2025-03-04 17:40:31 +01:00
Fred Navruzov
6828cca5a6
docs/vmanomaly: release v1.20.0 ()
### Describe Your Changes

> ⚠️ Even if approved, please don't merge it on my behalf, I
still may apply a couple of re-phrasings before merging it on Monday
03.03.2025

- Aligned `vmanomaly` docs with release v1.20.0
- Re-structured root page of anomaly detection docs for clarity
- Added several sections to FAQ, e.g. on how to incorporate domain
knowledge into anomaly detection configs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-04 00:24:34 +02:00
Yury Molodov
974c094a52
vmui: fix infinite loader on downsampling page ()
### Describe Your Changes

This PR fixes an issue where the Downsampling filters debug page would
get stuck in an infinite loading state when labels had no matches. Now,
the case is properly handled.

Related issue: 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 16:48:11 +01:00
Max Kotliar
649f4d5e00
docs/downsampling: fix terminology. replace "deduplication" with "downsample". ()
### Describe Your Changes

I guess a "downsampling" term is more appropriate than "deduplication"
in the downsampling paragraph.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-03 14:12:36 +01:00
Nikolay
a55ac4495f
docs: update release guide ()
* move testing step after assets upload
* add LTS build step before testing step

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-03-03 14:10:06 +01:00
Hui Wang
52988ebdc8
vmalert: add time buckets stats pipe check for vlogs expression ()
VictoriaLogs inserts `_time` field as a label in result when query with
[time buckets stats
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets),
making the result meaningless and may lead to cardinality issues.
>curl --location --request POST
'https://play-vmlogs.victoriametrics.com/select/logsql/stats_query?query=_time%3A1m%20%7C%20stats%20by%20(_time%3A10s)%20count%20()%20as%20total'

>{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"total","_time":"2025-01-24T12:31:30Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:10Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:00Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:20Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:50Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:40Z"},"value":[1737721904.4476516,"9"]}]}}%

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 14:09:34 +01:00
hagen1778
065a3d068c
docs/victorialogs/vmalert: apply consistent formatting and fix typos
* use unified numeric list;
* apply line width limits;
* remove time filter from quantile examples, as we suggest to
not use time filters.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 13:57:30 +01:00
Artem Fetishev
43a742ba0c
docs: bump last LTS versions
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-03 11:44:37 +01:00
Artem Fetishev
e69c744dd4
docs/CHANGELOG.md: update changelog with LTS release notes
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-03 11:43:00 +01:00
Max Kotliar
7451a3631a
docs: fix typo in docs ()
### Describe Your Changes

Just a small typo in the docs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-03 10:26:41 +01:00
Hui Wang
281f1a94e4
vmalert-tool: clean up the temporary storage path when process is ter… ()
…minated by SIGTERM or SIGINT
2025-02-28 15:17:05 +01:00
Zakhar Bessarab
7f6156e46d
docs: fix link to quay docker image for single-node ()
### Describe Your Changes

Previous link was incorrectly created by adding repository name to
organization link.
All other links are using proper schema.

This is a follow-up for 6ff61a1c.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-28 15:13:52 +01:00
Hui Wang
4015db18bc
vmalert-tool: add flag -httpListenPort to specify the port used dur… ()
…ing testing and make the temporary storage path more unique

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8393

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-02-28 11:25:35 +01:00
Zakhar Bessarab
99de272b72
lib/promrelabel/scrape_url: properly parse IPv6 address from __address__ label
Fix parsing of IPv6 addresses after discovery. Previously, it could lead
to target being discovered and discarded afterwards.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8374

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-28 14:19:10 +04:00
Zakhar Bessarab
6ff61a1c09
docs/all: add refs to Quay mirror for docker images ()
### Describe Your Changes

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4116

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-02-28 11:14:10 +01:00
Zakhar Bessarab
ed0db37d1d
app/vmbackupmanager: fix error propagation for policy apply
Previously error text was silently discarded.

This commit properly formats error message and add actual error cause to it.
2025-02-28 10:40:50 +01:00
Hui Wang
a36c8a9679
app/vmgateway: fix data query in rate limiter
Properly format datasource query request in rate limiter
2025-02-28 10:40:46 +01:00
hagen1778
e10b963a4d
docs: link to internal docs for alerting and recording rules
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-28 09:46:57 +01:00
hagen1778
3a04d9aca8
docs: fix markdown in stream aggregation docs
Availability mark was breaking the line.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-28 09:07:51 +01:00
Aliaksandr Valialkin
bfa9fee424
docs/VictoriaLogs/CHANGELOG.md: typo fix: filter -> filters 2025-02-28 00:09:00 +01:00
Aliaksandr Valialkin
d7431e69fd
deployment/docker: update VictoriaLogs Docker image tag from v1.14.0-victorialogs to v1.15.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.15.0-victorialogs
2025-02-28 00:02:55 +01:00
Aliaksandr Valialkin
0e5e90fe74
docs/VictoriaLogs/CHANGELOG.md: cut v1.15.0-victorialogs release 2025-02-27 22:58:18 +01:00
Aliaksandr Valialkin
744ac496bd
lib/logstorage: add ability to specify field name prefixes inside fields (...) lists passed to pack_json and pack_logfmt pipes 2025-02-27 22:54:18 +01:00
Aliaksandr Valialkin
03a04b4408
docs/changelog/CHANGELOG.md: upgrading from Go1.23.6 to Go1.24.0 doesn't fix any know security issues
The upgrade from Go1.23.6 to Go1.24.0 is a feature, not a security fix.

This is a follow-up for ea6ed4232a
2025-02-27 22:54:18 +01:00
Zhu Jiekun
117b7ce6b7
bugfix: negative rate result when lookbehind window longer than search.maxLookback ()
### Describe Your Changes

 

fix negative rate result when the lookbehind window is longer than
`-search.maxLookback` or `-search.maxStalenessInterval` and data
contains gap.

This issue was introduced since
[v1.110.0](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-27 22:52:39 +01:00
Aliaksandr Valialkin
84d5771b41
lib/logstorage: allow passing * at in(*), contains_any(*) and contains_all(*)
Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for Grafana.

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673
2025-02-27 11:37:43 +01:00
Roman Khavronenko
f39ce2aeef
dashboards: update victorialogs dashboard ()
* stats panels:
* * add ingested logs/bytes over 24h
* * fix correctness compression ratio and disk space usage panels
* overview panels:
* * add requests errors rate, logs rate, query duration panels
* add troubleshooting section:
* * add non-default falgs, dropped logs panels
* update descriptions of the panels and version requirements

Depends on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8396

--------------

<img width="1489" alt="image"
src="https://github.com/user-attachments/assets/5257b8e1-a2e7-4143-94c7-bb42b19e4206"
/>

<img width="1499" alt="image"
src="https://github.com/user-attachments/assets/1b521c58-6e12-4d38-b052-93abdac3b4fc"
/>

<img width="1468" alt="image"
src="https://github.com/user-attachments/assets/3010cf0a-8ca7-48d4-9967-328b37fdbc28"
/>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 11:15:52 +01:00
Roman Khavronenko
ba2bf9e73a
vlogs: add metrics to track select queries duration ()
This change would help to track slow queries via alerts or dashboards.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 11:11:26 +01:00
hagen1778
47d0bba8f0
docs: update never-firing alerts chapter
* add availability marker
* mention article https://victoriametrics.com/blog/never-firing-alerts/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 09:57:36 +01:00
f41gh7
ea6ed4232a
deployment/docker: update Go builder from 1.23.6 to 1.24.0
Changes: https://github.com/golang/go/issues?q=milestone%3AGo1.24
Release notes: https://tip.golang.org/doc/go1.24

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-26 19:14:49 +01:00
Andrii Chubatiuk
b27e9437a3
app/vminsert: properly apply relabelingConfig for scrapped metrics
Due to regression introduced at v1.108.0 release with the following commit:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7399/files#diff-0f7ec2fdf4bb6e94f786c37f19ec837995eca8c61529c3f3436ae332a4a93a5bR56

VictoriaMetrics single-node started to ignore global relabeling configuration.

 This commit properly checks if relabeling is needed.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8389
2025-02-26 16:05:26 +01:00
Zhu Jiekun
3d3480140c
lib/storage: properly cache extDB metricsID on search error
Previously, if indexDB search failed for some reason during search at previous indexDB (aka extDB), VictoriaMetrics stored empty search result at cache. It could cause incorrect search results at subsequent requests.

 This commit checks search error and stores request results only on success. 

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8345
2025-02-26 15:48:25 +01:00
Aliaksandr Valialkin
ae5e28524e
lib/logstorage: do not treat a string with leading zeros as a number at tryParseUint64
The "00123" string shouldn't be treated as 123 number.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-02-25 22:10:36 +01:00
Aliaksandr Valialkin
b484112f22
docs/VictoriaLogs/CHANGELOG.md: fix link to elasticsearch data ingestion docs 2025-02-25 21:31:16 +01:00
Aliaksandr Valialkin
1da04db33a
deployment/docker: update VictoriaLogs Docker image tag from v1.13.0-victorialogs to v1.14.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.14.0-victorialogs
2025-02-25 21:24:42 +01:00
Aliaksandr Valialkin
ef2415e0c5
docs/VictoriaLogs/CHANGELOG.md: cut v1.14.0-victorialogs release 2025-02-25 21:19:58 +01:00
Zhu Jiekun
521af4d7a3
fix: [stats_query_range] fix inconsistent result caused by shared variable ()
### Describe Your Changes

Previously, `timestamp` variable was shared by goroutines and modified
during `writeBlock`, verified in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8312.

This pull request create new variable within each goroutine to avoid
race condition.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ]  Test on sandbox.

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-25 20:56:03 +01:00
Roman Khavronenko
72bee659f5
docs: point github badge to the main action for master branch ()
Before, badge could have shown the latest build result. Even for open
PRs. Now, it shows the latest build result for `master` branch.

See
https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/monitoring-workflows/adding-a-workflow-status-badge

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 18:59:05 +01:00
hagen1778
49fff7989b
docs: fix link to release for search.disableImplicitConversion feature
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 18:54:44 +01:00
Aliaksandr Valialkin
128f6d78ff
docs/VictoriaLogs/FAQ.md: add a question on how to estimate the needed compute resources for the particular workload 2025-02-25 18:24:51 +01:00
Aliaksandr Valialkin
30974e7f3f
lib/logstorage: add le_field and lt_field filters
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.
2025-02-25 18:24:50 +01:00
Andrii Chubatiuk
7a1c84b6ec
vlinsert: accept ES ping requests to endpoint without trailing slash ()
### Describe Your Changes

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8353

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 15:48:40 +01:00
Roman Khavronenko
788c740d62
app/vmctl: respect time filter when exploring time series for influxdb ()
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8259


https://docs.influxdata.com/influxdb/v1/query_language/explore-schema/#run-a-show-series-query-limited-by-time

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 09:26:58 +01:00
Roman Khavronenko
cddccfde57
dashboards: add new panels to VictoriaMetrics dashboards ()
* FEATURE:
[dashboards/single](https://grafana.com/grafana/dashboards/10229) and
[dashboards/cluster](https://grafana.com/grafana/dashboards/11176): add
panel `Deduplication rate` that shows how many samples are
[deduplicated](https://docs.victoriametrics.com/#deduplication) during
merges or read queries by VictoriaMetrics components.
* FEATURE:
[dashboards/single](https://grafana.com/grafana/dashboards/10229) and
[dashboards/cluster](https://grafana.com/grafana/dashboards/11176): add
panel `Number of snapshots` that shows the max number of
[snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots)
across vmstorage nodes. This panel should help in disk usage
[troubleshooting](https://docs.victoriametrics.com/#snapshot-troubleshooting).
* FEATURE:
[dashboards/single](https://grafana.com/grafana/dashboards/10229) and
[dashboards/cluster](https://grafana.com/grafana/dashboards/11176):
account for samples dropped according to [relabeling
config](https://docs.victoriametrics.com/#relabeling) in `Samples
dropped for last 1h` panel.
* FEATURE:
[dashboards/single](https://grafana.com/grafana/dashboards/10229) and
[dashboards/cluster](https://grafana.com/grafana/dashboards/11176): show
number of parts in the last partition on `LSM parts max by type` panel.
Before, the resulting graph could be skewed by the max number of parts
across all partitions. Displaying parts for the latest partition is the
correct way to show if storage is currently impacted by merge delays.
* FEATURE:
[dashboards/cluster](https://grafana.com/grafana/dashboards/11176): add
panel `Partial query results` that shows the number of served [partial
responses](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability)
by vmselects.


-------------

Sorry, git diff is a mess. Dashboards switched to Grafana 11 and it
broke consistency of changes.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 09:25:34 +01:00
Roman Khavronenko
8f87427c81
deployment/rules: add alerting rule TooHighQueryLoad ()
TooHighQueryLoad should trigger when vmsingle or vmselect can't start
processing read queries for last 15min.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 09:22:25 +01:00
hagen1778
6a675bb69b
docs: put link to index-disabling-feature to a better place
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-24 17:03:45 +01:00
hagen1778
3828c01540
docs: properly update version tooltips
The version tooltips were incorrectly set to `#tip` instead of `#`,
so ` make docs-update-version` didn't catch them.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-24 17:00:01 +01:00
f41gh7
0314b79727
docs: update vm apps versions to the v1.112.0 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-24 15:55:06 +01:00
f41gh7
1e368add33
docs/changelog: mention LTS releases 2025-02-24 15:34:17 +01:00
f41gh7
1064c15454
CHANGELOG.md: cut v1.112.0 release 2025-02-24 15:34:17 +01:00
Aliaksandr Valialkin
d72c486d79
vendor: update github.com/VictoriaMetrics/metricsql from v0.83.1 to v0.84.0
The new version of github.com/VictoriaMetrics/metricsql handles $__interval and $__rate_interval
inside rollup functions in more correct way - it drops square brakets, so VictoriaMetrics
could automatically detect the needed lookbehind window depending on the time distance between real samples.

For example, rate(m[$__rate_interval]) is parsed and processed as rate(m) now.
2025-02-24 15:01:08 +01:00
Aliaksandr Valialkin
b8daa8afb4
docs/VictoriaLogs/LogsQL.md: typo fixes 2025-02-22 22:35:54 +01:00
Aliaksandr Valialkin
e977d6f63a
deployment/docker: update VictoriaLogs Docker image tag from v1.12.0-victorialogs to v1.13.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.13.0-victorialogs
2025-02-22 22:24:48 +01:00
Aliaksandr Valialkin
ce7e408c14
docs/VictoriaLogs/CHANGELOG.md: cut v1.13.0-victorialogs 2025-02-22 21:55:58 +01:00
Aliaksandr Valialkin
f1eac36a80
lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-22 21:55:58 +01:00
Aliaksandr Valialkin
c372e10937
lib/logstorage: add an ability to drop duplicate words at unpack_words pipe 2025-02-22 21:55:57 +01:00
Aliaksandr Valialkin
7da98b540b
lib/logstorage: rename unpack_tokens to unpack_words pipe
The LogsQL defines a word at https://docs.victoriametrics.com/victorialogs/logsql/#word ,
so it is more natural to use unpack_words instead of unpack_tokens name for the pipe.
2025-02-22 21:55:56 +01:00
Aliaksandr Valialkin
dfcfaba374
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-22 21:55:54 +01:00
Aliaksandr Valialkin
d33e24ab9b
lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
cd73c1bafb
lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
d32c697361
lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-22 21:55:52 +01:00
Fred Navruzov
965e3dc699
docs/vmanomaly: fix 404 in dashboard guide ()
### Describe Your Changes

- Fixed 404 links to anomaly score dashboard `.json` file on Github.
- Fixed broken markdown on QuickStart page.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-22 20:26:58 +02:00
Fred Navruzov
7c27538993
docs: vmanomaly - add anomaly score dashboard and guide ()
### Describe Your Changes

> (currently in DRAFT) This dashboard and the docs may look differently
after recent review and the feedback received.

As requested by our customers, this PR introduces `anomaly score`-based
dashboard for `vmanomaly` to improve the visual experience and ease the
drill down debugging process for respective anomaly detection setups.
Accompanying guide (under `default` preset mode) is provided as well.

- [For additional
benefits](https://docs.victoriametrics.com/victoriametrics-datasource/#motivation),
the dashboard is based on [VictoriaMetrics
datasource](https://docs.victoriametrics.com/victoriametrics-datasource/)
rather than on `Prometheus` datasource
- Tested locally and on our https://play.victoriametrics.com/
`anomaly_score` data (tenant ID 0)
- To check the guide, build the docs locally (`vmdocs`, `make run
local`) and follow to the
http://localhost:1313/anomaly-detection/presets/#default section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-22 20:09:56 +02:00
f41gh7
7e33efdb6b
docs/changelog: follow-up after 9ca74d1fff
Move vmalert BUGFIX record into current release.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-21 13:24:24 +01:00
Zakhar Bessarab
9ca74d1fff
app/vmalert: properly unregister exposed metrics for alerting rules
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package. 

 This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
2025-02-21 10:36:29 +01:00
Aliaksandr Valialkin
2318e57bbe
deployment/docker: update VictoriaLogs Docker image tag from v1.11.0-victorialogs to v1.12.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.12.0-victorialogs
2025-02-20 22:41:07 +01:00
Aliaksandr Valialkin
1ea3f72d50
lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-20 22:36:09 +01:00
Aliaksandr Valialkin
384a757c3d
docs/VictoriaLogs/CHANGELOG.md: cut v1.12.0-victorialogs 2025-02-20 20:49:54 +01:00
Aliaksandr Valialkin
ffbd0ebbae
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-20 19:04:51 +01:00
Aliaksandr Valialkin
dabe9ff389
docs/VictoriaLogs/keyConcepts.md: small clarifications 2025-02-20 19:04:50 +01:00
Jose Gómez-Sellés
8d389a6628
docs/cloud: fix image routes in getstarted ()
It looks like the version deployed works differently than in local, and
some paths are not taken correctly. This commit fixes that by adding
absolute paths to the 2 images included in this section.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 17:34:56 +01:00
Jose Gómez-Sellés
eabe3783c4
docs: fix vmanomaly image broken links ()
It looks like when using html image insertion, the path is changed. This
commit fixes that by redirecting to the previous dir in path.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 16:46:48 +01:00
Jose Gómez-Sellés
de94879515
docs/cloud: rework quickstart guide ()
This guide was outdated. the intention of these changes is to:
* Ease maintainability
* Ease user experience
* Encourage the user to quickly set up the product

Some images are removed because they were obsolete or to improve
readability

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 16:27:40 +01:00
Aliaksandr Valialkin
531ecdfa0d
docs/VictoriaLogs/CHANGELOG.md: move the description of the change at ca1d1bc12b to the appropriate release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8315
2025-02-20 16:14:19 +01:00
Andrii Chubatiuk
dc94ccf857
docs: set proper code syntax for snippets in cluster docs ()
### Describe Your Changes

apply proper syntax for code blocks

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 14:10:50 +01:00
Roman Khavronenko
1d4713f989
metricsql: support auto-format (prettify) for expressions that use quoted metric or label names ()
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 10:25:55 +01:00
Yury Molodov
086ca47884
vmui: improve Explore Cardinality page ()
### Describe Your Changes

This PR includes improvements to the **Explore Cardinality** page:  

- Improve number formatting for better readability.
- Add full error message display.
- Move values representing changes relative to the previous day to a
separate column for easier sorting.
- Refine UI styles for a better visual experience.

<details>

<summary>Demo UI</summary>

| Before | After |
|--------|-------|
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/a188c94c-3a06-498d-994d-453080c436a7"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/9a42624c-2779-457f-9c98-920fca116eeb"
/> |
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/cb4973ba-d7a0-475b-a7a4-f649607dc7b5"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/b396516e-2468-4835-b300-88b7a82deec0"
/> |

</details>

Related issue: 


### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 09:51:49 +01:00
hagen1778
b5c21337d3
docs: fix typos in recent change lines
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 09:14:15 +01:00
f41gh7
855dfb324d
app/vmselect: add query resource limits priority
This commit adds support for overriding vmstorage `maxUniqueTimeseries` with specific
resource limits:
1. `-search.maxLabelsAPISeries` for
[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
[/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
2. `-search. maxSeries` for
[/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series)
3. `-search.maxTSDBStatusSeries` for
[/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats)
4. `-search.maxDeleteSeries` for
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)

Currently, this limit priority logic cannot be applied to flags
`-search.maxFederateSeries` and `-search.maxExportSeries`, because they
share the same RPC `search_v7` with the /api/v1/query and
/api/v1/query_range APIs, preventing vmstorage from identifying the
actual API of the request. To address that, we need to add additional
information to the protocol between vmstorage and vmselect, which should
be introduced in the future when possible.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7857
2025-02-19 18:18:32 +01:00
Yury Molodov
a5ca5d370a
vmui: add legend customization options
Legend settings have been added to **Graph Settings**.  

#### **New Features:**  
1. **Table View** – Toggle to display the legend in a table format.  
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.


Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
2025-02-19 18:09:14 +01:00
Gunju Kim
10d5e979a6
app/vmalert: add command line flag -notifier.sendTimeout
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.

This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.


Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
2025-02-19 18:04:31 +01:00
Andrii Chubatiuk
58a80a9718
app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:01:54 +01:00
Zakhar Bessarab
f4b7e2e811
app/vmselect/filtersdebug: properly validate retention configuration
This commit properly validate retention flag value input for the enterprise debug UI.
API properly checks:
- presence of global retention configuration
- validate that retention is at least 1 day, same as vmstorage enforcement

Related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8343

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 17:52:22 +01:00
Andrii Chubatiuk
e97c530fdc
lib/protoparser/influx: add -influx.forceStreamMode flag to force parsing all Influx data in stream mode ()
Addresses 

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 17:05:02 +01:00
Roman Khavronenko
439af7927f
docs: update histogram recommendation for stream aggr ()
Use `rate_sum` instead of `total` output for the following reasons:
* `rate_sum` is far less sensitive for data delays than `total`, since
it represents the speed of change instead of absolute values.
* `rate_sum` remove need in using `rate` function for calculating final
results on query time.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 15:12:55 +01:00
hagen1778
9049731766
docs: re-order items in VictoriaMetrics navigation tab
* make QuickStart first, similarly to vlogs docs
* push release guide to bottom, as it is not for users

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:59:09 +01:00
Aliaksandr Valialkin
8d79b28b37
deployment: update VictoriaLogs Docker image tag from v1.10.1-victorialogs to v1.11.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.11.0-victorialogs
2025-02-19 13:34:11 +01:00
hagen1778
f927b08298
docs: clarify changelog line
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:27:22 +01:00
hagen1778
1dac6fcdc4
docs: rm duplicated changelog line
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:22:53 +01:00
hagen1778
3afd956041
docs: fix another markdown typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:22:15 +01:00
hagen1778
c77306616e
docs: fix markdown typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:21:42 +01:00
Zakhar Bessarab
eccef5f6ec
docs/guides: remove unneeded IDs for security context ()
### Describe Your Changes

Based on feedback at:
e4240e61c6 (r1961497609)

Thanks to @vrutkovs for the feedback.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 13:20:16 +01:00
Andrii Chubatiuk
c8fc903669
lib/streamaggr: added aggregation windows ()
### Describe Your Changes

By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.

For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.

Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:

- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
  It allows enabling aggregation windows for a specific aggregator.



### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:19:33 +01:00
Aliaksandr Valialkin
77aec3fe77
docs/VictoriaLogs/CHANGELOG.md: cut v1.11.0-victorialogs 2025-02-19 12:46:25 +01:00
Yury Molodov
b2a99d7c6f
vmui: optimize group view ()
### Describe Your Changes

This PR introduces several enhancements and optimizations for the Group
view:

1. **Disable hover effect:**  
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.

2. **Limit entries per Group:**  
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.

3. **Display group info:**  
Include the group number in the title along with the total count of
groups to improve clarity and navigation.

4. **Performance improvement:**  
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.
  
Related issue:   


<details>
  <summary>Demo UI</summary>
  
<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>

</details>
  
### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-19 12:42:54 +01:00
Aliaksandr Valialkin
b58e2ab214
lib/bytesutil: drop ByteBuffer.B when its capacity is bigger than 64KB at Reset
There is little sense in keeping too big buffers - they just waste RAM and do not reduce
the load on GC too much. So it is better dropping such buffers at Reset instead of keeping them around.
2025-02-19 12:35:06 +01:00
Zakhar Bessarab
616fed47a0
docs/guides: add guide about collecting OpenShift logs ()
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 14:42:27 +04:00
Aliaksandr Valialkin
9bb5ba5d2f
lib/logstorage: make sure that the data for every log field is stored in a separate file until the number of files is smaller than 256
This should improve query performance for logs with hundreds of fields (aka wide events).
Previously there was a high chance that the data for multiple log fields is stored in the same file.
This could result in query performance slowdown and/or increased disk read IO,
since the operating system could read unnecessary data for the fields, which aren't used in the query.

Now log fields are guaranteed to be stored in separate files until the number of fields exceeds 256.
After that multiple log fields start sharing files.
2025-02-19 01:48:14 +01:00
Jose Gómez-Sellés
95a42717b6
docs: create get-started cloud section ()
In this first step, content is moved from cloud docs root to a new
folder: get started. This one will allocate: overview, key features and
benefits, quickstart and guides and best practices.

### Changes Description

- In order to reuse content, the overview and README sections are
merged.
- A dummy Get started section is created, but needs further work.
- Next steps will also include reworking the quick start section and
creating the 2 new docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-18 21:08:35 +01:00
martin-kokos
13fa70a03a
docs: update Quick-Start.md ()
Add missing sudo's to binary install snippets

### Describe Your Changes

Some of the commands the need to be done as root are missing invocation
via sudo. This commit adds missing sudo invocations.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-18 17:11:28 +01:00
Andrii Chubatiuk
5bdbafb9b6
docs: disable operator, helm docs sync ()
### Describe Your Changes

exclude operator and helm charts docs sync as these repos will be synced
with vmdocs directly

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-18 14:24:14 +02:00
Artem Fetishev
f13f04db10
deployment/docker: update Alpine Docker image from 3.21.2 to 3.21.3
See https://alpinelinux.org/posts/Alpine-3.18.12-3.19.7-3.20.6-3.21.3-released.html

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-02-18 10:34:07 +01:00
Artem Fetishev
68a1e19d09
Include in changelog
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-02-18 10:03:15 +01:00
hagen1778
eba53b5275
docs: follow-up after ba0d7dc2fc
* change doc lines length for consistency
* add version availability mark
* add changelog line

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-18 09:41:46 +01:00
Nikolay
12c39b660d
lib/httpserver: properly check basic authorization
Commit 68791f9ccc introduced regression.
It performed basicAuth check before built-in routes. It made impossible
to bypass basic authorization with `authKey` param.

This commit fixeds that issue and removes unneeded check. It also adds
integration tests for this case.

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-17 15:52:22 +01:00
Github Actions
a91109fc97
Automatic update helm docs from VictoriaMetrics/helm-charts@a4683a6 ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2025-02-17 15:51:10 +01:00
gabyf
aaafceba25
docs: fix little typo
fixing a little typo from this page:
https://docs.victoriametrics.com/vmagent/index.html#troubleshooting
2025-02-17 15:50:42 +01:00
Zhu Jiekun
ca1d1bc12b
app/vlinsert: properly ingest journald logs with single-character name entity
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to  `^[A-Z_][A-Z0-9_]*`.

 It's needed to properly support entities with single-character
names.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
2025-02-17 15:49:20 +01:00
Eric Daniels
22dbaee600
docs: typo fix in keyConcepts.md ()
### Describe Your Changes

Just an extremely nit-picky grammatical fix.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-16 11:02:48 +01:00
Aliaksandr Valialkin
9619a84af6
docs/VictoriaLogs/CHANGELOG.md: typo fix 2025-02-14 16:20:06 +01:00
Aliaksandr Valialkin
35f3570abf
deployment: update VictoriaLogs Docker image tag from v1.10.0-victorialogs to v1.10.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.10.1-victorialogs
2025-02-14 15:39:20 +01:00
Aliaksandr Valialkin
a9e8661a67
docs/VictoriaLogs/CHANGELOG.md: cut v1.10.1-victorialogs release 2025-02-14 15:35:37 +01:00
Aliaksandr Valialkin
f768c15bb2
lib/logstorage: make sure that chunkedAllocator is isn't used from concurrently running goroutines
This is needed in order to avoid data races
2025-02-14 15:34:08 +01:00
Aliaksandr Valialkin
95422e8f7a
lib/logstorage: ensure that statsProcessor.updateStatsForAllRows() is called on non-empty blockResult
This eliminates a class of potential bugs with incorrect stats calculations when an additional filter
is applied to the blockResult before passing it to the stats function, and this filter removes
all the rows from blockResult.
2025-02-14 15:34:07 +01:00
Aliaksandr Valialkin
40bde6bd28
lib/logstorage: properly initialize minValue and maxValue at pipeLenProcessorShard and pipeHashProcessorShard
Previously this could result in incorrect 0 result of min() stats function applied to the len() results.

This is a follow-up for eddeccfcfb
2025-02-14 15:34:07 +01:00
Artem Fetishev
ba0d7dc2fc
Allow disabling per-day index ()
### Describe Your Changes

Allow disabling the per-day index using the `-disablePerDayIndex` flag.
This should significantly improve the ingestion rate and decrease the
disk space usage for the use cases that assume small or no churn rate.
See the docs added to `docs/README.md` for details.

Both improvements are due to no data written to the per-day index.
Benchmark results:

```shell
rm -Rf ./lib/storage/Benchmark*; go test ./lib/storage -run=NONE -bench=BenchmarkStorageInsertWithAndWithoutPerDayIndex --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/perDayIndexes-12                 1        3850268120 ns/op                39.56 data-MiB          28.20 indexdb-MiB           259722 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/noPerDayIndexes-12               1        2916865725 ns/op                39.57 data-MiB          25.73 indexdb-MiB           342834 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/perDayIndexes-12                   1        2218073474 ns/op                 9.772 data-MiB         13.73 indexdb-MiB           450842 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/noPerDayIndexes-12                 1        1295140898 ns/op                 9.771 data-MiB          0.3566 indexdb-MiB         772119 rows/s
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  11.421s
```

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-02-14 12:35:51 +01:00
Aliaksandr Valialkin
6d7f82ddb6
docs/VictoriaLogs/data-ingestion: remove configuration examples for ingesting logs into VictoriaLogs via Loki JSON protocol
There are many subtle issues while ingesting logs from popular log shippers into VictoriaLogs via Loki JSON protocol.
For example, the common issue is that structured logs are ingested as a JSON string at _msg field.
This is not what most users expect - they expect that fields in structured logs are ingested as separate log fields.
It is better removing examples with configs for Loki JSON protocol, since other supported protocols work much better
without any issues. This should reduce the confusion level for new users, who try Loki protocol and hit its issues.

- Elasticsearch ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api )
- JSON stream ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api )
2025-02-13 18:16:51 +01:00
Aliaksandr Valialkin
98241cfa78
docs/VictoriaLogs/README.md: mention high performance and resource efficiency points to the top of VictoriaLogs features 2025-02-13 15:55:17 +01:00
Maxim Evtush
7b0583eb5f
docs: fix typo in gateway doc () 2025-02-13 11:07:37 +01:00
Aliaksandr Valialkin
815ff805e6
docs/VictoriaLogs/LogsQL.md: add a chapter about subquery filters 2025-02-13 09:44:25 +01:00
Aliaksandr Valialkin
2234fa4211
docs/VictoriaLogs: refer to v1.10.0-victorialogs instead of v1.9.1-victorialogs
This is a follow-up for d755ad68c4, where these references weren't updated by an accident.
2025-02-12 22:53:54 +01:00
Aliaksandr Valialkin
d755ad68c4
deployment: update VictoriaLogs Docker image tag from v1.9.1-victorialogs to v1.10.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.10.0-victorialogs
2025-02-12 22:37:19 +01:00
Aliaksandr Valialkin
67d375261b
docs/VictoriaLogs/CHANGELOG.md: cut v1.10.0-victorialogs release 2025-02-12 22:15:32 +01:00
Aliaksandr Valialkin
5fdf6df804
app/vlinsert: add a link to the pull request at systemd repository, which enables compression support
This should simplify maintenance of this code in the future.
While at it, clarify the change at the docs/VictoriaLogs/CHANGELOG.md.

This is a follow-up commit for 3c9f9f49b0.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8264
Updates https://github.com/systemd/systemd/pull/34822
2025-02-12 22:07:35 +01:00
Aliaksandr Valialkin
c75cce1384
lib/logstorage: refactor bucketing code
1. Use distinct code paths for blockResult.getValues() and blockResult.getValuesBucketed().
   This should simplify debugging and maintenance of the resulting code.

2. Do not load column values if all the values in the block fit the same bucket.
   Use blockResultColumn.minValue and blockResultColumn.maxValue for determining whether
   column values must be loaded via blockResultColumn.getValuesEncoded().
   This signiciantly improves performance for big buckets, which cover all the column
   values in a block.

3. Properly calculate buckets for negative values.

4. Properly adjust weekly buckets by Monday.
2025-02-12 21:46:12 +01:00
Yury Molodov
778bcea30d
vmui/logs: fix autocomplete ignoring selected tenant ()
### Describe Your Changes

Fix autocomplete not passing `AccountID` and `ProjectID` headers when
fetching suggestions in VictoriaLogs UI.

Related: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-12 12:26:42 +01:00
Github Actions
d4632aceac
Automatic update helm docs from VictoriaMetrics/helm-charts@ab4b3d4 ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2025-02-12 10:48:37 +01:00
Andrii Chubatiuk
3a27073634
app/vlinsert: add OpenTelemetry ingested logs trace_id and span_id
This commit parses additional optional fields from OpenTelemetry logs protocol.

Related issue: 

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8255
2025-02-12 10:47:38 +01:00
Andrii Chubatiuk
3c9f9f49b0
app/vlinsert: add journald content negotiation, which enables compression on a client
Adding Accept-Encoding response header to support content negotiation,
which was introduced in [this
PR](https://github.com/systemd/systemd/pull/34822) and enables
compression on journald
2025-02-12 10:45:40 +01:00
Phuong Le
adf5d46f8c
lib/protoparser/clusternative: properly return error from parsing callback
Previously the `callbackErr` is silently ignored in clusternative parser, which is used at vminsert for parsing clusterNative requests and at vmstorage for parsing vminsert requests.

 This commit  fixes that by properly return callbackError after reading all block metrics. This aligns
with other parsers in `lib/protoparser`.
2025-02-12 10:33:10 +01:00
Yury Molodov
71160e64b1
vmui: update default step calculation for instant queries ()
### Describe Your Changes

Updated default `step` calculation for `Table` and `JSON` views.

- When switching to the `Table` or `JSON` view, the default step is now
automatically set to `end - start`, where `end` and `start` correspond
to the selected time range.
- If the user has manually set a step value, it will remain unchanged
when switching views or changing the time range.
- The UI now explicitly indicates when the step is automatically
calculated.

<details> <summary>Demo</summary>


https://github.com/user-attachments/assets/2540de24-36ed-4764-a047-1c6b48a80ed4

</details>

**Note:** These views use the
[`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query)
endpoint for instant queries.

Related issue: 


### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-11 22:55:17 +01:00
hagen1778
5f9257a6d2
docs: add year 2025 changelog
The 2025 changelog was in the parent directory - a default page
that opens for /changelog.
But it seems like it was confusing for users, so add 2025 that mirrors
/changelog page.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-11 22:39:57 +01:00
hagen1778
7f5d41dda7
docs: fix broken changelog link
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-11 22:28:07 +01:00
hagen1778
73da21c1ea
docs: separate security and feature changes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-11 22:26:52 +01:00
Phuong Le
23147c8339
docs: search.lookback-delta -> query.lookback-delta () 2025-02-11 22:15:39 +01:00
Evgeny
4e9fb93acc
fix race where the same list is used from 2 goroutines
### Describe Your Changes

There is an issue described in  this should fix it
- The alerts slice is shared across multiple goroutines (since send() is
called concurrently).
- `alerts[:0]` creates a new slice header, but it still references the
same underlying array.
- Appending (append(alertsToSend, a)) modifies the underlying array,
which may also be used by another goroutine.
Solution: Use a separate slice copy for each goroutine.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Evgeny Kuzin <evgeny@hudson-trading.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
2025-02-10 21:40:00 +04:00
Zakhar Bessarab
04943907e3
{docs,deployment}: update references to the latest release ()
### Describe Your Changes

- Update references to the latest release - v1.110.1
- Update links to the latest LTS releases

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-10 21:14:49 +04:00
Aliaksandr Valialkin
d142d43ae9
deployment/docker: update VictoriaLogs image from v1.9.0-victorialogs to v1.9.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.9.1-victorialogs
2025-02-10 17:35:37 +01:00
Aliaksandr Valialkin
4600724985
app/vlselect/vmui: run make vmui-logs-update after 6a8fa799c6
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8177
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7778
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8178
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8176
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8174

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8153
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8152
2025-02-10 17:30:24 +01:00
Zakhar Bessarab
6ed31b9180
docs/changelog: mention LTS releases, update release date
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-10 20:23:02 +04:00
Aliaksandr Valialkin
00fb218269
deployment/docker: update VictoriaLogs docker image from v1.8.0-victorialogs to v1.9.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.9.0-victorialogs
2025-02-10 15:29:52 +01:00
Aliaksandr Valialkin
aa0ac1d0ed
docs/VictoriaLogs/CHANGELOG.md: cut v1.9.0-victorialogs release 2025-02-10 15:21:56 +01:00
Aliaksandr Valialkin
81d359507d
lib/logstorage: properly compare RFC3339 timestamps with sub-second precision in lessString()
Previously RFC3339 timestamps with sub-second precision could be incorrectly compared by lessString().
For example, 2025-01-20T10:20:30.1Z was incorrectly treated as smaller than 2025-01-20T10:20:30.09Z,
because the first timestamp has smaller decimal number after the last dot than the second timestamp.
2025-02-10 15:00:59 +01:00
Aliaksandr Valialkin
ebac07bcf6
app/vlinsert: continue parsing JSON lines in the input stream after parse errors
Previosly the parsing of the input stream was stopped after the first parse error.
This isn't what most users expect when ingesting JSON lines in a stream where some JSON lines may be invalid.
2025-02-10 15:00:58 +01:00
Github Actions
92bfb7ba15
Automatic update helm docs from VictoriaMetrics/helm-charts@0fe31fd ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: zekker6 <1367798+zekker6@users.noreply.github.com>
2025-02-10 13:22:42 +01:00
Aliaksandr Valialkin
5e7866c3a0
deployment/docker: update Go builder from Go1.23.5 to Go1.23.6
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.6+label%3ACherryPickApproved
2025-02-09 22:50:04 +01:00
Aliaksandr Valialkin
c26cbf57dd
app/vlinsert: accept timestamps with microsecond and nanosecond precision at _time field 2025-02-09 22:41:38 +01:00
Zakhar Bessarab
ee8f852a83
docs: cut v1.111.0
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-07 19:42:11 +04:00
Zakhar Bessarab
80a5f6863b
docs: update refs to new release
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-07 18:53:01 +04:00
Aliaksandr Valialkin
48602a1ae8
lib/logstorage: optimize performance for stats, top and uniq pipes a bit
Split unique values (groups) into shards according to the configured concurrency
during processing of the matching rows if the number of unique values exceeds the hardcoded threshold.
Previously this splitting was performed unconditionally at the merge stage when merging independently
calculated per-CPU states into a single state. It is faster to perform the split during rows processing
if the number of unique values is big.

This gives up to 30% perfromance improvements when these pipes are applied to big number of unique values (groups).
2025-02-06 13:46:32 +01:00
Roman Khavronenko
0a6f6d0bba
app/vmselect/netstorage: stop exposing `vm_index_search_duration_seconds metric
This metric records time spent on search operations in the index.
It was introduced in
[v1.56.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.56.0).
However, this metric was used neither in dashboards nor in alerting
rules.
It also has high cardinality because index search operations latency can
differ by 3 orders of magnitude.

See
[example](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality?date=2025-02-05&match=vm_index_search_duration_seconds_bucket&topN=10&focusLabel=).

 Hence, dropping it as unused.

---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-06 13:40:52 +01:00
Aliaksandr Valialkin
171d4019cd
lib/logstorage: properly limit the number of concurrent workers at stats, top and uniq pipes according to the provided options(concurrency=N)
The number of worker shards per each pipe processor is created during query initialization.
This number equals to the `options(concurrency=N)` if this option is set or to the number of available CPU cores.
This means that all the pipes must adhere the given concurrency when passing data blocks
to the next pipe.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8201

The bug has been introduced in 0214aa328e
2025-02-06 09:16:56 +01:00
Github Actions
54f60d804a
Automatic update operator docs from VictoriaMetrics/operator@96519dc ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2025-02-05 22:48:03 +01:00
Nikolay
78dc9533fc
app/vmauth: allow to serve internal API and different address
vmauth uses 'lib/httpserver' for serving HTTP requests. This server
unconditionally defines built-in routes (such as '/metrics',
'/health', etc). It makes impossible to proxy `HTTP` requests to  backends with the same routes.
Since vmauth's httpserver matches built-in route and return local
response.

 This commit adds new flag `httpInternalListenAddr` with
default empty value. Which removes internal API routes from public
router and exposes it at separate http server.

For example given configuration disables private routes at `0.0.0.0:8427` address and serves it at `0.0.0.0:8426`:

`./bin/vmauth --auth.config=config.yaml --httpListenAddr=:8427 --httpInternalListenAddr=127.0.0.1:8426`

Related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6468
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345
2025-02-05 17:10:11 +01:00
Joost Buskermolen
088e06c2c2
app/vmselect: expose /-/healthy and /-/ready endpoints on full Prometheus path
This commit improves integration with third-party solutions who rely on non-root
endpoints (i.e. MinIO) when the vmselect path has been specified in the
configured Prometheus URL like:
`http://vmselect.:8481/select/0/prometheus`

Comparable change has been done before
(b885a3b6e9), however only takes care of
the root path. This means endpoints `-/healthy` and `-/ready` are still
not available on full vmselect Prometheus paths, resulting in
unsupported path requests.

This change makes these endpoints available on the full paths like:
`/select/0/prometheus/-/healthy` and `/select/0/prometheus/-/ready`,
thus achieving full Prometheus compatibility for external dependencies.

Related issues:
- https://github.com/minio/console/issues/2829
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833

---

Signed-off-by: Joost Buskermolen <j.buskermolen@cloudmeesters.com>
2025-02-05 17:00:52 +01:00
hagen1778
553e95e293
docs: fix copy&paste typo in change type
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-05 15:29:58 +01:00
Github Actions
46fbb95b16
Automatic update operator docs from VictoriaMetrics/operator@8192989 ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2025-02-05 15:24:10 +01:00
Github Actions
fbe592f495
Automatic update helm docs from VictoriaMetrics/helm-charts@0155ba2 ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2025-02-05 15:23:10 +01:00
Artem Fetishev
631b736bc2
lib/storage: fix cardinality limiting for cases when insertion takes fast path ()
### Describe Your Changes

The cardinality limiter in this case does not receive the actual
metricID but some other value found in r.TSID.MetricID and is not
initialized. Depending on the system and/or go runtime implementation,
this value can be 0 or some garbage value (which shouldn't have too wide
a range). Thus, there basically no limit for inserted metricIDs.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-05 15:22:34 +01:00
Fred Navruzov
ceefa406cc
docs/vmanomaly: fix 404 links ()
### Describe Your Changes

- fixed recently found 404 under `/anomaly-detection/` docs section
- get rid of remaining .html / .index.html endings in links under
`/anomaly-detection/` docs section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-04 20:37:46 +02:00
Zakhar Bessarab
2748681f40
{docs,app/vmctl}: clarify how vm-rate-limit is applied
### Describe Your Changes

Explicitly mention that `--vm-rate-limit` is used for each individual
worker defined by `--vm-concurrency`.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-04 18:22:54 +04:00
Zakhar Bessarab
e2cfd351c0
docs/streaming-aggregation: fix typo in metric name
### Describe Your Changes

Fix metric name labels set not being "closed" by `}`.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-04 13:09:11 +04:00
f41gh7
a2b49a1843
app/vmagent/kafka: properly close producer client
At v1.107.0 release with commit 21a7f06fc4beeb6ad32b9f7fd88704ed33674905 was introduced regression.

It prevents kafka producer client from graceful shutdown. This change also removed kafka message headers.

 This commit properly closes client.

 It also addresses the following issues:

* properly add headers to kafka message
* add url_id to the exported metrics, which prevents possible metrics clash if multiple remote write targets have
 the same url and topic name.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-03 14:21:52 +01:00
hagen1778
a70e8e0379
docs: mention tlast_over_time in influx guide
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8188

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-01 23:17:24 +01:00
hagen1778
894a2f91e6
docs: fix typo in image name
Follow-up after 35d77a3bed

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-01 23:14:08 +01:00
Roman Khavronenko
fa2107bbec
metricsql: bump to v0.83.0 ()
metricsql: bump to v0.83.0

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703

The update also returns an error if metric name is specified twice in
metrics selector.
For example, `foo{__name__="bar"}` is not allowed anymore. It would
successfully parse before
this change, but it won't satisfy the search filter any way. So it had
no sense in supporting this. This is why some test cases were removed.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-01 22:22:29 +01:00
hagen1778
5561970db0
app/vmalert: mention that remoteWrite.concurrency depends on CPU
Mnetion explicitly that `remoteWrite.concurrency` deopends on number
of available CPU cores. Updated docs to rm auto-printed default value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8151

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-01 22:10:47 +01:00
Github Actions
4d42e29967
Automatic update operator docs from VictoriaMetrics/operator@7930b3a ()
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2025-01-31 16:46:45 +01:00
Roman Khavronenko
72837919ae
app/vmselect/promql: fix discrepancies when using or binary operator
The change covers various corner cases when using `or` binary operator.
See corresponding issues and pull request here to see the cases:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7770

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7759
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7640


---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-31 14:14:17 +01:00
Zakhar Bessarab
6166149474
docs/release-guide: add a note for vmui build ()
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-01-31 16:57:54 +04:00
Github Actions
f34e82eb9b
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@de3daa7 () 2025-01-31 16:57:30 +04:00