Commit graph

3705 commits

Author SHA1 Message Date
f41gh7
0afa51a156
fixes test after 1572e1e5c3 2025-04-24 23:32:37 +03:00
Artur Minchukou
fdfa68bc2d
app/vmui/logs: fix zoom behavior on VictoriaLogs chart
Fix zoom behavior on VictoriaLogs chart. The problem was that after
zooming, the chart was recreated and the cursor position on the graph
was lost, so further zooming occurred relative to the zero position. So
to fix this problem:
- removed unnecessary dependencies from useEffect to avoid recreating
the chart;
- moved the chart with the legend to a separate component to optimize
the code when the graph is hidden;
- also this PR contains changes from [this
PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8718), which
preserve zoom selection on autorefresh

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8557
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8558
2025-04-24 20:25:48 +03:00
Zhu Jiekun
1572e1e5c3
app/vmagent: add consistent hashing for the remote write sharding
This commit adds the following changes:
* use consistent hashing  for the remote write sharding.
* properly count metric of remote write samples drop rate  when `shardByURL` was
enabled.

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8546
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8702
2025-04-24 20:22:59 +03:00
Aliaksandr Valialkin
55d75b911c
app/vlstorage: add /internal/force_flush endpoint for flusing all the recently ingested logs to searchable parts
This endpoint is needed for integration tests, which need querying the recently ingested logs.
2025-04-24 18:17:30 +02:00
Aliaksandr Valialkin
f9e312f35a
app/vlselect/logsql: allow passing multiple extra_filters and extra_stream_filters query args to all the querying APIs
This should help the proper implementation of extra filters in VictoriaLogs Plugin for Grafana and
in VictoriaLogs web UI. See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/293#issuecomment-2827285583
2025-04-24 17:55:33 +02:00
Yury Molodov
1c39853164
vmui: show read usage in Cardinality Explorer ()
### Describe Your Changes

Two new columns were added to the **Cardinality Explorer** table:

`Requests count`
- Displays how many times a metric was queried since stats collection
began
- Highlighted **red** when the value is `0` or missing

`Last request`
- Shows when the metric was last queried
- Highlighted:
  - **Red** if older than **30 days**
  - **Yellow** if between **7 and 30 days**
- Shows exact timestamp on hover in `YYYY-MM-DD HH:mm:ss` format

Related issue: 


![image](https://github.com/user-attachments/assets/fcf6f33c-e2d0-45c0-9f20-177cf6fa9dde)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-24 14:07:29 +02:00
Andrii Chubatiuk
7390f99897
app/vmalert: fixed rule group ID in UI ()
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
2025-04-24 13:46:42 +02:00
Yury Molodov
899d70a93e
vmui: fix /flags API requests to respect custom URL prefixes ()
### Describe Your Changes

Fixes incorrect /flags request in vmui when using URL prefixes.
 
 Related issue:  

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-24 11:33:43 +02:00
Yury Molodov
1d13718fcf
vmui/logs: fix oversized Query History button on mobile ()
### Describe Your Changes

Fixes oversized `Query History` button on mobile in logs UI.
Related issue:  

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-24 11:25:17 +02:00
Yury Molodov
fa4708178f
vmui: hide duplicate legend items in Raw Query ()
### Describe Your Changes

Hide identical series in the legend on the Raw Query page when
deduplication is disabled.

Related issue: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-24 11:20:41 +02:00
Yury Molodov
a9f03605d3
vmui/logs: handle ANSI escape sequences ()
### Describe Your Changes

1. Added a toggle to enable parsing of ANSI escape sequences (e.g.,
color codes).
    Related issue:  
2. Fixed the display of raw fields when the `_msg` field is missing. 
    Related issue: 
3. Fixed log display in Group view when Markdown parsing is enabled. 
    Related issue: 
4. Moved the Markdown parsing toggle to **Group View Settings**.
5. Configured testing setup.

<details>
<summary>Demo UI:</summary>

<br/>

**Disabled ANSI parsing:**

![image](https://github.com/user-attachments/assets/7f466051-ea6c-4ff7-b386-a3df4bb503c9)

**Enabled ANSI parsing:**

![image](https://github.com/user-attachments/assets/04074b6e-b1c2-4d5e-a8c2-5f58ca88494b)


</details>


### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-23 15:25:07 +02:00
Hui Wang
7756046865
vmalert: correctly update the debug param for recording rule when u… ()
…pdating the rule group

Refactor the `TestUpdateWith` a bit in preparation for
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8658
2025-04-23 14:32:07 +02:00
Zakhar Bessarab
f224b60410
fix: compatibility for FIPS builds
### Describe Your Changes

Fixes which are required in order to build FIPS-compliant binaries.
These changes were originally added for enterprise version and synced to
opensource for consistency and easier maintenance.

- consistently use `hash/fnv` at `app/vmalert` when calculating
checksums. Usage of md5 is not allowed in FIPS mode.
- increase encryption keys size used in testing in order to allow tests
to successfully run in FIPS mode

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-23 09:43:07 +04:00
Aliaksandr Valialkin
84d2922f5e
all: consistently use Grafana dashboard links ending with dashboard ID
The suffix after the ID may change in dashboard settings.
If it changes, the link becomes broken (dubious decision at grafana.com/grafana/dashboards/ ).
That's why it is better to drop all the suffixes and use links for Grafana dashbooards ending with IDs.
In this case they are automatically redirected to the url with the proper suffix.

This is a follow-up for 3e4c38c56c

See also the previous commits 9c0863babc and 0a5ffb3bc1
2025-04-22 14:20:15 +02:00
Aliaksandr Valialkin
54bb834c16
app/vlselect/vmui: run make vmui-logs-update after faf95b4a58 2025-04-22 13:53:40 +02:00
Roman Khavronenko
787e6a9b24
app/vmalert/remotewrite: rm noisy shutdown logs
On shutdown, rw client printed two log messages about shutting down and
how many series it flushed on shut down.

This commit removes these log messages for the following reasons:
1. These messages were printed for each `client.cc`, which is set to x2
of available CPUs. This behavior generated a lot of useless logs on
shutdown.
2. Number of flushed series on shutdown doesn't matter to the user.

Instead, it now prints only two log messages: when shutdown starts and
when it ends:
```
^C2025-04-22T07:37:11.932Z      info    VictoriaMetrics/app/vmalert/main.go:189 service received signal interrupt
2025-04-22T07:37:11.932Z        info    VictoriaMetrics/app/vmalert/remotewrite/client.go:152   shutting down remote write client: flushing remained series
2025-04-22T07:37:11.933Z        info    VictoriaMetrics/app/vmalert/remotewrite/client.go:155   shutting down remote write client: finished in 210.917µs
```

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-22 11:50:23 +03:00
f41gh7
ec6ed3cb48
lib/storage/metricnamestats: allow regex for match_pattern
This commit allows regex syntax for match_pattern query param.
It improves API usability.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
2025-04-22 11:35:23 +03:00
Yury Molodov
faf95b4a58
vmui/logs: update button styles to improve hover performance
Refactors button styles to fix high CPU usage on hover in the logs UI.

 Related issue: #
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8135
2025-04-22 11:28:19 +03:00
Artur Minchukou
a7106040de
app/vmui/logs: add 0 label to the Y axis
Added 0 label to the Y axis on VictoriaLogs chart.


Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8409
2025-04-22 11:26:34 +03:00
Phuong Le
792d0f8bb8
vmagent: reduce log noise from remote write retries
Throttle remote write retry warning logs to one message per 5 seconds.
This reduces log noise during connectivity issues.

Users can monitor `vmagent_remotewrite_retries_count_total` and
`vmagent_remotewrite_errors_total` metrics for detailed retry rates per
destination if needed. This change prioritizes reducing log volume over
immediate destination-specific error logging in the retry warnings.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8723
2025-04-22 11:20:33 +03:00
Georgy Torquemada
89237a04b9
fix: add missed severity levels (warn) for protobuff parser
Closes
[8647](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8647)

### Describe Your Changes

Added missed OTEL severities levels, added test for severity, fix some
severity in given tests

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-21 16:39:10 +04:00
Aliaksandr Valialkin
024a40a799
app/{vmselect,vlselect} run make vmui-update vmui-logs-update after 5fa40ee631 2025-04-17 20:08:28 +02:00
Yury Molodov
5fa40ee631
vmui/logs: fix incorrect table sorting for numeric ()
### Describe Your Changes

Fixed table sorting logic and added unit tests for descendingComparator.
Values are now correctly sorted by type: number, date, or string.

Related issue: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:40:46 +02:00
Artur Minchukou
53814b1ea6
app/vmui: add query history to VictoriaLogs ()
### Describe Your Changes
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8500

Added query history to VictoriaLogs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:34:15 +02:00
Artur Minchukou
9ca2a246a9
app/vmui: fix mobile layout on the VictoriaLogs page, fix width of groups and table settings modal ()
### Describe Your Changes
 - Fix mobile layout on the VictoriaLogs page
<img width="361" alt="image"
src="https://github.com/user-attachments/assets/2e09b999-34d5-4059-ba09-95a21b3e8ab3"
/>
<img width="353" alt="image"
src="https://github.com/user-attachments/assets/b9048d41-5972-4290-988f-e9b0a0472b99"
/>
 
- Fix width of groups settings modal
<img width="372" alt="image"
src="https://github.com/user-attachments/assets/e1cb1902-dc93-4bfd-8b32-eaf0d8e54253"
/>
<img width="352" alt="image"
src="https://github.com/user-attachments/assets/a7c7000f-2c4a-41d9-a3c1-a515fd077b9b"
/>

- Fix width of table settings modal
<img width="361" alt="image"
src="https://github.com/user-attachments/assets/12921936-6824-47e9-aff8-d0bb4de2e7f7"
/>
<img width="352" alt="image"
src="https://github.com/user-attachments/assets/77c857ee-85f4-4992-8113-4e252b40f1a6"
/>

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:27:10 +02:00
Artur Minchukou
4517425da8
app/vmui: add export logs button to VictoriaLogs ()
### Describe Your Changes
Related issue:
[](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8604)

Added a download logs button to VictoriaLogs, which allows you to export
logs in the following formats: csv, json.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:14:41 +02:00
Yury Molodov
953ed46680
vmui: update package.json
### Describe Your Changes

Bumped project dependencies in `package.json` to the latest stable
versions.
2025-04-16 20:19:17 +02:00
f41gh7
795d3fe722
lib/storage: enhance TSDB status response
This commit adds new fields - `requestsCount` and `lastRequestTimestamp`
to series count be metric names stats.
It allows to display an additional stats at explore cardinality page.
Stats will only be added if `storage.trackMetricNameStats` flag is set.

 This change requires an update to RPC protocol in order to properly
marshal data.

 In addition, this commit adds integration tests to TSDB stats API.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
2025-04-16 20:09:35 +02:00
f41gh7
536b40c06c
make linter happy after a113516588 2025-04-16 19:55:54 +02:00
Roman Khavronenko
a113516588
app/vmselect: log search query request stats
This commit adds `search.logSlowQueryStats=<duration>` cmd-line flag on vmselect.
It reads stats from eval function, and doesn't slow down the query execution.

 Log line has the following structure:

 vm_slow_query_stats type=%s query=%q query_hash=%d start_ms=%d end_ms=%d step_ms=%d range_ms=%d tenant=%q execution_duration_ms=%v series_fetched=%d samples_fetched=%d bytes=%d memory_estimated_bytes=%d

 This feature is only available for enterprise version.
2025-04-16 19:39:13 +02:00
Max Kotliar
477635e00f
Follow up to "vmagent/client: Use VictoriaMetrics remote write protocol by default, downgrade to Prometheus if needed" ()
### Describe Your Changes

Follow-up to
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462

Addressed review comments:
- Log panic with FATAL prefix to indicate possible on-disk data
corruption
- Moved version bump line to the tip block (v1.114.0 has already been
released) in changelog
- Removed duplicate vmagent entry from targets list from Makefile


### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-04-14 11:55:01 +03:00
Max Kotliar
2ab53acce4
vmagent/client: Use VictoriaMetrics remote write protocol by default, downgrade to Prometheus if needed ()
### Describe Your Changes

This commit improves how vmagent selects the remote write protocol.
Previously, vmagent [performed a handshake
probe](0ff1a3b154/lib/protoparser/protoparserutil/vmproto_handshake.go (L11))
at
[startup](0ff1a3b154/app/vmagent/remotewrite/client.go (L173)):

- If the probe succeeded, it used the VictoriaMetrics (VM) protocol.

- If the probe failed, it downgraded to the Prometheus protocol.

- No protocol changes occurred after the initial probe at runtime.

However, this approach had limitations:

- If vmstorage was unavailable during vmagent startup, vmagent would
immediately downgrade to the Prometheus protocol, leading to higher
network usage unitl vmagent restarted. This case has been reported in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615.

- If the remote write server was updated or downgraded (e.g., during a
fallback or migration), vmagent would not detect the protocol change. It
would continue retrying failed requests and eventually drop them.
Require a restart of vmagent to pick up the new protocol.

This commit introduces a more adaptive mechanism.
vmagent always starts with the VM protocol and downgrades to the
Prometheus protocol only if an unsupported media type or bad request
response is received.
When this happens, the protocol is downgraded for all future requests.
In-flight requests are re-packed from Zstd to Snappy and retried
immediately.
Snappy-encoded requests are dropped if an unsupported media type or bad
request is received (no retrying).

Additionally, the in-memory and persisted queues could mix snappy and
zstd encoded blocks. The proper encoding is decided before sending by
encoding.IsZstd function.

TODO:
* [x] Add tests
* [x] Update documentation
* [x] Changelog
* [x] Research on
[content-type](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054),
[accept-encoding](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786923382)

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615#top
issue.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-12 10:39:47 +03:00
Aliaksandr Valialkin
7a46af3920
victorialogs: add cluster mode
Cluster mode is enabled when -storageNode command-line flag is passed to VictoriaLogs.
In this mode it spreads the ingested logs among storage nodes specified in the -storageNode flag.
It also queries storage nodes during `select` queries.

Cluster mode allows building multi-level cluster setup when top-level select node can query multiple lower-level clusters
and get global querying view.

See https://docs.victoriametrics.com/victorialogs/cluster/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5077
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7950
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8223
2025-04-10 16:55:23 +02:00
Zakhar Bessarab
582160f566
make: fix make package for vmalert-tool
`make package` relies on presence of `APP_NAME/deployment/Dockerfile`
which was missing for vmalert-tool.
2025-04-08 16:13:28 +02:00
Max Kotliar
3f5bf4bd03
vmagent/remotewrite: set content encoding header based on actual body
Improve remote write handling in vmagent by setting the
`Content-Encoding` header based on the actual request body, rather than
relying on configuration.

- Detects Zstd compression via the Zstd magic number.
- Falls back to Snappy if Zstd is not detected.
- Persistent queue may now contain mixed-encoding content.
- Add basic vmagent integration tests

Follow up on
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5344 and
12cd32fd75.

Extracted from
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301
2025-04-08 16:12:06 +02:00
f41gh7
aa5edbc706
make vmui-update 2025-04-04 14:20:37 +02:00
Zakhar Bessarab
aff1580a1d
app/vmauth: return non-OK response for timeouts and request cancellation
Currently, requests failing due to network timeout would receive "200
OK" while producing a warning log message about the timeout. This
behaviour is confusing and might produce unexpected issues as it is not
possible to retry errors properly.

Change this to return "502 Bad Gateway" response so that error can be
handled by the client.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8621

Config for testing:
```
unauthorized_user:
  url_prefix: "http://example.com:9800"
```

Before the change:
```
*   Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
/* NOTE: 30 seconds timeout passes */
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 08:54:05 GMT
Date: Tue, 01 Apr 2025 08:54:05 GMT
<

* Connection  to host 127.0.0.1 left intact
```

After:
```
*   Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 09:13:57 GMT
Date: Tue, 01 Apr 2025 09:13:57 GMT
< Content-Length: 109
Content-Length: 109
<

* Connection  to host 127.0.0.1 left intact
```

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-03 13:44:51 +04:00
Emre Yazıcı
a9736a5bfb
app/vmalert: show partial responses in debug logs ()
### Describe Your Changes

Log when the data response from vmselect is partial during
rule(recording, alertingrule) evaluations.

vmselect returns `isPartial: true` in case data is not fully fetched
from scattered vmstorages. At the time of rule evals, it may be drifting
apart from real values due to missing points. This is an important event
that should be logged to inform users to see how often that happens as
it may lead to false positive alerts.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: emreya <emre.yazici@adyen.com>
Signed-off-by: emreya <e.yazici1990@gmail.com>
Signed-off-by: Emre Yazici <e.yazici1990@gmail.com>
(cherry picked from commit 56f60e8be9)
2025-04-03 09:47:34 +01:00
Aliaksandr Valialkin
635c5c9feb
app/vlselect: run /select/logsql/tail queries without concurrency limit
The concurrency limit is intended for short-running queries. If it is applied to tail queries,
then this can affect short-running queries.
2025-04-02 20:22:27 +02:00
Aliaksandr Valialkin
4e1260e189
app/vlselect: do not log canceled requests, since they are expected and legal 2025-04-02 19:14:44 +02:00
Aliaksandr Valialkin
ca3910748f
deployment: update Go builder from Go1.24.1 to Go1.24.2
See https://github.com/golang/go/issues?q=milestone%3AGo1.24.2+label%3ACherryPickApproved
2025-04-02 18:01:21 +02:00
Artem Fetishev
2f0796ff40
lib/storage: When creating and listing snapshots, panic instead of returning an error ()
When creating and listing snapshots, panic instead of returning an error
since errors are not recoverable anyway.
Also do not cleanup the filesystem on panic. Leave as is for further
manual inspection.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-02 15:47:23 +02:00
Aliaksandr Valialkin
f18daaeac5
app/vmui: replace old-style links to https://docs.victoriametrics.com/MetricsQL.html with https://docs.victoriametrics.com/metricsql/
Replace also https://docs.victoriametrics.com/keyConcepts.html with https://docs.victoriametrics.com/keyconcepts/

This is the follow-up for the commit ee1da35071
2025-04-02 13:22:58 +02:00
Aliaksandr Valialkin
bf024d3dce
app/vmagent: increase the default GOGC from 30 to 50
This reduces CPU usage by up to 30% in exchange of the increased RAM usage by 10%
when scraping thousands of targets, which expose millions of metrics in summary.

This looks like a good tradeoff after the commit edac875179 ,
which reduced RAM usage by more than 10%, so the final RAM usage for vmagent
is still lower than the RAM usage at v1.114.0 by ~15%, while CPU usage drops by 30%.
2025-04-01 21:04:28 +02:00
Max Kotliar
b1d1f1f461
vmagent/remotewrite: fix golangci-lint code style issue
### Describe Your Changes

Fixes golangci-lint issues introduced in
98f1e32e39

```
--- a/app/vmagent/remotewrite/pendingseries.go
+++ b/app/vmagent/remotewrite/pendingseries.go
@@ -202,7 +202,7 @@ func (wr *writeRequest) copyTimeSeries(dst, src *prompbmarshal.TimeSeries) {

 	// Pre-allocate memory for labels.
 	labelsLen := len(wr.labels)
-	wr.labels = slicesutil.SetLength(wr.labels, labelsLen + len(labelsSrc))
+	wr.labels = slicesutil.SetLength(wr.labels, labelsLen+len(labelsSrc))
 	labelsDst := wr.labels[labelsLen:]

 	// Pre-allocate memory for byte slice needed for storing label names and values.
@@ -212,7 +212,7 @@ func (wr *writeRequest) copyTimeSeries(dst, src *prompbmarshal.TimeSeries) {
 		neededBufLen += len(label.Name) + len(label.Value)
 	}
 	bufLen := len(wr.buf)
-	wr.buf = slicesutil.SetLength(wr.buf, bufLen + neededBufLen)
+	wr.buf = slicesutil.SetLength(wr.buf, bufLen+neededBufLen) buf := wr.buf[:bufLen]

 	// Copy labels

```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-01 18:37:02 +04:00
Aliaksandr Valialkin
98f1e32e39
app/vmagent/remotewrite: optimize writeRequest.copyTimeSeries a bit
Pre-allocate memory for labels and for the needed byte buffer used
for holding the copied label names and values.
2025-04-01 15:58:04 +02:00
Phuong Le
346db8a606
vmui: fix auto-suggestion doesn't work inside functions ()
Fixes 

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-31 16:22:25 +02:00
hagen1778
e2535fcb28
app/vmui: fix path to metricsql doc
This is follow-up after f152021521

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-31 14:47:38 +02:00
Roman Khavronenko
a2ba37be68
lib/promscrape: support filtering targets via scrapePool GET param in /api/v1/targets API ()
This improves compatibility with Prometheus `/api/v1/targets` API.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5343

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-31 14:26:25 +02:00
Zakhar Bessarab
f863b331c1
app/vmalert/rule: follow-up for d8fe739aba
Remove tenancy-related part of the commit as it is not relevant to OS vmalert version.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-28 18:29:41 +04:00
Hui Wang
d8fe739aba
vmalert: properly attach tenant labels vm_account_id and `vm_projec… ()
* vmalert: properly attach tenant labels `vm_account_id` and `vm_project_id` to alerting rules when enabling `-clusterMode`

Previously, these labels were lost in alert messages to Alertmanager. Bug was introduced in [v1.112.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.112.0).
2025-03-28 12:26:55 +01:00
Aliaksandr Valialkin
35b31f904d
lib/httputil: automatically initialize data transfer metrics for the created HTTP transports via NewTransport() 2025-03-27 15:03:52 +01:00
Dmytro Kozlov
7c05ec42fe
app/vmctl: fix show logs for prometheus migration mode ()
### Describe Your Changes
Fixed issue an issue with show stats at the end of the process. Please
check the images below
Before the fix

![image](https://github.com/user-attachments/assets/d549c327-ed2b-46c5-965c-4f3581f54d83)


After the fix


![image](https://github.com/user-attachments/assets/c3200aff-dd50-40cf-92a9-b09800a25834)

I fixed it by moving logic to the function. Now it works correctly.

Added the tests for the Prometheus migration mode (make tests great
again).

The main discussion was introduced in this
[PR](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7863).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 14:12:10 +01:00
Yury Molodov
53a1c6162d
vmui: update dependencies to latest versions ()
### Describe Your Changes

Update dependencies to latest versions

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-27 10:18:32 +01:00
Aliaksandr Valialkin
2f86ef95a1
lib/{httputil,promauth}: move functions, which create TLS config and TLS-based HTTP transport, from lib/httputil to lib/promauth
- Move lib/httputil.Transport to lib/promauth.NewTLSTransport. Remove the first arg to this function (URL),
  since it has zero relation to the created transport.

- Move lib/httputil.TLSConfig to lib/promauth.NewTLSConfig. Re-use the existing functionality
  from lib/promauth.Config for creating TLS config. This enables the following features:
  - Ability to load key, cert and CA files from http urls.
  - Ability to change the key, cert and CA files without the need to restart the service.
    It automatically re-loads the new files after they change.
2025-03-26 20:14:17 +01:00
Aliaksandr Valialkin
e5f4826964
lib/httputil: add NewTransport() function for creating pre-initialized net/http.Transport 2025-03-26 18:57:17 +01:00
Aliaksandr Valialkin
cd94593383
app/vmctl: rename app/vmctl/utils to app/vmctl/vmctutil for the sake of consistency naming of *util packages 2025-03-26 18:14:30 +01:00
Aliaksandr Valialkin
5d7f68726d
app/vmalert: rename app/vmalert/utils to app/vmalert/vmalertutil for the sake of consistency of *util package naming 2025-03-26 18:07:45 +01:00
Aliaksandr Valialkin
e3349be39e
lib: rename lib/influxutils to lib/influxutil for the sake of consistency naming of *util packages 2025-03-26 17:38:25 +01:00
Aliaksandr Valialkin
0e0432db6c
lib: rename lib/promutils to lib/promutil for the sake of consistency for *util package naming 2025-03-26 17:32:06 +01:00
Aliaksandr Valialkin
b43c28ccdf
lib/protoparser: rename lib/protoparser/datadogutils to lib/protoparser/datadogutil for the sake of consistency for *util package naming 2025-03-26 17:13:28 +01:00
Aliaksandr Valialkin
0f3f49f9b9
app/vlinsert: rename app/vlinsert/insertutils to app/vlinsert/insertutil for the sake of consistency for *util package naming 2025-03-26 17:09:01 +01:00
Aliaksandr Valialkin
2880a290fc
app/vmselect: rename app/vmselect/searchutils to app/vmselect/searchutil for the sake of consistency for *util package naming 2025-03-26 17:05:00 +01:00
Aliaksandr Valialkin
e9c4769baf
lib: rename lib/httputils to lib/httputil for the sake of consistency for *util package naming 2025-03-26 16:44:21 +01:00
Zakhar Bessarab
7195862a51
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-21 16:37:34 +04:00
Yury Molodov
db66ab1852
vmui: show hidden common labels in group name ()
### Describe Your Changes

Changes:
1. When `hide common labels` is enabled, they will now be displayed in
the group name.
2. Legend settings toggles have been moved below the graph for better
accessibility.


![image](https://github.com/user-attachments/assets/fc8c7f4c-c155-4056-8862-301ad375d7ae)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-20 09:41:15 +01:00
Yury Molodov
31f662a0f7
vmui/logs: implement nanosecond log sorting ()
### Describe Your Changes

This PR adds nanosecond precision to log sorting, ensuring accurate
ordering of entries with sub-millisecond differences.

Related issue:  

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-20 09:30:57 +01:00
Hui Wang
bd11e00a59
app/vmalert: properly register group and rules metrics
Commit 9ca74d1fff introduced an issue with metrics registration. Due to metrics.Summary type always registered at the global state of metrics package, vmalert had increased memory and CPU usage after multiple configuration reloads.

 This commit addresses this issue and properly registers metrics.Summary metric. Now metrics for group and rules must be explicitly registered before group.Start with group.Init method. It simplifies metrics usage an ensures that all needed metrics were registered and group is ready to start.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
2025-03-19 13:57:34 +01:00
Aliaksandr Valialkin
348991d1b3
app/vlselect: move the code responsible for limiting the number of concurrently executed requests, into separate functions
This improves code readability a bit.
2025-03-19 13:28:03 +01:00
Hui Wang
b97cacad45
app/vmalert: fix possible data race on group checksum
1. fix possible data race on group checksum when reload is called
concurrently. Before, it didn't affect much but might update the group
one more time.
2. remove the unnecessary g.mu.RLock() and compute group.id at newGroup creation. Changes to group.ID()
indicate that type and interval have changed, and the group is new.

Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8540
2025-03-19 12:58:51 +01:00
Hui Wang
dec237b7d6
app/vmalert: fix memory leak with -notifier.blackhole
Previous commit 9ca74d1fff added a regression for notifier's metrics exposed by vmalert. vmalert returned new notifier instances for the blackhole notifier type. And it registered new metrics each get notifiers function was called. It registered duplicate metrics and lead to OOM crash.

 This commit properly init blachole notifier instances and add metrics for it only once, during application start.

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
2025-03-19 10:43:57 +01:00
Aliaksandr Valialkin
f645479b5e
lib/protoparser: rename lib/protoparser/common to lib/protoparser/protoparserutil
This improves readability of the code, which uses this package.
2025-03-18 16:24:51 +01:00
Aliaksandr Valialkin
1aa14f3b6d
app/vlinsert: do not start background flusher in the LogMessageProcessor used for synchornous processing of a single data block
This should reduce CPU usage a bit for data ingestion protocols,
which process a single message per every request without streaming.
2025-03-18 11:17:08 +01:00
Yury Molodov
44a54e4590
vmui/logs: fix endless group expansion loop bug ()
### Describe Your Changes

Fix endless group expansion loop.
Related issue: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-03-17 16:39:41 +01:00
Guillem Jover
76d205feae
spelling and grammar fixes via codespell ()
### Describe Your Changes

Fix many spelling errors and some grammar, including misspellings in
filenames. 

The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards. 
So it seems to have low impact on users.

The change also deprecates `cspell` as it is much heavier and less usable. 
---------

Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
2025-03-17 16:32:10 +01:00
Aliaksandr Valialkin
39082103a6
app/vlinsert/opentelemetry: follow-up for a884949aba
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8511
2025-03-16 01:09:07 +01:00
Aliaksandr Valialkin
23c4e4cdb2
app/vlinsert: send 204 No Content response code at /insert/loki/api/v1/push endpoint
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8505
2025-03-15 23:34:49 +01:00
Aliaksandr Valialkin
0451a1c9e0
app/vlinsert: follow-up for 37ed1842ab
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
  Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
  when Content-Encoding header is missing.

- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
  This eliminates the churn in tests for these functions. This also fixes broken
  benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
  the whole request body on the first iteration and do nothing on subsequent iterations.

- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
  versions of VictoriaMetrics and VictoriaLogs.

- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
  at VictoriaLogs and VictoriaMetrics.

- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
  the passed reader arg in order to properly deal with it.

- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
  This allows removing repeated code across request-based protocol parsers without streaming mode.

- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
  Previously this wasn't the case for all the supported protocols.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
2025-03-15 00:03:03 +01:00
Aliaksandr Valialkin
c60b4175bb
app/vlinsert: add an ability to ignore log fields starting with the given prefixes
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
2025-03-15 00:03:02 +01:00
Roman Khavronenko
dc1f7ef0d0
app/vmselect/promql: optimize binary operator or for common cases ()
The optimization touches 2 things:
1. Reduces amount of allocations when comparing canonical metric names
between left and right parts of expressions.
2. Adds fast path for cases when right part of expression returns
scalar: `series_selector or on() vector(1)`, which is a typical
expression.

```
benchcmp old.txt new.txt
benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark                                             old ns/op     new ns/op     delta
BenchmarkBinaryOpOr/tss:1_or_tss:1-14                 291           272           -6.56%
BenchmarkBinaryOpOr/tss:1_or_tss:1000-14              44590         28592         -35.88%
BenchmarkBinaryOpOr/tss:1000_or_tss:1-14              103124        39563         -61.64%
BenchmarkBinaryOpOr/tss:1000_or_tss:1000-14           20386150      1859335       -90.88%
BenchmarkBinaryOpOr/tss:1000_or_on()_vector(0)-14     91382         36805         -59.72%
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8382

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 12:07:22 +01:00
Zhu Jiekun
9548b7e442
docs: revert doc change for on-disk persistence and move new content to another section ()
### Describe Your Changes

revert doc change in
815bad3687
and move new content to another section.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-14 11:50:17 +01:00
Aliaksandr Valialkin
b6e4abb31e
app/vlogsgenerator: increase write buffer size in order to reduce the number of send() syscalls
This increases data ingestion performance, which can be achieved by the vlogsgenerator
2025-03-14 03:14:01 +01:00
Andrii Chubatiuk
37ed1842ab
lib/protoparser: support zstd in all logs http ingestion, datadog and otel metrics protocols ()
This commit introduces common readers for multiple compression encoding algorithms. 

Currently, supported encodings are:
* zstd
* gzip
* deflat
* snappy

 It adds new common reader to the all VictoriaLogs ingestion protocols.
And updates opentelemetry metrics parsing for VictoriaMetrics components.

Also, it ports zstd stream parses from cluster branch.

Related issues:
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300

---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-14 00:39:52 +01:00
Zhu Jiekun
815bad3687
app/vmagent: prevent dropping persistent queue if -remoteWrite.showURL changed
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
 Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
 Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.

 This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
 It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision. 

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.


### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-13 23:50:01 +01:00
Aliaksandr Valialkin
19165c436f
app/vlinsert/loki: automatically parse JSON-encoded log fields from the plaintext log message
Loki doesn't support well high-cardinality log fields (e.g. fields with big number of unique values).
That's why Promtail, Grafana Agent and Grafana Alloy encode such fields into a JSON and push them
as a plaintext log message to the remote storage. This isn't an efficient way to store high-cardinality
log fields in VictoriaLogs, since it is optimized for storing and querying such fields when they are stored
distinctly as a regular log fields according to VictoriaLogs data model ( https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ).

This commit enables automatic parsing of JSON-encoded log fields at plaintext log message received over Loki protocol
and storing them as a separate log fields. This should improve data compression ratio and reduce disk space usage.
This should also improve query performance when the parsed log fields are used in queries for filtering and aggregation.
The old behaviour can be restored by passing -loki.disableMessageParsing command-line flag to VictoriaLogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8486
2025-03-12 21:23:28 +01:00
Aliaksandr Valialkin
cb240eda70
app/vlinsert: follow-up for 67f8fa66ed
- Properly handle negative timestamps (e.g. timestamps before 1970-01-01)

- Optimize parsing floating-point timestamps by eliminating the memory allocation
  needed for returning an error from strconv.ParseInt. Instead, check whether the string contains a dot,
  and then parse it as a floating-point number.

- Add tests for ParseUnixTimestamp function.

- Make the code easier to understand and maintain by removing unneeded generic function toNano().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8472
2025-03-12 16:28:40 +01:00
Emre Yazıcı
cfd2c6e5e7
app/vmalert: add vmalert_alerts_send_duration_seconds metric ()
### Describe Your Changes

Add `vmalert_alerts_send_latency_seconds` metric for
alertmanager.notifier.

To measure the time for alertmanager calls to send alerts per notifier.
This is needed to see the latency for each notifier from vmalert calls.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: emreya <e.yazici1990@gmail.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-03-12 14:30:54 +01:00
alicja-karasiewicz
d47d329ce7
feat: make topN limit configurable from CLI
Implement changes mentioned in 

Allow the administrator to specify the limit of returned TSDB series in
`/api/v1/status/tsdb` by making a TopN limit configurable from CLI.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: alicja-karasiewicz <alicja.karasiewicz@allegro.com>
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-03-12 11:33:30 +04:00
Aliaksandr Valialkin
edecc433ff
deployment: update Go builder from Go1.24.0 to Go1.24.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.24.1+label%3ACherryPickApproved
2025-03-10 18:59:59 +01:00
Andrii Chubatiuk
67f8fa66ed
app/vlinsert: support floats for elasticseach timestamps ()
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-10 13:48:48 +01:00
Artem Fetishev
c4fd62188a
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-07 14:15:38 +01:00
Hui Wang
e8e2ef54a0
vmalert: allow chaining groups with eval_offset ()
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/860,
see
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/change-evaloffset-behavior/docs/vmalert.md#chaining-groups

Also related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8154
2025-03-07 09:45:16 +01:00
Nikolay
b85b28d30a
lib/storage: add tracker for time series metric names statistics
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.

This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.

  New API added to the select component:

* /api/v1/status/metric_names_stats - which returns a JSON
object
    with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
    state of the tracker and reset tsid/cache.

   New metrics were added for this feature:

  * vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
  * vm_cache_size{type="storage/metricNamesUsageTracker"}
  * vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}

  Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-06 22:06:50 +01:00
Zakhar Bessarab
7dfaef9088
app/vmselect/promql: fix panic with using @ with series which is not present at the start of the query ()
### Describe Your Changes

Previously, "selector @ another_selector" assumed that
"another_selector" metric is supposed to exist since "start" used in the
query.

If the query was evaluated in the following case (timestamps):
- start - 2, end - 10
- "another_selector" 5,6,7,8,9,10
- "selector" The resulting "at" timestamp would be taken from NaN (as
`int64(NaN * 1000)`), causing a panic or invalid behavior later.

Note that type cast of `NaN` to int64 is also platform-dependent, so
value of `int64(math.NaN() * 1000)` can produce `0` or max int64 on
different platforms and versions of Go.

This commit changes this and checks for the first non-NaN value. This
makes it easier to use for users as series are not always aligned and
returning an error in this case would disallow using this for some time
ranges.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8444

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-06 16:42:19 +01:00
Andrii Chubatiuk
26fba57cfa
lib/protoparser/opentelemetry: properly marshal nested attributes into JSON
Previously, opentelemetry attribute parsed added extra field names according to 
golang JSON parser spec for structs:

```
struct AnyValue{
 StringValue string
}
```
 Was serialized into:
```
{"StringValue": "some-string"}
```
 While opentelemetry-collector serializes it as
```
"some-string"
```

 This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
2025-03-05 16:35:07 +01:00
hagen1778
fc5d495900
app/vmui: update error message for no matched rules
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 16:56:15 +01:00
Yury Molodov
974c094a52
vmui: fix infinite loader on downsampling page ()
### Describe Your Changes

This PR fixes an issue where the Downsampling filters debug page would
get stuck in an infinite loading state when labels had no matches. Now,
the case is properly handled.

Related issue: 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 16:48:11 +01:00
Hui Wang
52988ebdc8
vmalert: add time buckets stats pipe check for vlogs expression ()
VictoriaLogs inserts `_time` field as a label in result when query with
[time buckets stats
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets),
making the result meaningless and may lead to cardinality issues.
>curl --location --request POST
'https://play-vmlogs.victoriametrics.com/select/logsql/stats_query?query=_time%3A1m%20%7C%20stats%20by%20(_time%3A10s)%20count%20()%20as%20total'

>{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"total","_time":"2025-01-24T12:31:30Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:10Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:00Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:20Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:50Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:40Z"},"value":[1737721904.4476516,"9"]}]}}%

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 14:09:34 +01:00
Roman Khavronenko
63f6ac3ff8
lib/promutils: move time-related funcs from promutils to timeutil ()
Since funcs `ParseDuration` and `ParseTimeMsec` are used in vlogs,
vmalert, victoriametrics and other components, importing promutils only
for this reason makes them to export irrelevant
`vm_rows_invalid_total{type="prometheus"}` metric.

This change removes `vm_rows_invalid_total{type="prometheus"}` metric
from /metrics page for these components.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 10:25:42 +01:00
Hui Wang
281f1a94e4
vmalert-tool: clean up the temporary storage path when process is ter… ()
…minated by SIGTERM or SIGINT
2025-02-28 15:17:05 +01:00