### Describe Your Changes
Changes:
1. When `hide common labels` is enabled, they will now be displayed in
the group name.
2. Legend settings toggles have been moved below the graph for better
accessibility.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit db66ab1852)
### Describe Your Changes
This PR adds nanosecond precision to log sorting, ensuring accurate
ordering of entries with sub-millisecond differences.
Related issue: #8346
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 31f662a0f7)
Commit 9ca74d1fff introduced an issue with metrics registration. Due to metrics.Summary type always registered at the global state of metrics package, vmalert had increased memory and CPU usage after multiple configuration reloads.
This commit addresses this issue and properly registers metrics.Summary metric. Now metrics for group and rules must be explicitly registered before group.Start with group.Init method. It simplifies metrics usage an ensures that all needed metrics were registered and group is ready to start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
1. fix possible data race on group checksum when reload is called
concurrently. Before, it didn't affect much but might update the group
one more time.
2. remove the unnecessary g.mu.RLock() and compute group.id at newGroup creation. Changes to group.ID()
indicate that type and interval have changed, and the group is new.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8540
Previous commit 9ca74d1fff added a regression for notifier's metrics exposed by vmalert. vmalert returned new notifier instances for the blackhole notifier type. And it registered new metrics each get notifiers function was called. It registered duplicate metrics and lead to OOM crash.
This commit properly init blachole notifier instances and add metrics for it only once, during application start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
### Describe Your Changes
Fix endless group expansion loop.
Related issue: #8347
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit 44a54e4590)
### Describe Your Changes
Fix many spelling errors and some grammar, including misspellings in
filenames.
The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards.
So it seems to have low impact on users.
The change also deprecates `cspell` as it is much heavier and less usable.
---------
Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
(cherry picked from commit 76d205feae)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
when Content-Encoding header is missing.
- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
This eliminates the churn in tests for these functions. This also fixes broken
benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
the whole request body on the first iteration and do nothing on subsequent iterations.
- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
versions of VictoriaMetrics and VictoriaLogs.
- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
at VictoriaLogs and VictoriaMetrics.
- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
the passed reader arg in order to properly deal with it.
- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
This allows removing repeated code across request-based protocol parsers without streaming mode.
- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
Previously this wasn't the case for all the supported protocols.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
The optimization touches 2 things:
1. Reduces amount of allocations when comparing canonical metric names
between left and right parts of expressions.
2. Adds fast path for cases when right part of expression returns
scalar: `series_selector or on() vector(1)`, which is a typical
expression.
```
benchcmp old.txt new.txt
benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark old ns/op new ns/op delta
BenchmarkBinaryOpOr/tss:1_or_tss:1-14 291 272 -6.56%
BenchmarkBinaryOpOr/tss:1_or_tss:1000-14 44590 28592 -35.88%
BenchmarkBinaryOpOr/tss:1000_or_tss:1-14 103124 39563 -61.64%
BenchmarkBinaryOpOr/tss:1000_or_tss:1000-14 20386150 1859335 -90.88%
BenchmarkBinaryOpOr/tss:1000_or_on()_vector(0)-14 91382 36805 -59.72%
```
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8382
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit dc1f7ef0d0)
### Describe Your Changes
revert doc change in
815bad3687
and move new content to another section.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9548b7e442)
Previously, vmselect didn't stop multitenant query execution if it
receives error from vmstorage. Such as limit error or any other. It
continued to execute queries until it did it for all tenants. It leads
to the potential waste of resources.
In addition, callback error was incorrectly reference and can be updated by
subsequent callback call.
This commit returns error earlier, cancels sub-sequent requests for
tenants and properly return storageNode request error.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8461
This commit introduces common readers for multiple compression encoding algorithms.
Currently, supported encodings are:
* zstd
* gzip
* deflat
* snappy
It adds new common reader to the all VictoriaLogs ingestion protocols.
And updates opentelemetry metrics parsing for VictoriaMetrics components.
Also, it ports zstd stream parses from cluster branch.
Related issues:
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.
This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Loki doesn't support well high-cardinality log fields (e.g. fields with big number of unique values).
That's why Promtail, Grafana Agent and Grafana Alloy encode such fields into a JSON and push them
as a plaintext log message to the remote storage. This isn't an efficient way to store high-cardinality
log fields in VictoriaLogs, since it is optimized for storing and querying such fields when they are stored
distinctly as a regular log fields according to VictoriaLogs data model ( https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ).
This commit enables automatic parsing of JSON-encoded log fields at plaintext log message received over Loki protocol
and storing them as a separate log fields. This should improve data compression ratio and reduce disk space usage.
This should also improve query performance when the parsed log fields are used in queries for filtering and aggregation.
The old behaviour can be restored by passing -loki.disableMessageParsing command-line flag to VictoriaLogs.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8486
- Properly handle negative timestamps (e.g. timestamps before 1970-01-01)
- Optimize parsing floating-point timestamps by eliminating the memory allocation
needed for returning an error from strconv.ParseInt. Instead, check whether the string contains a dot,
and then parse it as a floating-point number.
- Add tests for ParseUnixTimestamp function.
- Make the code easier to understand and maintain by removing unneeded generic function toNano().
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8472
### Describe Your Changes
Add `vmalert_alerts_send_latency_seconds` metric for
alertmanager.notifier.
To measure the time for alertmanager calls to send alerts per notifier.
This is needed to see the latency for each notifier from vmalert calls.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: emreya <e.yazici1990@gmail.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
(cherry picked from commit cfd2c6e5e7)
### Describe Your Changes
Implement changes mentioned in #6898
Allow the administrator to specify the limit of returned TSDB series in
`/api/v1/status/tsdb` by making a TopN limit configurable from CLI.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: alicja-karasiewicz <alicja.karasiewicz@allegro.com>
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.
This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.
New API added to the select component:
* /api/v1/status/metric_names_stats - which returns a JSON
object
with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
state of the tracker and reset tsid/cache.
New metrics were added for this feature:
* vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
* vm_cache_size{type="storage/metricNamesUsageTracker"}
* vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Previously, "selector @ another_selector" assumed that
"another_selector" metric is supposed to exist since "start" used in the
query.
If the query was evaluated in the following case (timestamps):
- start - 2, end - 10
- "another_selector" 5,6,7,8,9,10
- "selector" The resulting "at" timestamp would be taken from NaN (as
`int64(NaN * 1000)`), causing a panic or invalid behavior later.
Note that type cast of `NaN` to int64 is also platform-dependent, so
value of `int64(math.NaN() * 1000)` can produce `0` or max int64 on
different platforms and versions of Go.
This commit changes this and checks for the first non-NaN value. This
makes it easier to use for users as series are not always aligned and
returning an error in this case would disallow using this for some time
ranges.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8444
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7dfaef9088)
Previously, opentelemetry attribute parsed added extra field names according to
golang JSON parser spec for structs:
```
struct AnyValue{
StringValue string
}
```
Was serialized into:
```
{"StringValue": "some-string"}
```
While opentelemetry-collector serializes it as
```
"some-string"
```
This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
### Describe Your Changes
This PR fixes an issue where the Downsampling filters debug page would
get stuck in an infinite loading state when labels had no matches. Now,
the case is properly handled.
Related issue: #8339
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 974c094a52)
VictoriaLogs inserts `_time` field as a label in result when query with
[time buckets stats
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets),
making the result meaningless and may lead to cardinality issues.
>curl --location --request POST
'https://play-vmlogs.victoriametrics.com/select/logsql/stats_query?query=_time%3A1m%20%7C%20stats%20by%20(_time%3A10s)%20count%20()%20as%20total'
>{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"total","_time":"2025-01-24T12:31:30Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:10Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:00Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:20Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:50Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:40Z"},"value":[1737721904.4476516,"9"]}]}}%
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 52988ebdc8)
Since funcs `ParseDuration` and `ParseTimeMsec` are used in vlogs,
vmalert, victoriametrics and other components, importing promutils only
for this reason makes them to export irrelevant
`vm_rows_invalid_total{type="prometheus"}` metric.
This change removes `vm_rows_invalid_total{type="prometheus"}` metric
from /metrics page for these components.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 63f6ac3ff8)
### Describe Your Changes
#8342
fix negative rate result when the lookbehind window is longer than
`-search.maxLookback` or `-search.maxStalenessInterval` and data
contains gap.
This issue was introduced since
[v1.110.0](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072).
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
`MustParsePromMetrics` imports `lib/protoparser/prometheus`, and this
package exposes the following metrics:
```
vm_protoparser_rows_read_total{type="promscrape"}
vm_rows_invalid_total{type="prometheus"}
```
It means every package that uses `lib/prompbmarshal` will start exposing
these metrics. For example, vlogs imports `lib/protoparser/common` which
uses `lib/prompbmarshal.Label`. And only because of this vlogs starts
exposing unrelated prometheus metrics on /metrics page.
Moving `MustParsePromMetrics` to `lib/protoparser/prometheus` seems like
the leas intrusive change.
-----------
Depends on another change
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8403
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This change would help to track slow queries via alerts or dashboards.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ba2bf9e73a)
### Describe Your Changes
Previously, `timestamp` variable was shared by goroutines and modified
during `writeBlock`, verified in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8312.
This pull request create new variable within each goroutine to avoid
race condition.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] Test on sandbox.
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package.
This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
Legend settings have been added to **Graph Settings**.
#### **New Features:**
1. **Table View** – Toggle to display the legend in a table format.
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.
Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.
This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
