This commit adds `search.logSlowQueryStats=<duration>` cmd-line flag on vmselect.
It reads stats from eval function, and doesn't slow down the query execution.
Log line has the following structure:
vm_slow_query_stats type=%s query=%q query_hash=%d start_ms=%d end_ms=%d step_ms=%d range_ms=%d tenant=%q execution_duration_ms=%v series_fetched=%d samples_fetched=%d bytes=%d memory_estimated_bytes=%d
This feature is only available for enterprise version.
### Describe Your Changes
Follow-up to
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462
Addressed review comments:
- Log panic with FATAL prefix to indicate possible on-disk data
corruption
- Moved version bump line to the tip block (v1.114.0 has already been
released) in changelog
- Removed duplicate vmagent entry from targets list from Makefile
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
This commit improves how vmagent selects the remote write protocol.
Previously, vmagent [performed a handshake
probe](0ff1a3b154/lib/protoparser/protoparserutil/vmproto_handshake.go (L11))
at
[startup](0ff1a3b154/app/vmagent/remotewrite/client.go (L173)):
- If the probe succeeded, it used the VictoriaMetrics (VM) protocol.
- If the probe failed, it downgraded to the Prometheus protocol.
- No protocol changes occurred after the initial probe at runtime.
However, this approach had limitations:
- If vmstorage was unavailable during vmagent startup, vmagent would
immediately downgrade to the Prometheus protocol, leading to higher
network usage unitl vmagent restarted. This case has been reported in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615.
- If the remote write server was updated or downgraded (e.g., during a
fallback or migration), vmagent would not detect the protocol change. It
would continue retrying failed requests and eventually drop them.
Require a restart of vmagent to pick up the new protocol.
This commit introduces a more adaptive mechanism.
vmagent always starts with the VM protocol and downgrades to the
Prometheus protocol only if an unsupported media type or bad request
response is received.
When this happens, the protocol is downgraded for all future requests.
In-flight requests are re-packed from Zstd to Snappy and retried
immediately.
Snappy-encoded requests are dropped if an unsupported media type or bad
request is received (no retrying).
Additionally, the in-memory and persisted queues could mix snappy and
zstd encoded blocks. The proper encoding is decided before sending by
encoding.IsZstd function.
TODO:
* [x] Add tests
* [x] Update documentation
* [x] Changelog
* [x] Research on
[content-type](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054),
[accept-encoding](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786923382)
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615#top
issue.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Currently, requests failing due to network timeout would receive "200
OK" while producing a warning log message about the timeout. This
behaviour is confusing and might produce unexpected issues as it is not
possible to retry errors properly.
Change this to return "502 Bad Gateway" response so that error can be
handled by the client.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8621
Config for testing:
```
unauthorized_user:
url_prefix: "http://example.com:9800"
```
Before the change:
```
* Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
/* NOTE: 30 seconds timeout passes */
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 08:54:05 GMT
Date: Tue, 01 Apr 2025 08:54:05 GMT
<
* Connection #0 to host 127.0.0.1 left intact
```
After:
```
* Trying 127.0.0.1:8427...
* Connected to 127.0.0.1 (127.0.0.1) port 8427
* using HTTP/1.x
> HEAD /api/v1/query HTTP/1.1
> Host: 127.0.0.1:8427
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Server-Hostname: pc
X-Server-Hostname: pc
< Date: Tue, 01 Apr 2025 09:13:57 GMT
Date: Tue, 01 Apr 2025 09:13:57 GMT
< Content-Length: 109
Content-Length: 109
<
* Connection #0 to host 127.0.0.1 left intact
```
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Log when the data response from vmselect is partial during
rule(recording, alertingrule) evaluations.
vmselect returns `isPartial: true` in case data is not fully fetched
from scattered vmstorages. At the time of rule evals, it may be drifting
apart from real values due to missing points. This is an important event
that should be logged to inform users to see how often that happens as
it may lead to false positive alerts.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: emreya <emre.yazici@adyen.com>
Signed-off-by: emreya <e.yazici1990@gmail.com>
Signed-off-by: Emre Yazici <e.yazici1990@gmail.com>
(cherry picked from commit 56f60e8be9)
When creating and listing snapshots, panic instead of returning an error
since errors are not recoverable anyway.
Also do not cleanup the filesystem on panic. Leave as is for further
manual inspection.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
This reduces CPU usage by up to 30% in exchange of the increased RAM usage by 10%
when scraping thousands of targets, which expose millions of metrics in summary.
This looks like a good tradeoff after the commit edac875179 ,
which reduced RAM usage by more than 10%, so the final RAM usage for vmagent
is still lower than the RAM usage at v1.114.0 by ~15%, while CPU usage drops by 30%.
* vmalert: properly attach tenant labels `vm_account_id` and `vm_project_id` to alerting rules when enabling `-clusterMode`
Previously, these labels were lost in alert messages to Alertmanager. Bug was introduced in [v1.112.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.112.0).
### Describe Your Changes
Update dependencies to latest versions
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- Move lib/httputil.Transport to lib/promauth.NewTLSTransport. Remove the first arg to this function (URL),
since it has zero relation to the created transport.
- Move lib/httputil.TLSConfig to lib/promauth.NewTLSConfig. Re-use the existing functionality
from lib/promauth.Config for creating TLS config. This enables the following features:
- Ability to load key, cert and CA files from http urls.
- Ability to change the key, cert and CA files without the need to restart the service.
It automatically re-loads the new files after they change.
### Describe Your Changes
Changes:
1. When `hide common labels` is enabled, they will now be displayed in
the group name.
2. Legend settings toggles have been moved below the graph for better
accessibility.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
This PR adds nanosecond precision to log sorting, ensuring accurate
ordering of entries with sub-millisecond differences.
Related issue: #8346
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Commit 9ca74d1fff introduced an issue with metrics registration. Due to metrics.Summary type always registered at the global state of metrics package, vmalert had increased memory and CPU usage after multiple configuration reloads.
This commit addresses this issue and properly registers metrics.Summary metric. Now metrics for group and rules must be explicitly registered before group.Start with group.Init method. It simplifies metrics usage an ensures that all needed metrics were registered and group is ready to start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
1. fix possible data race on group checksum when reload is called
concurrently. Before, it didn't affect much but might update the group
one more time.
2. remove the unnecessary g.mu.RLock() and compute group.id at newGroup creation. Changes to group.ID()
indicate that type and interval have changed, and the group is new.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8540
Previous commit 9ca74d1fff added a regression for notifier's metrics exposed by vmalert. vmalert returned new notifier instances for the blackhole notifier type. And it registered new metrics each get notifiers function was called. It registered duplicate metrics and lead to OOM crash.
This commit properly init blachole notifier instances and add metrics for it only once, during application start.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8532
### Describe Your Changes
Fix endless group expansion loop.
Related issue: #8347
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Fix many spelling errors and some grammar, including misspellings in
filenames.
The change also fixes a typo in metric `vm_mmaped_files` to `vm_mmapped_files`.
While this is a breaking change, this metric isn't used in alerts or dashboards.
So it seems to have low impact on users.
The change also deprecates `cspell` as it is much heavier and less usable.
---------
Co-authored-by: Andrii Chubatiuk <achubatiuk@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
when Content-Encoding header is missing.
- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
This eliminates the churn in tests for these functions. This also fixes broken
benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
the whole request body on the first iteration and do nothing on subsequent iterations.
- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
versions of VictoriaMetrics and VictoriaLogs.
- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
at VictoriaLogs and VictoriaMetrics.
- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
the passed reader arg in order to properly deal with it.
- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
This allows removing repeated code across request-based protocol parsers without streaming mode.
- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
Previously this wasn't the case for all the supported protocols.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
The optimization touches 2 things:
1. Reduces amount of allocations when comparing canonical metric names
between left and right parts of expressions.
2. Adds fast path for cases when right part of expression returns
scalar: `series_selector or on() vector(1)`, which is a typical
expression.
```
benchcmp old.txt new.txt
benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark old ns/op new ns/op delta
BenchmarkBinaryOpOr/tss:1_or_tss:1-14 291 272 -6.56%
BenchmarkBinaryOpOr/tss:1_or_tss:1000-14 44590 28592 -35.88%
BenchmarkBinaryOpOr/tss:1000_or_tss:1-14 103124 39563 -61.64%
BenchmarkBinaryOpOr/tss:1000_or_tss:1000-14 20386150 1859335 -90.88%
BenchmarkBinaryOpOr/tss:1000_or_on()_vector(0)-14 91382 36805 -59.72%
```
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8382
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
revert doc change in
815bad3687
and move new content to another section.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
This commit introduces common readers for multiple compression encoding algorithms.
Currently, supported encodings are:
* zstd
* gzip
* deflat
* snappy
It adds new common reader to the all VictoriaLogs ingestion protocols.
And updates opentelemetry metrics parsing for VictoriaMetrics components.
Also, it ports zstd stream parses from cluster branch.
Related issues:
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.
This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Loki doesn't support well high-cardinality log fields (e.g. fields with big number of unique values).
That's why Promtail, Grafana Agent and Grafana Alloy encode such fields into a JSON and push them
as a plaintext log message to the remote storage. This isn't an efficient way to store high-cardinality
log fields in VictoriaLogs, since it is optimized for storing and querying such fields when they are stored
distinctly as a regular log fields according to VictoriaLogs data model ( https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ).
This commit enables automatic parsing of JSON-encoded log fields at plaintext log message received over Loki protocol
and storing them as a separate log fields. This should improve data compression ratio and reduce disk space usage.
This should also improve query performance when the parsed log fields are used in queries for filtering and aggregation.
The old behaviour can be restored by passing -loki.disableMessageParsing command-line flag to VictoriaLogs.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8486
- Properly handle negative timestamps (e.g. timestamps before 1970-01-01)
- Optimize parsing floating-point timestamps by eliminating the memory allocation
needed for returning an error from strconv.ParseInt. Instead, check whether the string contains a dot,
and then parse it as a floating-point number.
- Add tests for ParseUnixTimestamp function.
- Make the code easier to understand and maintain by removing unneeded generic function toNano().
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8472
### Describe Your Changes
Add `vmalert_alerts_send_latency_seconds` metric for
alertmanager.notifier.
To measure the time for alertmanager calls to send alerts per notifier.
This is needed to see the latency for each notifier from vmalert calls.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: emreya <e.yazici1990@gmail.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Implement changes mentioned in #6898
Allow the administrator to specify the limit of returned TSDB series in
`/api/v1/status/tsdb` by making a TopN limit configurable from CLI.
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: alicja-karasiewicz <alicja.karasiewicz@allegro.com>
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.
This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.
New API added to the select component:
* /api/v1/status/metric_names_stats - which returns a JSON
object
with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
state of the tracker and reset tsid/cache.
New metrics were added for this feature:
* vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
* vm_cache_size{type="storage/metricNamesUsageTracker"}
* vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Previously, "selector @ another_selector" assumed that
"another_selector" metric is supposed to exist since "start" used in the
query.
If the query was evaluated in the following case (timestamps):
- start - 2, end - 10
- "another_selector" 5,6,7,8,9,10
- "selector" The resulting "at" timestamp would be taken from NaN (as
`int64(NaN * 1000)`), causing a panic or invalid behavior later.
Note that type cast of `NaN` to int64 is also platform-dependent, so
value of `int64(math.NaN() * 1000)` can produce `0` or max int64 on
different platforms and versions of Go.
This commit changes this and checks for the first non-NaN value. This
makes it easier to use for users as series are not always aligned and
returning an error in this case would disallow using this for some time
ranges.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8444
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Previously, opentelemetry attribute parsed added extra field names according to
golang JSON parser spec for structs:
```
struct AnyValue{
StringValue string
}
```
Was serialized into:
```
{"StringValue": "some-string"}
```
While opentelemetry-collector serializes it as
```
"some-string"
```
This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
### Describe Your Changes
This PR fixes an issue where the Downsampling filters debug page would
get stuck in an infinite loading state when labels had no matches. Now,
the case is properly handled.
Related issue: #8339
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
VictoriaLogs inserts `_time` field as a label in result when query with
[time buckets stats
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets),
making the result meaningless and may lead to cardinality issues.
>curl --location --request POST
'https://play-vmlogs.victoriametrics.com/select/logsql/stats_query?query=_time%3A1m%20%7C%20stats%20by%20(_time%3A10s)%20count%20()%20as%20total'
>{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"total","_time":"2025-01-24T12:31:30Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:10Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:00Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:31:20Z"},"value":[1737721904.4476516,"12"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:50Z"},"value":[1737721904.4476516,"10"]},{"metric":{"__name__":"total","_time":"2025-01-24T12:30:40Z"},"value":[1737721904.4476516,"9"]}]}}%
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Since funcs `ParseDuration` and `ParseTimeMsec` are used in vlogs,
vmalert, victoriametrics and other components, importing promutils only
for this reason makes them to export irrelevant
`vm_rows_invalid_total{type="prometheus"}` metric.
This change removes `vm_rows_invalid_total{type="prometheus"}` metric
from /metrics page for these components.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
#8342
fix negative rate result when the lookbehind window is longer than
`-search.maxLookback` or `-search.maxStalenessInterval` and data
contains gap.
This issue was introduced since
[v1.110.0](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072).
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
`MustParsePromMetrics` imports `lib/protoparser/prometheus`, and this
package exposes the following metrics:
```
vm_protoparser_rows_read_total{type="promscrape"}
vm_rows_invalid_total{type="prometheus"}
```
It means every package that uses `lib/prompbmarshal` will start exposing
these metrics. For example, vlogs imports `lib/protoparser/common` which
uses `lib/prompbmarshal.Label`. And only because of this vlogs starts
exposing unrelated prometheus metrics on /metrics page.
Moving `MustParsePromMetrics` to `lib/protoparser/prometheus` seems like
the leas intrusive change.
-----------
Depends on another change
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8403
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Previously, `timestamp` variable was shared by goroutines and modified
during `writeBlock`, verified in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8312.
This pull request create new variable within each goroutine to avoid
race condition.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ] Test on sandbox.
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package.
This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
Legend settings have been added to **Graph Settings**.
#### **New Features:**
1. **Table View** – Toggle to display the legend in a table format.
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.
Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.
This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
### Describe Your Changes
By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.
For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.
Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:
- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
It allows enabling aggregation windows for a specific aggregator.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
This PR introduces several enhancements and optimizations for the Group
view:
1. **Disable hover effect:**
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.
2. **Limit entries per Group:**
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.
3. **Display group info:**
Include the group number in the title along with the total count of
groups to improve clarity and navigation.
4. **Performance improvement:**
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.
Related issue: #8135
<details>
<summary>Demo UI</summary>
<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>
</details>
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to `^[A-Z_][A-Z0-9_]*`.
It's needed to properly support entities with single-character
names.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
### Describe Your Changes
Allow disabling the per-day index using the `-disablePerDayIndex` flag.
This should significantly improve the ingestion rate and decrease the
disk space usage for the use cases that assume small or no churn rate.
See the docs added to `docs/README.md` for details.
Both improvements are due to no data written to the per-day index.
Benchmark results:
```shell
rm -Rf ./lib/storage/Benchmark*; go test ./lib/storage -run=NONE -bench=BenchmarkStorageInsertWithAndWithoutPerDayIndex --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/perDayIndexes-12 1 3850268120 ns/op 39.56 data-MiB 28.20 indexdb-MiB 259722 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/noPerDayIndexes-12 1 2916865725 ns/op 39.57 data-MiB 25.73 indexdb-MiB 342834 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/perDayIndexes-12 1 2218073474 ns/op 9.772 data-MiB 13.73 indexdb-MiB 450842 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/noPerDayIndexes-12 1 1295140898 ns/op 9.771 data-MiB 0.3566 indexdb-MiB 772119 rows/s
PASS
ok github.com/VictoriaMetrics/VictoriaMetrics/lib/storage 11.421s
```
Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
MustOpenStorage function may accept variable number of optional
arguments. This commit combines optional arguments into dedicated OpenOptions
struct. It reduces complexity of adding new optional arguments.
Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8118
### Describe Your Changes
Fix autocomplete not passing `AccountID` and `ProjectID` headers when
fetching suggestions in VictoriaLogs UI.
Related: #8042
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Adding Accept-Encoding response header to support content negotiation,
which was introduced in [this
PR](https://github.com/systemd/systemd/pull/34822) and enables
compression on journald
### Describe Your Changes
Updated default `step` calculation for `Table` and `JSON` views.
- When switching to the `Table` or `JSON` view, the default step is now
automatically set to `end - start`, where `end` and `start` correspond
to the selected time range.
- If the user has manually set a step value, it will remain unchanged
when switching views or changing the time range.
- The UI now explicitly indicates when the step is automatically
calculated.
<details> <summary>Demo</summary>
https://github.com/user-attachments/assets/2540de24-36ed-4764-a047-1c6b48a80ed4
</details>
**Note:** These views use the
[`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query)
endpoint for instant queries.
Related issue: #8240
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
There is an issue described in #8040 this should fix it
- The alerts slice is shared across multiple goroutines (since send() is
called concurrently).
- `alerts[:0]` creates a new slice header, but it still references the
same underlying array.
- Appending (append(alertsToSend, a)) modifies the underlying array,
which may also be used by another goroutine.
Solution: Use a separate slice copy for each goroutine.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Evgeny Kuzin <evgeny@hudson-trading.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
Previosly the parsing of the input stream was stopped after the first parse error.
This isn't what most users expect when ingesting JSON lines in a stream where some JSON lines may be invalid.
### Describe Your Changes
- Migrated build process from Webpack (CRA) to Vite
Reason for migration: `create-react-app` has been
[deprecated](b532a58792)
and contains outdated dependencies that haven’t been updated for over
two years, leading to security vulnerabilities. Additionally, build
speed improved by more than 2x.
- Updated dependencies and fixed TypeScript typings in accordance with
the updates
b532a58792
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
vmauth uses 'lib/httpserver' for serving HTTP requests. This server
unconditionally defines built-in routes (such as '/metrics',
'/health', etc). It makes impossible to proxy `HTTP` requests to backends with the same routes.
Since vmauth's httpserver matches built-in route and return local
response.
This commit adds new flag `httpInternalListenAddr` with
default empty value. Which removes internal API routes from public
router and exposes it at separate http server.
For example given configuration disables private routes at `0.0.0.0:8427` address and serves it at `0.0.0.0:8426`:
`./bin/vmauth --auth.config=config.yaml --httpListenAddr=:8427 --httpInternalListenAddr=127.0.0.1:8426`
Related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6468
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345
### Describe Your Changes
Explicitly mention that `--vm-rate-limit` is used for each individual
worker defined by `--vm-concurrency`.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
metricsql: bump to v0.83.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703
The update also returns an error if metric name is specified twice in
metrics selector.
For example, `foo{__name__="bar"}` is not allowed anymore. It would
successfully parse before
this change, but it won't satisfy the search filter any way. So it had
no sense in supporting this. This is why some test cases were removed.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Mnetion explicitly that `remoteWrite.concurrency` deopends on number
of available CPU cores. Updated docs to rm auto-printed default value.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8151
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Fixed an issue where dropdown menus were not visible in the Group View
settings.
Ref issue: #8153
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
At this time `bufferedwriter` [silently ignores connection close
errors](78eaa056c0/lib/bufferedwriter/bufferedwriter.go (L67)).
It may be very convenient in some situations (to not log such
unimportant errors), but it's too implicit and unsafe for the others.
For example, if you close [export
API](https://docs.victoriametrics.com/#how-to-export-time-series) client
connection in the middle of communication, VictoriaMetrics won't notice
it and will start to hog CPU by exporting all the data into nowhere
until it process all of them. If you'll make a few retries, it will be
effectively a DoS on the server.
This commit replaces this implicit error suppressing with explicit error
handling which fixes the issue with export API.
Issue was introduced at e78f3ac8ac
### Describe Your Changes
- Added the `fields_limit` parameter for the `hits` query to limit the
number of returned fields. This reduces the response size and decreases
memory usage.
#### Legend Menu:
A context menu has been added for legend items, which includes:
1. Metric name
2. **Copy _stream name** - copies the full stream name in the format
`{field1="value1", ..., fieldN="valueN"}`.
3. **Add _stream to filter** - adds the full stream value to the current
filter:
`_stream: {field1="value1", ..., fieldN="valueN"} AND (old_expr)`.
4. **Exclude _stream from filter** - excludes the stream from the
current filter:
`(NOT _stream: {field1="value1", ..., fieldN="valueN"}) AND (old_expr)`.
5. List of fields with options:
- Copy as `field: "value"`.
- Add to filter: `field: "value" AND (old_expr)`.
- Exclude from filter: `-field: "value" AND (old_expr)`.
6. Total number of hits for the stream.
Related issue: #7552
<details>
<summary>UI Demo - Legend Menu</summary>
<img width="400"
src="https://github.com/user-attachments/assets/ee1954b2-fdce-44b4-a2dc-aa73096a5414"/>
<img width="400"
src="https://github.com/user-attachments/assets/19d71f04-c207-4143-a176-c5f221592e3d"/>
</details>
---
#### Legend:
1. Displays the total number of hits for the stream.
2. Added hints below the legend for total hits and graph interactions.
3. Click behavior is now the same as in other vmui charts:
- `click` - shows only the selected series.
- `click + ctrl/cmd` - hides the selected series.
<details>
<summary>UI Demo - Legend</summary>
before:
<img
src="https://github.com/user-attachments/assets/18270842-0c39-4f63-bcda-da62e15c3c73"/>
after:
<img
src="https://github.com/user-attachments/assets/351cad3a-f763-4b1d-b3be-b569b5472a7c"/>
</details>
---
#### Tooltip:
1. The `other` label is moved to the end, and others are sorted by
value.
2. Values are aligned to the right.
3. Labels are truncated and always shown in a single line for better
readability; the full name is available in the legend.
<details>
<summary>UI Demo - Tooltip</summary>
| before | after |
|----------|----------|
| <img
src="https://github.com/user-attachments/assets/adccff38-e2e6-46e4-a69e-21381982489c"/>
| <img
src="https://github.com/user-attachments/assets/81008897-d816-4aed-92cb-749ea7e0ff1e"/>
|
</details>
---
#### Group View (tab Group):
Groups are now sorted by the number of records in descending order.
<details>
<summary>UI Demo - Group View</summary>
before:
<img width="800"
src="https://github.com/user-attachments/assets/15b4ca72-7e5d-421f-913b-c5ff22c340cb"/>
after:
<img width="800"
src="https://github.com/user-attachments/assets/32ff627b-6f30-4195-bfe7-8c9b4aa11f6b"/>
</details>
### Describe Your Changes
Added saving column settings in the URL for the table view. See #7662
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
This commit fixes incorrect behaviour when pressing `Enter` did not execute the query, and
`Shift+Enter` did not insert a new line.
- The issue occurred when autocomplete was disabled.
- This problem affected the query editor in both the VictoriaMetrics UI
and VictoriaLogs UI.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8058
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8072
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Also always initialize Query.timestamp with the timestamp from the lexer.
This should avoid potential problems with relative timestamps inside inner queries.
For example, the `_time:1h` filter in the following query is correctly executed
relative to the current timestamp:
foo:in(_time:1h | keep foo)
Sync.Pool for readTrackingBody was added in order to reduce potential
load on garbage collector. But golang net/http standard library does not
allow to reuse request body, since it closes body asynchronously after
return. Related issue: https://github.com/golang/go/issues/51907
This commit removes sync.Pool in order to fix potential panic and data
race at requests processing.
Affected releases:
- all releases after v1.97.7
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8051
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
optimize for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6226
for user who set `*AuthKey` flag, they will receive new response in
body:
```go
// query arg not set
The provided authKey '' doesn't match -search.resetCacheAuthKey
// incorrect query arg
The provided authKey '5dxd71hsz==' doesn't match -search.resetCacheAuthKey
```
previously, they receive:
```
The provided authKey doesn't match -search.resetCacheAuthKey
```
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
`doInternal` has adaptive staleness detection mechanism. It is
calculated using timestamp distance between samples in selected list of
samples. It is dynamic because VM can store signals from many sources
with different samples resolution. And while it works for most of cases,
there are edge cases for rollup functions that are comparing values
between windows: increase, increase_pure, delta.
The edge case 1.
There was a gap between series because of the missed scrape or two. In
this case staleness will trigger and increase-like functions will assume
the value they need to compare with is 0. In result, this could produce
spikes for a flappy data - see
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894 This
problem was solved by introducing a `realPrevValue` field -
1f19c167a4.
It stores the closest real sample value on selected interval and is used
for comparison when samples start after the gap.
The edge case 2.
`realPrevValue` doesn't respect staleness interval. In result, even if
gap between samples is huge (hours), the increase-like functions will
not consider it as a new series that started from 0. See
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8002.
Covering both edge cases is tricky, because `realPrevValue` has to
respect and not respect the staleness interval in the same time. In
other words, it should be able to ignore periodic missing gaps, but
reset if the gap is too big. While "too big gap" can't be figured out
empirically, I suggest using `-search.maxStalenessInterval` for this
purpose. If `-search.maxStalenessInterval` is set to 0 (default), then
`realPrevValue` ignores staleness interval. If
`-search.maxStalenessInterval` is > 0, then `realPrevValue` respects it
as a staleness interval.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
New version has additional checks and reduced resource consumption, so
it doesn't timeout for our internal repos.
To make linter happy, I addressed "redefinition of the built-in
function" lint error.
----
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously columns with negative int64 values were stored either as float64 or string
depending on whether the negative int64 values are bigger or smaller than -2^53.
If the integer values are smaller than -2^53, then they are stored as string, since float64 cannot
hold such values without precision loss. Now such values are stored as int64.
This should improve compression ratio and query performance over columns with negative int64 values.
1. Do not copy every line from LineReader.buf to LineReader.Line - just refer the line at LineReader.buf.
2. Do not copy the next found line to the beginning of LineReader.buf - just track the next line start index with LineReader.bufOffset.
This reduces memory copying when many lines are read into LineReader.buf by a single read() syscall.
When vmselect process a rollup function it fetches all the raw samples
on requested `start-end` interval of the query. It then loops through
the raw samples, picks the range of the samples based on provided `step`
interval and invokes a rollup function for each of the picked ranges of
samples.
During this processing, vmselect always populates the `realPrevValue`
field with the closest previous raw sample value before the picked range
of samples. This `realPrevValue` is used by rollup functions like
increase_pure or delta to decide whether the counter change happened or
not. For example, we get the counter value == 1. If we've seen this
counter before and its value was also 1 - then no change happened. If we
didn't see it before, then this counter should have started with value=0
and we need to account for `1-0=1` change. All this is required to deal
with situations when scrapes are missing or `step` is too small.
However, vmselect doesn't check how "old" is the `realPrevValue`. In
other words, it doesn't respect the staleness interval when picking it.
In result, depending on the `start` and `end` params, vmselect can use
`realPrevValue` which is a couple of hours old and is unlikely to be a
temporary scrape fail. In result, some increases can be incorrectly
ingnored by vmselect.
This change makes sure that vmselect doesn't populate `realPrevValue`
with samples that are older than staleness interval.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
-------------------
To reproduce, create a dataset with one metric `foo` which has samples
with value=1 on interval of couple of hours and resolution 15s, and a
gap for an hour in the middle:
<img width="769" alt="image"
src="https://github.com/user-attachments/assets/a39b2740-b741-45f8-ad18-093b7c57c3b3"
/>
Then run `increase(foo[1m])` expression on this time range (disable
cache):
<img width="1472" alt="image"
src="https://github.com/user-attachments/assets/463cece1-f359-4c75-a96c-60092a31cab2"
/>
In result, there will be one increase on the beginning of the series.
And no increase after the gap. Then change the time range so it starts
in the middle of the gap:
<img width="1505" alt="image"
src="https://github.com/user-attachments/assets/f4a460c3-9fd1-4ec7-ab47-15e716ec1019"
/>
Now, there is an increase>0 because the `realPrevValue` wasn't
populated. This is wrong, because it hides the increase of the series.
With the fix, the original increase query on full time range should show
2 increases:
<img width="1492" alt="image"
src="https://github.com/user-attachments/assets/aa9d8a6b-7b22-41f6-9eb9-83b3113a6982"
/>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Hint allows to choose type of cache to be used for index search:
- in-memory parts are storing recently ingested samples and should use
main cache. This improves ingestion speed and cache hit ration for
queries accessing recently ingested samples.
- merges of file parts is performed in background, using a separate
cache allows avoiding pollution of the main cache with irrelevant
entries.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
This commit makes configurable interval for checking if final dedup
process for the historical data should be started. It allows to spread
resource utilisation for multiple vmstorage/vmsingle instances in time.
Since final dedup may add additional preasure on disk, backup systems
and make cluster less stable. Storage unconditionally adds 25% jitter to
the provided value, it should simplify configuration management at
Kubernetes ecosystem. Because Kubernetes application pods must have the
same configuration.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7880
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
commitL c7fc0d0d2f enabled skipping alerts
in case there is no labels present for an alert. This made clause which
was adding a comma for the JSON list incorrect as it is not possible to
determine if the next alert will be skipped or not.
This fix renders all alert labels in advance allowing properly format
JSON payload for Alertmanager notification.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7985
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
