mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

vmctl: add --backoff-retries, --backoff-factor, --backoff-min-duration global command-line flags (#6639 )

### Describe Your Changes

Added `--vm-backoff-retries`, `--vm-backoff-factor`,
`--vm-backoff-min-duration` and `--vm-native-backoff-retries`,
`--vm-native-backoff-factor`, `--vm-native-backoff-min-duration`
command-line flags to the `vmctl` app. Those changes will help to
configure the retry backoff policy for different situations.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6622
### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

2024-08-03 19:12:48 +02:00

174 KiB

Raw Blame History

sort

weight

title

aliases

100

CHANGELOG

docs

identifier	parent	weight
vm-changelog	victoriametrics	100

/CHANGELOG.html

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation runs under the constant load generated by prometheus-benchmark and is used for testing latest releases.

tip

FEATURE: vmctl: add --vm-backoff-retries, --vm-backoff-factor, --vm-backoff-min-duration and --vm-native-backoff-retries, --vm-native-backoff-factor, --vm-native-backoff-min-duration command-line flags. These flags allow to change backoff policy config for import requests to VictoriaMetrics. See this issue.

v1.102.1

Released at 2024-08-01

v1.102.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.102.x line will be supported for at least 12 months since v1.102.0 release

Update note 1: vmauth HTTP response code has changed from 503 to 502 for a case when all upstream backends were not available. This was changed to align vmauth behaviour with other well-known reverse-proxies behaviour.

SECURITY: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2. See alpine 3.20.2 release notes.
FEATURE: vmauth: add keep_original_host option, which can be used for proxying the original Host header from client request to the backend. By default, the backend host is used as Host header when proxying requests to the configured backends. See these docs.
FEATURE: vmauth: now returns HTTP 502 status code when all upstream backends are not available. Previously, it returned HTTP 503 status code. This change aligns vmauth behavior with other well-known reverse-proxies behavior.
FEATURE: vmagent dashboard: add Scrape duration 0.99 quantile panel to show the 99th quantile of scrape duration in seconds. This should help identifying vmagent instances that experiences too high scraping durations.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: Fixes panic if incorrect metrisql expression passed to the prettify-query API. See this issue for details.
BUGFIX: all VictoriaMetrics components: validate files specified via -tlsKeyFile and -tlsCertFile cmd-line flags on the process start-up. Previously, validation happened on the first connection accepted by HTTP server. See this issue for the details. Thanks to @yincongcyincong for the pull request.
BUGFIX: vmauth: properly proxy requests to backend urls ending with / if the original request path equals to /. Previously the trailing / at the backend path was incorrectly removed. For example, if the request to http://vmauth/ is configured to be proxied to url_prefix=http://backend/foo/, then it was proxied to http://backend/foo, while it should go to http://backend/foo/.
BUGFIX: vmauth: fix cannot read data after closing the reader error when proxying HTTP requests without body (aka GET requests). The issue has been introduced in v1.102.0 in this commit.
BUGFIX: vmui: fix auto-completion triggers for metric and label names, and disable it after specific characters. See this issue and these comments.
BUGFIX: vmui: fix automatic addition of quotes when inserting label values from autocomplete suggestions. See this issue.

v1.102.0

Released at 2024-07-17

Update note 1: support for snap packages was removed due to lack of interest from community. See this pull request for details. Please read about supported package types here.

Update note 2: stream aggregation config now prevents setting multiple identical outputs. For example, outputs: [total, total] will fail the validation phase. In addition, outputs: ["quantiles(0.5)", "quantiles(0.9)"] will fail the validation as well - use outputs: ["quantiles(0.5, 0.9)"] instead.

SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in Go1.22.5.
SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See alpine 3.20.1 release notes.
FEATURE: streaming aggregation: expose the following metrics at /metrics page of vmagent and single-node VictoriaMetrics:
- vm_streamaggr_matched_samples_total - the number of input samples matched by the corresponding aggregation rule
- vm_streamaggr_output_samples_total - the number of output samples produced by the corresponding aggregation rule
- vm_streamaggr_samples_lag_seconds - histogram with the lag between the current time and the timestamp seen in the aggregated input samples
FEATURE: steaming aggregation: add new labels to vm_streamaggr_* metrics:
- name - the name of the streaming aggregation rule, which can be configured via name option - see these docs.
- url - -remoteWrite.url for the corresponding -remoteWrite.streamAggr.config
- path - path to the corresponding streaming aggregation config file
- position - the position of the aggregation rule in the corresponding streaming aggregation config file
FEATURE: streaming aggregation: prevent having duplicated aggregation function as outputs in one aggregation config. It also prevents using outputs: ["quantiles(0.5)", "quantiles(0.9)"] instead of outputs: ["quantiles(0.5, 0.9)"], as the former has higher computation cost for producing the same result.
FEATURE: vmagent and Single-node VictoriaMetrics: add -graphite.sanitizeMetricName command-line flag for sanitizing metrics ingested via Graphite protocol. See this issue.
FEATURE: vmagent: yandexcloud_sd_configs: add support for obtaining IAM token in GCE format additionally to the deprecated Amazon EC2 IMDSv1 format. See this issue.
FEATURE: vmalert: make -replay.timeTo optional in replay mode. When omitted, the current timestamp will be used. See this issue.
FEATURE: vmauth: reduce CPU usage when proxying data ingestion requests.
FEATURE: vmauth: allow disabling request body caching with -maxRequestBodySizeToRetry=0. See this issue. Thanks to @shichanglin5 for the pull request.
FEATURE: vmauth: allow overriding Host header with backend host before sending the request to the configured backend. See these docs and this issue.
FEATURE: VictoriaMetrics cluster: do not retry RPC calls to vmstorage nodes if complexity limits were exceeded.
FEATURE: vmbackup and vmrestore: add support for Azure Managed Identity and default credentials lookup. See these docs and this issue for the details. Thanks to @justinrush for the pull request.
FEATURE: vmbackup and vmrestore: allow overriding Azure storage domain when performing backups via AZURE_STORAGE_DOMAIN environment variable. See these docs and this issue. Thanks to @justinrush for the pull request.
FEATURE: vmui: show compacted result in the JSON tab for query results. See this issue.
FEATURE: dashboards: add Grafana dashboard and alerting rules for vmauth dashboard. See this issue for details.
FEATURE: vmagent dashboard: stream aggregation section: add graphs based on newly exposed streaming aggregation metrics.
BUGFIX: MetricsQL: properly calculate histogram_quantile over Prometheus buckets with inconsistent values. See this comment and this pull request. Updates this issue.
BUGFIX: vmagent: fix panic when using multiple topics with the same name when ingesting metrics from Kafka. See this issue for the details.
BUGFIX: vmalert: fix incorrect redirection in WebUI of vmalert. See this issue and this pull request.
BUGFIX: vmgateway: properly apply read and write based rate limits. See this issue for details.
BUGFIX: vmbackupmanager: fix vm_backup_last_run_failed metric not being properly initialized during startup. Previously, it could imply an error even if the backup have been completed successfully. See this issue for the details.
BUGFIX: vmui: fix input cursor position reset in modal settings. See this pull request.
BUGFIX: docker-compose: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana.

v1.102.0-rc2

Released at 2024-06-24

Update note 1: the --vm-disable-progress-bar command-line flag at vmctl was deprecated. Use --disable-progress-bar instead.

Update note 2: *.passwordFile and similar flags are no longer trimming trailing whitespaces at the end of content. Make sure to update the templating of password files or HTTP endpoints to not include trailing whitespaces before the upgrade. See this PR for the details.

FEATURE: alerts-vmagent: add new alerting rules StreamAggrFlushTimeout and StreamAggrDedupFlushTimeout to notify about issues during stream aggregation.
FEATURE: dashboards/vmagent: add row Streaming aggregation with panels related to streaming aggregation process.
FEATURE: vmauth: add -idleConnTimeout flag set to 50s by default. It should reduce the probability of broken pipe or connection reset by peer errors in vmauth logs.
FEATURE: vmauth: add automatic retry for requests to backend for trivial network errors, such as broken pipe and connection reset for requests to the configured backends.
FEATURE: vmagent: increase default value of -promscrape.maxDroppedTargets command-line flag to 10_000 from 1000. This makes it easier to track down large number of dropped targets. See this issue.
FEATURE: vmagent: add max_scrape_size option to scrape config for setting custom limit on the response size target can send. The new automatically generated metric scrape_response_size_bytes is added to reflect the response size of the target. See these issues: 1, 2, 3, 4.
FEATURE: vmsingle: check for ranged vector arguments in non-rollup expressions when -search.disableImplicitConversion or -search.logImplicitConversion are enabled. For example, sum(up[5m]) or absent(up[5m]) will fail to execute if these flags are set.
FEATURE: vmsingle: validate that rollup expressions has ranged vector arguments passed when -search.disableImplicitConversion or -search.logImplicitConversion are enabled. For example, rate(metric) or count_over_time(metric) will fail to execute if these flags are set.
FEATURE: vmalert-tool: support file path with hierarchical patterns and regexpes, and http url in unittest cmd-line flag -files, e.g. -files="http://<some-server-addr>/path/to/rules" or -files="dir/**/*.yaml".
BUGFIX: all VictoriaMetrics components: properly calculate process_resident_memory_bytes metric for OS with non-default value of memory page size. See this issue for details.
BUGFIX: all VictoriaMetrics components: prioritize -configAuthKey and -reloadAuthKey over -httpAuth.* settings. This change aligns behavior of mentioned flags with other auth flags like -metricsAuthKey, -flagsAuthKey, -pprofAuthKey. Check this issue.
BUGFIX: all VictoriaMetrics components: do not trim trailing spaces when reading content from *.passwordFile and similar flags. Previously, trailing spaces were trimmed from the content of the password file, which could lead to unexpected authentication errors.
BUGFIX: vmctl: add --disable-progress-bar global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. --vm-disable-progress-bar command-line flag is deprecated and will be removed in the future releases. See this issue.
BUGFIX: stream aggregation: prevent rate_sum and rate_avg producing NaN results for stale time series. Before, when series matched for aggregation became stale or weren't updated during aggregation interval, the rate_sum or rate_avg could produce data point with NaN value. During visualization, such aggregation results would be displayed as gaps in time series.
BUGFIX: vmalert: fix path for system links printed on default vmalert's UI page when -http.pathPrefix is set. See this issue.
BUGFIX: vmalert: exit replay mode with non-zero code if generated samples are not successfully written into remoteWrite url. See this issue.
BUGFIX: vmbackup: properly configure authentication with S3 when -configFilePath cmd-line flag is specified. See this issue.
BUGFIX: vmalert enterprise: properly configure authentication with S3 when -s3.configFilePath cmd-line flag is specified for reading rule configs.
BUGFIX: vmalert: properly specify oauth2 ClientSecret when configuring authentication for notifier.url. See this issue for details. Thanks to @yincongcyincong for the pull request.
BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: add validation for the max value specified for -retentionPeriod. See this issue for details.
BUGFIX: vmui: copy row button in Table view produces unexpected result. See this issue and pull request.
BUGFIX: vmalert-tool: prevent hanging when processing groups without rules. See this issue.
BUGFIX: vmauth: fix discovering backend IPs when url_prefix contains hostname with srv+ prefix. Thanks to @shichanglin5 for the pull request.

v1.102.0-rc1

Released at 2024-06-07

Update note 1: the -remoteWrite.multitenantURL command-line flag at vmagent was removed starting from this release. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

Update note 2: the -streamAggr.dropInputLabels command-line flag at vmagent was renamed to -remoteWrite.streamAggr.dropInputLabels. -streamAggr.dropInputLabels is now used for global streaming aggregation.

Update note 3: the -maxLabelValueLen command-line flag default value was changed from 16KiB to 4KiB. It may lead to truncating of labels with too long values.

SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.
SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.
SECURITY: add release images built from scratch image. Such images could be more preferable for using in environments with higher security standards. See this pull request.
FEATURE: dashboards/single: support selecting of multiple instances on the dashboard. See this issue for details.
FEATURE: dashboards/single: properly display version in the Stats row for the custom builds of VictoriaMetrics.
FEATURE: dashboards/single: add Network Usage panel to Resource Usage row.
FEATURE: dashboards/operator, dashboards/backupmanager and dashboard/tenant-statistic: update dashboard to be compatible with Grafana 10+ version.
FEATURE: dashboards/cluster: add new panel Concurrent selects to vmstorage row. The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details.
FEATURE: dashboards: use $__interval variable for offsets and look-behind windows in annotations. This should improve precision of restarts and version change annotations when zooming-in/zooming-out on the dashboards.
FEATURE: vmagent: support aggregation and deduplication configs before replicating data to configured -remoteWrite.url destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See this issue.
FEATURE: vmagent: add service discovery support for Vultr. See these docs and this issue.
FEATURE: vmagent: allow specifying -remoteWrite.disableOnDiskQueue command-line flag per each -remoteWrite.url. If multiple -remoteWrite.disableOnDiskQueue command-line flags are configured, then the -remoteWrite.dropSamplesOnOverload is automatically set to true, so samples are automatically dropped if they cannot be sent to the corresponding -remoteWrite.url in a timely manner. See this pull request. Thanks to @rbizos for implementation!
FEATURE: vmagent: add path and url labels to vmagent_remotewrite_push_failures_total and vmagent_remotewrite_samples_dropped_total metrics, so the number of failed pushes and dropped samples can be tracked per each -remoteWrite.url.
FEATURE: stream aggregation: add rate_sum and rate_avg aggregation outputs.
FEATURE: stream aggregation: reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See this pull request.
FEATURE: vmalert: add -datasource.idleConnTimeout, -remoteWrite.idleConnTimeout and -remoteRead.idleConnTimeout flags. These flags are set to 50s by default and should reduce the probability of broken pipe or connection reset by peer errors in vmalert logs. See this issue for details.
FEATURE: vmalert: add auto request retry for trivial network errors, such as broken pipe and connection reset for requests to remoteRead, remoteWrite and datasource URLs. See this issue for details.
FEATURE: vmalert: reduce CPU usage when evaluating high number of alerting and recording rules.
FEATURE: vmalert: speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See this issue.
FEATURE: vmalert: support reading DNS SRV records in -datasource.url, -remoteWrite.url and -remoteRead.url command-line option. For example, -remoteWrite.url=http://srv+victoria-metrics automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See this feature request.
FEATURE: vmbackup, vmrestore, vmbackupmanager: add -s3TLSInsecureSkipVerify command-line flag for skipping TLS certificates verification when connecting to S3 endpoint.
FEATURE: expose metric vm_indexdb_items_dropped_total to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in reason label of the exposed metric. This change also comes with a new alerting rule to track changes of this metric.
FEATURE: alerts-health: add new alerting rules TooLongLabelValues and TooLongLabelNames to notify about truncation of label values or names respectively.
FEATURE: stream aggregation: expose vm_streamaggr_ignored_samples_total counters at /metrics page, which can be used for detecting amount of too old or NaN valued ignored samples. Expose also vm_streamaggr_samples_lag_seconds histogram to monitor aggregated samples lag.
FEATURE: stream aggregation: improve filtering speed of the received data samples if match field is matching only metric name.
FEATURE: VictoriaMetrics cluster: adds -relabelConfigCheckInterval flag and /-/reload endpoint for reloading relabeling rules. See this pull request for details.
BUGFIX: vmui: fix bug that prevents the first query trace from expanding on click event. See this issue. The issue was introduced in v1.100.0 release.
BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.
BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.
BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.
BUGFIX: vmagent: prevent potential panic during stream aggregation when more than one -remoteWrite.url command-line flags are passed to vmagent together with non-zero -remoteWrite.streamAggr.dedupInterval command-line flag. See this issue.
BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.
BUGFIX: stream aggregation: set correct suffix <output>_prometheus for aggregation outputs increase_prometheus and total_prometheus. Before, outputs total and total_prometheus or increase and increase_prometheus had the same suffix.
BUGFIX: stream aggregation: prevent from excessive resource usage when stream aggregation config file is empty.
BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.
BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it's set to minimum supported value 1s.
BUGFIX: Single-node VictoriaMetrics and vminsert in VictoriaMetrics cluster: reduce the default value for -maxLabelValueLen command-line flag from 16KiB to 1KiB. This should prevent from issues like this one when time series with too long labels are ingested into VictoriaMetrics.
BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.
BUGFIX: dashboards: fix AnnotationQueryRunner error in Grafana when executing annotations query against Prometheus backend. See this issue for details.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.
BUGFIX: vmalert-tool: fix float values template in input_series. See this issue.
BUGFIX: vmalert: fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested.
BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.
DEPRECATION: vmagent: removed deprecated -remoteWrite.multitenantURL flag from vmagent. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

v1.101.0

Released at 2024-04-26

FEATURE: MetricsQL: support filtering by multiple numeric constants via q == (C1, ..., CN) and q != (C1, ..., CN) syntax. For example, status_code == (200, 201, 300) returns status_code metrics with any of 200, 201 or 300 values, while status_code != (400, 404, 500) returns status_code metrics with all the values except of 400, 404 and 500.
FEATURE: VictoriaMetrics cluster: add support for fault domain awareness to vmselect. It can be configured to return full responses if up to -globalReplicationFactor - 1 fault domains (aka vmstorage groups) are unavailable. See this feature request and these docs.
FEATURE: all VictoriaMetrics enterprise components: add support for automatic issuing of TLS certificates for HTTPS server at -httpListenAddr via Let's Encrypt service. See these docs and this feature request.
FEATURE: vmsingle: support disable or log implicit conversions for subquery with cmd-line flags -search.disableImplicitConversion and -search.logImplicitConversion. See this feature request.
FEATURE: vmagent: support data replication additionally to sharding among remote storage systems if -remoteWrite.shardByURLReplicas=N command-line flag is set additionally to -remoteWrite.shardByURL command-line flag, where N is desired replication factor. This allows setting up data replication among failure domains when the replication factor is smaller than the number of failure domains. See these docs and this feature request.
FEATURE: vmagent: reduce CPU usage when sharding among remote storage systems is enabled.
FEATURE: stream aggregation: reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data.
FEATURE: vmagent: support DNS SRV addresses in -remoteWrite.url command-line option and in scrape target urls. For example, -remoteWrite.url=http://srv+victoria-metrics/api/v1/write automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See these docs and this feature request.
FEATURE: stream aggregation: allow skipping first N aggregation intervals via cmd-line flag -streamAggr.ignoreFirstIntervals for Single-node VictoriaMetrics or -remoteWrite.streamAggr.ignoreFirstIntervals for vmagent. See more details here.
FEATURE: vmauth: support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside url_prefix urls in the form http://srv+addr/path, where the addr is the DNS SRV address, which is automatically resolved to hostnames with TCP ports. See these docs for details.
FEATURE: vmauth: support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See these docs.
FEATURE: vmauth: support regex matching when routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.
FEATURE: vmui: optimize auto-suggestion performance for metric names when the database contains big number of unique time series.
FEATURE: vmui: in the Select component, user-entered values are now preserved on blur if they match options in the list.
FEATURE: vmui: auto-suggestion triggers at any cursor position in the query input. See this issue.
FEATURE: vmui: update error messages on the Query page for enhanced clarity. See this pull request.
BUGFIX: downsampling: skip unnecessary index lookups if downsampling wasn't set for ENT versions of VictoriaMetrics. Before, users of VictoriaMetrics ENT could have experience elevated CPU usage even if no downsampling was configured. The issue was introduced in v1.100.0.
BUGFIX: downsampling: properly populate downsampling metadata for data parts created by VictoriaMetrics ENT versions lower than v1.100.0. The bug could trigger the downsampling actions for parts that were downsampled already. This bug doesn't have any negative effect apart from spending extra CPU resources on the repeated downsampling. The issue was introduced in v1.100.0.
BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.
BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.
BUGFIX: vmalert: fix links with anchors in vmalert's UI. Starting from v1.99.0 vmalert could ignore anchors pointing to specific rule groups if search param was present in URL.
BUGFIX: vmauth: don't treat concurrency limit hit as an error of the backend. Previously, hitting the concurrency limit would increment both vmauth_concurrent_requests_limit_reached_total and vmauth_user_request_backend_errors_total counters. Now, only concurrency limit counter is incremented. Updates this issue.

v1.100.1

Released at 2024-04-11

Update note 1: This release contains the issue which could lead to extra CPU usage of storage component in ENT distribution of VictoriaMetrics. The issue is caused by downsampling per distinct sets of time series feature introduced in 1.100.0 ENT version. Please, upgrade to v1.101.0 or rollback to v1.98.0 ENT version if you're affected.

Update note 2: When upgrading to this release from v1.99.0 or v1.100.0 it is recommended to reset caches stored on disk according to these docs.

FEATURE: vmbackupmanager: allow specifying custom backup interval via -backupInterval command-line flag. See this feature request.
BUGFIX: properly register new entries in IndexDB when many new time series are ingested into VictoriaMetrics in a short period of time. See this and this issues. The bug has been introduced in v1.99.0.
BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.100.0

Released at 2024-04-04

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

Update note 1: the -datasource.lookback command-line flag at vmalert is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay option. See this issue for more details.

SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.
FEATURE: downsampling: add ability to configure distinct downsampling per distinct sets of time series and/or tenants. See this feature request and these docs.
FEATURE: vmauth: allow discovering ip addresses for backend instances hidden behind a shared hostname, via discover_backend_ips: true option. This allows evenly spreading load among backend instances. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP request headers via src_headers option at url_map. See these docs.
FEATURE: vmauth: add ability to read auth tokens from arbitrary HTTP request headers via -httpAuthHeader command-line flag. Previously auth tokens were read only from Authorization HTTP request header. See these docs for details.
FEATURE: vmauth: add ability to authorize by opaque HTTP request header value via auth_token option in -auth.config.
FEATURE: stream aggregation: reduce memory usage by up to 5x when aggregating over big number of unique time series. The memory usage reduction is most visible when stream deduplication is enabled.
FEATURE: stream aggregation: allow using -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval command-line flags without the need to specify -streamAggr.config and -remoteWrite.streamAggr.config. See these docs.
FEATURE: stream aggregation: add -streamAggr.dropInputLabels command-line flag, which can be used for dropping the listed labels from input samples before applying stream de-duplication and aggregation. This is faster and easier to use alternative to input_relabel_configs. See these docs.
FEATURE: stream aggregation: add dedup_interval option, which allows configuring individual deduplication intervals per each stream aggregation config.
FEATURE: stream aggregation: use the same logic in stream deduplication as in the deduplication at VictoriaMetrics. See this issue.
FEATURE: stream aggregation: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs. Thanks to @edma2 for the pull request.
FEATURE: stream aggregation: add an ability to ignore input samples with old timestamps outside the current aggregation interval. See these docs for details.
FEATURE: stream aggregation: add keep_metric_names option, which can be set at stream aggregation config in order to keep the original metric names in the output aggregated samples instead of using the default output metric naming scheme.
FEATURE: stream aggregation: align the time of aggregated data flush to the specified aggregation interval. For example, if interval is set to 1m, then the aggregated data will be flushed at the end of every minute. The alignment can be disabled by setting no_align_flush_to_interval: true option at stream aggregation config. See these docs for details.
FEATURE: stream aggregation: add unique_samples output, which can be used for calculating the number of unique sample values over the given interval.
FEATURE: stream aggregation: add increase_prometheus and total_prometheus outputs, which can be used for increase and total aggregations when the first sample of every new time series must be ignored.
FEATURE: stream aggregation: expose vm_streamaggr_flush_timeouts_total and vm_streamaggr_dedup_flush_timeouts_total counters at /metrics page, which can be used for detecting flush timeouts for stream aggregation states. Expose also vm_streamaggr_flush_duration_seconds and vm_streamaggr_dedup_flush_duration_seconds histograms for monitoring the real flush durations of stream aggregation states.
FEATURE: vmui: improve trace display for better visual separation of branches. See this pull request.
FEATURE: vmagent: ability to limit the ingestion rate via -maxIngestionRate command-line flag. See this pull request.
FEATURE: vmagent: use the provided -remoteWrite.tlsServerName as Host header in requests to -remoteWrite.url. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and the pull request.
FEATURE: vmagent: add -remoteWrite.shardByURL.ignoreLabels command-line flag, which can be used for specifying the ignored list of labels when sharding by -remoteWrite.url is enabled. Thanks to @edma2 for the idea and the pull request.
FEATURE: vmagent: automatically reload updated root CA certificates from files without the need to restart vmagent. See this issue.
FEATURE: optimize /api/v1/labels and /api/v1/label/.../values when match[] filters contains metric name. For example, /api/v1/label/instance/values?match[]=up now works much faster than before. See this issue.
FEATURE: vmagent: accept Prometheus remote write protocol data at /api/v1/push endpoint. This simplifies writing data to vmagent from Grafana Mimir. Thanks to @edma2 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for native protocol. See this feature request. Thanks to @khushijain21 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for VictoriaMetrics destination specified via --vm-* cmd-line flags used in InfluxDB, Remote Read protocol, OpenTSDB, Prometheus and Promscale migration modes.
FEATURE: vmctl: split explore phase in vm-native mode by time intervals when --vm-native-step-interval is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See this issue.
FEATURE: vmgateway: add -logInvalidAuthTokens command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See this feature request.
FEATURE: graphite: add support for aggregateSeriesLists, diffSeriesLists, multiplySeriesLists and sumSeriesLists functions. Thanks to @rbizos for the pull request.
FEATURE: OpenTelemetry: add -opentelemetry.usePrometheusNaming command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See these docs and this issue.
BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. The probability of this bug has been increased significantly after v1.99.0 because of optimizations related to registering new time series. See this and this issue.
BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.
BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.
BUGFIX: do not drop match[] filter at /api/v1/series if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set, since missing match[] filter breaks /api/v1/series requests.
BUGFIX: vmagent: return proper responses for AWS Firehose requests according to these docs. See this pull request and this issue.
BUGFIX: vmctl: properly parse TLS key and CA files for InfluxDB and OpenTSDB migration modes.
BUGFIX: vmui: fix VictoriaLogs UI query handling to correctly apply _time filter across all queries. See this issue.
BUGFIX: vmui: fix issue where step parameter wasn't updated after moving the focus outside the step input field in Firefox.
BUGFIX: vmui: fix freezing when pasting a query with autocomplete enabled. See this issue.
BUGFIX: vmui: fix auto-suggestion trigger issue after whitespace char. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with -search.maxLabelsAPIDuration duration. Before, -search.maxExportDuration value was used by mistake. The bug has been introduced in v1.99.0. Thanks to @kbweave for the pull request.
BUGFIX: properly wait for force merge to be completed during the shutdown. See this issue for the details.
BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.
BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.
BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.

v1.99.0

Released at 2024-03-01

This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please rollback to v1.98.0 or upgrade to v1.101.0.

FEATURE: MetricsQL: propagate label filters via all the label manipulation functions. For example, label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"} is now transformed to label_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}. This should reduce the amounts of time series processed during query execution.
FEATURE: MetricsQL: add count_values_over_time function. See this feature request.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_last_partition_parts metrics, which show the number of parts in the latest partition. These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_snapshots metric, which shows the current number of snapshots created via snapshot API.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: add -search.ignoreExtraFiltersAtLabelsAPI command-line flag, which can be used for reducing load on VictoriaMetrics when /api/v1/labels, /api/v1/label/.../values or /api/v1/series are queried with too broad extra_filters or extra_label, which match many time series. See these docs for details.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: allow limiting CPU and RAM usage at /api/v1/labels, /api/v1/label/.../values and /api/v1/series on systems with high churn rate via -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries command-line flags. See these docs for details.
FEATURE: vmagent: support reading Amazon CloudWatch metrics in OpenTelemetry format from Amazon Data Firehose.
FEATURE: vmagent: add support for client_id option into kuma_sd_configs in the same way as Prometheus does. See this pull request.
FEATURE: vmagent: add support for enable_compression option in scrape_configs in order to be compatible with Prometheus scrape configs. See this pull request and this feature request. Note that vmagent was always supporting disable_compression option before Prometheus added enable_compression option.
FEATURE: vmctl: support client-side TLS configuration for InfluxDB, Remote Read protocol and OpenTSDB. See this feature request. Thanks to @khushijain21 for pull requests 1, 2, 3.
FEATURE: vmui: preserve WITH templates when clicking the prettify query button at the right side of query input field. See this feature request.
FEATURE: vmui: allow filling gaps on graphs with interpolated lines as Grafana does. See this feature request and this pull request.
FEATURE: vmalert: support filtering by group, rule or labels in vmalert's UI for /groups and /alerts pages. See the pull request by @victoramsantos.
FEATURE: docker-compose: create a separate docker-compose environment for VictoriaLogs installation, including fluentbit and VictoriaLogs Grafana datasource. See these docs for details.
FEATURE: vmbackupmanager: wait for up 30 seconds before making a snapshot for backup if vmstorage is temporarily unavailable. This should prevent from vmbackupmanager termination in this case. See this feature request.
BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.98.0 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
BUGFIX: vmui: fix Popper display issues. See this pull request.
BUGFIX: vmselect: format time with milliseconds precision when __timestamp__:rfc3339 is specified for /api/v1/export/csv API response. See this issue for details.
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.98.0

Released at 2024-02-14

SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at -httpListenAddr (aka mTLS). See these docs and this feature request.
FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple -httpListenAddr command-line flags. For example, ./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428 starts VictoriaMetrics, which accepts incoming http requests at both some-host:12345 and localhost:8428. See this feature request.
FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, -remoteWrite.sendTimeout=',20s,' specifies three -remoteWrite.sendTimeout values - the first one and the last one have default values (30s in this case), while the second one is set to 20s.
FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.
FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka /api/beta/sketches endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request.
FEATURE: vmagent: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning the timeout needed for establishing TLS connections to -remoteWrite.url. Bigger tls handshake timeouts should reduce the probability of http: TLS handshake error from ...: EOF errors at the remote storage side under high load. See this issue.
FEATURE: VictoriaMetrics cluster: add -disableReroutingOnUnavailable command-line flag to vminsert, which can be used for reducing resource usage spikes at vmstorage nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request.
FEATURE: add -search.resetRollupResultCacheOnStartup command-line flag for resetting query cache on startup. See this feature request.
FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example, label_set(q1, "a", "b") + q2{c="d"} is automatically transformed to label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"} now. This should improve performance for such queries. See this issue.
FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.
FEATURE: dashboards/vmagent: add Targets scraped/s stat panel showing the number of targets scraped by the vmagent per-second.
FEATURE: dashboards/all: add new panel CPU spent on GC. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed.
FEATURE: vmalert: support filtering for /api/v1/rules API. See the pull request by @victoramsantos.
FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via -snapshot.tls* cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request.
FEATURE: vmui: add a time picker to the "Logs Explorer" page. See this issue.
BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from /-/reload endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.
BUGFIX: vmui: fix handling invalid timezone. See this issue.
BUGFIX: vmui: fix the bug where the select does not open. See this issue.
BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.
BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.
BUGFIX: dashboards: update Storage full ETA panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request.
BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.

v1.97.6

Released at 2024-07-17

v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release

SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in Go1.22.5.
SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See alpine 3.20.1 release notes.
BUGFIX: vmagent: fix panic when using multiple topics with the same name when ingesting metrics from Kafka. See this issue for the details.
BUGFIX: all VictoriaMetrics components: properly calculate process_resident_memory_bytes metric for OS with non-default value of memory page size. See this issue for details.
BUGFIX: vmalert: fix path for system links printed on default vmalert's UI page when -http.pathPrefix is set.
BUGFIX: vmbackup: properly configure authentication with S3 when -configFilePath cmd-line flag is specified.
BUGFIX: vmalert enterprise: properly configure authentication with S3 when -s3.configFilePath cmd-line flag is specified for reading rule configs.
BUGFIX: vmalert: properly specify oauth2 ClientSecret when configuring authentication for notifier.url. See this issue for details. Thanks to @yincongcyincong for the pull request.
BUGFIX: vmalert: fix incorrect redirection in WebUI of vmalert. See this issue and this pull request.
BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: add validation for the max value specified for -retentionPeriod. See this issue for details.
BUGFIX: vmui: copy row button in Table view produces unexpected result. See this issue and pull request.
BUGFIX: vmalert-tool: prevent hanging when processing groups without rules. See this issue.
BUGFIX: vmui: fix input cursor position reset in modal settings. See this pull request.
BUGFIX: MetricsQL: properly calculate histogram_quantile over Prometheus buckets with inconsistent values. See this comment and this pull request. Updates this issue.

v1.97.5

Released at 2024-06-07

SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.
SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.
BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.
BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.
BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.
BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.
BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.
BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it's set to minimum supported value 1s.
BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.
BUGFIX: vmalert: fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested.
BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.

v1.97.4

Released at 2024-04-19

SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.
BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. See this and this issue.
BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.
BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.
BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.
BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.
BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.
BUGFIX: vmalert: supported any status codes from the range 200-299 from alertmanager. Previously, only 200 status code considered a successful action. See this issue.
BUGFIX: vmalert: avoid blocking /api/v1/rules, /api/v1/alerts, /metrics APIs when alerting rule uses template functions query, which could takes a while to execute. See this issue.
BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.

v1.97.3

Released at 2024-03-01

BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
BUGFIX: take into account -search.maxUniqueTimeseries limit at /api/v1/labels, /api/v1/label/.../values and /api/v1/series. This limit was ignored at these endpoints starting from v1.97.0 when trying to address this issue. It has been appeared that this can result in high CPU usage when heavyweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return limit exceeded error to heavyweight queries instead of burning CPU.
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.97.2 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
BUGFIX: vmalert: reduce memory usage for ENT version of vmalert for configurations with high number of groups with enabled multitenancy.
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.

v1.97.2

Released at 2024-02-14

SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
BUGFIX: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.

v1.97.1

Released at 2024-02-01

FEATURE: vmagent: add support for username_file option at basic_auth section in scrape_configs. See this feature request. Thanks to @wasim-nihal for the initial implementation.
BUGFIX: vmagent: fix the increased CPU usage when sending the data to remote storage. The issue has been introduced in v1.97.0.
BUGFIX: fix runtime error: slice bounds out of range panic, which can occur during query execution. See this issue. The bug has been introduced in v1.97.0.
BUGFIX: MetricsQL: properly handle avg_over_time({some_filter}[d]) keep_metric_names queries, where some_filter matches multiple time series with multiple names, while d is bigger or equal to 3h. See this issue.
BUGFIX: dashboards/single: fix typo in query for version annotation which falsely produced many version change events.

v1.97.0

Released at 2024-01-30

SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.
FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.
FEATURE: graphite: add support for negative index in groupByNode and aliasByNode functions. Thanks to @rbizos for the pull request.
FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.
FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.
FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each -remoteWrite.url via the command-line flag -remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.
FEATURE: vmagent: add ability to set attach_metadata.node=true option for all the kubernetes_sd_configs defined at -promscrape.config via -promscrape.kubernetes.attachNodeMetadataAll command-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation.
FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying flush_on_shutdown: true setting in streaming aggregation config. See more details here.
FEATURE: streaming aggregation: expand %{ENV_VAR} placeholders in config files with the corresponding environment variable values.
FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
- -datasource.oauth2.endpointParams for -datasource.url
- -notifier.oauth2.endpointParams for -notifier.url
- -remoteRead.oauth2.endpointParams for -remoteRead.url
- -remoteWrite.oauth2.endpointParams for -remoteWrite.url
FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via src_hosts option at url_map. See these docs.
FEATURE: vmauth: expose vmauth_user_request_backend_errors_total and vmauth_unauthorized_user_request_backend_errors_total metrics, which track the number of failed requests because of backend errors. See this feature request.
FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via metric_labels section. See this feature request.
FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at -httpListenAddr already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.
FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to -pushmetrics.url. See push metrics docs.
FEATURE: all VictoriaMetrics components: add -metrics.exposeMetadata command-line flag, which allows displaying TYPE and HELP metadata at /metrics page exposed at -httpListenAddr. This may be needed when the /metrics page is scraped by collector, which requires the TYPE and HELP metadata such as Google Cloud Managed Prometheus.
FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using file:///path/to/file or http://host/path syntax for the following command-line flags: -configAuthKey, -deleteAuthKey, -flagsAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -httpAuth.password, -metricsAuthKey, -pprofAuthKey, -reloadAuthKey, -search.resetCacheAuthKey, -snapshotAuthKey. For example, -httpAuth.password=file:///path/to/password. See these docs for details.
FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
FEATURE: dashboards/single: change dashboard title from VictoriaMetrics to VictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.
FEATURE: vmctl: rename cmd-line flag vm-native-disable-retries to vm-native-disable-per-metric-migration to better reflect its meaning.
FEATURE: vmctl: add -vm-native-src-insecure-skip-verify and -vm-native-dst-insecure-skip-verify command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.
FEATURE: Alerting rules for VictoriaMetrics: add job label to DiskRunsOutOfSpace alerting rule, so it is easier to understand to which installation the triggered instance belongs.
FEATURE: vmstorage: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by -maxLabelsPerTimeseries or -maxLabelValueLen command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.
FEATURE: vmui: add the ability to export and import query reports:
- add a Query Analyzer page that allows you to build graphs from JSON data containing the results of executing a query request.
- add an Export query button to the graph that saves the result of executing the query in JSON. See this pull request.
FEATURE: vmui: add -vmui.defaultTimezone flag to set a default timezone. See this issue and these docs.
FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.
FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.
FEATURE: add VictoriaMetrics datasource to docker compose environment. See this pull request.
BUGFIX: properly return the list of matching label names and label values from /api/v1/labels and /api/v1/label/.../values when the database contains more than -search.maxUniqueTimeseries unique time series on the selected time range. Previously VictoriaMetrics could return the number of matching timeseries exceeds ... error in this case. See this issue.
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
BUGFIX: VictoriaMetrics cluster: properly return full results when -search.skipSlowReplicas command-line flag is passed to vmselect and when vmstorage groups are in use. Previously partial results could be returned in this case.
BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.
BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.
BUGFIX: vmalert: automatically add exported_ prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue.
BUGFIX: vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0. Previously, ALERTS_FOR_STATE was generated only for alerts with for > 0. The change aligns with Prometheus behavior. See more details in this issue.
BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with context cancelled error.
BUGFIX: MetricsQL: consistently sort results for q1 or q2 query, so they do not change colors with each refresh in Grafana. See this issue.
BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
BUGFIX: vmagent: exit if there is config syntax error in scrape_config_files when -promscrape.config.strictParse=true. See this issue.
BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.
BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.
BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in kubernetes_sd_configs. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.
BUGFIX: vmagent: properly follow redirects to another hostnames when follow_redirects: true option is set in scrape_configs while stream parsing mode is disabled. See this issue and this pull request.
BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.
BUGFIX: vmui: fix the display of autocomplete results and cache the results. See this issue and this issue.
BUGFIX: vmui: send step param for instant queries. The change reverts this issue due to reasons explained in this comment.
BUGFIX: vmui: fix the Enter key press issue in the Query field when using autocomplete. See this issue.
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
BUGFIX: vmselect: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.
BUGFIX: vmselect: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example, sum(avg_over_time(up[24h])). This bug has been introduced in v1.95.0. See this and this issues.

v1.96.0

See changes here

v1.95.1

See changes here

v1.95.0

See changes here