From b8f06e9636f34e1d9c9bcec211bcfb60d326d383 Mon Sep 17 00:00:00 2001 From: AndrewChubatiuk Date: Sat, 10 Aug 2024 08:12:38 +0300 Subject: [PATCH] docs: stream aggregation updated structure, added common mistakes section --- docs/CHANGELOG.md | 42 +- docs/CHANGELOG_2023.md | 10 +- .../README.md | 6 +- .../vmagent-grafana-dash.webp | Bin 0 -> 44904 bytes .../data-sample-in-influx.webp | Bin 0 -> 6094 bytes docs/stream-aggregation.md | 1208 ----------------- docs/stream-aggregation/README.md | 12 + docs/stream-aggregation/_index.md | 13 + docs/stream-aggregation/common-mistakes.md | 115 ++ .../configuration/README.md | 116 ++ .../configuration/_index.md | 14 + .../configuration/outputs/README.md | 553 ++++++++ .../configuration/outputs/_index.md | 14 + .../configuration/outputs/avg.webp} | Bin .../configuration/outputs/increase.webp} | Bin .../configuration/outputs/max.webp} | Bin .../configuration/outputs/min.webp} | Bin .../configuration/outputs/stdvar.webp} | Bin .../configuration/outputs/sum-samples.webp} | Bin .../configuration/outputs/total-reset.webp} | Bin .../configuration/outputs/total.webp} | Bin docs/stream-aggregation/key-concepts.md | 243 ++++ docs/stream-aggregation/troubleshooting.md | 74 + docs/stream-aggregation/use-cases.md | 305 +++++ docs/vmagent.md | 22 +- 25 files changed, 1499 insertions(+), 1248 deletions(-) create mode 100644 docs/guides/k8s-monitoring-via-vm-cluster/vmagent-grafana-dash.webp create mode 100644 docs/guides/migrate-from-influx/data-sample-in-influx.webp delete mode 100644 docs/stream-aggregation.md create mode 100644 docs/stream-aggregation/README.md create mode 100644 docs/stream-aggregation/_index.md create mode 100644 docs/stream-aggregation/common-mistakes.md create mode 100644 docs/stream-aggregation/configuration/README.md create mode 100644 docs/stream-aggregation/configuration/_index.md create mode 100644 docs/stream-aggregation/configuration/outputs/README.md create mode 100644 docs/stream-aggregation/configuration/outputs/_index.md rename docs/{stream-aggregation-check-avg.webp => stream-aggregation/configuration/outputs/avg.webp} (100%) rename docs/{stream-aggregation-check-increase.webp => stream-aggregation/configuration/outputs/increase.webp} (100%) rename docs/{stream-aggregation-check-max.webp => stream-aggregation/configuration/outputs/max.webp} (100%) rename docs/{stream-aggregation-check-min.webp => stream-aggregation/configuration/outputs/min.webp} (100%) rename docs/{stream-aggregation-check-stdvar.webp => stream-aggregation/configuration/outputs/stdvar.webp} (100%) rename docs/{stream-aggregation-check-sum-samples.webp => stream-aggregation/configuration/outputs/sum-samples.webp} (100%) rename docs/{stream-aggregation-check-total-reset.webp => stream-aggregation/configuration/outputs/total-reset.webp} (100%) rename docs/{stream-aggregation-check-total.webp => stream-aggregation/configuration/outputs/total.webp} (100%) create mode 100644 docs/stream-aggregation/key-concepts.md create mode 100644 docs/stream-aggregation/troubleshooting.md create mode 100644 docs/stream-aggregation/use-cases.md diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index e67371ae7..b8bb48bbf 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -73,7 +73,7 @@ The v1.102.x line will be supported for at least 12 months since [v1.102.0](http **Update note 1: support for snap packages was removed due to lack of interest from community. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543) for details. Please read about supported package types [here](https://docs.victoriametrics.com/#install).** -**Update note 2: [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) now prevents setting multiple identical outputs. For example, `outputs: [total, total]` will fail the validation phase. In addition, `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` will fail the validation as well - use `outputs: ["quantiles(0.5, 0.9)"]` instead.** +**Update note 2: [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/configuration) now prevents setting multiple identical outputs. For example, `outputs: [total, total]` will fail the validation phase. In addition, `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` will fail the validation as well - use `outputs: ["quantiles(0.5, 0.9)"]` instead.** * SECURITY: upgrade Go builder from Go1.22.4 to Go1.22.5. See the list of issues addressed in [Go1.22.5](https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved). * SECURITY: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1. See [alpine 3.20.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html). @@ -83,11 +83,11 @@ The v1.102.x line will be supported for at least 12 months since [v1.102.0](http * `vm_streamaggr_output_samples_total` - the number of output samples produced by the corresponding aggregation rule * `vm_streamaggr_samples_lag_seconds` - [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) with the lag between the current time and the timestamp seen in the aggregated input samples * FEATURE: [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): add new labels to `vm_streamaggr_*` metrics: - * `name` - the name of the streaming aggregation rule, which can be configured via `name` option - see [these docs](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). + * `name` - the name of the streaming aggregation rule, which can be configured via `name` option - see [these docs](https://docs.victoriametrics.com/stream-aggregation/configuration/). * `url` - `-remoteWrite.url` for the corresponding `-remoteWrite.streamAggr.config` * `path` - path to the corresponding streaming aggregation config file * `position` - the position of the aggregation rule in the corresponding streaming aggregation config file -* FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent having duplicated aggregation function as `outputs` in one [aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). It also prevents using `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` instead of `outputs: ["quantiles(0.5, 0.9)"]`, as the former has higher computation cost for producing the same result. +* FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent having duplicated aggregation function as `outputs` in one [aggregation config](https://docs.victoriametrics.com/stream-aggregation/configuration/). It also prevents using `outputs: ["quantiles(0.5)", "quantiles(0.9)"]` instead of `outputs: ["quantiles(0.5, 0.9)"]`, as the former has higher computation cost for producing the same result. * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/) and [Single-node VictoriaMetrics](https://docs.victoriametrics.com/): add `-graphite.sanitizeMetricName` command-line flag for sanitizing metrics ingested via [Graphite protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): [`yandexcloud_sd_configs`](https://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs): add support for obtaining IAM token in [GCE format](https://yandex.cloud/en-ru/docs/compute/operations/vm-connect/auth-inside-vm#auth-inside-vm) additionally to the [deprecated Amazon EC2 IMDSv1 format](https://yandex.cloud/en/docs/security/standard/authentication#aws-token). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513). * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): make `-replay.timeTo` optional in [replay mode](https://docs.victoriametrics.com/vmalert/#rules-backfilling). When omitted, the current timestamp will be used. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6492). @@ -131,7 +131,7 @@ Released at 2024-06-24 * BUGFIX: all VictoriaMetrics components: prioritize `-configAuthKey` and `-reloadAuthKey` over `-httpAuth.*` settings. This change aligns behavior of mentioned flags with other auth flags like `-metricsAuthKey`, `-flagsAuthKey`, `-pprofAuthKey`. Check [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329). * BUGFIX: all VictoriaMetrics components: do not trim trailing spaces when reading content from `*.passwordFile` and similar flags. Previously, trailing spaces were trimmed from the content of the password file, which could lead to unexpected authentication errors. * BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--disable-progress-bar` global command-line flag. It can be used for disabling dynamic progress bar for all migration modes. `--vm-disable-progress-bar` command-line flag is deprecated and will be removed in the future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6367). -* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent [rate_sum](https://docs.victoriametrics.com/stream-aggregation/#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/#rate_avg) producing `NaN` results for stale time series. Before, when series matched for aggregation became stale or weren't updated during aggregation interval, the `rate_sum` or `rate_avg` could produce data point with `NaN` value. During visualization, such aggregation results would be displayed as gaps in time series. +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent [rate_sum](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#rate_avg) producing `NaN` results for stale time series. Before, when series matched for aggregation became stale or weren't updated during aggregation interval, the `rate_sum` or `rate_avg` could produce data point with `NaN` value. During visualization, such aggregation results would be displayed as gaps in time series. * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): fix path for system links printed on default vmalert's UI page when `-http.pathPrefix` is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6433). * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert/): exit replay mode with non-zero code if generated samples are not successfully written into remoteWrite url. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512). * BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): properly configure authentication with S3 when `-configFilePath` cmd-line flag is specified. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6353). @@ -166,7 +166,7 @@ Released at 2024-06-07 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add service discovery support for [Vultr](https://www.vultr.com/). See [these docs](https://docs.victoriametrics.com/sd_configs/#vultr_sd_configs) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6041). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying `-remoteWrite.disableOnDiskQueue` command-line flag per each `-remoteWrite.url`. If multiple `-remoteWrite.disableOnDiskQueue` command-line flags are configured, then the `-remoteWrite.dropSamplesOnOverload` is automatically set to true, so samples are automatically dropped if they cannot be sent to the corresponding `-remoteWrite.url` in a timely manner. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6065). Thanks to @rbizos for implementation! * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `path` and `url` labels to `vmagent_remotewrite_push_failures_total` and `vmagent_remotewrite_samples_dropped_total` [metrics](https://docs.victoriametrics.com/vmagent/#monitoring), so the number of failed pushes and dropped samples can be tracked per each `-remoteWrite.url`. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [rate_sum](https://docs.victoriametrics.com/stream-aggregation/#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/#rate_avg) aggregation outputs. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [rate_sum](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#rate_sum) and [rate_avg](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#rate_avg) aggregation outputs. * FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6402). * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add `-datasource.idleConnTimeout`, `-remoteWrite.idleConnTimeout` and `-remoteRead.idleConnTimeout` flags. These flags are set to 50s by default and should reduce the probability of `broken pipe` or `connection reset by peer` errors in vmalert logs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert/): add auto request retry for trivial network errors, such as `broken pipe` and `connection reset` for requests to `remoteRead`, `remoteWrite` and `datasource` URLs. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5661) for details. @@ -177,16 +177,16 @@ Released at 2024-06-07 * FEATURE: expose metric `vm_indexdb_items_dropped_total` to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in `reason` label of the exposed metric. This change also comes with a new [alerting rule](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml) to track changes of this metric. * FEATURE: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): add new alerting rules `TooLongLabelValues` and `TooLongLabelNames` to notify about truncation of label values or names respectively. * FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): expose `vm_streamaggr_ignored_samples_total` [counters](https://docs.victoriametrics.com/keyconcepts/#counter) at [`/metrics` page](https://docs.victoriametrics.com/#monitoring), which can be used for detecting amount of too old or NaN valued ignored samples. Expose also `vm_streamaggr_samples_lag_seconds` [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) to monitor aggregated samples lag. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): improve filtering speed of the received data samples if [match](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) field is matching only [metric name](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): improve filtering speed of the received data samples if [match](https://docs.victoriametrics.com/stream-aggregation/configuration) field is matching only [metric name](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric). * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): adds `-relabelConfigCheckInterval` flag and `/-/reload` endpoint for reloading relabeling rules. See this [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3923) for details. * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix bug that prevents the first query trace from expanding on click event. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6186). The issue was introduced in [v1.100.0](https://docs.victoriametrics.com/changelog/#v11000) release. * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix calendar display when `UTC+00:00` timezone is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6239). * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): remove redundant requests on the `Explore Cardinality` page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6240). * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling of URL params for browser history navigation (back and forward buttons). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6126) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5516#issuecomment-1867507232). -* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent potential panic during [stream aggregation](https://docs.victoriametrics.com/stream-aggregation.html) when more than one `-remoteWrite.url` command-line flags are passed to `vmagent` together with non-zero `-remoteWrite.streamAggr.dedupInterval` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205). +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): prevent potential panic during [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) when more than one `-remoteWrite.url` command-line flags are passed to `vmagent` together with non-zero `-remoteWrite.streamAggr.dedupInterval` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6205). * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241). -* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): set correct suffix `_prometheus` for aggregation outputs [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus). Before, outputs `total` and `total_prometheus` or `increase` and `increase_prometheus` had the same suffix. +* BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): set correct suffix `_prometheus` for aggregation outputs [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/configuration/output#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#total_prometheus). Before, outputs `total` and `total_prometheus` or `increase` and `increase_prometheus` had the same suffix. * BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): prevent from excessive resource usage when stream aggregation config file is empty. * BUGFIX: properly estimate the needed memory for query execution if it has the format [`aggr_func`](https://docs.victoriametrics.com/metricsql/#aggregate-functions)([`rollup_func[d]`](https://docs.victoriametrics.com/metricsql/#rollup-functions) (for example, `sum(rate(request_duration_seconds_bucket[5m]))`). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/5138eaeea0791caa34bcfab410e0ca9cd253cd8f) in [v1.83.0](https://docs.victoriametrics.com/changelog_2022/#v1830). * BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): correctly apply `-inmemoryDataFlushInterval` when it's set to minimum supported value 1s. @@ -212,7 +212,7 @@ Released at 2024-04-26 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce CPU usage when [sharding among remote storage systems](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) is enabled. * FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce memory usage during stream aggregation if multiple aggregation configs are used for the same set of data. * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): support [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) addresses in `-remoteWrite.url` command-line option and in scrape target urls. For example, `-remoteWrite.url=http://srv+victoria-metrics/api/v1/write` automatically resolves the `victoria-metrics` DNS SRV to a list of hostnames with TCP ports and then sends the collected metrics to these TCP addresses. See [these docs](https://docs.victoriametrics.com/vmagent/#srv-urls) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6053). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow skipping first N aggregation intervals via cmd-line flag `-streamAggr.ignoreFirstIntervals` for [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) or `-remoteWrite.streamAggr.ignoreFirstIntervals` for [vmagent](https://docs.victoriametrics.com/vmagent/). See more details [here](https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow skipping first N aggregation intervals via cmd-line flag `-streamAggr.ignoreFirstIntervals` for [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) or `-remoteWrite.streamAggr.ignoreFirstIntervals` for [vmagent](https://docs.victoriametrics.com/vmagent/). See more details [here](https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignore-aggregation-intervals-on-start). * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support automatic discovering and load balancing for TCP addresses behind DNS SRV addresses. These addresses can be put inside `url_prefix` urls in the form `http://srv+addr/path`, where the `addr` is the [DNS SRV](https://en.wikipedia.org/wiki/SRV_record) address, which is automatically resolved to hostnames with TCP ports. See [these docs](https://docs.victoriametrics.com/vmauth/#srv-urls) for details. * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support specifying client TLS certificates and TLS ServerName for requests to HTTPS backends. See [these docs](https://docs.victoriametrics.com/vmauth/#backend-tls-setup). * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): support regex matching when routing incoming requests based on HTTP [query args](https://en.wikipedia.org/wiki/Query_string) via `src_query_args` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070). @@ -257,17 +257,17 @@ Released at 2024-04-04 * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): allow routing incoming requests based on HTTP request headers via `src_headers` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth/#generic-http-proxy-for-different-backends). * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to read auth tokens from arbitrary HTTP request headers via `-httpAuthHeader` command-line flag. Previously auth tokens were read only from `Authorization` HTTP request header. See [these docs](https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers) for details. * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth/): add ability to authorize by opaque HTTP request header value via `auth_token` option in [-auth.config](https://docs.victoriametrics.com/vmauth/#auth-config). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce memory usage by up to 5x when aggregating over big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). The memory usage reduction is most visible when [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) is enabled. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow using `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line flags without the need to specify `-streamAggr.config` and `-remoteWrite.streamAggr.config`. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#deduplication). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `-streamAggr.dropInputLabels` command-line flag, which can be used for dropping the listed labels from input samples before applying stream [de-duplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) and aggregation. This is faster and easier to use alternative to [input_relabel_configs](https://docs.victoriametrics.com/stream-aggregation/#relabeling). See [these docs](https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `dedup_interval` option, which allows configuring individual [deduplication intervals](https://docs.victoriametrics.com/stream-aggregation/#deduplication) per each [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): use the same logic in [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/#deduplication) as in [the deduplication at VictoriaMetrics](https://docs.victoriametrics.com/#deduplication). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): ignore out of order samples when calculating [`increase`](https://docs.victoriametrics.com/stream-aggregation/#increase), [`increase_prometheus`](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus), [`total`](https://docs.victoriametrics.com/stream-aggregation/#total) and [`total_prometheus`](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus) outputs. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add an ability to ignore input samples with old timestamps outside the current aggregation interval. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples) for details. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `keep_metric_names` option, which can be set at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config) in order to keep the original metric names in the output aggregated samples instead of using [the default output metric naming scheme](https://docs.victoriametrics.com/stream-aggregation/#output-metric-names). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): align the time of aggregated data flush to the specified aggregation `interval`. For example, if `interval` is set to `1m`, then the aggregated data will be flushed at the end of every minute. The alignment can be disabled by setting `no_align_flush_to_interval: true` option at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). See [these docs](https://docs.victoriametrics.com/stream-aggregation/#flush-time-alignment) for details. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [unique_samples](https://docs.victoriametrics.com/stream-aggregation/#unique_samples) output, which can be used for calculating the number of unique sample values over the given `interval`. -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/#total_prometheus) outputs, which can be used for `increase` and `total` aggregations when the first sample of every new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) must be ignored. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): reduce memory usage by up to 5x when aggregating over big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). The memory usage reduction is most visible when [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication) is enabled. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow using `-streamAggr.dedupInterval` and `-remoteWrite.streamAggr.dedupInterval` command-line flags without the need to specify `-streamAggr.config` and `-remoteWrite.streamAggr.config`. See [these docs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `-streamAggr.dropInputLabels` command-line flag, which can be used for dropping the listed labels from input samples before applying stream [de-duplication](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication) and aggregation. This is faster and easier to use alternative to [input_relabel_configs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#relabeling). See [these docs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#dropping-unneeded-labels). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `dedup_interval` option, which allows configuring individual [deduplication intervals](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication) per each [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/configuration). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): use the same logic in [stream deduplication](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication) as in [the deduplication at VictoriaMetrics](https://docs.victoriametrics.com/#deduplication). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5643). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): ignore out of order samples when calculating [`increase`](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#increase), [`increase_prometheus`](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#increase_prometheus), [`total`](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#total) and [`total_prometheus`](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#total_prometheus) outputs. Thanks to @edma2 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5931). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add an ability to ignore input samples with old timestamps outside the current aggregation interval. See [these docs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignoring-old-samples) for details. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add `keep_metric_names` option, which can be set at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/configuration) in order to keep the original metric names in the output aggregated samples instead of using [the default output metric naming scheme](https://docs.victoriametrics.com/stream-aggregation/key-concepts#output-metric-names). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): align the time of aggregated data flush to the specified aggregation `interval`. For example, if `interval` is set to `1m`, then the aggregated data will be flushed at the end of every minute. The alignment can be disabled by setting `no_align_flush_to_interval: true` option at [stream aggregation config](https://docs.victoriametrics.com/stream-aggregation/configuration). See [these docs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#flush-time-alignment) for details. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [unique_samples](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#unique_samples) output, which can be used for calculating the number of unique sample values over the given `interval`. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): add [increase_prometheus](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#increase_prometheus) and [total_prometheus](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs#total_prometheus) outputs, which can be used for `increase` and `total` aggregations when the first sample of every new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) must be ignored. * FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): expose `vm_streamaggr_flush_timeouts_total` and `vm_streamaggr_dedup_flush_timeouts_total` [counters](https://docs.victoriametrics.com/keyconcepts/#counter) at [`/metrics` page](https://docs.victoriametrics.com/#monitoring), which can be used for detecting flush timeouts for stream aggregation states. Expose also `vm_streamaggr_flush_duration_seconds` and `vm_streamaggr_dedup_flush_duration_seconds` [histograms](https://docs.victoriametrics.com/keyconcepts/#histogram) for monitoring the real flush durations of stream aggregation states. * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve trace display for better visual separation of branches. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5926). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): ability to limit the ingestion rate via `-maxIngestionRate` command-line flag. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5900). @@ -512,7 +512,7 @@ The v1.97.x line will be supported for at least 12 months since [v1.97.0](https: * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce initial memory usage when scraping targets, which return huge responses (for example, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): expose ability to set OAuth2 endpoint parameters per each `-remoteWrite.url` via the command-line flag `-remoteWrite.oauth2.endpointParams`. See [these docs](https://docs.victoriametrics.com/vmagent/#advanced-usage). Thanks to @mhill-holoplot for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add ability to set `attach_metadata.node=true` option for all the [`kubernetes_sd_configs`](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) defined at [`-promscrape.config`](https://docs.victoriametrics.com/vmagent/#quick-start) via `-promscrape.kubernetes.attachNodeMetadataAll` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640). Thanks to @wasim-nihal for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593). -* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): do not send unfinished [aggregation state](https://docs.victoriametrics.com/stream-aggregation/) on shutdown or [hot config reload](https://docs.victoriametrics.com/stream-aggregation/#configuration-update) by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config. See more details [here](https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): do not send unfinished [aggregation state](https://docs.victoriametrics.com/stream-aggregation/) on shutdown or [hot config reload](https://docs.victoriametrics.com/stream-aggregation/configuration#configuration-update) by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config. See more details [here](https://docs.victoriametrics.com/stream-aggregation/configuration/outputs). * FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation/): expand `%{ENV_VAR}` placeholders in config files with the corresponding environment variable values. * FEATURE: [vmalert](https://docs.victoriametrics.com/vmagent/): expose ability to set OAuth2 endpoint parameters via the following command-line flags: * `-datasource.oauth2.endpointParams` for `-datasource.url` diff --git a/docs/CHANGELOG_2023.md b/docs/CHANGELOG_2023.md index ddd85c182..b2665ffb3 100644 --- a/docs/CHANGELOG_2023.md +++ b/docs/CHANGELOG_2023.md @@ -331,7 +331,7 @@ The v1.93.x line will be supported for at least 12 months since [v1.93.0](https: * BUGFIX: prevent from possible data loss during `indexdb` rotation. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4873) for details. * BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form `-boolFlagName value`, since this leads to silent incomplete flags' parsing. This form should be replaced with `-boolFlagName=value`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4845). -* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set labels from `-remoteWrite.label` command-line flag just before sending samples to the configured `-remoteWrite.url` according to [these docs](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics). Previously these labels were incorrectly set before [the relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) configured via `-remoteWrite.urlRelabelConfigs` and [the stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configured via `-remoteWrite.streamAggr.config`, so these labels could be lost or incorrectly transformed before sending the samples to remote storage. The fix allows using `-remoteWrite.label` for identifying `vmagent` instances in [cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247) and [these docs](https://docs.victoriametrics.com/stream-aggregation/#cluster-mode) for more details. +* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/): properly set labels from `-remoteWrite.label` command-line flag just before sending samples to the configured `-remoteWrite.url` according to [these docs](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics). Previously these labels were incorrectly set before [the relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) configured via `-remoteWrite.urlRelabelConfigs` and [the stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configured via `-remoteWrite.streamAggr.config`, so these labels could be lost or incorrectly transformed before sending the samples to remote storage. The fix allows using `-remoteWrite.label` for identifying `vmagent` instances in [cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247) and [these docs](https://docs.victoriametrics.com/stream-aggregation/troubleshooting#cluster-mode) for more details. * BUGFIX: remove `DEBUG` logging when parsing `if` filters inside [relabeling rules](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements) and when parsing `match` filters inside [stream aggregation rules](https://docs.victoriametrics.com/stream-aggregation/). * BUGFIX: properly replace `:` chars in label names with `_` when `-usePromCompatibleNaming` command-line flag is passed to `vmagent`, `vminsert` or single-node VictoriaMetrics. This addresses [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113#issuecomment-1275077071). * BUGFIX: [vmbackup](https://docs.victoriametrics.com/vmbackup/): correctly check if specified `-dst` belongs to specified `-storageDataPath`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837). @@ -395,7 +395,7 @@ so rolling back to the previous versions of VictoriaMetrics may result in partia the following samples to the configured remote storage by default: - aggregated samples; -- the original input samples, which match zero `match` options from the provided [config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). +- the original input samples, which match zero `match` options from the provided [config](https://docs.victoriametrics.com/stream-aggregation/configuration). Previously only aggregated samples were written to the storage by default. The previous behavior can be restored in the following ways: @@ -422,8 +422,8 @@ The previous behavior can be restored in the following ways: * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/), when samples for the same time series must be aggregated by the same `vmagent` instance at the second level. See [these docs](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637) for details. * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow configuring staleness interval in [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) config. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667) for details. * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow specifying a list of [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) inside `if` option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow specifying a list of [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) inside `match` option of [stream aggregation configs](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). The input sample is aggregated when at least a single series selector matches. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635). -* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): preserve input samples, which match zero `match` options from the [configured aggregations](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config). Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics or by passing `-remoteWrite.streamAggr.dropInput` command-line flag to `vmagent`. +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): allow specifying a list of [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) inside `match` option of [stream aggregation configs](https://docs.victoriametrics.com/stream-aggregation/configuration). The input sample is aggregated when at least a single series selector matches. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635). +* FEATURE: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): preserve input samples, which match zero `match` options from the [configured aggregations](https://docs.victoriametrics.com/stream-aggregation/configuration). Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing `-streamAggr.dropInput` command-line flag to single-node VictoriaMetrics or by passing `-remoteWrite.streamAggr.dropInput` command-line flag to `vmagent`. * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add verbose output for docker installations or when TTY isn't available. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4081). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4442). * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4402). @@ -587,7 +587,7 @@ created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or n * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add `-kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add support for [Kafka producer and consumer](https://docs.victoriametrics.com/vmagent/#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): delete unused buffered data at `-remoteWrite.tmpDataPath` directory when there is no matching `-remoteWrite.url` to send this data to. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014). -* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add the ability for hot reloading of [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configs. See [these docs](https://docs.victoriametrics.com/stream-aggregation/#configuration-update) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3969). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): add the ability for hot reloading of [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) configs. See [these docs](https://docs.victoriametrics.com/stream-aggregation/configuration#configuration-update) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3969). * FEATURE: check the contents of `-relabelConfig` and `-streamAggr.config` files additionally to `-promscrape.config` when single-node VictoriaMetrics runs with `-dryRun` command-line flag. This aligns the behaviour of single-node VictoriaMetrics with [vmagent](https://docs.victoriametrics.com/vmagent/) behaviour for `-dryRun` command-line flag. * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): automatically draw a heatmap graph when the query selects a single [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram). This simplifies analyzing histograms. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384). * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3971). diff --git a/docs/guides/getting-started-with-opentelemetry/README.md b/docs/guides/getting-started-with-opentelemetry/README.md index 563d82bc0..35a6fe638 100644 --- a/docs/guides/getting-started-with-opentelemetry/README.md +++ b/docs/guides/getting-started-with-opentelemetry/README.md @@ -100,7 +100,7 @@ Metrics could be sent to VictoriaMetrics via OpenTelemetry instrumentation libra In our example, we'll create a WEB server in [Golang](https://go.dev/) and instrument it with metrics. ### Building the Go application instrumented with metrics -Copy the go file from [here](/guides/app.go-collector.example). This will give you a basic implementation of a dice roll WEB server with the urls for opentelemetry-collector pointing to localhost:4318. +Copy the go file from [here](app.go-collector.example). This will give you a basic implementation of a dice roll WEB server with the urls for opentelemetry-collector pointing to localhost:4318. In the same directory run the following command to create the `go.mod` file: ```sh go mod init vm/otel @@ -167,7 +167,7 @@ In our example, we'll create a WEB server in [Golang](https://go.dev/) and instr ### Building the Go application instrumented with metrics -See the full source code of the example [here](/guides/app.go.example). +See the full source code of the example [here](app.go.example). The list of OpenTelemetry dependencies for `go.mod` is the following: @@ -314,7 +314,7 @@ func newMetricsController(ctx context.Context) (*controller.Controller, error) { This controller will collect and push collected metrics to VictoriaMetrics address with interval of `10s`. -See the full source code of the example [here](/guides/app.go.example). +See the full source code of the example [here](app.go.example). ### Test metrics ingestion diff --git a/docs/guides/k8s-monitoring-via-vm-cluster/vmagent-grafana-dash.webp b/docs/guides/k8s-monitoring-via-vm-cluster/vmagent-grafana-dash.webp new file mode 100644 index 0000000000000000000000000000000000000000..2a41fd0176a81507267d1c255b7a2b216d7ae050 GIT binary patch literal 44904 zcmagF1y~$iwziGCySux)LvVL@hXf58B)Ge~6Wj^z?(Xgo+`T`NIq%FnbLRgKS6|&- zyQ-<(d#}Bo`&m`3EF~^(SOo;6@kvBcU6G4O_2WGf5m+WL4I^YesDKw&s&olmabCGs zns5eU$o>~VQj;V^k;bD#H57wDh{OSwCWeeBWg9_rYu2T7_TYo%YwEd@IdIB_taj>k zXWm-)xxtZ7*c|tSK@gA6(8gEm5WnChIjU7!AHb1!p6-#FXC5&13(AB z3PAck`>HV{*k`Z=NWJRc-|Pdl12j$)t|n^$Menmu_-7U`K4*Zq_uyCN_raI!o#Y|L z1Hw_kB_C}7zZqXOXNH3JLE%^p@FVEC5_U@oUwyLX6J=;k?g{j}L(NA|C))H@IK~09*~$1Ob3^vHf1^ zDpZ+Y$_n;Hy>ZK*f@*jOOoCIFQ}RL7>U&?X*IBIRZQ}yk1%(xt<=m};%~ayA?=pdt z=XhgJ14%6KYf(M=I#My_-fVRK0$Rw<+Jx~_*p;sgkC?fQR8A6>@y7CG z?m1MI?sZ zKteN|=eF4RCPWH~a$|eVuX=7iP0BU@IZL86b8Lmo0;i&G=uV|MT`Xc(t=ho=M`VC1YS1_PF6cks8mZ2V^k_Cu+u$^C5w8D_9YHl4DDw^V&F$# zw7`4yeE6c(cvI^oC)jl?Vo!>lI6|=p%KmphS!#j`zP5CM+?FE2ofem}Bg&RU-&R69 z0_Pq3LZVdlPE6}-ZU0axKZd)buvN}1b4Zs^O9+t{{3g8ft(3aWnYKF`f;{DtJd+M* zC1>Hu;-+1`JpxGq1X6?~6|B@?A1vxet9f-`9EP3YVwXL~RxqOzy?XbNpmPr}EqOVm z-sV11sxFYOoO%uzYoz+}o=3^rc%HaJ>FR&$z6&0QO^`}DPicGYK!^(t;d+rf6Ppix zT|{cN`(_u6B41!-o#k1sNHhbTn^tF5Mw$9Vw@tCg4-dbKarPC1qnsb4u)@Pb(^#Tl z#(hHHYBg28^n!ddTwUQp+l7BQ$UB*j5SDYf(R2cJ>?kG=y{hH5E)wx7SYQXwL#9Q1?cXcC{p z@SPxyuiE-{mymRcQ>3f5Kk}^T&!+6Kj4Bm?&%1>d;2e`?z>aaC2p+LOra83aqA7N2 z3Pu{P+UI|$Mu6uh^bL<6LH44lHoT*ulhNd7AG+9MV%y86yg_3tZMZ<~bVGsYny-On z*kOUL8LMU3|Ix?a;c?aDe@<7&k1y%c;1WRi&8jakKu~@TPV_5Yekzy!P!}`iBPfd zedB=o=TZOXp#JZ8&amJT0-4$&A3WnV$oFp6QNmFvN}nfg(zI77Pn?!tYA0nz^DgV+1SHDrrcx}e2$Fo33DWA#8#xmV5FLX))wFoop0t{BQh`k%_ zxZmS3uucrgAQ%UHTiqT~-FcmuYs?6jwRAiRpw_%UOp!y%)}oU%H46eVE4q{Xf`Ny6 zz@Tvy8e%>e-IpZ(uSVveZm8ieSL-4p%k?;hfVvfgaP||#M@L_AF%a!Ov3U?w3O=vF zU!bh7h5LP})nAEk+30o)e)>V<-0&d=sViYaVXYTrYuz29$K58Y)){*H#VH$8NIyWg zpd!DfZj}8%6x&(-GGz3?ChWd0gbyXq(6e8KQDKju>g8srm8(^Y!yTe2$ALkYbNL?x{CzZf98pm zN34FCvTb5f+J1YBtl~i+z|f^)#cW_f97g0TBg6U^%rGV z)_(=8NhggVv1ynFnH&3qxZfk8>k%)d5@~XBlcu|eAw(>&K&@tsaBLyd$;FWWYSa$% z`!|j9weFo#FC6(jLkR6J$G$wcD7x%Annic86F)Bv?Y7zpMvC?+Kaw&~f2MQ${PSu3 z$C@b7>-@-j>MyJ}JDV#kIV5VIGl@2S%Sq1!6WoME#jcYc4EgWHhCsAa2<(B48OwQg zbaw#Q;Pp8@`N>SF)*Q-HF~&#tj%8Z^tB#1*lcn+WmrFC>LyLitzq0-X{}8h^*bd1x zn^#7pVDmR{2`s@GUyTGWXcG_17j7Ky(2v}kTQI`3_@QOv>mSQ`ZZwaB8}&oOfG;xC zCrKxIYb{4UZ@>}WOt8tURsx>|MH3P@9hV6U;(r>tIhd4(0>W%Ce;+TZ_e zaEYjc`Hn>xItBGgZ#&t#-g895TeiIIREN)$Rw?DUI3$vwv(LyU1w%tA@ZAvhRah># z;rq2U>c`yCg+Eu0pTw-uagh>cy##(2Vwv?Q;DkW{PmKG$ph#_5Afg74vT8NshPvSD z1ai-Mdi}giYY2yc%A|Ug+4U?e`Twl^B(hq@H_Zd+=#B@&j=`ccVCj#}K_Hh|v(D+y5v1uxF&}^Y-VJd1$>daIeH|dfS)I<<`@fzs4wq zRG>9T=sy&9^u=%81_xlB<;3x;o@TCSIiVggMYax&|1T0CcU7qtp$937u8S>xV4OcU zC|m1P>xDeBH@zxR``GFh)M)mua(vU@e{LVW^GrUuRG2@Pw^{4_w`O||xc#}$a1q*@ zDZ0T@&j~@EJ8uV>srvGrI|M^8BVZ1PG*|ys zdt-NI2^_9moY#`f1j(e)58PqfVf+h5h@VAfUMNiy*bxujF)C7Y_?<;OYLoC^+V_{L z<;Sy%-tr$30Bv=*jIDMf=$wTySo0a;wSWMq2v6|LH$ZPcXWJ?{S-Oai?{ZwSD^{A? z%Ew1xwNIzKq2gmuwD7>qSkD&2izGOAikTI+uspy`D`oBR*v8500nhpvvK>u2Qfj#2 zKEM8nnNgCE#Q)&HGXzbqT#Wf5BFDQy&^RiuTt`Dj)M5GY_ri2!Ex3N@1`l8ktJ}H9 zah)_E;6mGF#5RVlGKwTAI#-5*FRy%O4Z-|9X-ZqVWBI|)(>PmMpxK#P{GG{ug66vg zf(9Y#yxYM&^HV_7>d9<4S*s`$r_4wLxCZ8Di&I3TP73R ze;@~Q?B0K_peZ-H4H@4E!9J0M7z3>pi3cLg-6bFHpirDvxq5;%DlQDQcPDtvJ3R|4 z!ZlxeI-_^uvxS)JC<~*Eg@)o)T5|3{YNAoOKiP+Jf&CRN5k?9PrHw8E=%0-;C|X#+ zNq^ymGo2)I2{8e8U*|sCR$30La8T?|`;;gMnJmfGCRwpSM^UvYcgf+Uz#+8#Cu*T5 z;k^s5{JP#A-OW7AlNjQi*GL8WoY+8qt`|LD|AbG#yD>AbfkNs z5)d8^S17F;_RmaBg_EjBo6r94S8fg6kU$c`6#i1d-x?Uc12aHXB>?-=b+1XU;4_DT zXu>ONqP<#3{8ohs3G(TOvH3^V!mn%sS0!Ekmk;`>l94uW^mqSJqUj4#`nQje+1-7i zxnpqryLmFt;b+cDn*vN6XnO>?A;Q$7RIQT2%k^h2A&5Z)2i;Egd5g|#)oy+DS5}x4 zSO9mx2}T;;Iof~jgsB%jGt|;vO~;U3FEC7tN-n$eS4pgBt}&eu$4L)!uKSr&Jq9(5 zgpS=@7J=RA%d6a;&-}>|U&T&>&J^qZvIA+$hsv&QBuRsd`Zc#>l158^IwkKXS!dt=lFe2c|3?9ItJ1}F%E4y{lGgu642TlrdEu2T?V~{FfJOk#pbwF3~Img zVL1PECSC;(kGN+RZ84wLdBoAa)$|PO+La#n&g8{|SK_Om@T9D#)LbkRDE%L;T95Yn zOS7n_KQv3*PL)MKQ*WeWbf&O9l3neG-F^wGc7ipwenntaBUBgihx{<@h*iCorA9AmYAajMj(GhcArSDHfs*q7G`q(v^a% z(&`C4EtVNf34YZQ>lZQ&vm+2#%&MO;qLN3B@yD0>)A9G$@0%A>OJQqdKBx5?tkphr zbeu3_;hUQ=yl>CAKcY-q2yyqcb1l2?c21v%jb%uISZ3Xg{tXp=HzM+zR`bID166+S zw0r<2fl*Y7%3d5)ReB250<%>tocCd;5K_SK%%Xb`cXp0AI>tVCt6RIWXkpDSu#k|P zbn4Ig6DjzlJUq&`71Ez3zBfpt(mb-0Us@Vm6O{fLMQ}g5)S=mzKV!O;T?Q+fEdUSu zE-IzJ-YrHPcMh-t8JT)Za@E7}l`2sS%vnV2&JWp_5+YI82xdU`d>I)w{uG_qWC$S# z*JhCle`jCWCgO!2*C}dDYu^c|opqaJeyyH1`K*GBZaM-RuYtl~P0`cTTmv5>Yl2wRdY@%X9; zw7fOAygZ!aSmzKKAAn<9>1=7FvFG}5_JfJn+F11s z91iHyKogI|JrTlIhYd`(Ko{0p6*?P*7AUv2pc*fl}uuMDX z`(c#`3z_`nj9VaZuT$J{I|Z#xHhV-;qlUBR_h$8T6q(gUgC_NfI7651w)6LHQ#c2x zmlsqU$;Eb4qMOMYFyTA)L(j$Tk#CH&VOdHJwX6=g$(NI$iELFHR^z>ZcX6PWJ!WWA z@%jGBorIJJ?Ia%{++^go2gD?1Ef(+{%f~H(dNBkPbzk4HU}yb?x%#@RhwuzWrxX8z z&^=BIf=}3vyR=+x(MfXPp17Kz@1MF@~MwQhrqht;#)6C=eW$5I=SivcNO%Yw#-Q1 z>DR=zejT|grx$iFPE90b)Aspk@L1l1pP6z)m}5yTI_97dpB}2x7v$+OW@zx^7uGGE zTT*s;en0H?~geO9g!oFN`9j)&1E0nC;C)35y{Lchxk zABYw4wVTX<+)ZO=%q!5pYx)r$4y_fJlMqx{-A-1d+;Mbt+EV%ieCTNKTX0M6g0d5= z;`3;#<}qcV;$8VhvCR`%L0rj^VcH=WIX>IB{eb_V3;G#Wo{r0qLsLm&1Ss-s)jPMY zX$X+dVg&!S>}MI>eRhrx1OO9S<#PIS>w$C}GQ|4^i|M4`4{goEyEy>m( za8?{uE6vc}ojBcgf>Sk&HA$b(V|ciq1WM0l#82e9tc=cVX-WzkrT`37b3tc$c7#D& zdPCuAdIJZAmRuizK*Y0fh`UHA{={^tg5)<>14S?Mj~8JhvXt~~#A(;*c2uR>+I!W9 z&6A=?zjyL*dxu|+ zJZkvu%KG7&#bAo{3$)Qf<~mL?*tftXPC{o0lDk8bTPdhpsiR#x3RSP=8{<5Csz6B} z3v21*u3ub6nufd^X<~fNkQdQ6s}8%oHxhIX?h%MgS@wmao0xISYbaq_D{8L3RKLdv ze${EdkNxR%MrwE*II2Pvd*zEpk)X&CXmOGJbO8^a%$+NV$9cbxsp^Z%*zk|~&WFqU zOA`OGZy(b4$G+7JsctB~r%A*G07ur(6CRrxE`AL@Dvd@X;fFd2-mYq;gGFes%+(kt zmb!+txIx{??ADHtLmr~UI0~*!fks%c!<~Oac-Zom+ z<17{HO#x2wknKqVqFHOH8o@>fA0>G>X|)ebC3SuWsU!Q6-KB0CIr|7R_Q!9=aVl)l zvWOqQf}2aO3;LE1;}%~`+JN0&x}TZaVjyz(Mze0V60tC&aO3~>q1HMG}XL;tn7gyQy9PH;3fn&9Vg*mc(LXCPD`e)cPac{5W@29ze>`&z2m7T2 zU=)T3TWk8CxG1B>hL?Pso0zdPb2uar8G{njU(KQy@5cL|(&C@5T4AnWYY+5y(wvIx zf{@mOI7j)LnR6uqLaO*1{f_-$5s>@0qCH-jN3e59+bbx<{a0nM49?|!8QSGqyWfnM zc!nKD8*}18!|^YMG>MlP!9Be$Ogw$I5NKfsn=91z1xJlok-H{oYt*Pajz}B&#Uxsv zmVju(>Bth=xBPA5QS|$3v9#~_7M;972T8*Tfy3SO4~jyXIUuSG^Mgtz^LL;#>HCO& z^r5eE?RYgYCaXQaKk^$+t3tMq~OiI{9O5pf~`ycM^Q#SH2>YDoS!T48azQLHOwK7FX7J_%xE(bdRj6zz?y&X%)E3yE zPI?JoDF*qEze^vrN3KDRM1P7MOdw~HCbS3op6~t-2K_*}y{g|R#|Ze1a$~0^SllLFfsa;A*YK4)O4tCMX3PccgRmTkxJ*;V? z|EUhSG?e3z$`O=L^)%A5+M&_?W>WKsfZt5|f0C+1T23KMSo}dr6gQUTJ;uSym37}^YF^h<)-I**$TYv0^0NI zIdG92nIioi(~ytqv9-e9MA-nMpaWj&8&2GB%CweQzkx>jS|s9(;_U1TVF@T7BGsA_ zugi%HLu;uzC>Vm?`3k{}v^TslFVeNil@wHctLFw^Pk0{xSt+(rek4wyAR4l3o7&a`+U%5eTk4A1-z z!}P()4&0d($*_b&?dJIbbbhG zsPouMLO-yD!~lJXAqd?;z2rf&n80)96Q+_B3PE0*kVAeZ(+Fptdw%+&<6l41Qe01;_?f8XQsZEI8jpdz6LPkqS zURy75fk?*%*`Ae|jqNXr;89DK9kb;(Ym}FDYcX9r*)IrcSIlzqurEBd;cnW;ztQg*Aq9mZ5Z6I}GP zuIogr%s*8-T9SBs%lu>sFu0OAdl@RcMDb>S*$#jVzBTY=@x=({$EwykYP_9OBu|Ua ziC{P=Q$u1Qg!d1&029&L5>yZ+w8ucA?@uBqbwevX@~Lx7JLv7!S<;Z@)j#s2$#*f8 zoN#t#gRurqU}SqzwB9K4;O6}y%8YYpBP5ByLtKvR@#BJUDkD*Md6F;1Aq2}&uh%r^&c%^f@!I#&W?cI$5sx+woO@zcd76ie>RMzl{OS$GY9OqzGX>{p@S{WLs+G&r%8=%x(xp+H;gk6Sp!3aqEkyWxg^{tD6@K+3@7l^7{%0JJkIOM|&W$8vk z7{*C%BuwpZS!V8ML4aQBxwh^XCsN9FLr=<)f@t?1*c*Kf9QZu42BMHU^1Qd_caQZ* z(*ao_d-Pp1j3j2$kHVu8@{6bosQLsV+0O0pXWzS?u+{YSgrH_&nMn5tFp;gCFBgG< zIeYB1)1Wm~#0<{+Txy`*XZL=ZnUAi;D<(O7@y3h$6}nkgbxGUwVIg6qlK_H7(@rI5 zM&-82L|WfkCLm>6=jfU0RY1ZG(3J%8!P0(pvK4`(D{AhyIq~|s6Of`LLUqK^xqG}u zx5ib60$(kBB5I6-`ia$_U#Tp@xnuCx4LVZZ{dzW)^|fUb#<5>ud&3`TqN|u& z@nwCZousY})s3R`AA5rY^|2CpnUxc9%&hUr!`2^*wvx4VTeFNKudiFt2vHzH+dqdE zzHZ4)4pb&5#aKpG6~OSr^VIaB$1(+F{qD=3VS`9e%U7Xg(0SL@O#bzN@>7C9R(>i5 zl$B9vYFX?V1gNu~b0biXBgfal|rGxbLT>d`zJ#8-Gi&jd^h52=Sw?tQC zwk(;XL*oSs5Z_^FFokbJC8JZ4}lnR^c6w z3qnCdYL<@ttfdSqA-m#3 zTe$FSF~#lB0ZX)g-Li&c_-S| zxQn(qoeH9~Z@V*%pNhG4LIRob&Tncx+K?+mpx~?4=Zow+0z;~tdzfu#o4)Q>|G(~iB9*P0cL~fPmP0RG%!-iUX;tJzfvWh!b z<2mN6x9FMG)L@tLwfsk&1{e2H?Yea^u~x`T8CXw(yyQqprye4}M*ZCE%`>y6mQOAP z<&F1WQM{Gvkn91zD0LMchSzI9vdqY_Qns7VcF=55pP@D@>V1R-BGwTu)*;hd3rm7- z$@$+Uls_rU6yiK8M(vC5CyU6~9kW_s`@Tvd+D(+zDu;|Km|A8b+IY@R5o^U7RiTr- z1<&=coT5F_wWTAA-xsH~M}zm*3q$Npum<-lMj2edVIG}izRB!4DQ~MXLf^tWA;m@c zLPr4TI&p)der7g@PdK6{G^*1_TdZJKm28#XPbzJRPn-K;Q!F5>5rX|Z(OsddLK2SK zryWMAON58P5$i%G?ZydRl8ZH^l}!`sPzCfs#CiKN8@GVC=$R3HBsN6l#j^X+0iE3_ zDv3rG!+ogKF|LRMWMji$T!PTM4_)%NUS+`yKAuxnBZuFb_rUg~VJFUXbFRhhQXHYO zEmj;huVw-92#Z#9E{|mfv*fp5L2F&UsyI$Ejuv%;*ubQ)6>ga`*%aP?!HcL43~8rb`$fhB|M5J8tU!bWc2_at0jc@A)jGBJlwyxP9+dkw! zrgr!x!U*{teCB*CUd$k4i7tUR#*e@PlW*6d%Ex@W#%om<6wNiHEPo(}etz34?h= z;N0Verh!2*Za0CHaPs;z^xdKGf>HJeiu*|4eNAz@(=Fj@K2ZK1(Tr4dyGUvPIx1p2 zuhWzLbU8$=tOO4*w*hhgTuBw1C`WfX<9?7)NYHQlzpO0Ge9szPz2mt1imCw$j~NQ zeaGCagis{ilpQ|G9LMNcYrk^Vw;!x}Gc>~eX{v3ixJQ(`Ujv()E-J1z*};WhZFH0Q zIPSc#j}swVqbT27mltiyNFRhd!&`hLSFn3##u0LFls9^Bg))&EXUn ze5-VDtzzEwxO-T34=&i%LqBRiE07e)272ZL5t4*`EC3)N@PuvaZLN=s5Wg&rm{s;<}$`(^4*bU1IFX6sTy~c{pL##d^Z481xa#)VI6uG>4e$%fX;Xi>!!i zayY41$Y3OIQy><5OZUp7WI!F?8Ue|=i5g0ky#iq2NN`OC(`x!^j$p~oH$}9myb~Wt?+m(&*BxZ#h zAt4lTPL;H>F>kM+3LoK(h|X|yF214daEmQWa)sA%OwYTYgLr>$Q>28)2~gEFf2P@I z6&vo5{Dv1SWT+R6ESF*G&Tbe&R}cgBP2H~@JBy)e_X*EszN=YtH>5vVN%-|uUDHVg z7)?i$uVjGstI{-v2ESwMI7Nbx(tavS*2+;m2}F}h`*BV9YuDHARwtaTx*f3!ZHtnM zXFdARRsk}_ac|Sk^p#)NHv@E@L%)2P6KQEjh@#vpFv+wvwUeufB5r_@KW}^9G^30< zvx`^zsFgXP4l`aV^ujJ0A2HC80kz8na!JDn&EV5&#DxuaISk%d9bU{aAXz+x_=r+8 z=n;Y-j{C`V;xlmWumqo~lEdJh#S@iY_5mEn3=i+0Ip0XLDg(ZXwd2~ zr<2{1Z?vOC7gt{o-2F4-78rFdQQLk%ePf>7We!4WdhYIn1=yoRO1h(z8AJUNCP*5! zz^|?uuNlS)^`3$^3SlNxVu(D1U{4?pxYaAbDO#v z+?#LXM8Z^#jEDvZ9**fAUYvOjAME--QP^K}d@)(yq>k-9%p4-iDwZAG{tBmgg!r zLcZ+F`zfLA(}a1TkW*#Y{dhq<({P(Lpk{Q0`L5OH70ut{4~fVdQ$d5gx%uRAHzwS0 zhw<==^vGxsfeu1?n$)#<<(FEBp6XHr(Y@D6?_qF|HY!5Wwqhc3{LS zeDOu4nQR2_HCVLf%>?Bp_k zFWn^E0rRiRejmlSSt#rKAXK3#|E7r6;9{xbj@{^=|kyIPr$)o-!8N92I%U?Hi zHb~thRy>}yGd;g+Un#%dP7>nwT+Au)dUy!SC6U7t!YG1Rd@k|1Ub9d>*fZs!r14&o z`Stk%E3?lujs@*?2ZBCNix6;pQGr6k{+UXTM<64=25$fBv*NMd7`Wy_t1>KYkKM1; zAAGJ2zedkK>&vjE3kNf1xLgVaey?UPXzOEYGxO{CT4F3NaUEmIP^X~;R~8ONqc7m* zPYG-bFui`jXjp-fv86iJ<=+||UbzY<^%Wui;YXS5Hmdb0X)Q+l+#;&>{n`pJe1dG` zP)O2wJqqlGyDPj$;7SAUMksJtKqv3um9tSZOrs~os6ma)dGsFXKjt^FWOAo!~XI!SC(UMxF`MXAIKOuPD5Ws#u@ ziyduB#!y6B^&Oo)VSF}4Gm^@;2jd69bfWFp_3bO-832bb>{ zPo!Mgt(5a=__)AhLVTD6*U?r>U7I)oXt)5|@GC_cMo%Vn@&GOs&(@9|KS0z$ks1;)sugkY0INxeSF=XJCli)K@+9jP#Bc%3o z8SX(A5;9PwuAJ=Yx6UwHy%D;q5D$xjN^r*|Ef(VQQA$%Z{U63!N=SGQJ~B7Ic;BJtf(u&3$pY(S0cS}uEO zSRM=0+1e)lgSHvd--NS2rv1WHU9QE_heEUKNw`f#sHQMB#pcT42lq=F6e!%nSK0^$ z8T4?}gO6MJG%QcP7BAESN86Q(Gg9BW<$-Jbv?r)cIcE4Jy#!SAV;6|%`$jdyOG&z+ z3`mUqUgd_>H*moeK*=^IpWEuMli{rV1cbhd`Ej9vX|@f1H3glYbU}=Y4$b7=SX;6d z&ix!fl)rMZ#2d)1iI`SH1fg2aU_fK$Z(}{i$$U>mV4WAVm6Z2(2T>F*V(|Gu^4K4MigWwD5f;Zf#BhS&^>ojv&Jyp4S14vHCff`SjVsfF9w1p zpvS}eky_mlB1p;wgl$4k31Ddz8B8-b`$D}RANMMk9oUcwO09J>r1BhCUgE|zeF;^+ z(4}m{)Al>Pb=Osv{7l)rSrc2Wo4*IkF|wlQ%0{DacjNKmp1V%F{O^9raRUl|@8-Bg^2@vU6$`wtyNkK-h9|`#ai+{iuOYb|ld_<2mV4+Nf zn66oZnTQeEILBVEzJs=XEHi@rc2UsbiSlc|HSOUcszD+{pKRg?VdZ$M%NUt+&9;ilms zx}-p)eueRGvmyXz8p@PYL-lxw4vkElfs+%l+y)gzQ*l_cm2#I0dOUXvt zIy(Cb?=%bB6`T}xop<_({orPckx(Fo^%`U=>A@3}(sosvHJ}3v*=gO>ViHNo+VB*& zx-}-e<$-_vrtJHwj{PmYu&tnX`=Epp)nt^6=Bo1%sW+$yC9y~aE%d$ZqEY2Fx46j^ zGse;k)>b{iIkXUCe7c5;!x%qF^<7>39Z1%`kmC$OX#y35taqyZ=O2P2b}h2N>^%`L zLy+L#+o%@MQh&Cu@9Wc4?I5n^SEqFU0L8z%+P9V@`#oSEfB3zY_)P_G&sGjvxVA`R zLO%hQvJZ~nGtp<95-39}g;xR#$7YU{6Fc}3(dLj}4;PVO6KX}X-^XWIVd>Lf+EqE@ z9!{D4R065mowIN-2Y<>NoeEPnMhPemyMqpHN)FdNesgdxw~RaXvN14*h1EChFO z9ig(LV1|CF=up}j!BM>|U{c}!ZZf}gh8k9lZ#x^&79h5-oU6j^NqCRELpD`e9h9#L zMEiVQFm7UtdMLeh>(A91?QqsRdQ;5Xi-eLClqbfzP)*iL0hnlhn4^zikDEQ_JaEcL zz7#3zweP`}oC(dOKtR8^=2TM?+kvHR_dh`(O7uH5D+c%nq=oxgM=LefPiwh2eWjfM zEK<1G#YJ4POo{tPHc|?`UAE2MjQ5R~Zd!&fm5gydUI76s?h9uOBP#YG9IiYC)@9jA zuydhQ6*9=ZsjE;5!JzAm`BAZ3jT3@(9`7@aB*f80BPip@A@HywHR!1OLa>DpK?(?_#5TLf#RUJE{)L-FYWHwJF<-O=B;;#`UlA755FR)6<}FOyCEMN54S#? zIov~^ov-9;Z7W=kN^rO%t>nvXIMzM9LUhc2$tu4qAmEK1CNU!VWWz<}??-t^x3a2Dr-mUAbH8qR1Ub1u6&C zw@#5!cBGiV+lA-uP3uFzz)GyMcLVwjxP|&m5|vJchGobj&=2hwKI5}fjE>>*$GW&2 z%6iFDf6rU{X4Nc$PJc%CBE~M<>3-Yjk9Uf-311y$*mgr!U7g22X&sEu?i?H%s*J=D z{Gi6@?b+_ov5C@}*1=oB;TTK}eUC4PUG?Cml#NjQIUXJ@-+Cc(Ow|43w)f56d;xlV za1U*4jxgYQ%fSqSWW_9?keP3k?inkzjq7O+VM zEcN`a&$(5O2=h24(ASpmI*#zlmt5)(iYr!O7Ci{eJR>kx~sfsdaShb?3@c#y?EGx zs2I9H#3R50v85MzgjNE&LqQ_TF({W0yO+f>0)=TnmxFR+!1Hbjoz#&Gz8oZjapp6k z;I2Om{aTlldh$u9!!q{S_oiYH4MhHa+AOUKb3l&>a^d%*ukE6wWxIoZWEDNUmNFCv zt+>_suvfEbRv>!mO}oK^ZcJ&O^4>Wk3s$D3?6P4qN0QEC)uf(WuW*V~B9WWM_jK-?k@3j9^od`|9bTL3K5TlTQs!4!rwg!B=s ziX;_igov-fLH9&e4b{2I9S`iaXCS*2MeCup`-~O0eQ70u5{3Rz za&FzN35j4AFiFIG#>HmKZ@*wPHcB|s&pVx%BRXD)UYF?)v!n~(^g0S`*Q^0qPh>~p z$>$EQ#7^ydQQ;_sT5&&uYD5p=*046ss|Ki~ z#Dyc1F|g?;eXX)mtb2>A|;hPDQHogk-Pu3G}Lk7G3}_ z$r587&#l=_+Sk*alF;o@3eDq4vW#tSv!w6$vO8hm{`gr5r!Cgt{Gzz)p=?c(crSgv@xTIOoQMxImRmicvOb+&f0d* z3%|HS7-u#t$CKvatjy!inqMx(m}8iT51Hd~3#gHL@=|t1db$@0*y;tNa|iA6I4}Cq zZt~z@#TRuoqWmBncq+9l!keR4BUw^Q6qI*?YQGyVWU(13&KVV>&wDOlRnX)4_kh81 z(+sNQMK>UMi5;~$tH1?XFx&Dw@VeN!ZKRpU>Rb2UDnk3=3Qa>aftN8$S`w|QF?1oB zbSAf>8PHSm;N)JGAxfyBU5q%dP1~`Zp=iBx^6^67G68RYPh~+&@~69_egHJ*sj#bK zd_g1-vQFkYQ9H|p9we+4hIzpMWgU>M&#Se(8D03O$k*5iv4j?>-Dt8S@&uiek+ySu zlSXcpB%6?8=*}$(4lp;qOT+A^IC-ZrFs66pBEaif5u1BYEY^IHb%^49DYlhtCKJO=j1$sZ|^=$oYAJQG?bZ0spM zewP%p?S-MV%KE$>KvZkq*O=3j+yU&!Qq^)=Ilpqbf^?EWyn5<-M>YE%xb8m8cyr-ycz`8Y6;gpD?M~I^6X%g8u4rhcQ*aJ;i ze-f5youIbMK7<`KD}}bWRB_u7A&=#pL6_c-Ybm7sZw_g9(K}PLc)ZUZhP{m2-m$BB zt2f-i)U$ad5jUot&r2g7SKG=E0yq=HoOMNhEUL(Ss|8$e?=JL+gM@(?wAz_M>t3e)R zG$5PBT4~K^1zsfpJ=eXqWi(ZH#q`p2ltZVL);Vy{k`ox`%QrN(`Q>O`*or_l<#qs3 z$4riC*r!lC3hJ(nrLR#tI))jHB*lQvY(VSwisfoa5xc!uPx4d|IHBQ1`U)0WfXPJA zeTx|C5(fwfSj7BkS{C>c)D;N$mBI$IK>Eh^B99(aS~d!~PeSPh>t4?jHi&}|2{=S{ zFRnF$Nq_oD5=nynoy>nK{oqJ3PpUGPzuI(lzItVH)MUFuNs=^GS$f2WPMYHvquDBK z7ci>;g!vTGRZpT)Naqo1e>;+_X-M75o;w{!z*sOV9m<}l9U(M86|5}lA)nrBBrI!= zBHmA)bMT|Yba%nVg{PKtQ8w`CDwzXBN;yq%Z~2Rs1B_D0K=hU7a+M%@x^RHvN{HPw zM5#QF0z~214Py(=xI{$0;RPe%OA6~T9Qoqhj99~w{?g>dNzNJ~PUp6i(}dl2rHNfO z+NGZ2Rl%n^0GyD>i%`kV0CItmaSA>rlG-3dkI$G$?46vYF)f&|NU&?L$mPL&!|CW) z%d)A$G`+Kn9Ht*>=pjshmt$`VJ-2>_OY23WR2qiTd|aBV>*W$*ofKNP76|?WxEwGV zS_E_IAt^O7{RCB>7)^R)XSj))J%J+PxWEO8Up7zMIR_sv$;Gd#pZ64@d}gsxcbVR0 zVD?#Nm1Ug|)OO7)6@%7|fjuMT{Ava98P21EnhYLfRaHR8Oz8z|PNB}$`2lNUR8al6 zN6x&U3U&Aa`b0^P5aKk>)Z7(^N5dS~6t|?ZOV9Orc zwr$(CZQHhOoA=nZZQHi(d*;4*Kl3r^q|@2y>Oxm#*T!1KWVx1`R>u_>tshQl#!63m z^Aty9Sebzqj&|$m6dK1Z$uY4MfBGG6A>R{BJ`7({lgS&9Qsb=Gz0ee92=DZG zBy|1sju@A2V=@T}4zFvBlnN(Ah=dP?w(k3~=QUxv9pN)$sFNK|f7@(&OF((IoXaY& z)An8&N)l>BG7@XPwJX=edmN!$))u5%_C{(iHS;4Hp5C{T2@_wZ9$gyD{%ewF$N{e0 z7k}kb8Fy9fF)C4%7;M9`T4HALkj2Bu`L1IS5{<5e9tZBse}q9IvGM-&V>iMeLWYIp zRR@rH-F1Hm=qq@bByA7^oTC5IB9?E{EXZYoDtW^DEOIZp6_dYB7?bwH36Lk>^80GW z9N(sz6~Pw&G1taM_) z9sj-GWS%3PhaV0`OmJQy6aTV;Cr4=H2vyfC{{7C1qeB&()UksmM-eJje|idP%}R}- zV8}F<#rP7%7E# z?;3OW+CKe6nq1_v$W2c1AC=v@-?RM9b#t5XQ0BVZ-X`7x7 zwSq`ooIdYAvU*iMW1`Hxzn3800R|@%c>j+Kf9d;$N?b)WlpO!W*yf|J?-wlul3oip zbH72h!n7UDXMU)b@W(#_zkskZC9Wa^?&%^tviPJNX zyj1yj6GM`QZ46>(uoIDiAzW2860=-xf0n<KR+yzGU{qhVHp>1K5zg*pssNke12brP#{MeDQ{EIC%nv_L$uw3vNPkxJy#AQr2pr-WnLt<1e2Kc%7W zp?wJR*d^XKm9u%)sax`SGS3w(KwBmA`P4YTqE6$PV9d~!biZSdYSS((p!|rOVQ;oI z+8;_#ApHC~;MSZ6coZnIpQtVtGWg^isOkO!o<{4&U(n>D@Uqwcg+f;*~P=W<9AF~M9^jmkqnC;84_G{X}J}`=~dix z^I}7(YqalL)xGB{3f1<2sd@Xo<2lamw&)hj_{M}MTHo?XzlqK#5BwkUNEewYkS!~~P~T?YeEiI>JuLTs7Y2OO`Ac~tEHtJuij!`PQcUo25F&`@PIKn}uM+mcXOuT1QNb2H!q~ zrF0e@+j@@+nD`~{A>?$T>1&C8Z*EP7J(6=vWgc30?j3XMw}Edd0Q1YI^SYfbT!~Xr zV2Kr&Hm3G=jy4CvKSdjw0zBCeIOCMaSg{hLXG*N$i%eKIXvjzNCexXU^Kg(s>iGc@ zS+v(L|1n6K5nM7gInRdx3%^0Pd?oAZXUgn7MQz*vY2SkM|PCfSL zo|$7emRAqo-yO^i&TTyyDfH4&4Zi@%L`3~%QaBWSb=@0n&HiQrYofKvCyN<9!G0(} zTH=%vV2P0E!OCLVaIa5ToZI?BbVGF>-GBRQ5Rm|rymyvR(lj*o2({|ZG^ZpXN}g%+ z5%~vYT%D0ei#ql^^yi;Fz`S<(Buhg){=|^39BDGh(RIA+tXfj>E=3Ns&A9e3JMIwT zLitjrSU8jwz#pk1A33*BH?P)Z37ea?^-RjIT3AUaQ2Rn-sDz)42T7!Y@suj|L(lzIm!JN1WEdJdrZJCd^o49!!~6vMn77!GAc9`-}L?;QjU7L1^cZCtSFYxSYksbMquVI=@!VT=x z%s5NsOf-9Z-X4Dq3TJ8=XT&D*$mJk=C92qOjb2{>fH6V+^e#!$Zc72$>)R@_F&$+! z7>W`AS(<53zNib*igGF3^+_Y&Sh&aRIX4uA&+YSz=}p|xApl|j-cYxJqJuSnkeq1& z5DwGH8(x!R%tfY{Rm=?1F#!mN5nk`c%O@B>xP+6sL@11R*M8VRIX3{|Fx}hROR~j}RJ{t5`)?EeG zv8o{LXYm55-#e~ZECYZ1|1z|*Q2l8~cCf_;_Ah;aw1|dmwp+Erowh8lVs+KT&pc?* z&l0hAx!Wg47$y4kC3skx_J3TvT6f(~pN>rtR~lG{Q$YPCZ!=mYkodCvmg$LsUai$A z#&D6-vu_RK&40mS1fuT1+UVD5YK=Rxw66zut;#Iif4o}9=o1uSLi95GNsMg1iFa(9 zArFRYRjxsDz3m&&)nGLtWU09V^AXvgTjlV9$UMkJJPQ8t^|~nNd620lRmV#c9meg) zpoZ)=u(HORxE@Nn=+hxG?QVqse_o>1tVd1_Nl>2^#T-68U zDj~ih9T|udntU2#c+eF&h;nW7v%r82)o4 zYZOR@6};(la%~LAEF6>~DEkwPPu7!2rUNplrBWCh=>KFPS#GJ;VAK+bHnQhTTgh`xihQzA5VlH+RW$KK-nuA&&h?6J^^Af27fnz7wkD z=6%;`G!8b`Pq8GH1Zk?q2_Y;J-iA`b1{X`m!zC5Q7=*bFfu&)@ZS@FW*9*PW4+i;% zT{?G+wLEE`y$nf$5;izrJQ_-`=Z-BKD}c!w?@&*d$f0c{Dc7;e!`35$G{(9=35Yl0 zOnO6{iJ54a&u~~Ajen3fk18(sTa>fYDUV{>^{=TALmHxq?j~W`6KfKPa`!_ACF=l} zDS*?ifFtNlMaQJ%NJozqfF>0Fl8hPG~bW z0FiHGg_E&6Ht%Yr(PE92LiOr$^f^!OLhWadb0z zlhT2qS?_^Rqz{s4_&lybBxv=ij#2*X3f zL`_H8%l@F$QFv@oxe70g^yd#POoRIJq6jA?7N{MVpD`-{j~6k!PFanTcIWS$%^U0J zk@^@VY+>GjIaGs5O@+-T*?^xXQ>FeK`ToYOHu8n)YG9mR?)cxlfzyh-$sVYPv$X?Z zXzQ1UQDCZ0KRn-;kPVyuamis`DEN@k2MlRPjYhgbA*#=T=%hxA7k3-bOrcGogV4wV zJdVXW|JGaDKW3A|`WJbypDfBXc2{*XTG@J_m$CyWJpjPE_q~HuGqx-pnM_sDEr4v7 z&#ZDZW8Z!hCgXk>3{(fI!MDu7+ed-dV>8R_4-u>3OPa!MQgXo8^Y<-|2=R3G9QOIV zI}fydlmP4MmvDrXAKy6YgUUL1Ob>on!~q zm+>OJr+mS_IG1<_SRf<)im{Q2&5%K5!oZ!aw@6^U;+^mGvf;+V`u2u-fr@#h+hhQM zAv*twe0kpX;~UZ$tpl-vJwI}3~RX&rzZ1QKRYyrvO^WR zlbz#6VO|3bitk|)TCU8!;Hl^hrd@#iMNKb?n%a8$t>@5!6zu`P6T77EK=23Kuu^sE z&^S*-oK_N~+IlnHzM`jl1|wg9@^Q3igvj1EH#{uv86YgEzngN{K;6g53&rv1kLCBV z31!K~bw@*jV8f=n4UvN|A%)%PP!A+NFj84Y7^vUqX4AqIzy5#PLH_*X5I)I z#EuNF;V1cn>?Tv4^Vv=!@`=E4y27APp}z2gTrTf6nI5ytT1*epqu*TUU;o%M^+_6p68l|fA5lAb{JyWPffe^bu012vM>cB>`|fd z7$^Cs&$3*nM&A4qKyv#EdDiKv8{P?HXd(|2dgO)@5K?rjXi*HQV?w8_bwgBB+cg*} zePcsIDpDVep2gOLK3FvGem5VoC0{Sz`mi)EZ{V&$=5Bhb1iKMvd4W$3_P-L!57CY> z3)>f!*i$hzb&DbmiTdz~GkY${QHwwnE%*H7S*zPOCXl;;EtkXs2?mD6Xv(Nz{xbmafp_T@<^+kEvvQ9Xp zoN`y(<42P`$uo+Fx^JIx{#Ly;;ui>yracL{~r-#GEbN7+5;|z8%Z4(O)Ffj(Z5t{ETV*OWL@2*ochy2+M!_!zVzP z0BM!3O3}yWb1A#B?3+17E)QYw-LHf<31U~pT^iBe*Zjy(Su`@-mqJCkS9@x1z4`3e zFe-zd8qLlDmPBL3(#|O@p+DJ0e6$%)nCD4wOnP|@07J{i5y-RTXLl6WZM4HD4(RjEus!!fbJz?HV2wCfzznL0zbH5IiBs5l;i0CMgm} ztkRuFc;_@A^`9=kRsbhp|K#f`gTx3*9j2d|6OjLl9-0*uJ|=QWaOxgYVa5CN+fGC(4YU#^5s!FhF>YVH}e@qc(`=F=7>BkfC57#JKYtB zHh53%0bFLSv!MmeBY~lL*PS4ZEq-EHAI=VBnyngsFndfjfIpKqwa=&1wR&f3CsLJ& zh$eWwJk4{3Kg=VR^Y&K_%_Id1-D>sXNQao0QzD+hP97CJM2D`b`U~LNl#1aI6qRb<8YgpmZ zgUWgu7ZlW%ZxI;mFb4KA>sMYm-KOEXDXxyiC;+&@dcgy_!uc3rHAGHO3LI%9So_K_iW3}|T{tD9y6N7$H?(9)ZTChudDcuQQconqpNz3BlryQ(2g+r&& z2@2{TEEI7?`X=Slc&Nqd|6q_aNuBounRCnY(jNhLK|k&QcrkTO#oxaV*5&i-)NZ~W zARz1`a;RmmKEm3=U@I`euX*+5;La(3s2$`?`R!209f+$MVxN2JY zro?i1)aEd}`|3q-8mnaoy~B%!<}Q04PJBrOPvqCi8Br^*yqxc9r*IT-;a|PRHJreE zJf|tx&s@}_jYgvPy>s3~Dm-nOqB;5>BOEIM-PvPiB)$27*0A)Tc5HugEmrUx!tgX> zvyrv{*e*kFH>X%;z_T#TYtsBQGljk1DJRusHxs|kLpzX$U_y^;l93d60OQCknCQPF zlcm6SD%0hxhh7AAIIHK`djcL|-u719*ub9WfZ6H8=_m(ae)D)u?dY-o*VF(Ko?TF{ zbm7JH<1B4r!n@9Op2kwlbgU{CVB3C$+BT(AA_;lF?>yGXZ)k8wIXyiR)`k)0(isI3 z_8pGy^vyCK=K!2$5p8dEbHb9|Yp$LkIA{BTvk}u=wQ|Hzv0Y6C;Z0hE{R)H|TJgMA zm7Hf9TZDn!{+C^9-2yXBb1UwtD6el5a5|~+v8+f>PqyA^Yz_5mB2oZq9|?e`|G)z~ zvO^?X6Rmiq?tO)M&OQ1g639CBK^kFeWptNw*3OSj{8@ zW}6+t(Y8aMI&!jWNfzcx_16pnoOKWk0WpyG-gjdNSl1#@+iX4VAGl=mj)N3tR)xNb zsu$NSU0x}NRVUdRdayDz<;ua#9K8vU5A zD5w!>&|!$lodZERds&K-iV@$4fq~UJ40TID(I@}vANiFQ7=K;W;L)t?i^oSYzc z{d2*UsVpk#nJzpwCd=JI;86n$@jI^Y7g24zEwZ_BVnpoh0ghvC*VQ~wYhuusDoBe) zg;3s2=A#u*mz&M@(+i*&j|Tn*VA2x&83&5z=!)1x1eCz9dU} z!QGpVve5Q(J60tF@YHT4!N`5&LiT(Iy1{SGwu-4bJ~kcz-Ln;*7uPM-zU!lth<}rw z7D%@fNHgOiGeKUs)db<4Pgwb1ZX_^{g&;h4`iSf_ogb_#OGfv_`iquk2zjzfz=rTl z&VB$=fR8sGZnYI1HHE7Yi)GLCGtXa0szpV%Zf6@_LCiR8C?ml*i{Mt$H)Qgt??A)4O1N%M|jwIL7AOWo(Wft5KY?Uq z-;j<3o(L8QP)o!h8O?cp=sJ@LK{A^1deL1DBe)zhwlKZmKIZTF$I zq<03OXXZRHV-L!M)53oJP8DKGVJ5leE|!`yKvFf%wHK&)4CKh{+u#&oX@5_fnY8VhKdMAI^}Fj~+tdM}Ng;MvSRTOnqc( zCM#8K5GGIi2&~Fzn>)3i_!0v-dx=&-J`M5#`uo!B8)qmufhj8SRR!#rv@}9%gPJ0c zp*xWHYs5`PJOH7%KcmIFA66Yq6Uf*P-3b7YnIUyzYsJ^s-itW!&;R~2G+OE0V`W9^+dOBr7x6FU;puwXvZ9n5O9+>QIaHZ`FWi>ed;@FSfNm zN%B9X*5W&bD<*KzMs1OKS*6!86rus>XR*=g4t@H zg?R=PG)HsNJ5>?A-je}h>6FAn2gK4TjfM7$rBjO4&0YbpNGD2aip`U|iVfJ&7BVgRZMMzBurJL1pIVOeQi zHehw2kzYdez7*;~vNwEawUf&9r+dawja!=r$#(~Y9;IFK<=~zL6qwMPf*Z3mRt%a<bLz6XCtK@G>FKwUiBg!IpM*1(ijsz{UAGZ$`YX*wNC85AbZ` zOFL3trkKEQjJfGurz;x8o2zQ>%T1p~O6@cmki>6^kzAq$RVX;FRgTBkobzVb+~6#` z6RE+TGDJ*qNyVYKG!2H49xT3H`i$#Z3L;QB*A%kgAxG}d>PAVu%Yc|RUTwESxIFKYHTVgCP z(JSX*`sU*>=9%1T`N~-bdj@srsszF1cl1}TU?j8CQ!1_(RI$a{?YghCQjJH9>IetC z{fN+eY#GL4J=G|iDyn)TR3n^NQnFPtKz+1@>!5|gEQeMB4BC9LT_2HpLF=lKVMLHp8q`T2wwM)_P=To2O zwUNK15608YPxnSPepER~Ul+xUE|9*_gr#jQ6J(Prb$l}-BSq>gGQD=vmX!v-fIrNw-vxliOK80yPj2S&f`?Ys12_t?J;>R(M zWj1Y;?f3u+Q_L4t;4}`!sw79*%!X7nbJWp#L)_aez4aEtgFvQT5x z<^=G>s$h-k*wff5{{A9D*|3YLH*#6FoSU@2z!_@;sE`lv)l3Cx>=tqjHY_Ai-bZ!az=cz7m+r903=6i zI>4T=jg#GeWZQ`ux)wev^q)m#8H%PZ@A=Iy+v%7vp4tr06+wVn_rT8{@e8!HW|I1C zOHPe2Es!Devds0jLNn^7N3$xok=e1!)n%;-W9ovb_l#PJ)31NGbd8T4$$s2?T>KGR zVX*tlWCFEA@FbmuH73xn1PHe4yhm!m2S$4kmJD2)$%BS43NVm*t^D#oL#?hcW5;R# z?p{g5h~I|zaozuC`lU05lAiav@OD>mOqLOnwBwkU*qSBe&{6e`;d@n@l1I0871->W zOARFwTJ)6`ku>5FsfvI*?L}^GxVH$|mMJPg5ch0_$W~}RF(*tN6)RLrB!KypR&VsA zvJ%s+J>QzOrFE&Pl#++SCf#{9)!bI+WJXAUf?9Qw+Zn`e8JlA*T9dv>LA+dJ&$v=n zCMn>RDEv~d+q;$rv5_mZJlx&v4-N2&oyL=^QF)0k6jA`vsm8viq3d7XHo5#6`grUO zyfX4v^uc(%{$MT2DqSQ|q%p6NfUcnwQTq2`C(al#J~mYD`AeGMC+p^$FPqUQSdlE6 zZg11mJ(g+Jf~_|Tw3Rb6`Nb?76G|*#=zOS<* z+3gv7TDR&78F?sd(%mPMw9(Q`ubh6?a-p7)E4#&Z`i#;A#2D*Xi7hbLedSVtnn8Gy zj>2k_SU?EAa~emBa}+|515-5JbVa%s)t=BNtE1C>`owSSz?=GN`{DY|Uv<~)5;XJ9 zzoqKweF?FP%X3PmtWn^nWM%03OjU77G0$ol(`l2%l9?GgL*4(Gf-1aJgf zYEn$fgijrg8i}Fd;gI^tJJ1=Af|7gyle|hzr{;v|oIA#T<}Q#W{#Bhbi_y$G{>u2V za{d$F|M@)#$)_*lC`>V*RDpFR(rCK(J1!<<&v~2%RZy<~wARNpzT1WR%?EG%%US-m z(pDBi%^_veo79zuMK|>cP&Ux8JB*|p_BdWGS_lHBudUhV#{C_baR|c9ws&ukF`wgU zl2IIk#~prTH|dd$`lvPD8(cbr#nEC_np-Af_5$3QP8JyXx@C!VhQVKr)4{DAlhG9W zEG5;|0zJQaB(sWjM2@>JGz&U@_fp+r<<|W;Tc$v^)LQ#VvRPh4*F&9o3DjS8nkt#V z8@Z~K8P(Oo={=5GWnyF$LzoLIz!!p|_Eh+p-K;BN(%-1;EAt*}D*QFi_Xfcf3fa>> z-9)LrzPii?rp~sA-l$QXD534DN;eQW?!Q~j^}L9s)z#26F;@WgHEc}1GSQmw#Rn{! ztxne$9Aah@kOI7Wb}&pn(oe22`jfv24i};V2RC$vo+;erz~xMFGjKYM$K#n+7qtN3 zrun~`#b0xN!h_)M`Sy0SB3KV4c^ygVAF$@=<0e_o`YmJ4S7;QZTew9KSxVP=A_ z^t+|>EFi@gpEy?^ICnYckkZ+>BX#}3;nwynMe%|sE+)LZvX(}EIL&7FB>s284vx#= z%Ez};6l$t*;thmoaZb4RctGf97=svctOgOZRhGMzU36uf^0DI%E&=Ol_Q1}mVxswc zB@6@;?!gXc;SvH^jX6hA^#hZbRBkbk6B*8RR$?LoD0N)3UVwNa`%ww(RmOW%qW6HRGiwmiP0uuU#c? z56W>L!u_dI%Aqw;?bWusuz~M-h{}m#f7?W6ojH^qJ4B@5rf`m!{7IhaB^TAaH-f-c zivzUBgWB)jCZLBwD>M7t7&WmqyuU=({T9J{F9`kZO`gFRvw{vls1(X%k_Vw8%+mw3 zD3X~x5%!zM14mRxRn5KRb6CIA=m@r0)-xhEKlz-GkQ)? zU_I0cZLU)&8mb?~;hd)$ROXgeyjaI&mRMe|w_R<`6WiN^D3Hp(HQ=Yg3@t#`qz*as zoa+m{lCX&hGsQ48{Db&x?ASL?k3kS-w}0R#)@(=7r{BAG^{Qm^?EI*K&eju0_2{uT z$UpeuYEH@-fT(7TZ(@*Sc3&V@K(0Szw_H4H0=+Vpgah6Hdqa0&TAq+~lJnlvun;Xo zmE|@{5HSmGEFDZ_6Y%HBkE+6ThK*7hwqG2o7QwG-pUcSc3YJz=)laNbQ$rix;lP%D zaBBO;rv};DVE>W@%==opxF-gmC81t(sfpV*p3jW^p7aZ$ zHbXu@%LY!H_XCEPgU|(q!pFJ@B#}LvHpti{f)>X4^ee)A16k42f&bFo>j*yi94tn* z2RnPV7K=4wKXz?=cP{aLw_ns8YG`__G|?*`?zX`7rMGU9M0aRS3uL;>bKPD@u?M)F zB$t!a#rVQ2{iF20oEj*0n2^7CF`c=C(9bz!&W^aG+Uki_S^@HcCI1CD!;hpbFRy98 zv2{~%qk57u;}B|?*S(YO$W?uxX(3`Q*h}4=J6VlCIPXVFslssB!*-$@F^XFP%uxt+ z@Y)H6XJ#Yw|6tmOJ3o#_hQkU>QxQ%bXe|U%q=A1{!q4j8=DjW+*xf8nM8e(``A(8F z+Y=Sph1eKm!4he7xUbjZD$ZF%&uM;1QlPhiwP)mLUEON1ePIS$cJ!pY&q z3DcSIL)WpYNwMCV2w8H`u3OoVi+$hf6Qc!P|qu}c+@zQ zq&@243bJvsHrawh!c1QdGq*T{NI2~SUc{wnsIuK5*3+FK%Pz@g<3TqSG@bD0`UqpRgxXb74UGXbVmH1Iy-{+Q{e==Lg!@bD$n>e;%nONFm6O%vc) z=XmeNu?fAgsxt4RfIs2PhzsGAXD82U#sq}MZEDgf991x&UabYLd4rXBY|hCy^p{ZV zUSz-H<8vBhxi4Xd$fvZ9hg`TR!A#?;lB^WWBx@xc(gJ!5mswDMnmjAuRJatR zBV$6%5&XQ&?K6761Keo|SIT_6+y09cBLrCd(qB|bObhXg0obbTs{&hhUH=J!eS)`wtV3w?p>U%ICmh) z*tf{-Pi7$+Ds!Yy>fA=uJEnF!H-$l=t4OcZi8BAG6aPmgmO0=mz+Q$j$pcrFKzKL| zRmG3}I;u>sa%QeKtc@4iutgtjI<$iC5Lk_vl-P$?H<}-jVXYgVd!w*y<#h9`TJCYqK`2l z6o0~;dztV-hQHpWOIe10npgZf5_J}}4}Ns*d^8NsGV@Mh{V=Xh zHoLD0f%X8Vd#Ty8NGEhvw+VjG&$u}#d{x?et`p2h!r<&BzPrLM%mG5XNW5vEi+n6n zrdr0~MrdL~chU&i^+&a8|rVZb{_W1&rZOXy|uCY+aLNV3B- z#=qD1r3vbO@i^LoaT696>&GW+zcmy%%5E7*Ns}VK;rOmxJ01*!^US<67_bB=I0(W;W zk8#}OoZ*6-ivwP67l>%3-$g!t2EXI2D=opkNc&g-G1C8R6+(M}wN>@O@ zKWCxEoO5j$Y8o|i!>l0iQMCBWMjZ>>MLecK8H+Vt;A=W5V8!3;v8!2$3}O$Y-J4o8 zcaJzw5Q3|T>GFBWk0EvqyAL9YuII<=PF$Z4keM~hhOH&_v%Z}!-?ze$p6POzs2fDU z6JfcH+x)tIa_jFVI7GM6+J$!8D4uCcSZx#uPG~;RM7aUUN|t4nR&-Zo)S-}dP*y=S zxb>^rcmOB001w)#p+ke(9V@;n`3^HaJE9cb9YQW0&QcTWe|}ALom|3WvQ{|}^{5!9 z4w-(XIKL7jaOwd|qT^M?>PiSPX6lj6V?x(L>^CAS1)O#JL)zbWJQ&lF1X4)6REw{L z)601+Ys??1r1nac9<9gE$=5&qVE_OaZ%D*}xx{T2C3^;J*gzdAt#XGC`nNZ#r4AeE z&_CS}sz{%5veI=WL%y?VXtq21H(f)0hV_CyWN#69ga+QyGH+g#VBe41hsCKX=?GZh zh9sOgsIrnPl)J;HZP*}wbH;QT$oTc84TCXv@qbF5U{r?6Ni*Ze2Z~J$X}N!*O%in) z>{O34tl^p)&j*D49_fbX{g)(ZU*P&14O@AF(F*@|%2D}z{7MZ(qOna{L=~*_{~0)x z&~-H7$h&WtO$08zf9N)!^yyV?rWJu217ZbkI0@H446&Em|MOST-Z=tq)X9K1AFrV` zs62au*r=m4%T5*eVSmt#omF>P1Dy@0BL@H0M^8RVvB<2S=eizq8Dbe-9E@xEaLA^} zzOOR`jJMsL2Dn{|f7|ZXgRHnW@GT;MGynp)4`?TjLKx+#_I6_}lQlR4k03~bN2uo! z#BHa>;pQ`t{&TF4sO|>wD323Vqqqa*Z&L+<#@C8Ej5{F5W0jDfYDM%^o(&RZWITCT z?ITORDpPoJRTC)(rVc`T^ETVS~owww=<<%Jy6O5>ct zjG5sgsd4pDLc^)7$MKijgL%JA$R8Ht1-N#}w|r*%c(I-0vk2rBlxUNnbw+w@xFE(3 z67sTbMY|u4pt8ozQouaO^hq;QXvWWt_-7}f#;Zqh{Q!V&e46JI$OoD&w0>WU6U2G? zsgE!Z@Zl9s`B0M7iJ>L`AQ_h}{q9F<#9OO76;O|$oZNAk{uVLwF*CUIIF5)$@3($ z36w7&c#{LOzTGhwu#FMqM?`V&>1F>&2@Sd^;Lp(_jMY$%V$x{D_GMhc_q{DSSre&mz=-pg zg{7Z~!OG9W{Ho$@InTo#0iAvL;*IO)Jl4s|EaJ5@k~$glK%+hQ&HT}MwISk%az25- zDGp-?S;Mgb6GFd&v31VYdS>iFVr!fZp>W{WMVkG=MynKlF6nhg;D#H+XCZ!z=QRj8 z8{5g<;u3qFkIg40ykL%MIa<72s6PDd59Q>9XNdnr#={C=jnCHr1l-kDs`9hACN@U@C!k!oh+>Ol55wKXo*KjiE;wE2Bna$XrGoG@$7`UHMzL99KXv(I0mt;>$XJAWFr$k zyKh{Y3}j3N`B_;yv_5ptN^(P#yD>vSn=vYS2XoujsV*Xan`lwhuw?5mz3n=IvzVtt zFS+ALWbb}9R_bwN#0TvDx?37eA%gBoT?e|vJ+f7`z*qm<(}Vc?R>#9&<7Ztq?hNLjE$> z$hD~MZ&9dhV;jWjkmh_bVEU_1x%zvmUrXFj>4!3&wLAWYybUw&an#=^LXf(M<&Rt+ zH@bH;*L;IO8oFA!+g4siz9GGF0Dyofiv&--ha=7ekH4BuDg~kdsljwDuG|W`6|wZI zdK&r;^BB!d0_;W|g#2^DYwuJ=?%8Iht%+03JJMjaO2DH_=-O?5n^6}(a9y;^R zQ9HmXzq0P9J$_l8Evv|g4b3r5+&#(d=}`%N>u)X$knr4PM~U&&()6rqRa?e<5@P2c zCeY0$UnHsYVN$g^(R^?-P_)H+0KoJ0Qod&flGYV2%>s6^dpif|S#g%tOv54v9lgxy zR*H>;X?gO7`8MvfS_%KjaDomcob z-icDrT9mD_tfQq(xCyzts%e_zBK{HXL!F4}mNsY66&6ZGm(Vm&^c65cXmF;0)Ld0{ zsLJF!9-Mzvs$cAM9_3(}>iRoG$0z?q+y7UkS=5-nbU--eU6-u>IOr0%fYVjYsfVov z0%n}D39+*KFSdq4AKV$LH&5Nxpe=qd149Bjtw;eG0WwdxJ(V4;bb!m0zbYJg&E&lC z<#EAP3PzPMf7SRKC`^1iI5AdHxfxp>uRR5;S&jCBNnbdJ@{CCw87D#npRfbV<`J_h zbwGD>#?anCj9pAFJ}nJQ#-dL^qi{@&IylZlm73pdM+S*?Qe;xxke;m3rd^gV-CwM< z>VfK(8+Sr0gxe|(LIkX~5ckY3j>x2yABV1%-RXzwGi+sUj?pV+J8FvL7v_}~a||t8 zXEyBydxsp9^T}&_n=mE#{wXf!T!wa-n8kN5Io*-2c)Skq3zqWo%c`>AA!=|AxSJ^~ zE)b(N16aD6iyuo=?xIdu+fpfH1QGc!=}Lw@HDgCE;oJJAr;xcQSW70@i-LTX3I3YB z=~$x)yMf-;MG17FUVS?PJwr3R~D5@ zeBPZ#F_nzorX~rOJtDwHDlaRf+1AKHHrJj}Y#3Y=PS)KW7$Ne5A<~B-Hxg7xuF>PQ z08G|>Fj)_vggf#Yr5?U)=~p9Ay09x~nQm&f0}rht#nlpBpv+FWooOKirXBZC!G{nPil5D}Ho# zA@&3W-6XkSTm*g?I?K*>f~Lk@REeR&7l};o_$h&=`DH+1uL@S0U?-fiF*Cg+!RI9E zpQm^Q=^vmi@95~y;VB#+$nbP8zUVIneaxRKr7Anp`T%(Y+>t5kIOxQec@KR0o=7PQ~L z`S^bom&5h@hpn#5vp+!dNt%~iK@@z{OF)%~+2w(obNm4c@VjOR8+}fFRxqg*K;3az zkSYdLv=i2trESln-FU5_(MH-bzswXFoCfXQiht3>M8db;{j`$IOAS9k7?Dk}S(A(q zW;{*&zR-OU?7mcaK5*Md2BSA-tQdU-;1Xt!4Ak(_qxcT|jftw2`G}_55EWm~g;DtV z4qxmtag!!crZJu;NXf8-cYhtWh00gPy;lN6QYk;?1e&n#SBC#awVm_2cCT0}#p|xN zxcEz+5EL38AIi-auN=qVk}@enp0&r!OW#)&yw)E5?se_fAQV|J#yJPi0tROv_KJz* zapJJAVj&qDr)_x#(8XOkW(F-;+gL^ zze5fYNToOYP4KT5Fb!J{?p&{(LF=iEb*$d|5n&M|A3&7nzcds z!p|QwntASd20O)dB*k4f9riq`lU;1AIuFyFpV75KrNJlXrZ92gcWO|XItjzWR|`bu zZSbY(TOKEZwie;CzvDr`5-H&gvz4?i9KwyKIyHr+s%2n(*MH8^nN62X25y7c)ae;q zrd9)o2E5t>I=?6R-Zc@60daftzpm^G6^ggY#EdS=wlmeN7&A>)d3?Ln-`Gn|@h7nX zY*k^t{Og_ygPS;;xO(N$cF=B%@jgR~UQb5PyTBji&S_<&VeiJx#)hkg8aEL%GP@ew zm@bC42R8w~_uH*ZY6zz<4$>hhH8P&e#T=%9^D!a$C&6$3ma>O3V+AXZ-`cFqUu1y& zI{5Hkm2h^5by6VRf{_x zle#_Bu_$d5^QqfM{!Vy0?78oFL)@yUdxKn7glHUnwDMn|j}g~oqVn8a-eP6*FrQm#+S+pO+{>;jes1DEecE*DaBre9JFJXjExBe!Boy{b$|${G zQwu;M%&D&(s)iw9(KDXD4%H1Q3!lB}uA}1m7XZRnECnTRB#ZUu^sc_tWSQp$z+)|| z|L3HM1Rb$Z*ELF$&~8m{#tN@M81*kEngI5o(cegiR1YR{fGtr8$fw5^t?Or4Bsuc6_>z}aXx514`(NPq;*6?+l85w6ax?Vo^1 z?m0@133cZio;q8x)IKcQXazmXVjKuXVs7CQqw&PAR?Oq4{AiP-Ls734MS#!J`iM%9 zZidpFK6Uq9m??ZsRZK7Oto9`cMOEZ6(9C*g9n>^@afzO`CjuXM z%7hlK88zBbYr-S059M@okWl7t0u~iskfUY5AdO%fG5Oo92OUC^uC_N~FO0+J_RG3l z`?uLcawTF}OdN@}P*6H}7U&t=GJB=PSpor^^4A20xI=2B))ut(k(G48E5aE+=e|&& zYF_l|p4J6x8Nj@LI_n;9!OMQuUg@+47Xr5UiWcf~$`G>o@W;8!Gg81ad|wo^#F|~{ zAQ<0wt4hP6BtHvI19s~EYcb*k=XM%XlT$llFVEd*T_AX?3PHM>Rk<-%u|aNOJ$Zny z$++IHoi^RhK@rzE!&^qgFuvq`re}Yi`lj>yD>fHvAKkKG=Fq0Ui*mLC_lVZD?{`aL z_bqhpMD9^8cpXyGfb9?bZD<>WxOC=$e5n3QIUlmqPdzExWpdXQq>!DHa6K zeo8l7@!#!Y8Uq{wk2k+fSwmF**7pBQEYV{@B&`230;|-b2#}x_!qYSaPw6YLNpA6$ zs+ou_eDbG{>QZqpz}?JA=J3WbbIuj}+IxR>p0=YsX7yD1?m-9!DOLD#z)i`Vq?hzI zva!dDARrX{k}TK^Ie5nyB@u!A#Sj^jIQ8s2(LiS(5VIyh5ZP7#k1xoRX0M{IaPI)7 zE}#uZ#c|NZ>CU*!EdLDYl(&3qvKVpSzRIiiTF%vS^iDd@g&3lyP)7Dm3doHe)efgz z&4@}92@_`TuLz3Da97x8j`2Z03qfis!eVDRIq`5pCcU{9T&|a4_s%{SQk+1ccqi&M zP_lp`1m#*$=(9vtY){7#3ccL0@aI>U$7i-|6PF95Q-jL~@cP8oR@T{1L^aV9^^+;| z$cs(^*Po{PVv=}jo{Fv>APoV6vnNK|T8aAOT|N^R8~xNv#jsXk5tK|I%33{C5g0)6s)g zY|yHcD5e6hJD8R=`(kSS$AG`75bAriF%j{H`L!gv?Z|w!8vUn5*_Fy{miuN<8+Qs5fn`L2nrQH0`ppqv@<45o!38u{y1CYv5D?*pT(4s9&t_n{r{_RuH9VHz*V z?LQvoi6^rqWc_i1Z{4)E>sI z@v;wOqrqg6pT)?0rs78(tyAMvGl@TXuNTHXdtfog|5kq6&ZNlgdbcgbls_*Dg9HmX zqL~=9-K$+(H$eM5RJ}dghxq^|rk)6M*mk0CTgT;M1T-|?NUQQ!G_|R6;YIH7 z!|eGKLt+}jmH|puP};D!@Y&p0oW+kvtVm6=Pz|}hjg`g#kUefglXI5vmI!V+^FINw z!rF9-VMdO;V*xCrY$kFbOMcg}PyUp3jmLW?i7}R4XP>7MKi zLz|xaB1^kDhP;lKkX)xfDk-CY{l*$Rj3Y1Gjl9s@Ik~;38|Dr%_Gv%=N$>KlCZ2yV z(seRXa9LOjzjtrj_=ssZwC)De6x=;)(=uVf%&Mh`qf=>sJX5^%xgcQF=UB-w{+$3& ziDnsOS{)f z6b@Q_Q>$`{=id0ZaMio6cKB_6yfD3{xVmblVDXljFA(Hj|!zb!ldxHBLA9kg8OhCySXNxmg3FRfDs z;hIn|9K;q0sMnFaez&3+G;>XL)~Uu~=#(P}!^|ON1GSMW@>)M0!1nA3e-i?5`klEqq_FY@o(OSxWb1w z$jSEx4LM`WVIG%LAnVayIJ^L}tKD;9cv|IH6rW(SK>}T-f#Q1cW5H@9J_#&wzFEWW z{k>dZSVbZi1Q#$fUR!Y!SY;wl8(TCUfRfLy`OrF67(t`sEoxt78$r{`p;<^BR4E(| zMOC9k>k}H0tn$ufK6PWHeLWz$dcl1%)jm#d|JzM1^2g_5UJoP`fb+aC)=?EU+GW%t zK&AO?FxBzz&OPZPPU4rkJ_u>z=5JS!qJb$1E3jQ~(c$bWQgFg*^BYM+jRg(smHqpg zOTIaU_e1OOje-SJL^$Ktp@Hp(J}lLZ3MNWFf9zP|F7XL6ttajJC{?12EviRC#Vwj4 zyv8w=sa|{dicuKZ626D~QEgfI51|>)uY2EhtW{tr{TR4w# zJ|5ErT`?V!v&Uv$3gJ%LE0i0+wqadWt1I<{{kY19>;Z;L!^*_ou(d3=7RmDiM0& z^?#!y>wh-p6-&9uWRe?p9Z*WfHLu&D=MJeQ=QvmY{v#g*0U=4qu}P}GOrT>3kW!0v z?FMTbhuHI70-6yZR`Rf2aNOo9)W%F!ez7yLJ2@cO-kH1%Q2-j??qmhtBLc6>stQt2=wgDt`&k~!g!PVlkTpLM$!b%JNAHVyafycU0Tfd90NW!Q%=C^N19@E=9d! zjrXfclmtWIJ)4+`(R$AtXF_5)vsM7u#!^RMHwf4-mofK;J>!kBZ8@kho*v!(EUFbk z;hxV*0dIMQ>uXdpEgCz7Cb>E&)mvjMA!3F{}$4`8nA5!yN z)QBf5UB;NLUh{_a$#7GJXcmgi~|6JwjtIzq9plnAWnb;vo z=DQ+5;7cxaJqyqU&JD4sZYYcH8kY-y`u7e659;9LPK9ZI-cns zx|uef`xntpig=_k(d1jyBWp(*xT81&*!M|FI8W@D3F?3tO9j(UYfpv3>cQA)ZTJg7 zwyjHk*K4+iUBd>+mT6<(<_0NnE}Gqm_UF7ca9eB^Xfcy`Ur1eGI*E$LVjMs;NND*q zl6Zo^&c(7IGXqryD=w^PUgW{!epu&aI|DS70bLI>6gPk*VVd&=p_}}c+GwQ}+-tp)8%KjZuna*`K=zC(^u7AB{eTV&ZTp4C6YbNll zh7JYIE5Re=XML~kwuIANtA|_C$;N%tlBbRMP$F-rP!AcLdfohmbXJsjPlb1|n37P! zog29jq4$GtKrU3VEwkEnTJz_FHpE`;Gv0Fv)bnE4jk>rDaK&H;0U+k%R#RE4D$2i- zvTvLCfd>aCdJXQLYH7NGBGfa5<+v}agpE~B3(UqKk2sOYvz>xi!Tty`T;X*SVnn{^S&psXa;#7(%K`0>ez%6tXv%tZJh7(`)fh?DW(o~37cm^MVed1=P-6Bi<$*~`O}=ws<<#!}{EBlh?+OOFeBa^Ph3 z{1Pa-G0w%0#+;=A1el%NNCc~?%JNEA@|AG3pJThCc#Z`!cWS{9(za7-xtVrUHPs%g z&rmSgSLe3Ze*Pp^sPUFx5SB%Gi)v{L80w_**&@1S$6I1T-LJ>mj~`1X7AqHAvRgt0 z$>x7m5qvJ(+0cI9E+lUK!J8^?D+y+{Kfu;r$TkLZG8<(%UA;{SVHdr*14IJF}{lyv=N z5Iks$$SV9OdP5uz8sx`jJ$2z8VWC_^KXX)p8U|>N^yW5}Ime5ET76`~r9}a(Iv1Kg zJtlb0=oqPUq1OL_0uR{k1BYk4MuKLGfm3(q znSg(?SbhwBJ_e((N4PM^uK_bdGXSv#rjKzpdH_0Gt2Mfz6L!ke-JE;re_b%Shbmf6 zWn$}~AHMzadAVv_!~cCi0L`NA$!3aL`e)xhv8F$wJbHZFV-Xp%3yg0S8Cdue7fE-7 z-MxUcF8TUEd0eH&1MH@aU9awX@#WvoJ_WZ7fdOUkP{`j8+}klt#q#9Rf7;jS&D3Z7 zvY@MQ^4v=S5x`zW^{PU*Bj~MSKBOS}8`GTcdmFdLidP7?oLDxehoZr7Hp_BB6%6{y zZBMa=dWZa%xi2Z>v0yePNTfre0MqMK0nMgK<0|FG`+q8zfWHS-CW(sD81;Xa;t<#J z^rB}h^=`rt>{(mFt}G4q8ueK`pG386z`#z*A#IMGs+dgR9_F*1ZVaisOSwfL>Z{yS zu6}Of26nVb)kx|h#QYj&B|0U{cf8>>PV{gFuZEQ(&~M`BBnX{*3Bj>b_5pG zqKDS@Q~d1@w>XmXFE<@iZw$X9V?9B%a+{S~!oaSjj-#`o*r}5H&mEZi>oot8BvElz zym+0D?rZFhzJ$vz?{8vbk@~2j8GtT>bFK__49gRr0OR)mAvPl5zY(795bz{tf`^(k zrC0hN7iRE`!}YX2xhF*I>Ry(;LV&lqw~!n`zDmz&@C+fG=KW!}6NZP1Ad0v9Q&yp5 zEp%+JmAv`BcGUR5YYj&(gK!cu{@oHU>x@7TTi7?aA|xtk<K|WcCH{&=uE67M&0J_)D0sC{o}(;YpNH|VlrF(t2>EdKN|#phxw4e~_N zywkG*EBw8;B^qr=D(OvLLx?P%)>cp3$(N!f+|+0q|7mtuF!O6@C^u^K(PTO1YT@1o zG+7DXgRMGMSeKrQI7f;m7dR$QB-u^hE%xLRRo2YQ#f{9(JW^@1&0BaTGK~o*?k_Ow zo?rS^5&R{HFdVO_J}VE9%Oy9R?BoUq0c$yXl-&Jh6pPtUqR6g{)|17pzLEWj4WiTq z5@k^`=|}s)b=iC1IH^p$9O7qoLk6N}ybG8ZJS!R;fR(6qc!3ZX5Sc&xqturaB6sek zowH;9OOPcbcFUJJ>PhUj>MdQpQA+VfoH?UNFauOda{Keg$fP48k6)_O2i?~Wv&%-S zG0op;aLBepvCl*VPFhTdKoHktQq@KZUXZ&8#!Z%zQ!Y77+dvMjyGefavS58;;vMLf;&!@`AUUxYH zMN9fY+*!3H(sQpRFhB0+GBNujIk?*6W3Aq-+!U5mA-nfJdtChvNgNp(v=T>jQ$@FyQ+uq3^m?FzXh{zeCT+}fUCV{4S zt&Ean&$xyLU2UJ5-k3si5(8@VHEbS`k|nVuvlBL$8k~y)Csxq-NR)tQ=BO`pc}-|2 z2ekatI#mywCbMQPShK@ijoch=bkCR~U~GR|l=&DsQNv|%@;6R3im|_HX#Kq~gLLJB z={yGR(PlMKvfgrR?yMxh)^eWz^|724#veys8~y=}ZkikrrgM#8$un!+NB6~qCj0L! zdx1&G4u7Z~W1V9ldtKb;K8#_(q5K?ONecD41SGuk6p5a2Nc0AgeCk@Ah_SJJ4$a+e z3-h^nE|$pt4xZQgr>Yp!P=z16n%c_KHWlF7@%+jPDsTBZ$t?wKh8g~!=8 zV@l=v?(Q8NB{76S1Qy|RjrQG*!9Ehiu8+T@Hhf5k%Cf$*NORAPGa;C?i}><9@|Np( zI(T%`n1-bPts59I$iBY8vrPtGLTHX&BXtbhfS~_SHb~^tAqn{f|K_wv+~>pKy`D`F zecYA?i=YWB51S8!I6pxRP@gi%+ytY5_E(SDr|%b{wN)_?O`{ca9sTCyUtUcp1~%pX z1cr}qweNSASCP!_Iq_RoFxi+LgHR z#^z(tIm*2cS7`f2Gzd~AK5#B784-vW__!>KfFIkqS@&-16O`IO)?h!C#fUQQ^>^`R0;?1=BgTu`AL^@tmP}CE)LpEWp4|W0~VFW$KmVCmv=|vMy zIt1Z##ktSQnoskSPf>W(BJdtX?4JlY%PNzWq7Ef)7jW?~^(4VL-~Py04M}Gylx72D z_S@?@JLNo@#IFAl?C5CKG(sfqa#^z%-c#t$>^25abQhL{WJqUHWO}S{vX3Ljwqt9; z*>TEu?!&{)Xfn)b`-K_WrNy%`O|v2*i>xLa;J6EVLUO-Z#S2zr)QFTOgMYj>JdwG3 zj4gTk_bjg3G84#hDc#=3s`#GC#PCLB@0+)7M9N#pomNk45A1@~s7 zL2|_dXRh~*_b88PtCo%ao5WJ#w>pB;HE_Xc?y^DcDcaQ;M?qq*l?DC5XV>vhNopckR6U4uUE(-s zd!fCx7wD3ZAQXB50Oh!PQ*N5)xLLFM|A}^iexPj~ShcE#jIkcjBxRy`S+l0rKMPf< zIm_njyPng-tG!dbJ&IU4r>0xRZ}hn@>6OxL239Y0=jCJFum6dpQ*mf8Xm3#exm!zi z9Q`L?)g|6y-XG4qsgGoLhqerf8#5!X@855eWyYO@@ zD9N3LqR=zhFsCLyAT-}t2!+abs!9)9_neNu&Qf2E>;5IJ&HOU?7iE8&1|XR#O3;`D z>m0~q4X!=cc&tmse>T zzO*Yz5^=C`xxMs@~PHR!iw=caTs)b&iHUbZzYK< z4*7<*qGSX$N78hZc%>BWU^2UB&A;-LD2x|jLiVe^S+~m=&+Xl7@OSKCr!>ZJZgc+> z2mDIL*50ux)Gsye1;K@ocG*64fqTxU#Yvb<84~n7?yj80YS1}HKan_sAb^5APzpE! zdM*M+MVnhk;T4lNceWU?ohYmPs1AGe$u zqho!EjuC(1ptW=yta7bWeVzrw7*1v~K>}6OL6deQG@}Ad#{`m>EfZoPo3kkRVD_xe zo+x(|w+*}=+r#@MvtTn_RexPA2$mRNt9AI**J(##1rg&<3-VWA&femYGI3hg>@Hyh zVp&d8OKYRLlAA)|Uw339SwG@l7=6TZbQ?cx9he{`9QTcI^v#EhdI6bI`#ZQ4WQzY@P?Pxn8uu)i zDC-L8_|BBG9y$41m!yv_*mo_lS?KWa?)Lh8&w>(k9M_#9r;S%P4rm`yb?8>N3)hX^ zmbu2K-Z(qhahtnc}O;t~pPnzJ_?R(b2Cd#qi%LJ-G z3vYa(+FLA+9ITiRyRnCRMrpyn65fDDQ;~_K;E5mm_J_(+jb(8AH-AOy&R@*s5bRF> zutaP7EM;$t!9qUxG@P!FC~hmEh$#K&57_y(((lPXCd{G$?w{1$7FloY6U#|(R<$VB z3wkX(!ES0-xxD8=%z{)d-kaUus!R2XmO+FY)PTHxA^bO-3h4hyLnO1!FB`vBCGT)zukinT1~cIgAO% zAKF5}3qUSvIOXH|DJRv|GTy;P>7rmZ%L%bM-it>>H;B)HBq+};UU=Hd9=S;$y=b%P z4UoX_ggCD2y~cj)k2*MBQF9h9`+;rsO_L?MA&#$6M#jYdE6DLLF-5yEM1cC*rpOR$ z`v73}8Qe1N*E;enX~)#ope&d`6}%XRIh#ld3nl~tfTnt#Q2SOcKy)rt%ubWnbd<2iPbgapXXEz{@{NRcIaVp9^{&mh;i;~PiP#E*QD^H3WPE_ zm4ky63N1*?s#SuGSqn}sj)+MmMzz^G<4s~k<640CW>ZDPHuBM?zILb*s8d;jV+c-~ zC|xhY+&Gth?}8kGZ(o{4@dIgNQ)PsO(`i-tO2d=~5+{v|jNOl*01z;RT>2(oEVE~| zljvE$1=9jxV%V)TzK@lBM=51)v9<0$JLigMp6N&CFxQpwRL5osJ24|R%9(^C5IVid z>M=NH21)p}Wop>T5bd9xMt|>8ueuKk7GHH@SG@Ns1apADMA|2nNipjT$m}p=9T8JM)hh~!7J~!Z`bc$D@9HM!kqgwU(7SybRAhNgEnUqDqaFs0X}^=noFbz>W?A-BLm_& zTfP^!0G}ULazYI-oPw4ObF6hpCUrgWechz^NaxA=;{)_sLbRXC}E&M0tpQ(OQ ze_!^3#~)o9jeI}tSLgq-{^UjMoy=x~o$0<>E^BcXxMpXe29DRaI40 zRaI40w3p;WL_|bHGfy`b3dVqOaC;V%9HIZYi&k-S5J3O?yyEw)Y{DKax1j^6b*MQ* z|8n6ydGGzqTC0!UmGL5grfA~5 z-QD$Hkqpz##fFl>NUy9~vx}I52m4*-je~<#ND&YI<}F#pMM!SlMc8;qP$+A-`q&o| z&MsmIAOCloF)B&J9V=|&<{*Lo-vo!Mhk6M>9D+O%+|10)A3(n%BVc;$mWgSvGBiMU z2^~5#Q0nk1)Q5WP9Nu1I8rhO>#mEE~tM55nRe0seV!n0l`zew1av8iq)RP?sv*ac@o`4RZxjt$Nm&1y?<_<Ct^6rNAlD5ufp(_84`z1=EBSVp8-G7Rvzv3)f2aRIGC*+yZ>3ZWOxTI-A+#ia)*f9_({oLs~Y zu}?lu`DZ{EoRZ~~ldfbAJj~3@(ew-QB0d?JnVD3gwIS^W03=JyM>QAnz5CAR5h5Z5 zpfOk=aGv(0pD8I1-qU=_cG4F7#>$@3vp8&Yo~o(~k$gv1olK5hs~6K#kOg$28(M4o z64NN&*qh8hsWPBSCSHh>!5$+29cTs33G9QBEtRr!R?GMAf|0`FzGS3hg)lEMDzFg* z2__7_c)`SL?q#JF>dD?^pPKqI8?GY@gs7yLTD#Nee@!tOZsy$GY#2mwRXB4n zOn?Kzwihmf;(zO560c3s z0#1|9qqbR*UBc`bW7yPWfyEI2G!^;>FhTxp)|rUB47b|epkp(@dQ?;APS9C;hiE9t z@az^|q1p>CQ0)bmsCI(O)H^|C>IY!5^$yTkdR!0y0092+b#*u8Y?r^-`yy1BhFsF> zEj9WCojO0bMt2mP!8t9Q40e^4*wJ10^5c|)=mEQmZO?E@f0iQp+u)V+un$DMN=1R$ z>udGGEhIr?$-?Y^jL=k5D~SNw&4AaW*j(y zYkwea)%NpKrvJCVC6=GY(bH2KBIhVq0zs+S7U{DS#jcHu@^w=Kf&bq0KWYj{%!2Kt z5{eMkY#8~eYxckwhTq<3*xwg;xSjG3?O@;DhA14tl7&hzA>t0Zowl$hCyOyMrg73& zZy|Wq+pH2klO*4QoBe%$CqW)&bP_VE{9~tr*<)0gSR?It*HS?u*LqcyL1lElr6FH1 zkJJ~=3fxQEfUC*0O==-0CFA&LFZ~uO)7nd2MXISkLM@?YcRT>lKmeV)Fkyj}Hb5bS z4FtY-dbzdkB7CfjjV*`Lrkox^1J9$;{_YKg5dd8kpRr}}8?fg@RL40<_u!{BKCGk~ z@$so8hXFxFOb>}-cZCHNFgUIS3tA65B3Fmbt4+Nq-IikkumPoKXN%uPCb*@);Q*8= z#SOUHIy;@WA3Hl+V*?j-=ko^(^N$AR{@Z*ylAgLGJ^T6F!ErvQAZXCt3+!wqFn2Ff zDL!5T_H%9rEHVDN4^u4{YvQ?o|h_biU7vpGi^Lz znMeH;N00>X1Y~Ec_v~9YI3&mD5fPVk6p%XBTcv#K6@tg(!nz?^_NX~>)RzKpjV)`e zd!y`L_1gqsb$xHqiz3&7IF-~^24=bnqj`IFOAa-Q!eW*u%h*U^VL35x0!!OCt(f^` zY21^BM~|Rhu>fdiR(I?5$lyg-+;*qZuxLVaR6Eh(X9I61tcc-Tc4cJAu_GG;LD`P{ z{5e`Y9r@m%%xz=$v)dn|1_I-v(spdBRU6v*Ql3NkeMfeU=~qzBxoLj;OUkBe%r$CV z+DUuA=5oTZfZiC$b*HU-R(MTo$XJ zmU1#<1Jjke-!%4Pmek6;RwXbQ{H5~h`V;TgcLSc?`yqu@R@rz-!@?rR#|GW!I|k6B zF{fTD++~h6zFm9VUdJqnP6_{g=3rfFsjf-RIEZ>24UfP=I-*b@;f z>HitfAFJ)Hx7hUtXa@01Qm0gp(|Z=!i7czAY|xbHuqNidUqP5i_k2Ms!H4n7jvWbm z2_?9UitP_z`do|Ff4HK%3rOfR z&QGX<(a{K`*{%?a_FUg>GN;>1^HkR{Lbyuz|C+i`$siky5}_(Dqwp8WGg%9Ri;}K` z&yrlQ&fsn@`?e_@g$mhs1OJ(f#*ENJO|goQg%RNb!} z5L9+R3>hGNJFZ4|45L@%(6xb81U?c-gYNN^_V%G=QVT2fIFVV7lmWm$o(VKw=ks zGm)H&RN}BYU<4a#HzxXvCIw=^|Jm7Ftbl6hk>|*}xIQ&|D)uG59`IQgP9h#L6*$yOU!6p){S_!Nv6cs$kssU_`v=5)&A)vOQYG9sG( zX;ZQOKH~dEr#Ug0D%Iw41HuUr<&QSdktE$!Ryb_&R3RM(~7`3Fy(ITOkJY9Y}6ZButC}}Eu(i*vnCw!Sm(On{41@WvaL7#;@A8H7?7L)9#8k<7#JpfvGHw z1B`F%0gh+h*TzD*UB^O>zwR=sSPaTk?p-EfVbZ4EfZ$0hym^4`vpy6*)$xEYPy#8c zMg4021_91uyc^{a^uY$cHpM=qi$2^rQn0WRB|0oK*-oWS<I1#r%x z3JNMI@^$xFh*tw`?z~(_(^=TSIYC6h#o+K;AQte&M}! z=B>`#;%JmQ3^5mqYA?`y^4Vv4_ENA4S#C%7*Nr(OU6C++7SY98_{D%%1?ppxk?Jx- zNZu#7j?Dy7C(;|~EAQ-$0a6A60ROO}oDP8U>*tOdN7y&;P4*!jV{Z{;iQK>oM05A@)PNsRjWn?IE* znA8=ecER0ij+(}j&?G(G@z%P>Pq-dePH6qT?aeI+%1qg zH&&C7%@$ae&dihKLxU)MXs%Eh2|q|Tus)Hp1d|K74BUV+%n#R?%)ZqK1Y)kgRv(#& z01gluN?;_*%gc(N*~YflsjmIV*e^zqZd^M}2CV=DQVHWSrw{O6taY76AvhFTn(Dxu z#j2UUc^tGZY@HFUFpz7`i>&=m>b-eCAPBJk;36?2gnQ+G z-TeJF*l9IC1?SBIX!%PIn;|+!bRTR_poC;@}~{j zAV-$s#HWjzzqt{N;IC|5;BVEZ=Bb>D+1pA2C(=}U+vkp7G_A84u^#)#ZRw05k}&UM zy?W-Cb4`dkdPy@0mBQ3wE6lcCgz|_`he>yvB8I?XGLSFX-YZZ$aVt5j2B^V9#l=H-Ze;S zELj9d3V+AccWqlf+@_=_x42CG2W$)-$fn>~+#)1*I4wx5mY=-$x{`UgTcU znA8ZoJ05Fc*Shv6XmJA4B4XAizQ@wzl>0QkjW45hgXPQNwZ^YkiTmqjcl3rZ85Ins zH=L1u*{mokVzdI<&&lLL$p8#I{EO5(R0vmA`a+c8+LTS`A-ZNPW*ON?foDpG)4|mL zR|S9+!)t8vsxs!fYov!uCAUi!TPT&-jy49gb!$st+|pKLha5$@COFu)&xDc^6oLx| z&SIXVsWc4E-O(pDm9PPrOXoX3i#W(~%=8g2(7jy9>Ji@|M4!o;bAL zGzdvj;$G%jyC^1Ycswc-hX!W#m@zk0X_Gp0%v}}e_etU{RR=GQuK&j6>zz##Yd*5! zYFn2vWk#qte>@7+s!m1dRdUSnk#LE86BRV4)A4{`Euq*WxUSCbNn|2Klh#h1G~q2y z%{E^nhkFVj z)Fi4X{86>1V3eyak3pH;6;%P6IkE01*Mh2TUT>Y0v$~f*$}F}}_1J6vFjcTHj!XRv zSc!5d1;h`5Xw8?o5wuCq)||A|-kWvv;43b6J?Dx2V_=*+LM8>a@oF9gvIdJGz1{92 zZ862K^Sjd9WN2c8t`7(G<~&c4fb-?m{-J4!vkxW7bykW#Zt<%ajG`dpsKiE0A1a?A zx$zpm?%)g1kZ^T}t2=?f?fJm?ze`Bh!w9X8=^FLj?y9(c_nZ^|0Y2Y51`7_ z5~Tzwa4I`FJ$Y}mh;NCS#sGXc4sIIDfFx7HQN> zg-xts5E>W`aQa%!Rrl`7W${0r`=3$Oy|5@YZu_C&U7T z56c-HFWY>WLRVFOXBggV%!`)APP zvE)Qgj!Hb7J?fHErb^)tpn~-|95X4b6jO`qzV?fT@4axaU`u{-sXJ*K!i=0}He z_0p18o4|P3VJLmP0HAKx)q(#jn*DgVz&!E!+6y#`|J5EJzI#Q*(%nz}nB*Txw5B2L zk&9Xvm2xq!OK{cLC?I7Rtno?#MQ5rJaHA?tu^r7=_aR-`ct}~wC+o`wl{TqGU8N2FUV{V`D z?2uSgTM<$q2S69-_iDdb1(gr7fU8`L z8POUxo!-6q(ACg4h5mDgwuxe`T1rRpFtSjUFhJw7&svLxrBTXkg8dqjg%S`AJiE=L z#rzC$izIu5$|6ggjj5u9WDhY72i_T5(;r&t#q75t{+CY}b|oN%3ekikg))icuFkGZ zQ5;jUY|O}a3>yR`)-2-+pUCR9fsqYW{*D!MQc+fZAW3q#V*AwTYm)F46ArZMnR*y$syVe^Qr_7{X^Hl~N z#)QcZi(^J)Z0vW-HlrEI0L5mnuk02z+0J!n`wqhxis@6hT$b|c@cW%V+2+M)p4n2hA3;G_=dh|sml@LbDjvRQ(8 z$awdfx8&|BK4073xvTGu zVj%5jBHS0^bW^QmWL!JPXf&r(F&qqtAw|W9GnA!INZP5nLaXo-;#{D(^y{3XTUvXV Uw0cBM5#G9zr*i19000000GgKsX8-^I literal 0 HcmV?d00001 diff --git a/docs/stream-aggregation.md b/docs/stream-aggregation.md deleted file mode 100644 index 5107fd4fd..000000000 --- a/docs/stream-aggregation.md +++ /dev/null @@ -1,1208 +0,0 @@ ---- -weight: 98 -title: Streaming aggregation -menu: - docs: - parent: 'victoriametrics' - weight: 98 -aliases: -- /stream-aggregation.html ---- -[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) -can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage -(or local storage for single-node VictoriaMetrics). -The aggregation is applied to all the metrics received via any [supported data ingestion protocol](https://docs.victoriametrics.com/#how-to-import-time-series-data) -and/or scraped from [Prometheus-compatible targets](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) -after applying all the configured [relabeling stages](https://docs.victoriametrics.com/vmagent/#relabeling). - -**By default, stream aggregation ignores timestamps associated with the input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples). -It expects that the ingested samples have timestamps close to the current time. See [how to ignore old samples](#ignoring-old-samples).** - -## Configuration - -Stream aggregation can be configured via the following command-line flags: - -- `-streamAggr.config` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - and at [vmagent](https://docs.victoriametrics.com/vmagent/). -- `-remoteWrite.streamAggr.config` at [vmagent](https://docs.victoriametrics.com/vmagent/) only. This flag can be specified individually - per each `-remoteWrite.url`, so the aggregation happens independently per each remote storage destination. - This allows writing different aggregates to different remote storage systems. - -These flags must point to a file containing [stream aggregation config](#stream-aggregation-config). -The file may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values. - -By default, the following data is written to the storage when stream aggregation is enabled: - -- the aggregated samples; -- the raw input samples, which didn't match any `match` option in the provided [config](#stream-aggregation-config). - -This behaviour can be changed via the following command-line flags: - -- `-streamAggr.keepInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/) - `-remoteWrite.streamAggr.keepInput` flag can be specified individually per each `-remoteWrite.url`. - If one of these flags is set, then all the input samples are written to the storage alongside the aggregated samples. -- `-streamAggr.dropInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/) - `-remoteWrite.streamAggr.dropInput` flag can be specified individually per each `-remoteWrite.url`. - If one of these flags are set, then all the input samples are dropped, while only the aggregated samples are written to the storage. - -## Routing - -[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports relabeling, -deduplication and stream aggregation for all the received data, scraped or pushed. -The processed data is then stored in local storage and **can't be forwarded further**. - -[vmagent](https://docs.victoriametrics.com/vmagent/) supports relabeling, deduplication and stream aggregation for all -the received data, scraped or pushed. Then, the collected data will be forwarded to specified `-remoteWrite.url` destinations. -The data processing order is the following: - -1. all the received data is relabeled according to the specified [`-remoteWrite.relabelConfig`](https://docs.victoriametrics.com/vmagent/#relabeling) (if it is set) -1. all the received data is deduplicated according to specified [`-streamAggr.dedupInterval`](https://docs.victoriametrics.com/stream-aggregation/#deduplication) - (if it is set to duration bigger than 0) -1. all the received data is aggregated according to specified [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) (if it is set) -1. the resulting data is then replicated to each `-remoteWrite.url` -1. data sent to each `-remoteWrite.url` can be additionally relabeled according to the corresponding `-remoteWrite.urlRelabelConfig` (set individually per URL) -1. data sent to each `-remoteWrite.url` can be additionally deduplicated according to the corresponding `-remoteWrite.streamAggr.dedupInterval` (set individually per URL) -1. data sent to each `-remoteWrite.url` can be additionally aggregated according to the corresponding `-remoteWrite.streamAggr.config` (set individually per URL) - It isn't recommended using `-streamAggr.config` and `-remoteWrite.streamAggr.config` simultaneously, unless you understand the complications. - -Typical scenarios for data routing with `vmagent`: - -1. **Aggregate incoming data and replicate to N destinations**. Specify [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) command-line flag - to aggregate the incoming data before replicating it to all the configured `-remoteWrite.url` destinations. -2. **Individually aggregate incoming data for each destination**. Specify [`-remoteWrite.streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) - command-line flag for each `-remoteWrite.url` destination. [Relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) via `-remoteWrite.urlRelabelConfig` - can be used for routing only the selected metrics to each `-remoteWrite.url` destination. - -## Deduplication - -[vmagent](https://docs.victoriametrics.com/vmagent/) supports online [de-duplication](https://docs.victoriametrics.com/#deduplication) of samples -before sending them to the configured `-remoteWrite.url`. The de-duplication can be enabled via the following options: - -- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag for all received data - or via `-remoteWrite.streamAggr.dedupInterval` command-line flag for the particular `-remoteWrite.url` destination. - For example, `./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=30s` instructs `vmagent` to leave - only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds. - The de-deduplication is performed after applying [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) and - before performing the aggregation. - If the `-remoteWrite.streamAggr.config` and / or `-streamAggr.config` is set, then the de-duplication is performed individually per each - [stream aggregation config](#stream-aggregation-config) for the matching samples after applying [input_relabel_configs](#relabeling). - -- By specifying `dedup_interval` option individually per each [stream aggregation config](#stream-aggregation-config) - in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs. - -[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports two types of de-duplication: -- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](https://docs.victoriametrics.com/#deduplication) command-line option. -- Before storing the duplicate samples to local storage. This type of de-duplication can be enabled via the following options: - - By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag. - For example, `./victoria-metrics -streamAggr.dedupInterval=30s` instructs VictoriaMetrics to leave only the last sample per each - seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds. - The de-duplication is performed after applying `-relabelConfig` [relabeling](https://docs.victoriametrics.com/#relabeling). - - If the `-streamAggr.config` is set, then the de-duplication is performed individually per each [stream aggregation config](#stream-aggregation-config) - for the matching samples after applying [input_relabel_configs](#relabeling). - - - By specifying `dedup_interval` option individually per each [stream aggregation config](#stream-aggregation-config) at `-streamAggr.config`. - -It is possible to drop the given labels before applying the de-duplication. See [these docs](#dropping-unneeded-labels). - -The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](https://docs.victoriametrics.com/#deduplication) at VictoriaMetrics. - -## Ignoring old samples - -By default, all the input samples are taken into account during stream aggregation. If samples with old timestamps -outside the current [aggregation interval](#stream-aggregation-config) must be ignored, then the following options can be used: - -- To pass `-streamAggr.ignoreOldSamples` command-line flag to [single-node VictoriaMetrics](https://docs.victoriametrics.com/) - or to [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/) - `-remoteWrite.streamAggr.ignoreOldSamples` flag can be specified individually per each `-remoteWrite.url`. - This enables ignoring old samples for all the [aggregation configs](#stream-aggregation-config). - -- To set `ignore_old_samples: true` option at the particular [aggregation config](#stream-aggregation-config). - This enables ignoring old samples for that particular aggregation config. - -## Ignore aggregation intervals on start - -Streaming aggregation results may be incorrect for some time after the restart of [vmagent](https://docs.victoriametrics.com/vmagent/) -or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) until all the buffered [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) -are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent). -In this case it may be a good idea to drop the aggregated data during the first `N` [aggregation intervals](#stream-aggregation-config) -just after the restart of `vmagent` or single-node VictoriaMetrics. This can be done via the following options: - -- The `-streamAggr.ignoreFirstIntervals=N` command-line flag at `vmagent` and single-node VictoriaMetrics. This flag instructs skipping the first `N` - [aggregation intervals](#stream-aggregation-config) just after the restart across all the [configured stream aggregation configs](#configuration). - - The `-remoteWrite.streamAggr.ignoreFirstIntervals` command-line flag can be specified individually per each `-remoteWrite.url` at [vmagent](https://docs.victoriametrics.com/vmagent/). - -- The `ignore_first_intervals: N` option at the particular [aggregation config](#stream-aggregation-config). - -See also: - -- [Flush time alignment](#flush-time-alignment) -- [Ignoring old samples](#ignoring-old-samples) - -## Flush time alignment - -By default, the time for aggregated data flush is aligned by the `interval` option specified in [aggregate config](#stream-aggregation-config). - -For example: - -- if `interval: 1m` is set, then the aggregated data is flushed to the storage at the end of every minute -- if `interval: 1h` is set, then the aggregated data is flushed to the storage at the end of every hour - -If you do not need such an alignment, then set `no_align_flush_to_interval: true` option in the [aggregate config](#stream-aggregation-config). -In this case aggregated data flushes will be aligned to the `vmagent` start time or to [config reload](#configuration-update) time. - -The aggregated data on the first and the last interval is dropped during `vmagent` start, restart or [config reload](#configuration-update), -since the first and the last aggregation intervals are incomplete, so they usually contain incomplete confusing data. -If you need preserving the aggregated data on these intervals, then set `flush_on_shutdown: true` option in the [aggregate config](#stream-aggregation-config). - -See also: - -- [Ignore aggregation intervals on start](#ignore-aggregation-intervals-on-start) -- [Ignoring old samples](#ignoring-old-samples) - -## Use cases - -Stream aggregation can be used in the following cases: - -* [Statsd alternative](#statsd-alternative) -* [Recording rules alternative](#recording-rules-alternative) -* [Reducing the number of stored samples](#reducing-the-number-of-stored-samples) -* [Reducing the number of stored series](#reducing-the-number-of-stored-series) - -### Statsd alternative - -Stream aggregation can be used as [statsd](https://github.com/statsd/statsd) alternative in the following cases: - -* [Counting input samples](#counting-input-samples) -* [Summing input metrics](#summing-input-metrics) -* [Quantiles over input metrics](#quantiles-over-input-metrics) -* [Histograms over input metrics](#histograms-over-input-metrics) -* [Aggregating histograms](#aggregating-histograms) - -Currently, streaming aggregation is available only for [supported data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data) -and not available for [Statsd metrics format](https://github.com/statsd/statsd/blob/master/docs/metric_types.md). - -### Recording rules alternative - -Sometimes [alerting queries](https://docs.victoriametrics.com/vmalert/#alerting-rules) may require non-trivial amounts of CPU, RAM, -disk IO and network bandwidth at metrics storage side. For example, if `http_request_duration_seconds` histogram is generated by thousands -of application instances, then the alerting query `histogram_quantile(0.99, sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)) > 0.5` -can become slow, since it needs to scan too big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) -with `http_request_duration_seconds_bucket` name. This alerting query can be accelerated by pre-calculating -the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](https://docs.victoriametrics.com/vmalert/#recording-rules). -But this recording rule may take too much time to execute too. In this case the slow recording rule can be substituted -with the following [stream aggregation config](#stream-aggregation-config): - -```yaml -- match: 'http_request_duration_seconds_bucket' - interval: 5m - without: [instance] - outputs: [total] -``` - -This stream aggregation generates `http_request_duration_seconds_bucket:5m_without_instance_total` output series according to [output metric naming](#output-metric-names). -Then these series can be used in [alerting rules](https://docs.victoriametrics.com/vmalert/#alerting-rules): - -```metricsql -histogram_quantile(0.99, last_over_time(http_request_duration_seconds_bucket:5m_without_instance_total[5m])) > 0.5 -``` - -This query is executed much faster than the original query, because it needs to scan much lower number of time series. - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [aggregating by labels](#aggregating-by-labels). - -Field `interval` is recommended to be set to a value at least several times higher than your metrics collect interval. - - -### Reducing the number of stored samples - -If per-[series](https://docs.victoriametrics.com/keyconcepts/#time-series) samples are ingested at high frequency, -then this may result in high disk space usage, since too much data must be stored to disk. This also may result -in slow queries, since too much data must be processed during queries. - -This can be fixed with the stream aggregation by increasing the interval between per-series samples stored in the database. - -For example, the following [stream aggregation config](#stream-aggregation-config) reduces the frequency of input samples -to one sample per 5 minutes per each input time series (this operation is also known as downsampling): - -```yaml - # Aggregate metrics ending with _total with `total` output. - # See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs -- match: '{__name__=~".+_total"}' - interval: 5m - outputs: [total] - - # Downsample other metrics with `count_samples`, `sum_samples`, `min` and `max` outputs - # See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs -- match: '{__name__!~".+_total"}' - interval: 5m - outputs: [count_samples, sum_samples, min, max] -``` - -The aggregated output metrics have the following names according to [output metric naming](#output-metric-names): - -```text -# For input metrics ending with _total -some_metric_total:5m_total - -# For input metrics not ending with _total -some_metric:5m_count_samples -some_metric:5m_sum_samples -some_metric:5m_min -some_metric:5m_max -``` - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [aggregating histograms](#aggregating-histograms) and [aggregating by labels](#aggregating-by-labels). - -### Reducing the number of stored series - -Sometimes applications may generate too many [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). -For example, the `http_requests_total` metric may have `path` or `user` label with too big number of unique values. -In this case the following stream aggregation can be used for reducing the number metrics stored in VictoriaMetrics: - -```yaml -- match: 'http_requests_total' - interval: 30s - without: [path, user] - outputs: [total] -``` - -This config specifies labels, which must be removed from the aggregate output, in the `without` list. -See [these docs](#aggregating-by-labels) for more details. - -The aggregated output metric has the following name according to [output metric naming](#output-metric-names): - -```text -http_requests_total:30s_without_path_user_total -``` - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [aggregating histograms](#aggregating-histograms). - -### Counting input samples - -If the monitored application generates event-based metrics, then it may be useful to count the number of such metrics -at stream aggregation level. - -For example, if an advertising server generates `hits{some="labels"} 1` and `clicks{some="labels"} 1` metrics -per each incoming hit and click, then the following [stream aggregation config](#stream-aggregation-config) -can be used for counting these metrics per every 30 second interval: - -```yaml -- match: '{__name__=~"hits|clicks"}' - interval: 30s - outputs: [count_samples] -``` - -This config generates the following output metrics for `hits` and `clicks` input metrics -according to [output metric naming](#output-metric-names): - -```text -hits:30s_count_samples count1 -clicks:30s_count_samples count2 -``` - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [aggregating by labels](#aggregating-by-labels). - - -### Summing input metrics - -If the monitored application calculates some events and then sends the calculated number of events to VictoriaMetrics -at irregular intervals or at too high frequency, then stream aggregation can be used for summing such events -and writing the aggregate sums to the storage at regular intervals. - -For example, if an advertising server generates `hits{some="labels} N` and `clicks{some="labels"} M` metrics -at irregular intervals, then the following [stream aggregation config](#stream-aggregation-config) -can be used for summing these metrics per every minute: - -```yaml -- match: '{__name__=~"hits|clicks"}' - interval: 1m - outputs: [sum_samples] -``` - -This config generates the following output metrics according to [output metric naming](#output-metric-names): - -```text -hits:1m_sum_samples sum1 -clicks:1m_sum_samples sum2 -``` - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [aggregating by labels](#aggregating-by-labels). - - -### Quantiles over input metrics - -If the monitored application generates measurement metrics per each request, then it may be useful to calculate -the pre-defined set of [percentiles](https://en.wikipedia.org/wiki/Percentile) over these measurements. - -For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics -per each incoming request, then the following [stream aggregation config](#stream-aggregation-config) -can be used for calculating 50th and 99th percentiles for these metrics every 30 seconds: - -```yaml -- match: - - request_duration_seconds - - response_size_bytes - interval: 30s - outputs: ["quantiles(0.50, 0.99)"] -``` - -This config generates the following output metrics according to [output metric naming](#output-metric-names): - -```text -request_duration_seconds:30s_quantiles{quantile="0.50"} value1 -request_duration_seconds:30s_quantiles{quantile="0.99"} value2 - -response_size_bytes:30s_quantiles{quantile="0.50"} value1 -response_size_bytes:30s_quantiles{quantile="0.99"} value2 -``` - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [histograms over input metrics](#histograms-over-input-metrics) and [aggregating by labels](#aggregating-by-labels). - -### Histograms over input metrics - -If the monitored application generates measurement metrics per each request, then it may be useful to calculate -a [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) over these metrics. - -For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics -per each incoming request, then the following [stream aggregation config](#stream-aggregation-config) -can be used for calculating [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) -for these metrics every 60 seconds: - -```yaml -- match: - - request_duration_seconds - - response_size_bytes - interval: 60s - outputs: [histogram_bucket] -``` - -This config generates the following output metrics according to [output metric naming](#output-metric-names). - -```text -request_duration_seconds:60s_histogram_bucket{vmrange="start1...end1"} count1 -request_duration_seconds:60s_histogram_bucket{vmrange="start2...end2"} count2 -... -request_duration_seconds:60s_histogram_bucket{vmrange="startN...endN"} countN - -response_size_bytes:60s_histogram_bucket{vmrange="start1...end1"} count1 -response_size_bytes:60s_histogram_bucket{vmrange="start2...end2"} count2 -... -response_size_bytes:60s_histogram_bucket{vmrange="startN...endN"} countN -``` - -The resulting histogram buckets can be queried with [MetricsQL](https://docs.victoriametrics.com/metricsql/) in the following ways: - -1. An estimated 50th and 99th [percentiles](https://en.wikipedia.org/wiki/Percentile) of the request duration over the last hour: - - ```metricsql - histogram_quantiles("quantile", 0.50, 0.99, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) - ``` - - This query uses [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function. - -1. An estimated [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of the request duration over the last hour: - - ```metricsql - histogram_stddev(sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) - ``` - - This query uses [histogram_stddev](https://docs.victoriametrics.com/metricsql/#histogram_stddev) function. - -1. An estimated share of requests with the duration smaller than `0.5s` over the last hour: - - ```metricsql - histogram_share(0.5, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) - ``` - - This query uses [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share) function. - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [quantiles over input metrics](#quantiles-over-input-metrics) and [aggregating by labels](#aggregating-by-labels). - -### Aggregating histograms - -[Histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) is a set of [counter](https://docs.victoriametrics.com/keyconcepts/#counter) -metrics with different `vmrange` or `le` labels. As they're counters, the applicable aggregation output is -[total](https://docs.victoriametrics.com/stream-aggregation/#total): - -```yaml -- match: 'http_request_duration_seconds_bucket' - interval: 1m - without: [instance] - outputs: [total] -``` - -This config generates the following output metrics according to [output metric naming](#output-metric-names): - -```text -http_request_duration_seconds_bucket:1m_without_instance_total{le="0.1"} value1 -http_request_duration_seconds_bucket:1m_without_instance_total{le="0.2"} value2 -http_request_duration_seconds_bucket:1m_without_instance_total{le="0.4"} value3 -http_request_duration_seconds_bucket:1m_without_instance_total{le="1"} value4 -http_request_duration_seconds_bucket:1m_without_instance_total{le="3"} value5 -http_request_duration_seconds_bucket:1m_without_instance_total{le="+Inf" value6 -``` - -The resulting metrics can be passed to [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile) -function: - -```metricsql -histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket:1m_without_instance_total[5m])) by(le)) -``` - -Please note, histograms can be aggregated if their `le` labels are configured identically. -[VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) -have no such requirement. - -See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field. -See also [histograms over input metrics](#histograms-over-input-metrics) and [quantiles over input metrics](#quantiles-over-input-metrics). - -## Output metric names - -Output metric names for stream aggregation are constructed according to the following pattern: - -```text -:[_by_][_without_]_ -``` - -- `` is the original metric name. -- `` is the interval specified in the [stream aggregation config](#stream-aggregation-config). -- `` is `_`-delimited sorted list of `by` labels specified in the [stream aggregation config](#stream-aggregation-config). - If the `by` list is missing in the config, then the `_by_` part isn't included in the output metric name. -- `` is an optional `_`-delimited sorted list of `without` labels specified in the [stream aggregation config](#stream-aggregation-config). - If the `without` list is missing in the config, then the `_without_` part isn't included in the output metric name. -- `` is the aggregate used for constructing the output metric. The aggregate name is taken from the `outputs` list - at the corresponding [stream aggregation config](#stream-aggregation-config). - -Both input and output metric names can be modified if needed via relabeling according to [these docs](#relabeling). - -It is possible to leave the original metric name after the aggregation by specifying `keep_metric_names: true` option at [stream aggregation config](#stream-aggregation-config). -The `keep_metric_names` option can be used if only a single output is set in [`outputs` list](#aggregation-outputs). - -## Relabeling - -It is possible to apply [arbitrary relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) to input and output metrics -during stream aggregation via `input_relabel_configs` and `output_relabel_configs` options in [stream aggregation config](#stream-aggregation-config). - -Relabeling rules inside `input_relabel_configs` are applied to samples matching the `match` filters before optional [deduplication](#deduplication). -Relabeling rules inside `output_relabel_configs` are applied to aggregated samples before sending them to the remote storage. - -For example, the following config removes the `:1m_sum_samples` suffix added [to the output metric name](#output-metric-names): - -```yaml -- interval: 1m - outputs: [sum_samples] - output_relabel_configs: - - source_labels: [__name__] - target_label: __name__ - regex: "(.+):.+" -``` - -Another option to remove the suffix, which is added by stream aggregation, is to add `keep_metric_names: true` to the config: - -```yaml -- interval: 1m - outputs: [sum_samples] - keep_metric_names: true -``` - -See also [dropping unneeded labels](#dropping-unneeded-labels). - - -## Dropping unneeded labels - -If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication) -and [stream aggregation](#aggregation-outputs), then the following options exist: - -- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag - or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`. - For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples - before applying de-duplication and stream aggregation. - -- To specify `drop_input_labels` list with the labels to drop in [stream aggregation config](#stream-aggregation-config). - For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes` - before calculating the average over one minute: - - ```yaml - - match: process_resident_memory_bytes - interval: 1m - drop_input_labels: [replica] - outputs: [avg] - keep_metric_names: true - ``` - -Typical use case is to drop `replica` label from samples, which are received from high availability replicas. - -## Aggregation outputs - -The aggregations are calculated during the `interval` specified in the [config](#stream-aggregation-config) -and then sent to the storage once per `interval`. The aggregated samples are named according to [output metric naming](#output-metric-names). - -If `by` and `without` lists are specified in the [config](#stream-aggregation-config), -then the [aggregation by labels](#aggregating-by-labels) is performed additionally to aggregation by `interval`. - -Below are aggregation functions that can be put in the `outputs` list at [stream aggregation config](#stream-aggregation-config): - -* [avg](#avg) -* [count_samples](#count_samples) -* [count_series](#count_series) -* [histogram_bucket](#histogram_bucket) -* [increase](#increase) -* [increase_prometheus](#increase_prometheus) -* [last](#last) -* [max](#max) -* [min](#min) -* [rate_avg](#rate_avg) -* [rate_sum](#rate_sum) -* [stddev](#stddev) -* [stdvar](#stdvar) -* [sum_samples](#sum_samples) -* [total](#total) -* [total_prometheus](#total_prometheus) -* [unique_samples](#unique_samples) -* [quantiles](#quantiles) - -### avg - -`avg` returns the average over input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples). -`avg` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `avg` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(sum_over_time(some_metric[interval])) / sum(count_over_time(some_metric[interval])) -``` - -For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: - -![avg aggregation](stream-aggregation-check-avg.webp) - -See also: - -- [max](#max) -- [min](#min) -- [quantiles](#quantiles) -- [sum_samples](#sum_samples) -- [count_samples](#count_samples) - -### count_samples - -`count_samples` counts the number of input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. - -The results of `count_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(count_over_time(some_metric[interval])) -``` - -See also: - -- [count_series](#count_series) -- [sum_samples](#sum_samples) - -### count_series - -`count_series` counts the number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`. - -The results of `count_series` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -count(last_over_time(some_metric[interval])) -``` - -See also: - -- [count_samples](#count_samples) -- [unique_samples](#unique_samples) - -### histogram_bucket - -`histogram_bucket` returns [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) - for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. -`histogram_bucket` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). -See how to aggregate regular histograms [here](#aggregating-histograms). - -The results of `histogram_bucket` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) -or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. - -```metricsql -sum(histogram_over_time(some_histogram_bucket[interval])) by (vmrange) -``` - -See also: - -- [quantiles](#quantiles) -- [avg](#avg) -- [max](#max) -- [min](#min) - -### increase - -`increase` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given 'interval'. -`increase` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `increase` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(increase_pure(some_counter[interval])) -``` - -`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) -is `10`, then `increase` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, -then take a look at [increase_prometheus](#increase_prometheus). - -For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: - -![increase aggregation](stream-aggregation-check-increase.webp) - -Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) -or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. - -See also: - -- [increase_prometheus](#increase_prometheus) -- [total](#total) -- [rate_avg](#rate_avg) -- [rate_sum](#rate_sum) - -### increase_prometheus - -`increase_prometheus` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`. -`increase_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `increase_prometheus` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(increase_prometheus(some_counter[interval])) -``` - -`increase_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). -If you need taking into account the first sample per time series, then take a look at [increase](#increase). - -Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) -or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. - -See also: - -- [increase](#increase) -- [rate_avg](#rate_avg) -- [rate_sum](#rate_sum) -- [total](#total) -- [total_prometheus](#total_prometheus) - -### last - -`last` returns the last input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. - -The results of `last` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -last_over_time(some_metric[interval]) -``` - -See also: - -- [avg](#avg) -- [max](#max) -- [min](#min) -- [quantiles](#quantiles) - -### max - -`max` returns the maximum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. - -The results of `max` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -max(max_over_time(some_metric[interval])) -``` - -For example, see below time series produced by config with aggregation interval `1m` and the regular query: - -![total aggregation](stream-aggregation-check-max.webp) - -See also: - -- [min](#min) -- [avg](#avg) -- [last](#last) -- [quantiles](#quantiles) - -### min - -`min` returns the minimum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. - -The results of `min` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -min(min_over_time(some_metric[interval])) -``` - -For example, see below time series produced by config with aggregation interval `1m` and the regular query: - -![min aggregation](stream-aggregation-check-min.webp) - -See also: - -- [max](#max) -- [avg](#avg) -- [last](#last) -- [quantiles](#quantiles) - -### rate_avg - -`rate_avg` returns the average of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`. -`rate_avg` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `rate_avg` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -avg(rate(some_counter[interval])) -``` - -See also: - -- [rate_sum](#rate_sum) -- [increase](#increase) -- [total](#total) - -### rate_sum - -`rate_sum` returns the sum of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`. -`rate_sum` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `rate_sum` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(rate(some_counter[interval])) -``` - -See also: - -- [rate_avg](#rate_avg) -- [increase](#increase) -- [total](#total) - -### stddev - -`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) -over the given `interval`. -`stddev` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `stddev` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -histogram_stddev(sum(histogram_over_time(some_metric[interval])) by (vmrange)) -``` - -See also: - -- [stdvar](#stdvar) -- [avg](#avg) -- [quantiles](#quantiles) - -### stdvar - -`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) -over the given `interval`. -`stdvar` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `stdvar` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -histogram_stdvar(sum(histogram_over_time(some_metric[interval])) by (vmrange)) -``` - -For example, see below time series produced by config with aggregation interval `1m` and the regular query: - -![stdvar aggregation](stream-aggregation-check-stdvar.webp) - -See also: - -- [stddev](#stddev) -- [avg](#avg) -- [quantiles](#quantiles) - -### sum_samples - -`sum_samples` sums input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`. -`sum_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `sum_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(sum_over_time(some_metric[interval])) -``` - -For example, see below time series produced by config with aggregation interval `1m` and the regular query: - -![sum_samples aggregation](stream-aggregation-check-sum-samples.webp) - -See also: - -- [count_samples](#count_samples) -- [count_series](#count_series) - -### total - -`total` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`. -`total` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `total` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(running_sum(increase_pure(some_counter))) -``` - -`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) -is `10`, then `total` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, -then take a look at [total_prometheus](#total_prometheus). - -For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: - -![total aggregation](stream-aggregation-check-total.webp) - -`total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) - -it continues to increase monotonically with respect to the previous value. -The counters are most often reset when the application is restarted. - -For example: - -![total aggregation counter reset](stream-aggregation-check-total-reset.webp) - -The same behavior occurs when creating or deleting new series in an aggregation group - -`total` output increases monotonically considering the values of the series set. -An example of changing a set of series can be restarting a pod in the Kubernetes. -This changes pod name label, but the `total` accounts for such a scenario and doesn't reset the state of aggregated metric. - -Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) -or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. - -See also: - -- [total_prometheus](#total_prometheus) -- [increase](#increase) -- [increase_prometheus](#increase_prometheus) -- [rate_sum](#rate_sum) -- [rate_avg](#rate_avg) - -### total_prometheus - -`total_prometheus` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`. -`total_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter). - -The results of `total_prometheus` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -sum(running_sum(increase_prometheus(some_counter))) -``` - -`total_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). -If you need taking into account the first sample per time series, then take a look at [total](#total). - -`total_prometheus` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) - -it continues to increase monotonically with respect to the previous value. -The counters are most often reset when the application is restarted. - -Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) -or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. - -See also: - -- [total](#total) -- [increase](#increase) -- [increase_prometheus](#increase_prometheus) -- [rate_sum](#rate_sum) -- [rate_avg](#rate_avg) - -### unique_samples - -`unique_samples` counts the number of unique sample values over the given `interval`. -`unique_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `unique_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -count(count_values_over_time(some_metric[interval])) -``` - -See also: - -- [sum_samples](#sum_samples) -- [count_series](#count_series) - -### quantiles - -`quantiles(phi1, ..., phiN)` returns [percentiles](https://en.wikipedia.org/wiki/Percentile) for the given `phi*` -over the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given `interval`. -`phi` must be in the range `[0..1]`, where `0` means `0th` percentile, while `1` means `100th` percentile. -`quantiles(...)` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). - -The results of `quantiles(phi1, ..., phiN)` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query: - -```metricsql -histogram_quantiles("quantile", phi1, ..., phiN, sum(histogram_over_time(some_metric[interval])) by (vmrange)) -``` - -See also: - -- [histogram_bucket](#histogram_bucket) -- [avg](#avg) -- [max](#max) -- [min](#min) - - -## Aggregating by labels - -All the labels for the input metrics are preserved by default in the output metrics. For example, -the input metric `foo{app="bar",instance="host1"}` results to the output metric `foo:1m_sum_samples{app="bar",instance="host1"}` -when the following [stream aggregation config](#stream-aggregation-config) is used: - -```yaml -- interval: 1m - outputs: [sum_samples] -``` - -The input labels can be removed via `without` list specified in the config. For example, the following config -removes the `instance` label from output metrics by summing input samples across all the instances: - -```yaml -- interval: 1m - without: [instance] - outputs: [sum_samples] -``` - -In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1m_without_instance_sum_samples{app="bar"}` -output metric according to [output metric naming](#output-metric-names). - -It is possible specifying the exact list of labels in the output metrics via `by` list. -For example, the following config sums input samples by the `app` label: - -```yaml -- interval: 1m - by: [app] - outputs: [sum_samples] -``` - -In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1m_by_app_sum_samples{app="bar"}` -output metric according to [output metric naming](#output-metric-names). - -The labels used in `by` and `without` lists can be modified via `input_relabel_configs` section - see [these docs](#relabeling). - -See also [aggregation outputs](#aggregation-outputs). - - -## Stream aggregation config - -Below is the format for stream aggregation config file, which may be referred via `-streamAggr.config` command-line flag at -[single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/). -At [vmagent](https://docs.victoriametrics.com/vmagent/) `-remoteWrite.streamAggr.config` command-line flag can be -specified individually per each `-remoteWrite.url`: - -```yaml - - # name is an optional name of the given streaming aggregation config. - # - # If it is set, then it is used as `name` label in the exposed metrics - # for the given aggregation config at /metrics page. - # See https://docs.victoriametrics.com/vmagent/#monitoring and https://docs.victoriametrics.com/#monitoring -- name: 'foobar' - - # match is an optional filter for incoming samples to aggregate. - # It can contain arbitrary Prometheus series selector - # according to https://docs.victoriametrics.com/keyconcepts/#filtering . - # If match isn't set, then all the incoming samples are aggregated. - # - # match also can contain a list of series selectors. Then the incoming samples are aggregated - # if they match at least a single series selector. - # - match: 'http_request_duration_seconds_bucket{env=~"prod|staging"}' - - # interval is the interval for the aggregation. - # The aggregated stats is sent to remote storage once per interval. - # - interval: 1m - - # dedup_interval is an optional interval for de-duplication of input samples before the aggregation. - # Samples are de-duplicated on a per-series basis. See https://docs.victoriametrics.com/keyconcepts/#time-series - # and https://docs.victoriametrics.com/#deduplication - # The deduplication is performed after input_relabel_configs relabeling is applied. - # By default, the deduplication is disabled unless -remoteWrite.streamAggr.dedupInterval or -streamAggr.dedupInterval - # command-line flags are set. - # - # dedup_interval: 30s - - # staleness_interval is an optional interval for resetting the per-series state if no new samples - # are received during this interval for the following outputs: - # - histogram_bucket - # - increase - # - increase_prometheus - # - rate_avg - # - rate_sum - # - total - # - total_prometheus - # See https://docs.victoriametrics.com/stream-aggregation/#staleness for more details. - # - # staleness_interval: 2m - - # no_align_flush_to_interval disables aligning of flush times for the aggregated data to multiples of interval. - # By default, flush times for the aggregated data is aligned to multiples of interval. - # For example: - # - if `interval: 1m` is set, then flushes happen at the end of every minute, - # - if `interval: 1h` is set, then flushes happen at the end of every hour - # - # no_align_flush_to_interval: false - - # flush_on_shutdown instructs to flush aggregated data to the storage on the first and the last intervals - # during vmagent starts, restarts or configuration reloads. - # Incomplete aggregated data isn't flushed to the storage by default, since it is usually confusing. - # - # flush_on_shutdown: false - - # without is an optional list of labels, which must be removed from the output aggregation. - # See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels - # - # without: [instance] - - # by is an optional list of labels, which must be preserved in the output aggregation. - # See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels - # - # by: [job, vmrange] - - # outputs is the list of unique aggregations to perform on the input data. - # See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs - # - outputs: [total] - - # keep_metric_names instructs keeping the original metric names for the aggregated samples. - # This option can be set only if outputs list contains only a single output. - # By default, a special suffix is added to original metric names in the aggregated samples. - # See https://docs.victoriametrics.com/stream-aggregation/#output-metric-names - # - # keep_metric_names: false - - # ignore_old_samples instructs ignoring input samples with old timestamps outside the current aggregation interval. - # See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples - # See also -remoteWrite.streamAggr.ignoreOldSamples and -streamAggr.ignoreOldSamples command-line flag. - # - # ignore_old_samples: false - - # ignore_first_intervals instructs ignoring the first N aggregation intervals after process start. - # See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start - # See also -remoteWrite.streamAggr.ignoreFirstIntervals and -streamAggr.ignoreFirstIntervals command-line flags. - # - # ignore_first_intervals: N - - # drop_input_labels instructs dropping the given labels from input samples. - # The labels' dropping is performed before input_relabel_configs are applied. - # This also means that the labels are dropped before de-duplication ( https://docs.victoriametrics.com/stream-aggregation/#deduplication ) - # and stream aggregation. - # - # drop_input_labels: [replica, availability_zone] - - # input_relabel_configs is an optional relabeling rules, - # which are applied to the incoming samples after they pass the match filter - # and before being aggregated. - # See https://docs.victoriametrics.com/stream-aggregation/#relabeling - # - input_relabel_configs: - - target_label: vmaggr - replacement: before - - # output_relabel_configs is an optional relabeling rules, - # which are applied to the aggregated output metrics. - # - output_relabel_configs: - - target_label: vmaggr - replacement: after -``` - -The file can contain multiple aggregation configs. The aggregation is performed independently -per each specified config entry. - -### Configuration update - -[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) -support the following approaches for hot reloading stream aggregation configs from `-remoteWrite.streamAggr.config` and `-streamAggr.config`: - -* By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process: - - ```sh - kill -SIGHUP `pidof vmagent` - ``` - -* By sending HTTP request to `/-/reload` endpoint (e.g. `http://vmagent:8429/-/reload` or `http://victoria-metrics:8428/-/reload). - - -## Troubleshooting - -- [Unexpected spikes for `total` or `increase` outputs](#staleness). -- [Lower than expected values for `total_prometheus` and `increase_prometheus` outputs](#staleness). -- [High memory usage and CPU usage](#high-resource-usage). -- [Unexpected results in vmagent cluster mode](#cluster-mode). - -### Staleness - -The following outputs track the last seen per-series values in order to properly calculate output values: - -- [histogram_bucket](#histogram_bucket) -- [increase](#increase) -- [increase_prometheus](#increase_prometheus) -- [rate_avg](#rate_avg) -- [rate_sum](#rate_sum) -- [total](#total) -- [total_prometheus](#total_prometheus) - -The last seen per-series value is dropped if no new samples are received for the given time series during two consecutive aggregation -intervals specified in [stream aggregation config](#stream-aggregation-config) via `interval` option. -If a new sample for the existing time series is received after that, then it is treated as the first sample for a new time series. -This may lead to the following issues: - -- Lower than expected results for [total_prometheus](#total_prometheus) and [increase_prometheus](#increase_prometheus) outputs, - since they ignore the first sample in a new time series. -- Unexpected spikes for [total](#total) and [increase](#increase) outputs, since they assume that new time series start from 0. - -These issues can be fixed in the following ways: - -- By increasing the `interval` option at [stream aggregation config](#stream-aggregation-config), so it covers the expected - delays in data ingestion pipelines. -- By specifying the `staleness_interval` option at [stream aggregation config](#stream-aggregation-config), so it covers the expected - delays in data ingestion pipelines. By default, the `staleness_interval` equals to `2 x interval`. - -### High resource usage - -The following solutions can help reducing memory usage and CPU usage durting streaming aggregation: - -- To use more specific `match` filters at [streaming aggregation config](#stream-aggregation-config), so only the really needed - [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) are aggregated. -- To increase aggregation interval by specifying bigger duration for the `interval` option at [streaming aggregation config](#stream-aggregation-config). -- To generate lower number of output time series by using less specific [`by` list](#aggregating-by-labels) or more specific [`without` list](#aggregating-by-labels). -- To drop unneeded long labels in input samples via [input_relabel_configs](#relabeling). - -### Cluster mode - -If you use [vmagent in cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) for streaming aggregation -then be careful when using [`by` or `without` options](#aggregating-by-labels) or when modifying sample labels -via [relabeling](#relabeling), since incorrect usage may result in duplicates and data collision. - -For example, if more than one `vmagent` instance calculates [increase](#increase) for `http_requests_total` metric -with `by: [path]` option, then all the `vmagent` instances will aggregate samples to the same set of time series with different `path` labels. -The proper fix would be [adding an unique label](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics) for all the output samples -produced by each `vmagent`, so they are aggregated into distinct sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). -These time series then can be aggregated later as needed during querying. - -If `vmagent` instances run in Docker or Kubernetes, then you can refer `POD_NAME` or `HOSTNAME` environment variables -as an unique label value per each `vmagent` via `-remoteWrite.label=vmagent=%{HOSTNAME}` command-line flag. -See [these docs](https://docs.victoriametrics.com/#environment-variables) on how to refer environment variables in VictoriaMetrics components. diff --git a/docs/stream-aggregation/README.md b/docs/stream-aggregation/README.md new file mode 100644 index 000000000..f5460aed2 --- /dev/null +++ b/docs/stream-aggregation/README.md @@ -0,0 +1,12 @@ +[vmagent](https://docs.victoriametrics.com/vmagent) and [single-node VictoriaMetrics](https://docs.victoriametrics.com) +can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts#raw-samples) in streaming mode by time and by labels before data is written to remote storage +(or local storage for single-node VictoriaMetrics). +The aggregation is applied to all the metrics received via any [supported data ingestion protocol](https://docs.victoriametrics.com#how-to-import-time-series-data) +and/or scraped from [Prometheus-compatible targets](https://docs.victoriametrics.com#how-to-scrape-prometheus-exporters-such-as-node-exporter) +after applying all the configured [relabeling stages](https://docs.victoriametrics.com/vmagent#relabeling). + +_By default, stream aggregation ignores timestamps associated with the input [samples](https://docs.victoriametrics.com/keyconcepts#raw-samples). +It expects that the ingested samples have timestamps close to the current time. See [how to ignore old samples](./configuration/#ignoring-old-samples)._ + +## Next steps +{{% section %}} diff --git a/docs/stream-aggregation/_index.md b/docs/stream-aggregation/_index.md new file mode 100644 index 000000000..94d1704d8 --- /dev/null +++ b/docs/stream-aggregation/_index.md @@ -0,0 +1,13 @@ +--- +weight: 10 +title: Streaming aggregation +menu: + docs: + identifier: stream-aggregation + parent: 'victoriametrics' + weight: 10 +aliases: +- /stream-aggregation/ +- /stream-aggregation/index.html +--- +{{% content "README.md" %}} diff --git a/docs/stream-aggregation/common-mistakes.md b/docs/stream-aggregation/common-mistakes.md new file mode 100644 index 000000000..a0cf84f67 --- /dev/null +++ b/docs/stream-aggregation/common-mistakes.md @@ -0,0 +1,115 @@ +--- +sort: 5 +weight: 5 +title: Common mistakes +menu: + docs: + identifier: stream-aggregation-common-mistakes + parent: 'stream-aggregation' + weight: 5 +aliases: +- /stream-aggregation/common-mistakes/ +- /stream-aggregation/common-mistakes/index.html +--- + +## Place aggregation agents behind a load balancer + +Partial aggregation (only a subset of all data, which satisfies [`match`](./configuration/#match) expression was pushed to an aggregation agent) is not acceptable. +It produces wrong aggregations which are not usable and not comparable to equivalent [recording rules](https://docs.victoriametrics.com/vmalert#recording-rules). + +To keep aggregation results consistent, it should be either fully processed on a single VMAgent or data can be sharded across multiple VMAgents by metric name. + +## Create separate aggregator for each recording rule + +As was mentioned in [use case scenarios](./use-cases.md#recording-rules-alternative), stream aggregation can be considered as a substitution for [recording rules](https://docs.victoriametrics.com/vmalert#recording-rules), but straightforward conversion of recording rules to [stream aggregation config](./configuration/#configuration-file-reference) can lead to inefficient resource usage on the component it's configured on ([VMAgent](https://docs.victoriametrics.com/vmagent) or [VMSingle](https://docs.victoriametrics.com/vmsingle)). + +To optimize this, we recommend merging together aggregations which only differ in match expressions. E.g: + +Given list of recording rules: + +```yaml +- expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"}[3m])) BY (instance) + record: instance:node_cpu:rate:sum +- expr: sum(rate(node_network_receive_bytes_total[3m])) BY (instance) + record: instance:node_network_receive_bytes:rate:sum +- expr: sum(rate(node_network_transmit_bytes_total[3m])) BY (instance) + record: instance:node_network_transmit_bytes:rate:sum +- expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"}[5m])) + record: cluster:node_cpu:sum_rate5m +``` + +can be converted to aggregation rules: + +```yaml +- match: node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"} + interval: 3m + by: + - instance + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + replacement: instance:node_cpu:rate:sum +- match: node_network_receive_bytes_total + interval: 3m + by: + - instance + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + replacement: instance:node_network_receive_bytes:rate:sum +- match: node_network_transmit_bytes_total + interval: 3m + by: + - instance + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + replacement: instance:node_network_transmit_bytes:rate:sum +- match: node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"} + interval: 5m + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + replacement: cluster:node_cpu:sum_rate5m +``` + +note, that first 3 aggregation rules differ only in [`match`](./configuration/#match), so they can be merged together: + +```yaml +- match: + - node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"} + - node_network_receive_bytes_total + - node_network_transmit_bytes_total + interval: 3m + by: + - instance + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + regex: regex: "(.+)(_seconds)?(_total)?:.+" + replacement: cluster:node_cpu:sum_rate5m +- match: node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"} + interval: 5m + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + replacement: cluster:node_cpu:sum_rate5m +``` + +**Note**: having separate aggregator for a certain [`match`](./configuration/#match) expression can only be justified when aggregator cannot keep up with all +the data pushed to an aggregator within an aggregation interval + +## Use identical --remoteWrite.streamAggr.config for all remote writes + +As it's described in [previous](#create-separate-aggregator-for-each-recording-rule) case having many aggregators leads to increased resource usage so having `n` +identical aggregation configurations `-remoteWrite.streamAggr.config` for multiple `-remoteWrite.url` requires `n * x` resources. + +As an optimization, we suggest using `-streamAggr.config` as a replacement for `-remoteWrite.streamAggr.config`. +It places the global aggregator in front of all remote writes, which helps to reduce resource usage. + +## Treat aggregated metrics in the same manner as original ones + +Stream aggregation allows to keep for aggregation result the name of a source metric using [`keep_metric_names:`](./configuration/#keep-metric-names) `true`. +But graphs and alerts, which were previously used for a raw metric can become incorrect for aggregated one. + +Dashboards and alerts should be updated according to aggregation configurations. diff --git a/docs/stream-aggregation/configuration/README.md b/docs/stream-aggregation/configuration/README.md new file mode 100644 index 000000000..3242f99e8 --- /dev/null +++ b/docs/stream-aggregation/configuration/README.md @@ -0,0 +1,116 @@ +Stream aggregation can be configured via the following command-line flags: + +- `-streamAggr.config` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/) + and at [vmagent](https://docs.victoriametrics.com/vmagent). +- `-remoteWrite.streamAggr.config` at [vmagent](https://docs.victoriametrics.com/vmagent) only. + This flag can be specified individually per each `-remoteWrite.url` and aggregation will happen independently for each of them. + This allows writing different aggregates to different remote storage destinations. + +These flags must point to a file containing [stream aggregation config](#configuration-file-reference). +The file may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values. + +By default, the following data is written to the storage when stream aggregation is enabled: + +- the aggregated samples; +- the raw input samples, which didn't match any [`match`](#match) option in the provided [config](#configuration-file-reference) + +This behaviour can be changed via the following command-line flags: + +- `-streamAggr.keepInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com) + and [vmagent](https://docs.victoriametrics.com/vmagent). At [vmagent](https://docs.victoriametrics.com/vmagent) + `-remoteWrite.streamAggr.keepInput` flag can be specified individually per each `-remoteWrite.url`. + If one of these flags is set, then all the input samples are written to the storage alongside the aggregated samples. +- `-streamAggr.dropInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com) + and [vmagent](https://docs.victoriametrics.com/vmagent). At [vmagent](https://docs.victoriametrics.com/vmagent) + `-remoteWrite.streamAggr.dropInput` flag can be specified individually per each `-remoteWrite.url`. + If one of these flags are set, then all the input samples are dropped, while only the aggregated samples are written to the storage. + +## Configuration File Reference + +### Overview + +Stream aggregation config file contains YAML formatted configs and may be referred via +`-streamAggr.config` command-line flag at [single-node VictoriaMetrics](https://docs.victoriametrics.com) +and [vmagent](https://docs.victoriametrics.com/vmagent). At [vmagent](https://docs.victoriametrics.com/vmagent) `-remoteWrite.streamAggr.config` +command-line flag can be specified individually per each `-remoteWrite.url` or just once, which +enables same aggregation rules for each `-remoteWrite.url`. + +### Example configuration + +```yaml +- match: 'http_request_duration_seconds_bucket{env=~"prod|staging"}' + interval: 1m + by: [vmrange] + outputs: [total] +``` +* [`name`](?selfref=true#name) `(string: "none")` - name of the given streaming aggregation config. + If it is set, then it is used as `name` label in the exposed metrics for the given aggregation config at /metrics page. + See monitoring related information [here](https://docs.victoriametrics.com/vmagent#monitoring) and [here](https://docs.victoriametrics.com/#monitoring) +* [`match`](?selfref=true#match) `(list or string: [])` - an optional filter for incoming samples to aggregate. + It can contain arbitrary Prometheus series selector + according to [filtering concepts](https://docs.victoriametrics.com/keyconcepts#filtering). + If match isn't set, then all the incoming samples are aggregated. + match also can contain a list of series selectors. Then the incoming samples are aggregated + if they match at least a single series selector. +* [`interval`](?selfref=true#interval) `(string: "", required)` - interval for the aggregation. The aggregated stats are sent to + remote storage once per interval. +* [`dedup_interval`](?selfref=true#dedup_interval) `(string:"")` - interval for de-duplication of input samples before the aggregation. + Samples are de-duplicated on a per-series basis. See [timeseries](https://docs.victoriametrics.com/keyconcepts#time-series) and [deduplication](#deduplication) + The deduplication is performed after [`input_relabel_configs`](#input-relabel-configs) relabeling is applied. By default, the deduplication is disabled + unless `-remoteWrite.streamAggr.dedupInterval` or `-streamAggr.dedupInterval` command-line flags are set. +* [`staleness_interval`](?selfref=true#staleness_interval) `(string:2*interval)` - interval for resetting the per-series state if no new samples + are received during this interval for the following outputs: + * [`rate_avg`](./outputs/#rate_avg) + * [`rate_sum`](./outputs/#rate_sum) + * [`total`](./outputs/#total) + * [`total_prometheus`](./outputs/#total_prometheus) + * [`increase`](./outputs/#increase) + * [`increase_prometheus`](./outputs/#increase_prometheus) + * [`histogram_bucket`](./outputs/#histogram_bucket) + Check [staleness](#staleness) for more details. +* [`no_align_flush_to_interval`](?selfref=true#no_align_flush_to_interval) `(bool: false)` - disables aligning of flush times for the aggregated data to multiples of interval. + By default, flush times for the aggregated data is aligned to multiples of interval. + For example: + * if [`interval:`](#interval) `1m` is set, then flushes happen at the end of every minute, + * if [`interval:`](#interval) `1h` is set, then flushes happen at the end of every hour +* [`flush_on_shutdown`](?selfref=true#flush_on_shutdown) `(bool: false)` - instructs to flush aggregated data to the storage on the first and the last intervals + during vmagent starts, restarts or configuration reloads. + Incomplete aggregated data isn't flushed to the storage by default, since it is usually confusing. +* [`without`](?selfref=true#without) `(list: [])` - list of labels, which must be removed from the output aggregation. + See [aggregation by labels](#aggregating-by-labels) +* [`by`](?selfref=true#by) `(list: [])` - list of labels, which must be preserved in the output aggregation. + See [aggregation by labels](#aggregating-by-labels) +* [`outputs`](?selfref=true#outputs) `(list:[], required)` - list of aggregations to perform on the input data. + See [aggregation outputs](#outputs). +* [`keep_metric_names`](?selfref=true#keep_input_names) `(bool: false)` - instructs keeping the original metric names for the aggregated samples. + This option can be set only if outputs list contains only a single output. + By default, a special suffix is added to original metric names in the aggregated samples. + See [output metric names](../#output-metric-names) +* [`ignore_old_samples`](?selfref=true#ignore_old_samples) `(bool: false)` - instructs ignoring input samples with old timestamps outside the current aggregation interval. + See [ignoring old samples](../#ignoring-old-samples) + See also [-remoteWrite.streamAggr.ignoreOldSamples](#remote-write-ignore-old-samples-flag) or [-streamAggr.ignoreOldSamples](#ignore-old-samples-flag) command-line flag. +* [`ignore_first_intervals`](?selfref=true#ignore_first_intervals) `(int: 0)` - instructs ignoring the first N aggregation intervals after process start. + See [ignore first intervals on start](../#ignore-aggregation-intervals-on-start) + See also [-remoteWrite.streamAggr.ignoreFirstIntervals](#remote-write-ignore-first-intervals-flag) or [-streamAggr.ignoreFirstIntervals](#ignore-first-intervals-flag) command-line flags. +* [`drop_input_labels`](?selfref=true#drop_input_labels) `(bool: false)` - instructs dropping the given labels from input samples. + The labels' dropping is performed before [`input_relabel_configs`](#input-relabel-configs) are applied. + This also means that the labels are dropped before [deduplication](../#deduplication) and stream aggregation. +* [`input_relabel_configs`](?selfref=true#output_relabel_configs) `(array: [])` - relabeling rules, which are applied to the incoming samples + after they pass the match filter and before being aggregated. See [relabeling](../#relabeling) +* [`output_relabel_configs`](?selfref=true#output_relabel_configs) `(array: [])` - relabeling rules, which are applied to the aggregated output metrics. + +The file can contain multiple aggregation configs. The aggregation is performed independently +per each specified config entry. + +### Configuration update + +[vmagent](https://docs.victoriametrics.com/vmagent) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/vmagent) +support the following approaches for hot reloading stream aggregation configs from `-remoteWrite.streamAggr.config` and `-streamAggr.config`: + +* By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process: + + ```sh + kill -SIGHUP `pidof vmagent` + ``` + +* By sending HTTP request to `/-/reload` endpoint (e.g. `http://vmagent:8429/-/reload` or `http://victoria-metrics:8428/-/reload). diff --git a/docs/stream-aggregation/configuration/_index.md b/docs/stream-aggregation/configuration/_index.md new file mode 100644 index 000000000..0f9db62f1 --- /dev/null +++ b/docs/stream-aggregation/configuration/_index.md @@ -0,0 +1,14 @@ +--- +sort: 4 +weight: 4 +title: Configuration +menu: + docs: + identifier: stream-aggregation-config + parent: stream-aggregation + weight: 4 +aliases: +- /stream-aggregation/configuration/ +- /stream-aggregation/configuration/index.html +--- +{{% content "README.md" %}} diff --git a/docs/stream-aggregation/configuration/outputs/README.md b/docs/stream-aggregation/configuration/outputs/README.md new file mode 100644 index 000000000..d6d427749 --- /dev/null +++ b/docs/stream-aggregation/configuration/outputs/README.md @@ -0,0 +1,553 @@ +The aggregations are calculated during the [`interval`](../#interval) and then sent to the storage once +per [`interval`](../#interval). The aggregated samples are named according to [`output metric naming`](../#output-metric-names). +If [`by`](../#by) and [`without`](../#without) lists are set then the [`aggregation by labels`](../#aggregating-by-labels) is performed additionally to aggregation by `interval`. + +Below are aggregation functions that can be set in the [`outputs`](../#outputs): + +## avg +`avg` returns the average over input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples). +`avg` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +For example, see below time series produced by config with [`aggregation interval:`](../#interval) `1m` and [`by:`](../#by) `[instance]` and the regular query: + +![avg aggregation](./avg.webp) + +### Example + +```yaml +match: +- some_counter +interval: 1m +outputs: +- avg +``` + +### MetricsQL + +```metricsql +sum(sum_over_time(some_metric[interval])) / sum(count_over_time(some_metric[5m])) +``` + +### See also + +- [`min`](#min) +- [`max`](#max) +- [`sum_samples`](#sum_samples) +- [`count_samples`](#count_samples) + +## count_samples + +`count_samples` counts the number of input [samples](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given [`interval`](../#interval). + +### Example + +```yaml +match: +- some_counter +interval: 1m +outputs: +- count_samples +``` + +### MetricsQL + +```metricsql +sum(count_over_time(some_metric[interval])) +``` + +### See also + +- [`count_series`](#count_series) +- [`sum_samples`](#sum_samples) + +## count_series + +`count_series` counts the number of unique [time series](https://docs.victoriametrics.com/keyconcepts#time-series) over the given [`interval`](../#interval). + +### Example + +```yaml +match: +- some_counter +interval: 1m +outputs: +- count_series +``` + +### MetricsQL + +```metricsql +count(last_over_time(some_metric[interval])) +``` + +### See also + +- [`count_samples`](#count_samples) +- [`unique_samples`](#unique_samples) + +## histogram_bucket + +`histogram_bucket` returns [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) + for the input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given [`interval`](../#interval). +`histogram_bucket` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). +See how to aggregate regular histograms [here](#aggregating-histograms). + +The results of `histogram_bucket` is equal to the following [MetricsQL](../../MetricsQL.md) query: + +Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) +or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. + +### Example + +```yaml +match: +- some_histogram_bucket +interval: 1m +outputs: +- histogram_bucket +``` + +### MetricsQL + +```metricsql +sum(histogram_over_time(some_histogram_bucket[1m])) by (vmrange) +``` + +### See also + +- [quantiles](#quantiles) +- [min](#min) +- [max](#max) +- [avg](#avg) + +## increase + +`increase` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts#time-series) over the given [`interval`](../#interval). +`increase` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts#time-series) +is `10`, then `increase` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, +then take a look at [increase_prometheus](#increase_prometheus). + +For example, see below time series produced by config with [`aggregation interval`](../#interval) `1m` and `by: ["instance"]` and the regular query: + +![increase aggregation](./increase.webp) + +Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) +or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](../#staleness") option. + +### Example + +```yaml +match: +- some_counter +interval: 2m +outputs: +- increase +``` + +### MetricsQL + +```metricsql +sum(increase_pure(some_counter[2m])) +``` + +### See also + +- [`increase_prometheus`](#increase_prometheus) +- [`total`](#total) + +## increase_prometheus + +`increase_prometheus` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts#time-series) over the given [`interval`](../#interval). +`increase_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +`increase_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts#time-series). +If you need taking into account the first sample per time series, then take a look at [increase](../#increase). + +Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) +or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](../#staleness) option. + +### Example + +```yaml +match: +- some_counter +interval: 5m +outputs: +- increase_prometheus +``` + +### MetricsQL + +```metricsql +sum(increase_prometheus(some_counter[5m])) +``` + +### See also + +- [increase](#increase) +- [total](#total) +- [total_prometheus](#total_prometheus) + +## last +`last` returns the last input [sample value](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given [`interval`](../#interval). + +### Example + +```yaml +match: +- some_metric +interval: 5m +outputs: +- last +``` + +### MetricsQL + +```metricsql +last_over_time(some_metric[interval]) +``` + +### See also + +- [`min`](#min) +- [`max`](#max) +- [`avg`](#avg). + +## max + +`max` returns the maximum input [sample value](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given [`interval`](../#interval). + +For example, see below time series produced by config with [`aggregation interval:`](../#interval) `1m` and the regular query: + +![total aggregation](./max.webp) + +### Example + +```yaml +match: +- some_metric +interval: 5m +outputs: +- max +``` + +### MetricsQL + +```metricsql +max(max_over_time(some_metric[interval])) +``` + +### See also + +- [`min`](#min) +- [`avg`](#avg) + +## min + +`min` returns the minimum input [sample value](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given `interval`. + +For example, see below time series produced by config with [`aggregation interval:`](../#interval) `1m` and the regular query: + +![min aggregation](./min.webp) + +### Example + +```yaml +match: +- some_metric +interval: 5m +outputs: +- min +``` + +### MetricsQL + +```metricsql +min(min_over_time(some_metric[5m])) +``` + +### See also + +- [`max`](#max) +- [`avg`](#avg) + +## quantiles + +`quantiles(phi1, ..., phiN)` returns [percentiles](https://en.wikipedia.org/wiki/Percentile) for the given `phi*` +over the input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples) on the given `interval`. +`phi` must be in the range `[0..1]`, where `0` means `0th` percentile, while `1` means `100th` percentile. +`quantiles(...)` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +### Example + +```yaml +match: +- temperature +interval: 24h +outputs: +- quantiles(0.55, 0.99) +``` + +### MetricsQL + +```metricsql +histogram_quantiles("quantile", 0.55, 0.99, sum(histogram_over_time(temperature[24h])) by (vmrange)) +``` + +### See also + +- [`histogram_bucket`](#histogram_bucket) +- [`min`](#min) +- [`avg`](#avg) + +## rate_avg + +`rate_avg` returns the average of average per-second of input [time series](https://docs.victoriametrics.com/keyconcepts#time-series) over the given [`interval`](../#interval). +`rate_avg` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +See also [rate_sum](#rate_sum) and [total](#total) outputs. + +### Example + +```yaml +match: +- some_counter +interval: 5m +outputs: +- rate_avg +``` + +### MetricsQL + +```metricsql +avg(rate(some_counter[5m])) +``` + +## rate_sum + +`rate_sum` returns the sum of average per-second change of input [time series](https://docs.victoriametrics.com/keyconcepts#time-series) over the given [`interval`](../#interval). +`rate_sum` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +### Example + +```yaml +match: +- some_counter +interval: 2m +outputs: +- rate_sum +``` + +### MetricsQL + +```metricsql +sum(rate(some_counter[2m])) +``` + +### See also + +- [`rate_avg`](#rate_avg) +- [`total`](#total) + +## stddev +`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples) +over the given [`interval`](../#interval). +`stddev` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +### Example + +```yaml +match: +- some_metric +interval: 2m +outputs: +- stddev +``` + +### MetricsQL + +```metricsql +histogram_stddev(sum(histogram_over_time(some_metric[2m])) by (vmrange)) +``` + +### See also + +- [`stdvar`](#stdvar) +- [`avg`](#avg). + +## stdvar + +`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples) +over `he given [`interval`](../#interval). +`stdvar` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +For example, see below time series produced by config with [`aggregation interval`](../#interval) `1m` and the regular query: + +![stdvar aggregation](./stdvar.webp) + +### Example + +```yaml +match: +- some_metric +interval: 10m +outputs: +- stdvar +``` + +### MetricsQL + +```metricsql +histogram_stdvar(sum(histogram_over_time(some_metric[10m])) by (vmrange)) +``` + +### See also + +- [`stddev`](#stddev) +- [`avg`](#avg) + +## sum_samples + +`sum_samples` sums input [sample values](https://docs.victoriametrics.com/keyconcepts#raw-samples) over the given [`interval`](../#interval). +`sum_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +For example, see below time series produced by config with [`aggregation interval`](../#interval) `1m` and the regular query: + +![sum_samples aggregation](./sum-samples.webp) + +### Example + +```yaml +match: +- some_metric +interval: 5m +outputs: +- sum_samples +``` + +### MetricsQL + +```metricsql +sum(sum_over_time(some_metric[5m])) +``` + +### See also + +- [`count_samples`](#count_samples) +- [`count_series`](#count_series) + +## total + +`total` generates output [counter](https://docs.victoriametrics.com/keyconcepts#counter) by summing the input counters over the given [`interval`](../#interval). +`total` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts#time-series) +is `10`, then `total` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series, +then take a look at [total_prometheus](#total_prometheus). + +For example, see below time series produced by config with [`aggregation interval:`](../#interval) `1m` and [`by:`](../#by) `[instance]` and the regular query: + +![total aggregation](./total.webp) + +`total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts#counter) - +it continues to increase monotonically with respect to the previous value. +The counters are most often reset when the application is restarted. + +For example: + +![total aggregation counter reset](./total-reset.webp) + +The same behavior occurs when creating or deleting new series in an aggregation group - +`total` output increases monotonically considering the values of the series set. +An example of changing a set of series can be restarting a pod in the Kubernetes. +This changes pod name label, but the `total` accounts for such a scenario and doesn't reset the state of aggregated metric. + +Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) +or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](../#staleness) option. + +### Example + +```yaml +match: +- some_counter +interval: 1m +outputs: +- total +``` + +### MetricsQL + +```metricsql +sum(running_sum(increase_pure(some_counter[1m]))) +``` + +### See also + +- [`total_prometheus`](#total_prometheus) +- [`increase`](#increase) +- [`increase_prometheus`](#increase_prometheus) + +## total_prometheus + +`total_prometheus` generates output [counter](https://docs.victoriametrics.com/keyconcepts#counter) by summing the input counters over the given [`interval`](../#interval). +`total_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts#counter). + +`total_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts#time-series). +If you need taking into account the first sample per time series, then take a look at [total](#total). + +`total_prometheus` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts#counter) - +it continues to increase monotonically with respect to the previous value. +The counters are most often reset when the application is restarted. + +Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) +or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. + +### Example + +```yaml +match: +- some_counter +interval: 1m +outputs: +- total_prometheus +``` + +### MetricsQL + +```metricsql +sum(running_sum(increase_prometheus(some_counter[1m]))) +``` + +### See also + +- [`total`](#total) +- [`increase`](#increase) +- [`increase_prometheus`](#increase_prometheus) + +## unique_samples + +`unique_samples` counts the number of unique sample values over the given [`interval`](../#interval). +`unique_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts#gauge). + +### Example + +```yaml +match: +- some_metric +interval: 2m +outputs: +- unique_samples +``` + +### MetricsQL + +```metricsql +count(count_values_over_time(some_metric[2m])) +``` + +### See also + +- [`sum_samples`](#sum_samples) +- [`count_series`](#count_series) diff --git a/docs/stream-aggregation/configuration/outputs/_index.md b/docs/stream-aggregation/configuration/outputs/_index.md new file mode 100644 index 000000000..2d1a22885 --- /dev/null +++ b/docs/stream-aggregation/configuration/outputs/_index.md @@ -0,0 +1,14 @@ +--- +sort: 1 +weight: 1 +title: Outputs +menu: + docs: + identifier: stream-aggregation-config-outputs + parent: stream-aggregation-config + weight: 1 +aliases: +- /stream-aggregation/outputs/ +- /stream-aggregation/outputs/index.html +--- +{{% content "README.md" %}} diff --git a/docs/stream-aggregation-check-avg.webp b/docs/stream-aggregation/configuration/outputs/avg.webp similarity index 100% rename from docs/stream-aggregation-check-avg.webp rename to docs/stream-aggregation/configuration/outputs/avg.webp diff --git a/docs/stream-aggregation-check-increase.webp b/docs/stream-aggregation/configuration/outputs/increase.webp similarity index 100% rename from docs/stream-aggregation-check-increase.webp rename to docs/stream-aggregation/configuration/outputs/increase.webp diff --git a/docs/stream-aggregation-check-max.webp b/docs/stream-aggregation/configuration/outputs/max.webp similarity index 100% rename from docs/stream-aggregation-check-max.webp rename to docs/stream-aggregation/configuration/outputs/max.webp diff --git a/docs/stream-aggregation-check-min.webp b/docs/stream-aggregation/configuration/outputs/min.webp similarity index 100% rename from docs/stream-aggregation-check-min.webp rename to docs/stream-aggregation/configuration/outputs/min.webp diff --git a/docs/stream-aggregation-check-stdvar.webp b/docs/stream-aggregation/configuration/outputs/stdvar.webp similarity index 100% rename from docs/stream-aggregation-check-stdvar.webp rename to docs/stream-aggregation/configuration/outputs/stdvar.webp diff --git a/docs/stream-aggregation-check-sum-samples.webp b/docs/stream-aggregation/configuration/outputs/sum-samples.webp similarity index 100% rename from docs/stream-aggregation-check-sum-samples.webp rename to docs/stream-aggregation/configuration/outputs/sum-samples.webp diff --git a/docs/stream-aggregation-check-total-reset.webp b/docs/stream-aggregation/configuration/outputs/total-reset.webp similarity index 100% rename from docs/stream-aggregation-check-total-reset.webp rename to docs/stream-aggregation/configuration/outputs/total-reset.webp diff --git a/docs/stream-aggregation-check-total.webp b/docs/stream-aggregation/configuration/outputs/total.webp similarity index 100% rename from docs/stream-aggregation-check-total.webp rename to docs/stream-aggregation/configuration/outputs/total.webp diff --git a/docs/stream-aggregation/key-concepts.md b/docs/stream-aggregation/key-concepts.md new file mode 100644 index 000000000..c7301095b --- /dev/null +++ b/docs/stream-aggregation/key-concepts.md @@ -0,0 +1,243 @@ +--- +sort: 2 +weight: 2 +title: Key concepts +menu: + docs: + identifier: stream-aggregation-key-concepts + parent: 'stream-aggregation' + weight: 2 +aliases: +- /stream-aggregation/key-concepts/index.html +- /stream-aggretation/key-concepts/ +--- + +[Single-node VictoriaMetrics](https://docs.victoriametrics.com/) supports relabeling, +deduplication and stream aggregation for all the received data, scraped or pushed. +The processed data is then stored in local storage and **can't be forwarded further**. + +[vmagent](https://docs.victoriametrics.com/vmagent) supports relabeling, deduplication and stream aggregation for all +the received data, scraped or pushed. Then, the collected data will be forwarded to specified `-remoteWrite.url` destinations. +The data processing order is the following: +1. All the received data is [relabeled](https://docs.victoriametrics.com/vmagent#relabeling) according to + specified `-remoteWrite.relabelConfig`; +2. All the received data is [deduplicated](#deduplication) + according to specified `-streamAggr.dedupInterval`; +3. All the received data is aggregated according to specified `-streamAggr.config`; +4. The resulting data from p1 and p2 is then replicated to each `-remoteWrite.url`; +5. Data sent to each `-remoteWrite.url` can be additionally relabeled according to the + corresponding `-remoteWrite.urlRelabelConfig` (set individually per URL); +6. Data sent to each `-remoteWrite.url` can be additionally deduplicated according to the + corresponding `-remoteWrite.streamAggr.dedupInterval` (set individually per URL); +7. Data sent to each `-remoteWrite.url` can be additionally aggregated according to the + corresponding `-remoteWrite.streamAggr.config` (set individually per URL). Please note, it is not recommended + to use `-streamAggr.config` and `-remoteWrite.streamAggr.config` together, unless you understand the complications. + +Typical scenarios for data routing with vmagent: +1. **Aggregate incoming data and replicate to N destinations**. For this one should configure `-streamAggr.config` +to aggregate the incoming data before replicating it to all the configured `-remoteWrite.url` destinations. +2. **Individually aggregate incoming data for each destination**. For this on should configure `-remoteWrite.streamAggr.config` +for each `-remoteWrite.url` destination. [Relabeling](https://docs.victoriametrics.com/vmagent#relabeling) +via `-remoteWrite.urlRelabelConfig` can be used for routing only selected metrics to each `-remoteWrite.url` destination. + +## Deduplication + +[vmagent](https://docs.victoriametrics.com/vmagent) supports online [de-duplication](https://docs.victoriametrics.com#deduplication) of samples +before sending them to the configured `-remoteWrite.url`. The de-duplication can be enabled via the following options: + +- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag for all received data + or via `-remoteWrite.streamAggr.dedupInterval` command-line flag for the particular `-remoteWrite.url` destination. + For example, `./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=30s` instructs `vmagent` to leave + only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts#time-series) per every 30 seconds. + The de-deduplication is performed after applying [relabeling](https://docs.victoriametrics.com/vmagent#relabeling) and + before performing the aggregation. + If the `-remoteWrite.streamAggr.config` and / or `-streamAggr.config` is set, then the de-duplication is performed individually per each + [stream aggregation config](./configuration/#configuration-file-reference) for the matching samples after applying [input_relabel_configs](#relabeling). + +- By specifying `dedup_interval` option individually per each [stream aggregation config](./configuration/#configuration-file-reference) + in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs. + +[Single-node VictoriaMetrics](https://docs.victoriametrics.com/) supports two types of de-duplication: +- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](https://docs.victoriametrics.com/#deduplication) command-line option. +- Before storing the duplicate samples to local storage. This type of de-duplication can be enabled via the following options: + - By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag. + For example, `./victoria-metrics -streamAggr.dedupInterval=30s` instructs VictoriaMetrics to leave only the last sample per each + seen [time series](https://docs.victoriametrics.com/keyconcepts#time-series) per every 30 seconds. + The de-duplication is performed after applying [relabeling](https://docs.victoriametrics.com/#relabeling) and before performing the aggregation. + + If the `-remtoeWrite.streamAggr.config` and / or `-streamAggr.config` is set, then the de-duplication is performed individually per each + [stream aggregation config](./configuration/#configuration-file-reference) for the matching samples after applying [input_relabel_configs](#relabeling). + + - By specifying `dedup_interval` option individually per each [stream aggregation config](./configuration/#configuration-file-reference) + in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs. + +It is possible to drop the given labels before applying the de-duplication. See [these docs](#dropping-unneeded-labels). + +The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](https://docs.victoriametrics.com#deduplication) at VictoriaMetrics. + +## Ignoring old samples + +By default, all the input samples are taken into account during stream aggregation. If samples with old timestamps +outside the current [aggregation interval](./configuration/#interval) must be ignored, then the following options can be used: + +- To pass `-streamAggr.ignoreOldSamples` command-line flag to [single-node VictoriaMetrics](https://docs.victoriametrics.com) + or to [vmagent](https://docs.victoriametrics.com/vmagent). At [vmagent](https://docs.victoriametrics.com/vmagent) + `-remoteWrite.streamAggr.ignoreOldSamples` flag can be specified individually per each `-remoteWrite.url`. + This enables ignoring old samples for all the [aggregation configs](./configuration/#configuration-file-reference). + +- To set [`ignore_old_samples:`](./configuration/#ignore-old-samples) `true` option at the particular [aggregation config](./configuration/#configuration-file-reference). + This enables ignoring old samples for that particular aggregation config. + +## Ignore aggregation intervals on start + +Streaming aggregation results may be incorrect for some time after the restart of [vmagent](https://docs.victoriametrics.com/vmagent) +or [single-node VictoriaMetrics](https://docs.victoriametrics.com) until all the buffered [samples](https://docs.victoriametrics.com/keyconcepts#raw-samples) +are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](https://docs.victoriametrics.com/vmagent#how-to-push-data-to-vmagent). +In this case it may be a good idea to drop the aggregated data during the first `N` [aggregation intervals](./configuration/#interval) +just after the restart of `vmagent` or single-node VictoriaMetrics. This can be done via the following options: + +- Set `-streamAggr.ignoreFirstIntervals=` command-line flag to [single-node VictoriaMetrics](https://docs.victoriametrics.com) + or to [vmagent](https://docs.victoriametrics.com/vmagent) to skip first `` [aggregation intervals](./configuration/#interval) + from persisting to the storage. At [vmagent](https://docs.victoriametrics.com/vmagent) + `-remoteWrite.streamAggr.ignoreFirstIntervals=` flag can be specified individually per each `-remoteWrite.url`. + It is expected that all incomplete or queued data will be processed during specified `` + and all subsequent aggregation intervals will produce correct data. + +- Set `ignore_first_intervals: ` option individually per [aggregation config](./configuration/#configuration-file-reference). + This enables ignoring first `` aggregation intervals for that particular aggregation config. + +## Flush time alignment + +By default, the time for aggregated data flush is aligned by the [`interval`](./configuration/#interval) option. + +For example: + +- if `interval: 1m` is set, then the aggregated data is flushed to the storage at the end of every minute +- if `interval: 1h` is set, then the aggregated data is flushed to the storage at the end of every hour + +If you do not need such an alignment, then set [`no_align_flush_to_interval:`](./configuration/#no-align-flush-to-interval) `true` option in the [aggregate config](./configuration/#configuration-file-reference). +In this case aggregated data flushes will be aligned to the `vmagent` start time or to [config reload](./configuration/#configuration-update) time. + +The aggregated data on the first and the last interval is dropped during `vmagent` start, restart or [config reload](./configuration/#configuration-update), +since the first and the last aggregation intervals are incomplete, so they usually contain incomplete confusing data. +If you need preserving the aggregated data on these intervals, then set [`flush_on_shutdown:`](./configuration/#flush-on-shutdown) `true` option. + +See also: + +- [Ignore aggregation intervals on start](#ignore-aggregation-intervals-on-start) +- [Ignoring old samples](#ignoring-old-samples) + +## Output metric names + +Output metric names for stream aggregation are constructed according to the following pattern: + +```text +:[_by_][_without_]_ +``` + +- `` is the original metric name. +- `` is the [`interval`](./configuration/#interval) specified in the [stream aggregation config](./configuration/#configuration-file-reference). +- `` is `_`-delimited sorted list of [`by`](./configuration/#by) labels. + If the [`by`](./configuration/#by) list is missing in the config, then the `_by_` part isn't included in the output metric name. +- `` is an optional `_`-delimited sorted list of [`without`](./configuration/#without) labels specified in the [stream aggregation config](./configuration/#configuration-file-reference). + If the [`without`](./configuration/#without) list is missing in the config, then the `_without_` part isn't included in the output metric name. +- `` is the aggregate used for constructing the output metric. The aggregate name is taken from the [`outputs`](./configuration/outputs) list + at the corresponding [stream aggregation config](./configuration/#configuration-file-reference). + +Both input and output metric names can be modified if needed via relabeling according to [these docs](#relabeling). + +It is possible to leave the original metric name after the aggregation by specifying [`keep_metric_names:`](./configuration/#keep-metric-names) `true` option at [stream aggregation config](./configuration/#configuration-file-reference). +The [`keep_metric_names`](./configuration/#keep-metric-names) option can be used if only a single output is set in [`outputs`](./configuration/outputs) list. + +## Relabeling + +It is possible to apply [arbitrary relabeling](https://docs.victoriametrics.com/vmagent#relabeling) to input and output metrics +during stream aggregation via [`input_relabel_configs`](./configuration/#input-relabel-configs) and [`output_relabel_configs`](./configuration/#output-relabel-configs) options in [stream aggregation config](./configuration/#configuration-file-reference). + +Relabeling rules inside [`input_relabel_configs`](./configuration/#input-relabel-configs) are applied to samples matching the [`match`](./configuration/#match) filters before optional [deduplication](#deduplication). +Relabeling rules inside [`output_relabel_configs`](./configuration/#output-relabel-configs) are applied to aggregated samples before sending them to the remote storage. + +For example, the following config removes the `:1m_sum_samples` suffix added [to the output metric name](#output-metric-names): + +```yaml +- interval: 1m + outputs: [sum_samples] + output_relabel_configs: + - source_labels: [__name__] + target_label: __name__ + regex: "(.+):.+" +``` + +Another option to remove the suffix, which is added by stream aggregation, is to add [`keep_metric_names:`](./configuration/#keep-metric-names) `true` to the config: + +```yaml +- interval: 1m + outputs: [sum_samples] + keep_metric_names: true +``` + +See also [dropping unneeded labels](#dropping-unneeded-labels). + + +## Dropping unneeded labels + +If you need dropping some labels from input samples before [input relabeling](#relabeling), [de-duplication](#deduplication) +and stream aggregation, then the following options exist: + +- To specify comma-separated list of label names to drop in `-streamAggr.dropInputLabels` command-line flag + or via `-remoteWrite.streamAggr.dropInputLabels` individually per each `-remoteWrite.url`. + For example, `-streamAggr.dropInputLabels=replica,az` instructs to drop `replica` and `az` labels from input samples + before applying de-duplication and stream aggregation. + +- To specify [`drop_input_labels`](./configuration/#drop-input-labels) list with the labels to drop. + For example, the following config drops `replica` label from input samples with the name `process_resident_memory_bytes` + before calculating the average over one minute: + + ```yaml + - match: process_resident_memory_bytes + interval: 1m + drop_input_labels: [replica] + outputs: [avg] + keep_metric_names: true + ``` + +Typical use case is to drop `replica` label from samples, which are received from high availability replicas. + +## Aggregating by labels + +All the labels for the input metrics are preserved by default in the output metrics. For example, +the input metric `foo{app="bar",instance="host1"}` results to the output metric `foo:1m_sum_samples{app="bar",instance="host1"}` +when the following [stream aggregation config](./configuration/#configuration-file-reference) is used: + +```yaml +- interval: 1m + outputs: [sum_samples] +``` + +The input labels can be removed via [`without`](./configuration/#without) list specified in the config. For example, the following config +removes the `instance` label from output metrics by summing input samples across all the instances: + +```yaml +- interval: 1m + without: [instance] + outputs: [sum_samples] +``` + +In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1m_without_instance_sum_samples{app="bar"}` +output metric according to [output metric naming](#output-metric-names). + +It is possible specifying the exact list of labels in the output metrics via [`by`](./configuration/#by) list. +For example, the following config sums input samples by the `app` label: + +```yaml +- interval: 1m + by: [app] + outputs: [sum_samples] +``` + +In this case the `foo{app="bar",instance="..."}` input metrics are transformed into `foo:1m_by_app_sum_samples{app="bar"}` +output metric according to [output metric naming](#output-metric-names). + +The labels used in [`by`](./configuration/#by) and [`without`](./configuration/#without) lists can be modified via [`input_relabel_configs`](./configuration/#input-relabel-configs) section - see [these docs](#relabeling). + +See also [aggregation outputs](./configuration/outputs/). diff --git a/docs/stream-aggregation/troubleshooting.md b/docs/stream-aggregation/troubleshooting.md new file mode 100644 index 000000000..945060051 --- /dev/null +++ b/docs/stream-aggregation/troubleshooting.md @@ -0,0 +1,74 @@ +--- +sort: 5 +weight: 5 +title: Troubleshooting +menu: + docs: + identifier: stream-aggregation-troubleshooting + parent: 'stream-aggregation' + weight: 5 +aliases: +- /stream-aggregation/troubleshooting/ +- /stream-aggregation/troubleshooting/index.html +--- + +## Known scenarios + +- [Unexpected spikes for `total` or `increase` outputs](#staleness). +- [Lower than expected values for `total_prometheus` and `increase_prometheus` outputs](#staleness). +- [High memory usage and CPU usage](#high-resource-usage). +- [Unexpected results in vmagent cluster mode](#cluster-mode). + +### Staleness + +The following outputs track the last seen per-series values in order to properly calculate output values: + +- [rate_sum](./configuration/outputs/#rate_sum) +- [rate_avg](./configuration/outputs/#rate_avg) +- [total](./configuration/outputs/#total) +- [total_prometheus](./configuration/outputs/#total_prometheus) +- [increase](./configuration/outputs/#increase) +- [increase_prometheus](./configuration/outputs/#increase_prometheus) +- [histogram_bucket](./configuration/outputs/#histogram_bucket) + +The last seen per-series value is dropped if no new samples are received for the given time series during two consecutive aggregation +intervals specified in [stream aggregation config](./configuration/README.md) via `interval` option. +If a new sample for the existing time series is received after that, then it is treated as the first sample for a new time series. +This may lead to the following issues: + +- Lower than expected results for [total_prometheus](./configuration/outputs/#total_prometheus) and [increase_prometheus](./configuration/outputs/#increase_prometheus) outputs, + since they ignore the first sample in a new time series. +- Unexpected spikes for [total](./configuration/outputs/#total) and [increase](./configuration/outputs/#increase) outputs, since they assume that new time series start from 0. + +These issues can be fixed in the following ways: + +- By increasing the `interval` option at [stream aggregation config](./configuration/README.md), so it covers the expected + delays in data ingestion pipelines. +- By specifying the `staleness_interval` option at [stream aggregation config](./configuration/README.md), so it covers the expected + delays in data ingestion pipelines. By default, the `staleness_interval` equals to `2 x interval`. + +### High resource usage + +The following solutions can help reducing memory usage and CPU usage durting streaming aggregation: + +- To use more specific `match` filters at [streaming aggregation config](./configuration/README.md), so only the really needed + [raw samples](https://docs.victoriametrics.com/keyconcepts#raw-samples) are aggregated. +- To increase aggregation interval by specifying bigger duration for the `interval` option at [streaming aggregation config](./configuration/README.md). +- To generate lower number of output time series by using less specific [`by` list](#aggregating-by-labels) or more specific [`without` list](#aggregating-by-labels). +- To drop unneeded long labels in input samples via [input_relabel_configs](#relabeling). + +### Cluster mode + +If you use [vmagent in cluster mode](https://docs.victoriametrics.com/vmagent#scraping-big-number-of-targets) for streaming aggregation +then be careful when using [`by` or `without` options](#aggregating-by-labels) or when modifying sample labels +via [relabeling](#relabeling), since incorrect usage may result in duplicates and data collision. + +For example, if more than one `vmagent` instance calculates [increase](./configuration/outputs/#increase) for `http_requests_total` metric +with `by: [path]` option, then all the `vmagent` instances will aggregate samples to the same set of time series with different `path` labels. +The proper fix would be [adding an unique label](https://docs.victoriametrics.com/vmagent#adding-labels-to-metrics) for all the output samples +produced by each `vmagent`, so they are aggregated into distinct sets of [time series](https://docs.victoriametrics.com/keyconcepts#time-series). +These time series then can be aggregated later as needed during querying. + +If `vmagent` instances run in Docker or Kubernetes, then you can refer `POD_NAME` or `HOSTNAME` environment variables +as an unique label value per each `vmagent` via `-remoteWrite.label=vmagent=%{HOSTNAME}` command-line flag. +See [these docs](https://docs.victoriametrics.com/#environment-variables) on how to refer environment variables in VictoriaMetrics components. diff --git a/docs/stream-aggregation/use-cases.md b/docs/stream-aggregation/use-cases.md new file mode 100644 index 000000000..afcc9e78f --- /dev/null +++ b/docs/stream-aggregation/use-cases.md @@ -0,0 +1,305 @@ +--- +sort: 1 +weight: 1 +title: Use cases +menu: + docs: + identifier: stream-aggregation-use-cases + parent: 'stream-aggregation' + weight: 1 +aliases: +- /stream-aggregation/use-cases/ +- /stream-aggregation/use-cases/index.html +--- + +## Statsd alternative + +Stream aggregation can be used as [statsd](https://github.com/statsd/statsd) alternative in the following cases: + +* [Counting input samples](#counting-input-samples) +* [Summing input metrics](#summing-input-metrics) +* [Quantiles over input metrics](#quantiles-over-input-metrics) +* [Histograms over input metrics](#histograms-over-input-metrics) +* [Aggregating histograms](#aggregating-histograms) + +Currently, streaming aggregation is available only for [supported data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data) +and not available for [Statsd metrics format](https://github.com/statsd/statsd/blob/master/docs/metric_types.md). + +## Recording rules alternative + +Sometimes [alerting queries](https://docs.victoriametrics.com/vmalert#alerting-rules) may require non-trivial amounts of CPU, RAM, +disk IO and network bandwidth at metrics storage side. For example, if `http_request_duration_seconds` histogram is generated by thousands +of application instances, then the alerting query `histogram_quantile(0.99, sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)) > 0.5` +can become slow, since it needs to scan too big number of unique [time series](https://docs.victoriametrics.com/keyconcepts#time-series) +with `http_request_duration_seconds_bucket` name. This alerting query can be accelerated by pre-calculating +the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](https://docs.victoriametrics.com/vmalert#recording-rules). +But this recording rule may take too much time to execute too. In this case the slow recording rule can be substituted +with the following [stream aggregation config](./configuration/#configuration-file-reference): + +```yaml +- match: 'http_request_duration_seconds_bucket' + interval: 5m + without: [instance] + outputs: [total] +``` + +This stream aggregation generates `http_request_duration_seconds_bucket:5m_without_instance_total` output series according to [output metric naming](./key-concepts.md#output-metric-names). +Then these series can be used in [alerting rules](https://docs.victoriametrics.com/vmalert#alerting-rules): + +```metricsql +histogram_quantile(0.99, last_over_time(http_request_duration_seconds_bucket:5m_without_instance_total[5m])) > 0.5 +``` + +This query is executed much faster than the original query, because it needs to scan much lower number of time series. + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [aggregating by labels](./key-concepts.md/#aggregating-by-labels). + +Field `interval` is recommended to be set to a value at least several times higher than your metrics collect interval. + +## Reducing the number of stored samples + +If per-[series](https://docs.victoriametrics.com/keyconcepts#time-series) samples are ingested at high frequency, +then this may result in high disk space usage, since too much data must be stored to disk. This also may result +in slow queries, since too much data must be processed during queries. + +This can be fixed with the stream aggregation by increasing the interval between per-series samples stored in the database. + +For example, the following [stream aggregation config](./configuration/#configuration-file-reference) reduces the frequency of input samples +to one sample per 5 minutes per each input time series (this operation is also known as downsampling): + +```yaml + # Aggregate metrics ending with _total with `total` output. + # See {{% ref "./configuration/outputs" %}} +- match: '{__name__=~".+_total"}' + interval: 5m + outputs: [total] + + # Downsample other metrics with `count_samples`, `sum_samples`, `min` and `max` outputs + # See {{% ref "./configuration/outputs" %}} +- match: '{__name__!~".+_total"}' + interval: 5m + outputs: [count_samples, sum_samples, min, max] +``` + +The aggregated output metrics have the following names according to [output metric naming](./key-concepts.md#output-metric-names): + +```text +# For input metrics ending with _total +some_metric_total:5m_total + +# For input metrics not ending with _total +some_metric:5m_count_samples +some_metric:5m_sum_samples +some_metric:5m_min +some_metric:5m_max +``` + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [aggregating histograms](#aggregating-histograms) and [aggregating by labels](./key-concepts.md#aggregating-by-labels). + +## Reducing the number of stored series + +Sometimes applications may generate too many [time series](https://docs.victoriametrics.com/keyconcepts#time-series). +For example, the `http_requests_total` metric may have `path` or `user` label with too big number of unique values. +In this case the following stream aggregation can be used for reducing the number metrics stored in VictoriaMetrics: + +```yaml +- match: 'http_requests_total' + interval: 30s + without: [path, user] + outputs: [total] +``` + +This config specifies labels, which must be removed from the aggregate output, in the `without` list. +See [these docs](./key-concepts.md#aggregating-by-labels) for more details. + +The aggregated output metric has the following name according to [output metric naming](./key-concepts.md#output-metric-names): + +```text +http_requests_total:30s_without_path_user_total +``` + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [aggregating histograms](#aggregating-histograms). + +## Counting input samples + +If the monitored application generates event-based metrics, then it may be useful to count the number of such metrics +at stream aggregation level. + +For example, if an advertising server generates `hits{some="labels"} 1` and `clicks{some="labels"} 1` metrics +per each incoming hit and click, then the following [stream aggregation config](./configuration/#configuration-file-reference) +can be used for counting these metrics per 30 second interval: + +```yaml +- match: '{__name__=~"hits|clicks"}' + interval: 30s + outputs: [count_samples] +``` + +This config generates the following output metrics for `hits` and `clicks` input metrics +according to [output metric naming](./key-concepts.md#output-metric-names): + +```text +hits:30s_count_samples count1 +clicks:30s_count_samples count2 +``` + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [aggregating by labels](./key-concepts.md#aggregating-by-labels). + +## Summing input metrics + +If the monitored application calculates some events and then sends the calculated number of events to VictoriaMetrics +at irregular intervals or at too high frequency, then stream aggregation can be used for summing such events +and writing the aggregate sums to the storage at regular intervals. + +For example, if an advertising server generates `hits{some="labels} N` and `clicks{some="labels"} M` metrics +at irregular intervals, then the following [stream aggregation config](./configuration/#configuration-file-reference) +can be used for summing these metrics per minute: + +```yaml +- match: '{__name__=~"hits|clicks"}' + interval: 1m + outputs: [sum_samples] +``` + +This config generates the following output metrics according to [output metric naming](https://docs.victoriametrics.com/keyconcepts#output-metric-names): + +```text +hits:1m_sum_samples sum1 +clicks:1m_sum_samples sum2 +``` + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [aggregating by labels](./key-concepts.md#aggregating-by-labels). + +## Quantiles over input metrics + +If the monitored application generates measurement metrics per request, then it may be useful to calculate +the pre-defined set of [percentiles](https://en.wikipedia.org/wiki/Percentile) over these measurements. + +For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics +per each incoming request, then the following [stream aggregation config](./configuration/#configuration-file-reference) +can be used for calculating 50th and 99th percentiles for these metrics every 30 seconds: + +```yaml +- match: + - request_duration_seconds + - response_size_bytes + interval: 30s + outputs: ["quantiles(0.50, 0.99)"] +``` + +This config generates the following output metrics according to [output metric naming](./key-concepts.md#output-metric-names): + +```text +request_duration_seconds:30s_quantiles{quantile="0.50"} value1 +request_duration_seconds:30s_quantiles{quantile="0.99"} value2 + +response_size_bytes:30s_quantiles{quantile="0.50"} value1 +response_size_bytes:30s_quantiles{quantile="0.99"} value2 +``` + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [histograms over input metrics](#histograms-over-input-metrics) and [aggregating by labels](./key-concepts.md#aggregating-by-labels). + +## Histograms over input metrics + +If the monitored application generates measurement metrics per request, then it may be useful to calculate +a [histogram](https://docs.victoriametrics.com/keyconcepts#histogram) over these metrics. + +For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics +per each incoming request, then the following [stream aggregation config](./configuration/#configuration-file-reference) +can be used for calculating [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) +for these metrics every 60 seconds: + +```yaml +- match: + - request_duration_seconds + - response_size_bytes + interval: 60s + outputs: [histogram_bucket] +``` + +This config generates the following output metrics according to [output metric naming](./key-concepts.md#output-metric-names). + +```text +request_duration_seconds:60s_histogram_bucket{vmrange="start1...end1"} count1 +request_duration_seconds:60s_histogram_bucket{vmrange="start2...end2"} count2 +... +request_duration_seconds:60s_histogram_bucket{vmrange="startN...endN"} countN + +response_size_bytes:60s_histogram_bucket{vmrange="start1...end1"} count1 +response_size_bytes:60s_histogram_bucket{vmrange="start2...end2"} count2 +... +response_size_bytes:60s_histogram_bucket{vmrange="startN...endN"} countN +``` + +The resulting histogram buckets can be queried with [MetricsQL](https://docs.victoriametrics.com/metricsql/) in the following ways: + +1. An estimated 50th and 99th [percentiles](https://en.wikipedia.org/wiki/Percentile) of the request duration over the last hour: + + ```metricsql + histogram_quantiles("quantile", 0.50, 0.99, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) + ``` + + This query uses [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function. + +1. An estimated [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of the request duration over the last hour: + + ```metricsql + histogram_stddev(sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) + ``` + + This query uses [histogram_stddev](https://docs.victoriametrics.com/metricsql/#histogram_stddev) function. + +1. An estimated share of requests with the duration smaller than `0.5s` over the last hour: + + ```metricsql + histogram_share(0.5, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange)) + ``` + + This query uses [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share) function. + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [quantiles over input metrics](#quantiles-over-input-metrics) and [aggregating by labels](./key-concepts.md#aggregating-by-labels). + +## Aggregating histograms + +[Histogram](https://docs.victoriametrics.com/keyconcepts#histogram) is a set of [counter](https://docs.victoriametrics.com/keyconcepts#counter) +metrics with different `vmrange` or `le` labels. As they're counters, the applicable aggregation output is +[total](./configuration/outputs/#total): + +```yaml +- match: 'http_request_duration_seconds_bucket' + interval: 1m + without: [instance] + outputs: [total] +``` + +This config generates the following output metrics according to [output metric naming](./key-concepts.md#output-metric-names): + +```text +http_request_duration_seconds_bucket:1m_without_instance_total{le="0.1"} value1 +http_request_duration_seconds_bucket:1m_without_instance_total{le="0.2"} value2 +http_request_duration_seconds_bucket:1m_without_instance_total{le="0.4"} value3 +http_request_duration_seconds_bucket:1m_without_instance_total{le="1"} value4 +http_request_duration_seconds_bucket:1m_without_instance_total{le="3"} value5 +http_request_duration_seconds_bucket:1m_without_instance_total{le="+Inf" value6 +``` + +The resulting metrics can be passed to [histogram_quantile](https://docs.victoriametrics.com/metricsql#histogram_quantile) +function: + +```metricsql +histogram_quantile(0.9, sum(rate(http_request_duration_seconds_bucket:1m_without_instance_total[5m])) by(le)) +``` + +Please note, histograms can be aggregated if their `le` labels are configured identically. +[VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) +have no such requirement. + +See [the list of aggregate output](./configuration/outputs), which can be specified at `output` field. +See also [histograms over input metrics](#histograms-over-input-metrics) and [quantiles over input metrics](#quantiles-over-input-metrics). diff --git a/docs/vmagent.md b/docs/vmagent.md index 30b1b452c..7c6ed1c0d 100644 --- a/docs/vmagent.md +++ b/docs/vmagent.md @@ -153,7 +153,7 @@ See [these docs](#how-to-collect-metrics-in-prometheus-format) for details. `vmagent` can be used as an alternative to [statsd](https://github.com/statsd/statsd) when [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) is enabled. -See [these docs](https://docs.victoriametrics.com/stream-aggregation/#statsd-alternative) for details. +See [these docs](https://docs.victoriametrics.com/stream-aggregation/key-concepts#statsd-alternative) for details. ### Flexible metrics relay @@ -263,10 +263,10 @@ There is also support for multitenant writes. See [these docs](#multitenancy). ### Flexible deduplication -[Deduplication at stream aggregation](https://docs.victoriametrics.com/stream-aggregation/#deduplication) allows setting up arbitrary complex de-duplication schemes +[Deduplication at stream aggregation](https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication) allows setting up arbitrary complex de-duplication schemes for the collected samples. Examples: -- The following command instructs `vmagent` to send only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 60 seconds: +- The following command instructs `vmagent` to send only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) every 60 seconds: ``` ./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=60s ``` @@ -2199,7 +2199,7 @@ See the docs at https://docs.victoriametrics.com/vmagent/ . Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -remoteWrite.streamAggr.dedupInterval array - Input samples are de-duplicated with this interval before optional aggregation with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication (default 0s) + Input samples are de-duplicated with this interval before optional aggregation with -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication (default 0s) Supports array of values separated by comma or specified via multiple flags. Empty values are set to default value. -remoteWrite.streamAggr.dropInput array @@ -2207,13 +2207,13 @@ See the docs at https://docs.victoriametrics.com/vmagent/ . Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -remoteWrite.streamAggr.dropInputLabels array - An optional list of labels to drop from samples before stream de-duplication and aggregation with -remoteWrite.streamAggr.config and -remoteWrite.streamAggr.dedupInterval at the corresponding -remoteWrite.url. See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels + An optional list of labels to drop from samples before stream de-duplication and aggregation with -remoteWrite.streamAggr.config and -remoteWrite.streamAggr.dedupInterval at the corresponding -remoteWrite.url. See https://docs.victoriametrics.com/stream-aggregation/key-concepts#dropping-unneeded-labels Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -remoteWrite.streamAggr.ignoreFirstIntervals array - Number of aggregation intervals to skip after the start for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. Increase this value if you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving bufferred delayed data from clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start + Number of aggregation intervals to skip after the start for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. Increase this value if you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving bufferred delayed data from clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignore-aggregation-intervals-on-start -remoteWrite.streamAggr.ignoreOldSamples array - Whether to ignore input samples with old timestamps outside the current aggregation interval for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples + Whether to ignore input samples with old timestamps outside the current aggregation interval for the corresponding -remoteWrite.streamAggr.config at the corresponding -remoteWrite.url. See https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignoring-old-samples Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -remoteWrite.streamAggr.keepInput array @@ -2261,18 +2261,18 @@ See the docs at https://docs.victoriametrics.com/vmagent/ . -streamAggr.config string Optional path to file with stream aggregation config. See https://docs.victoriametrics.com/stream-aggregation/ . See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval -streamAggr.dedupInterval value - Input samples are de-duplicated with this interval on aggregator before optional aggregation with -streamAggr.config . See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication + Input samples are de-duplicated with this interval on aggregator before optional aggregation with -streamAggr.config . See also -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/key-concepts#deduplication The following optional suffixes are supported: s (second), m (minute), h (hour), d (day), w (week), y (year). If suffix isn't set, then the duration is counted in months (default 0s) -streamAggr.dropInput Whether to drop all the input samples after the aggregation with -remoteWrite.streamAggr.config. By default, only aggregates samples are dropped, while the remaining samples are written to remote storages write. See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation/ -streamAggr.dropInputLabels array - An optional list of labels to drop from samples for aggregator before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels + An optional list of labels to drop from samples for aggregator before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/key-concepts#dropping-unneeded-labels Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -streamAggr.ignoreFirstIntervals int - Number of aggregation intervals to skip after the start for aggregator. Increase this value if you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving unordered delayed data from clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start + Number of aggregation intervals to skip after the start for aggregator. Increase this value if you observe incorrect aggregation results after vmagent restarts. It could be caused by receiving unordered delayed data from clients pushing data into the vmagent. See https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignore-aggregation-intervals-on-start -streamAggr.ignoreOldSamples - Whether to ignore input samples with old timestamps outside the current aggregation interval for aggregator. See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples + Whether to ignore input samples with old timestamps outside the current aggregation interval for aggregator. See https://docs.victoriametrics.com/stream-aggregation/key-concepts#ignoring-old-samples -streamAggr.keepInput Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregates samples are dropped, while the remaining samples are written to remote storages write. See also -streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation/ -tls array