VictoriaMetrics/app/vminsert
Andrii Chubatiuk c8fc903669
lib/streamaggr: added aggregation windows ()
### Describe Your Changes

By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.

For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.

Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:

- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
  It allows enabling aggregation windows for a specific aggregator.



### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:19:33 +01:00
..
common lib/streamaggr: added aggregation windows () 2025-02-19 13:19:33 +01:00
csvimport app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
datadogsketches app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
datadogv1 app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
datadogv2 app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
graphite app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
influx app/vminsert: properly ingest influx metrics 2024-12-26 12:14:42 +01:00
native app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
newrelic app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
opentelemetry app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
opentsdb app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
opentsdbhttp app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
prometheusimport app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
prompush app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
promremotewrite app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
relabel refactoring: changed prompb to prompbmarshal everythere where internal series transformations are happening () 2024-11-26 12:45:17 +01:00
static/css all: follow-up after 8edb390e21 2022-06-07 00:57:09 +03:00
vmimport app/vminsert: properly apply relabeling at ingestion 2024-12-18 22:27:51 +01:00
main.go app/{vminsert,vmagent}: drop time series on exceeding labels limits. 2024-12-10 21:19:16 +01:00
README.md all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00

vminsert routes the ingested data to vmstorage.