diff --git a/app/vmalert/README.md b/app/vmalert/README.md index 52e2a6d0f1..3b65798e7b 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -390,6 +390,10 @@ Alertmanagers. To avoid recording rules results and alerts state duplication in VictoriaMetrics server don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). The recommended value for `-dedup.minScrapeInterval` must be greater or equal to vmalert's `evaluation_interval`. +If you observe inconsistent or "jumping" values in series produced by vmalert, try disabling `-datasource.queryTimeAlignment` +command line flag. Because of alignment, two or more vmalert HA pairs will produce results with the same timestamps. +But due of backfilling (data delivered to the datasource with some delay) values of such results may differ, +which would affect deduplication logic and result into "jumping" datapoints. Alertmanager will automatically deduplicate alerts with identical labels, so ensure that all `vmalert`s are having the same config. diff --git a/docs/vmalert.md b/docs/vmalert.md index 75166636d6..e11cb61b76 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -394,6 +394,10 @@ Alertmanagers. To avoid recording rules results and alerts state duplication in VictoriaMetrics server don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication). The recommended value for `-dedup.minScrapeInterval` must be greater or equal to vmalert's `evaluation_interval`. +If you observe inconsistent or "jumping" values in series produced by vmalert, try disabling `-datasource.queryTimeAlignment` +command line flag. Because of alignment, two or more vmalert HA pairs will produce results with the same timestamps. +But due of backfilling (data delivered to the datasource with some delay) values of such results may differ, +which would affect deduplication logic and result into "jumping" datapoints. Alertmanager will automatically deduplicate alerts with identical labels, so ensure that all `vmalert`s are having the same config.