From 03f04afedab4440181fc4804044d66a42c222f42 Mon Sep 17 00:00:00 2001 From: hagen1778 Date: Thu, 27 Jul 2023 11:29:17 +0200 Subject: [PATCH] vmalert: clarify docs for state restore with additional details The important change is to highlight that restore procedure happens only once and only for already loaded rules. Config hot-reload doesn't trigger the restore procedure. Signed-off-by: hagen1778 --- app/vmalert/README.md | 19 ++++++++++++------- docs/vmalert.md | 19 ++++++++++++------- 2 files changed, 24 insertions(+), 14 deletions(-) diff --git a/app/vmalert/README.md b/app/vmalert/README.md index 43332d03d..6e06c822b 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -357,19 +357,24 @@ For recording rules to work `-remoteWrite.url` must be specified. ### Alerts state on restarts -`vmalert` has no local storage, so alerts state is stored in the process memory. Hence, after restart of `vmalert` -the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: +`vmalert` is stateless, it holds alerts state in the process memory. Restarting of `vmalert` process +will reset alerts state in memory. To prevent `vmalert` from losing alerts state it should be configured +to persist the state to the remote destination via the following flags: * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state - into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. - These are regular time series and maybe queried from VM just as any other time series. - The state is stored to the configured address on every rule evaluation. + to the configured address in the form of [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series) + `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. + These time series can be queried from VictoriaMetrics just as any other time series. + The state will be persisted to the configured address on each evaluation. * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state - from configured address by querying time series with name `ALERTS_FOR_STATE`. + from the configured address by querying time series with name `ALERTS_FOR_STATE`. The restore happens only once when + `vmalert` process starts, and only for the configured rules. Config [hot reload](#hot-config-reload) doesn't trigger + state restore. Both flags are required for proper state restoration. Restore process may fail if time series are missing in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) -or received state doesn't match current `vmalert` rules configuration. +or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules +with `restored` label in [web UI](#WEB). ### Multitenancy diff --git a/docs/vmalert.md b/docs/vmalert.md index 8585af0e8..2e2704815 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -368,19 +368,24 @@ For recording rules to work `-remoteWrite.url` must be specified. ### Alerts state on restarts -`vmalert` has no local storage, so alerts state is stored in the process memory. Hence, after restart of `vmalert` -the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: +`vmalert` is stateless, it holds alerts state in the process memory. Restarting of `vmalert` process +will reset alerts state in memory. To prevent `vmalert` from losing alerts state it should be configured +to persist the state to the remote destination via the following flags: * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state - into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. - These are regular time series and maybe queried from VM just as any other time series. - The state is stored to the configured address on every rule evaluation. + to the configured address in the form of [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series) + `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. + These time series can be queried from VictoriaMetrics just as any other time series. + The state will be persisted to the configured address on each evaluation. * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state - from configured address by querying time series with name `ALERTS_FOR_STATE`. + from the configured address by querying time series with name `ALERTS_FOR_STATE`. The restore happens only once when + `vmalert` process starts, and only for the configured rules. Config [hot reload](#hot-config-reload) doesn't trigger + state restore. Both flags are required for proper state restoration. Restore process may fail if time series are missing in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) -or received state doesn't match current `vmalert` rules configuration. +or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules +with `restored` label in [web UI](#WEB). ### Multitenancy