github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

vmalert: clarify docs for state restore with additional details

Browse source 
The important change is to highlight that restore procedure happens
only once and only for already loaded rules. Config hot-reload
doesn't trigger the restore procedure.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
This commit is contained in:
hagen1778 2023-07-27 11:29:17 +02:00
parent 7fe81fe612
commit 03f04afeda
No known key found for this signature in database
GPG key ID: 3BF75F3741CA9640
2 changed files with 24 additions and 14 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

19
app/vmalert/README.md
View file

@ -357,19 +357,24 @@ For recording rules to work `-remoteWrite.url` must be specified.
### Alerts state on restarts ### Alerts state on restarts
`vmalert` has no local storage, so alerts state is stored in the process memory. Hence, after restart of `vmalert` `vmalert` is stateless, it holds alerts state in the process memory. Restarting of `vmalert` process
the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: will reset alerts state in memory. To prevent `vmalert` from losing alerts state it should be configured
to persist the state to the remote destination via the following flags:
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. to the configured address in the form of [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
These are regular time series and maybe queried from VM just as any other time series. `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
The state is stored to the configured address on every rule evaluation. These time series can be queried from VictoriaMetrics just as any other time series.
The state will be persisted to the configured address on each evaluation.
* `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
from configured address by querying time series with name `ALERTS_FOR_STATE`. from the configured address by querying time series with name `ALERTS_FOR_STATE`. The restore happens only once when
`vmalert` process starts, and only for the configured rules. Config [hot reload](#hot-config-reload) doesn't trigger
state restore.
Both flags are required for proper state restoration. Restore process may fail if time series are missing Both flags are required for proper state restoration. Restore process may fail if time series are missing
in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`)
or received state doesn't match current `vmalert` rules configuration. or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules
with `restored` label in [web UI](#WEB).
### Multitenancy ### Multitenancy

19
docs/vmalert.md
View file

@ -368,19 +368,24 @@ For recording rules to work `-remoteWrite.url` must be specified.
### Alerts state on restarts ### Alerts state on restarts
`vmalert` has no local storage, so alerts state is stored in the process memory. Hence, after restart of `vmalert` `vmalert` is stateless, it holds alerts state in the process memory. Restarting of `vmalert` process
the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags: will reset alerts state in memory. To prevent `vmalert` from losing alerts state it should be configured
to persist the state to the remote destination via the following flags:
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol. to the configured address in the form of [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
These are regular time series and maybe queried from VM just as any other time series. `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
The state is stored to the configured address on every rule evaluation. These time series can be queried from VictoriaMetrics just as any other time series.
The state will be persisted to the configured address on each evaluation.
* `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
from configured address by querying time series with name `ALERTS_FOR_STATE`. from the configured address by querying time series with name `ALERTS_FOR_STATE`. The restore happens only once when
`vmalert` process starts, and only for the configured rules. Config [hot reload](#hot-config-reload) doesn't trigger
state restore.
Both flags are required for proper state restoration. Restore process may fail if time series are missing Both flags are required for proper state restoration. Restore process may fail if time series are missing
in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`)
or received state doesn't match current `vmalert` rules configuration. or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules
with `restored` label in [web UI](#WEB).
### Multitenancy ### Multitenancy