mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
Vmalert docs upd (#1890)
* vmalert: add topology examples in docs * vmalert: docs typo fix
This commit is contained in:
Roman Khavronenko
GitHub
parent
2c41f25fb8
commit
866b6a842b
14 changed files with 2953 additions and 40 deletions
|
|
@ -2,7 +2,11 @@
|
|||
|
||||
`vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
|
||||
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
|
||||
rules against configured address. It is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
|
||||
rules against configured `-datasource.url`. For sending alerting notifications
|
||||
vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag.
|
||||
Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
|
||||
protocol and require `-remoteWrite.url` to be configured.
|
||||
Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
|
||||
implementation and aims to be compatible with its syntax.
|
||||
|
||||
## Features
|
||||
|
|
@ -18,12 +22,12 @@ implementation and aims to be compatible with its syntax.
|
|||
* Lightweight without extra dependencies.
|
||||
|
||||
## Limitations
|
||||
* `vmalert` execute queries against remote datasource which has reliability risks because of network.
|
||||
It is recommended to configure alerts thresholds and rules expressions with understanding that network request
|
||||
may fail;
|
||||
* `vmalert` execute queries against remote datasource which has reliability risks because of the network.
|
||||
It is recommended to configure alerts thresholds and rules expressions with the understanding that network
|
||||
requests may fail;
|
||||
* by default, rules execution is sequential within one group, but persistence of execution results to remote
|
||||
storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous
|
||||
recording rule is reused in next one;
|
||||
recording rule is reused in the next one;
|
||||
|
||||
## QuickStart
|
||||
|
||||
|
|
@ -33,13 +37,13 @@ git clone https://github.com/VictoriaMetrics/VictoriaMetrics
|
|||
cd VictoriaMetrics
|
||||
make vmalert
|
||||
```
|
||||
The build binary will be placed to `VictoriaMetrics/bin` folder.
|
||||
The build binary will be placed in `VictoriaMetrics/bin` folder.
|
||||
|
||||
To start using `vmalert` you will need the following things:
|
||||
* list of rules - PromQL/MetricsQL expressions to execute;
|
||||
* datasource address - reachable MetricsQL endpoint to run queries against;
|
||||
* notifier address [optional] - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing,
|
||||
aggregating alerts and sending notifications.
|
||||
aggregating alerts, and sending notifications.
|
||||
* remote write address [optional] - [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
|
||||
compatible storage to persist rules and alerts state info;
|
||||
* remote read address [optional] - MetricsQL compatible datasource to restore alerts state from.
|
||||
|
|
@ -58,7 +62,7 @@ Then configure `vmalert` accordingly:
|
|||
|
||||
Note there's a separate `remoteRead.url` to allow writing results of
|
||||
alerting/recording rules into a different storage than the initial data that's
|
||||
queried. This allows using `vmalert` to aggregate data from a short-term,
|
||||
queried. This allows using `vmalert` to aggregate data from a short-term,
|
||||
high-frequency, high-cardinality storage into a long-term storage with
|
||||
decreased cardinality and a bigger interval between samples.
|
||||
|
||||
|
|
@ -120,14 +124,14 @@ expression and then act according to the Rule type.
|
|||
|
||||
There are two types of Rules:
|
||||
* [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) -
|
||||
Alerting rules allow to define alert conditions via `expr` field and to send notifications to
|
||||
Alerting rules allow defining alert conditions via `expr` field and to send notifications to
|
||||
[Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty.
|
||||
* [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) -
|
||||
Recording rules allow to define `expr` which result will be then backfilled to configured
|
||||
Recording rules allow defining `expr` which result will be then backfilled to configured
|
||||
`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally
|
||||
expensive expressions and save their result as a new set of time series.
|
||||
|
||||
`vmalert` forbids defining duplicates - rules with the same combination of name, expression and labels
|
||||
`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels
|
||||
within one group.
|
||||
|
||||
#### Alerting rules
|
||||
|
|
@ -143,7 +147,7 @@ alert: <string>
|
|||
expr: <string>
|
||||
|
||||
# Alerts are considered firing once they have been returned for this long.
|
||||
# Alerts which have not yet fired for long enough are considered pending.
|
||||
# Alerts which have not yet been fired for long enough are considered pending.
|
||||
# If param is omitted or set to 0 then alerts will be immediately considered
|
||||
# as firing once they return.
|
||||
[ for: <duration> | default = 0s ]
|
||||
|
|
@ -188,19 +192,19 @@ For recording rules to work `-remoteWrite.url` must be specified.
|
|||
the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags:
|
||||
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
|
||||
into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
|
||||
These are regular time series and may be queried from VM just as any other time series.
|
||||
These are regular time series and maybe queried from VM just as any other time series.
|
||||
The state is stored to the configured address on every rule evaluation.
|
||||
* `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
|
||||
from configured address by querying time series with name `ALERTS_FOR_STATE`.
|
||||
|
||||
Both flags are required for proper state restoring. Restore process may fail if time series are missing
|
||||
Both flags are required for proper state restoration. Restore process may fail if time series are missing
|
||||
in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`)
|
||||
or received state doesn't match current `vmalert` rules configuration.
|
||||
|
||||
|
||||
### Multitenancy
|
||||
|
||||
The are the following approaches exist for alerting and recording rules across
|
||||
There are the following approaches exist for alerting and recording rules across
|
||||
[multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy):
|
||||
|
||||
* To run a separate `vmalert` instance per each tenant.
|
||||
|
|
@ -240,6 +244,106 @@ The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz`
|
|||
at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise`
|
||||
tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags).
|
||||
|
||||
### Topology examples
|
||||
|
||||
The following sections are showing how `vmalert` may be used and configured
|
||||
for different scenarios.
|
||||
|
||||
Please note, not all flags in examples are required:
|
||||
* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if
|
||||
you have recording rules or want to store [alerts state](#alerts-state-on-restarts) on `vmalert` restarts;
|
||||
* `-notifier.url` is optional and is needed only if you have alerting rules.
|
||||
|
||||
#### Single-node VictoriaMetrics
|
||||
|
||||
<img alt="vmalert single" src="vmalert_single.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
|
||||
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
|
||||
```
|
||||
|
||||
The simplest configuration where one single-node VM server is used for
|
||||
rules execution, storing recording rules results and alerts state.
|
||||
|
||||
#### Cluster VictoriaMetrics
|
||||
|
||||
<img alt="vmalert cluster" src="vmalert_cluster.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
|
||||
-remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
|
||||
```
|
||||
|
||||
In [cluster mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html)
|
||||
VictoriaMetrics has separate components for writing and reading path:
|
||||
`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions
|
||||
and `vminsert` is used to persist recording rules results and alerts state.
|
||||
Cluster mode could have multiple `vminsert` and `vmselect` components. In case when you want
|
||||
to spread the load on these components - add balancers before them and configure
|
||||
`vmalert` with balancer's addresses. Please, see more about VM's cluster architecture
|
||||
[here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview).
|
||||
|
||||
#### vmalert high availability
|
||||
|
||||
<img alt="vmalert ha" src="vmalert_ha.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
|
||||
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger
|
||||
-notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers
|
||||
```
|
||||
|
||||
For HA user can run multiple identically configured `vmalert` instances.
|
||||
It means all of them will execute the same rules, write state and results to
|
||||
the same destinations, and send alert notifications to multiple configured
|
||||
Alertmanagers.
|
||||
|
||||
To avoid recording rules results and alerts state duplication in VictoriaMetrics server
|
||||
don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).
|
||||
|
||||
Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
|
||||
all `vmalert`s are having the same config.
|
||||
|
||||
Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
for Alertmanagers for better reliability.
|
||||
|
||||
This example uses single-node VM server for the sake of simplicity.
|
||||
Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics) if needed.
|
||||
|
||||
#### Downsampling and aggregation via vmalert
|
||||
|
||||
<img alt="vmalert multi cluster" src="vmalert_multicluster.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions
|
||||
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results
|
||||
```
|
||||
|
||||
Example shows how to build a topology where `vmalert` will process data from one cluster
|
||||
and write results into another. Such clusters may be called as "hot" (low retention,
|
||||
high-speed disks, used for operative monitoring) and "cold" (long term retention,
|
||||
slower/cheaper disks, low resolution data). With help of `vmalert`, user can setup
|
||||
recording rules to process raw data from "hot" cluster (by applying additional transformations
|
||||
or reducing resolution) and push results to "cold" cluster.
|
||||
|
||||
Please note, [replay](#rules-backfilling) feature may be used for transforming historical data.
|
||||
|
||||
Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used.
|
||||
|
||||
### Web
|
||||
|
||||
|
|
@ -259,7 +363,7 @@ vmalert sends requests to `<-datasource.url>/render?format=json` during evaluati
|
|||
if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render`
|
||||
implements [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) for `format=json`.
|
||||
When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget
|
||||
to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on query type.
|
||||
to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on the query type.
|
||||
|
||||
## Rules backfilling
|
||||
|
||||
|
|
@ -318,11 +422,11 @@ to prevent cache pollution and unwanted time range boundaries adjustment during
|
|||
|
||||
#### Recording rules
|
||||
|
||||
Result of recording rules `replay` should match with results of normal rules evaluation.
|
||||
The result of recording rules `replay` should match with results of normal rules evaluation.
|
||||
|
||||
#### Alerting rules
|
||||
|
||||
Result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
|
||||
The result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
|
||||
To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression:
|
||||
```
|
||||
ALERTS{alertname="your_alertname", alertstate="firing"}
|
||||
|
|
@ -335,7 +439,7 @@ There are following non-required `replay` flags:
|
|||
|
||||
* `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request.
|
||||
In two words, it affects the max time range for every `/query_range` request. The higher the value,
|
||||
the less requests will be issued during `replay`.
|
||||
the fewer requests will be issued during `replay`.
|
||||
* `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries
|
||||
per rule before giving up.
|
||||
* `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining
|
||||
|
|
@ -357,7 +461,7 @@ See full description for these flags in `./vmalert --help`.
|
|||
We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported
|
||||
metrics may be analyzed later.
|
||||
|
||||
Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
|
||||
Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
|
||||
If you have suggestions for improvements or have found a bug - please open an issue on github or add
|
||||
a review to the dashboard.
|
||||
|
||||
|
|
|
|||
525
app/vmalert/vmalert_cluster.excalidraw
Normal file
525
app/vmalert/vmalert_cluster.excalidraw
Normal file
|
|
@ -0,0 +1,525 @@
|
|||
{
|
||||
"type": "excalidraw",
|
||||
"version": 2,
|
||||
"source": "https://excalidraw.com",
|
||||
"elements": [
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 794,
|
||||
"versionNonce": 1855937036,
|
||||
"isDeleted": false,
|
||||
"id": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 422.3502197265625,
|
||||
"y": 215.55953979492188,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 123.7601318359375,
|
||||
"height": 72.13211059570312,
|
||||
"seed": 1194011660,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"miEbzHxOPXe4PEYvXiJp5",
|
||||
"rcmiQfIWtfbTTlwxqr1sl",
|
||||
"P-dpWlSTtnsux-zr5oqgF",
|
||||
"oAToSPttH7aWoD_AqXGFX",
|
||||
"Bpy5by47XGKB4yS99ZkuA",
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348083348
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 659,
|
||||
"versionNonce": 247957684,
|
||||
"isDeleted": false,
|
||||
"id": "e9TDm09y-GhPm84XWt0Jv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 443.89678955078125,
|
||||
"y": 236.64378356933594,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 82,
|
||||
"height": 24,
|
||||
"seed": 327273100,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347948032,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalert",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "middle"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1670,
|
||||
"versionNonce": 2021681972,
|
||||
"isDeleted": false,
|
||||
"id": "dd52BjHfPMPRji9Tws7U-",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 750.3317260742188,
|
||||
"y": 226.5509033203125,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1779959692,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"Bpy5by47XGKB4yS99ZkuA"
|
||||
],
|
||||
"updated": 1638348054411
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 1311,
|
||||
"versionNonce": 1283453068,
|
||||
"isDeleted": false,
|
||||
"id": "9TEzv0sVCHAkc46ou0oNF",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 759.2862243652344,
|
||||
"y": 238.68240356445312,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 152,
|
||||
"height": 24,
|
||||
"seed": 1617178804,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348054411,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vminsert:8480",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 897,
|
||||
"versionNonce": 1983434892,
|
||||
"isDeleted": false,
|
||||
"id": "Sa4OBd1ZjD6itohm7Ll8z",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 61.744873046875,
|
||||
"y": 224.9600830078125,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 126267060,
|
||||
"groupIds": [
|
||||
"ek-pq3umtz1yN-J_-preq"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh"
|
||||
],
|
||||
"updated": 1638348077724
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 719,
|
||||
"versionNonce": 457402292,
|
||||
"isDeleted": false,
|
||||
"id": "we766A079lfGYu2_aC4Pl",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 70.69937133789062,
|
||||
"y": 237.33523559570312,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 152,
|
||||
"height": 24,
|
||||
"seed": 478660236,
|
||||
"groupIds": [
|
||||
"ek-pq3umtz1yN-J_-preq"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348077725,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmselect:8481",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1098,
|
||||
"versionNonce": 1480603788,
|
||||
"isDeleted": false,
|
||||
"id": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 371.7434387207031,
|
||||
"y": 398.50787353515625,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 240.10644531249997,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 99322124,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348083348
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 864,
|
||||
"versionNonce": 1115813900,
|
||||
"isDeleted": false,
|
||||
"id": "GUs816aggGqUSdoEsSmea",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 393.73809814453125,
|
||||
"y": 410.5976257324219,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 199,
|
||||
"height": 24,
|
||||
"seed": 1194745268,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348009180,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "alertmanager:9093",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 2405,
|
||||
"versionNonce": 959767732,
|
||||
"isDeleted": false,
|
||||
"id": "Bpy5by47XGKB4yS99ZkuA",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 556.6860961914062,
|
||||
"y": 252.2582773408825,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 184.9195556640625,
|
||||
"height": 1.6022679018915937,
|
||||
"seed": 357577356,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348054411,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": 0.0344528515859526,
|
||||
"gap": 10.57574462890625
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "dd52BjHfPMPRji9Tws7U-",
|
||||
"focus": -0.039393828258510157,
|
||||
"gap": 8.72607421875
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
184.9195556640625,
|
||||
-1.6022679018915937
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 1173,
|
||||
"versionNonce": 1248255756,
|
||||
"isDeleted": false,
|
||||
"id": "wRO0q9xKPHc8e8XPPsQWh",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 406.0439244722469,
|
||||
"y": 246.80533728741074,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 157.86774649373126,
|
||||
"height": 0.1417938392881979,
|
||||
"seed": 656189364,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348077725,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": 0.13736472619498497,
|
||||
"gap": 16.306295254315614
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "Sa4OBd1ZjD6itohm7Ll8z",
|
||||
"focus": -0.013200835330936087,
|
||||
"gap": 14.437713623046875
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-157.86774649373126,
|
||||
0.1417938392881979
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 557,
|
||||
"versionNonce": 995289780,
|
||||
"isDeleted": false,
|
||||
"id": "RbVSa4PnOgAMtzoKb-DhW",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 552.4987182617188,
|
||||
"y": 212.27996826171875,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 188,
|
||||
"height": 76,
|
||||
"seed": 1989838604,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348059525,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "persist alerts state\n\n\nand recording rules",
|
||||
"baseline": 72,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 803,
|
||||
"versionNonce": 1576507444,
|
||||
"isDeleted": false,
|
||||
"id": "ia2QzZNl_tuvfY3ymLjyJ",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 290.34130859375,
|
||||
"y": 210.56927490234375,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 122,
|
||||
"height": 19,
|
||||
"seed": 157304972,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh"
|
||||
],
|
||||
"updated": 1638347948032,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "execute rules",
|
||||
"baseline": 15,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 1471,
|
||||
"versionNonce": 1361321140,
|
||||
"isDeleted": false,
|
||||
"id": "sxEhnxlbT7ldlSsmHDUHp",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 484.18669893674246,
|
||||
"y": 302.3424013553929,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 1.0484739253853945,
|
||||
"height": 84.72775855671654,
|
||||
"seed": 1818348300,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348083348,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": 0.010768924644894236,
|
||||
"gap": 14.650750964767894
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"focus": -0.051051952959743775,
|
||||
"gap": 11.437713623046818
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
1.0484739253853945,
|
||||
84.72775855671654
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 576,
|
||||
"versionNonce": 2112088460,
|
||||
"isDeleted": false,
|
||||
"id": "E9Run6wCm2chQ6JHrmc_y",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 530.5612182617188,
|
||||
"y": 318.60687255859375,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 122,
|
||||
"height": 38,
|
||||
"seed": 1836541708,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348023735,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "send alert \nnotifications",
|
||||
"baseline": 34,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 480,
|
||||
"versionNonce": 1835119500,
|
||||
"isDeleted": false,
|
||||
"id": "ff5OkfgmkKLifS13_TFj3",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 291.37474060058594,
|
||||
"y": 261.4861297607422,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 122,
|
||||
"height": 19,
|
||||
"seed": 264004620,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh"
|
||||
],
|
||||
"updated": 1638347948032,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "restore state",
|
||||
"baseline": 15,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
}
|
||||
],
|
||||
"appState": {
|
||||
"gridSize": null,
|
||||
"viewBackgroundColor": "#ffffff"
|
||||
},
|
||||
"files": {}
|
||||
}
|
||||
BIN
app/vmalert/vmalert_cluster.png
Normal file
BIN
app/vmalert/vmalert_cluster.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
911
app/vmalert/vmalert_ha.excalidraw
Normal file
911
app/vmalert/vmalert_ha.excalidraw
Normal file
|
|
@ -0,0 +1,911 @@
|
|||
{
|
||||
"type": "excalidraw",
|
||||
"version": 2,
|
||||
"source": "https://excalidraw.com",
|
||||
"elements": [
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 906,
|
||||
"versionNonce": 448716468,
|
||||
"isDeleted": false,
|
||||
"id": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 302.2676696777344,
|
||||
"y": 275.59356689453125,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 123.7601318359375,
|
||||
"height": 72.13211059570312,
|
||||
"seed": 1194011660,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"miEbzHxOPXe4PEYvXiJp5",
|
||||
"rcmiQfIWtfbTTlwxqr1sl",
|
||||
"P-dpWlSTtnsux-zr5oqgF",
|
||||
"oAToSPttH7aWoD_AqXGFX",
|
||||
"Bpy5by47XGKB4yS99ZkuA",
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"sxEhnxlbT7ldlSsmHDUHp",
|
||||
"m9_BptFOFxbV2sS_xJDu2",
|
||||
"fsGFp4NW4JlrCdF0HR3uA",
|
||||
"OTKoeHmKtqCxFArDbY-sP"
|
||||
],
|
||||
"updated": 1638355221749
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 762,
|
||||
"versionNonce": 223660724,
|
||||
"isDeleted": false,
|
||||
"id": "e9TDm09y-GhPm84XWt0Jv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 318.0538024902344,
|
||||
"y": 296.6778106689453,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 327273100,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"m9_BptFOFxbV2sS_xJDu2"
|
||||
],
|
||||
"updated": 1638355102070,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalert1",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 2067,
|
||||
"versionNonce": 817347852,
|
||||
"isDeleted": false,
|
||||
"id": "dd52BjHfPMPRji9Tws7U-",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 329.6426086425781,
|
||||
"y": 90.3275146484375,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 269.336669921875,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1779959692,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"Bpy5by47XGKB4yS99ZkuA",
|
||||
"m9_BptFOFxbV2sS_xJDu2",
|
||||
"S_dOHQrhGmu8SFJzobJK7"
|
||||
],
|
||||
"updated": 1638355058507
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 1717,
|
||||
"versionNonce": 2040797452,
|
||||
"isDeleted": false,
|
||||
"id": "9TEzv0sVCHAkc46ou0oNF",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 349.01177978515625,
|
||||
"y": 102.45901489257812,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 234,
|
||||
"height": 24,
|
||||
"seed": 1617178804,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"m9_BptFOFxbV2sS_xJDu2",
|
||||
"S_dOHQrhGmu8SFJzobJK7"
|
||||
],
|
||||
"updated": 1638355040938,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "victoriametrics:8428",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1553,
|
||||
"versionNonce": 39381044,
|
||||
"isDeleted": false,
|
||||
"id": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 173.67257690429688,
|
||||
"y": 470.5100402832031,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 240.10644531249997,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 99322124,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp",
|
||||
"OTKoeHmKtqCxFArDbY-sP",
|
||||
"XhPgLRBk-YhWAFcSQi9TJ",
|
||||
"D6fkQH1E_MFbCuL697ArO"
|
||||
],
|
||||
"updated": 1638355221749
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 1309,
|
||||
"versionNonce": 1112872844,
|
||||
"isDeleted": false,
|
||||
"id": "GUs816aggGqUSdoEsSmea",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 189.667236328125,
|
||||
"y": 482.59979248046875,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 211,
|
||||
"height": 24,
|
||||
"seed": 1194745268,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355040938,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "alertmanager1:9093",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 835,
|
||||
"versionNonce": 2036483596,
|
||||
"isDeleted": false,
|
||||
"id": "RbVSa4PnOgAMtzoKb-DhW",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 525.83837890625,
|
||||
"y": 147.33470153808594,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 188,
|
||||
"height": 95,
|
||||
"seed": 1989838604,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355040939,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n",
|
||||
"baseline": 91,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 777,
|
||||
"versionNonce": 990468492,
|
||||
"isDeleted": false,
|
||||
"id": "E9Run6wCm2chQ6JHrmc_y",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 617.0690307617188,
|
||||
"y": 376.9822692871094,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 122,
|
||||
"height": 38,
|
||||
"seed": 1836541708,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638355227582,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "send alert \nnotifications",
|
||||
"baseline": 34,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1112,
|
||||
"versionNonce": 999136652,
|
||||
"isDeleted": false,
|
||||
"id": "mIu-d0lmShCxzMLD5iA_p",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 492.29505920410156,
|
||||
"y": 272.3052215576172,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 123.7601318359375,
|
||||
"height": 72.13211059570312,
|
||||
"seed": 756416780,
|
||||
"groupIds": [
|
||||
"jbBot-UNdMoWy2jPXC1u5"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"miEbzHxOPXe4PEYvXiJp5",
|
||||
"rcmiQfIWtfbTTlwxqr1sl",
|
||||
"P-dpWlSTtnsux-zr5oqgF",
|
||||
"oAToSPttH7aWoD_AqXGFX",
|
||||
"Bpy5by47XGKB4yS99ZkuA",
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"sxEhnxlbT7ldlSsmHDUHp",
|
||||
"S_dOHQrhGmu8SFJzobJK7",
|
||||
"XhPgLRBk-YhWAFcSQi9TJ",
|
||||
"Ar-hcDLlzVSoTPs2MaywO"
|
||||
],
|
||||
"updated": 1638355153683
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 970,
|
||||
"versionNonce": 430760500,
|
||||
"isDeleted": false,
|
||||
"id": "ZqIR6SaLNDQl8s0zZbdnE",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 508.08119201660156,
|
||||
"y": 293.38946533203125,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 1477404084,
|
||||
"groupIds": [
|
||||
"jbBot-UNdMoWy2jPXC1u5"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355105020,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalertN",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"id": "qZFezRGU4Chwxgvb2451t",
|
||||
"type": "line",
|
||||
"x": 449.93653869628906,
|
||||
"y": 306.30010986328125,
|
||||
"width": 19.48321533203125,
|
||||
"height": 0.3865966796875,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 1833426828,
|
||||
"version": 69,
|
||||
"versionNonce": 871948300,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355040939,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
19.48321533203125,
|
||||
0.3865966796875
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": null,
|
||||
"endBinding": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": null
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1649,
|
||||
"versionNonce": 93981708,
|
||||
"isDeleted": false,
|
||||
"id": "gNHiZJKo0ap69ALDobtZ-",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 518.5596466064453,
|
||||
"y": 471.4539489746094,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 240.10644531249997,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 454422412,
|
||||
"groupIds": [
|
||||
"fEAIeQ0DxLnI_rPlKPZqW"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp",
|
||||
"fsGFp4NW4JlrCdF0HR3uA",
|
||||
"Ar-hcDLlzVSoTPs2MaywO",
|
||||
"D6fkQH1E_MFbCuL697ArO"
|
||||
],
|
||||
"updated": 1638355153683
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 1412,
|
||||
"versionNonce": 75038348,
|
||||
"isDeleted": false,
|
||||
"id": "O-zgjZBvt4RI1PrkBNQnb",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 534.3148040771484,
|
||||
"y": 483.543701171875,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 211,
|
||||
"height": 24,
|
||||
"seed": 1026268980,
|
||||
"groupIds": [
|
||||
"fEAIeQ0DxLnI_rPlKPZqW"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355040939,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "alertmanagerN:9093",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"id": "m9_BptFOFxbV2sS_xJDu2",
|
||||
"type": "arrow",
|
||||
"x": 378.57185562792466,
|
||||
"y": 262.1596984863281,
|
||||
"width": 79.96146539019026,
|
||||
"height": 111.53411865234375,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 141224884,
|
||||
"version": 521,
|
||||
"versionNonce": 283254540,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355102070,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
79.96146539019026,
|
||||
-111.53411865234375
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": -0.23996018586441184,
|
||||
"gap": 13.433868408203125
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "dd52BjHfPMPRji9Tws7U-",
|
||||
"focus": -0.14207100087207922,
|
||||
"gap": 15.550811767578125
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "S_dOHQrhGmu8SFJzobJK7",
|
||||
"type": "arrow",
|
||||
"x": 558.1990515577129,
|
||||
"y": 263.4271240234375,
|
||||
"width": 88.18940317574186,
|
||||
"height": 114.15780639648438,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 1004789812,
|
||||
"version": 314,
|
||||
"versionNonce": 1259171724,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355105019,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-88.18940317574186,
|
||||
-114.15780639648438
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "mIu-d0lmShCxzMLD5iA_p",
|
||||
"focus": 0.4315094049079832,
|
||||
"gap": 8.878097534179688
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "dd52BjHfPMPRji9Tws7U-",
|
||||
"focus": 0.14840834302306677,
|
||||
"gap": 14.194549560546875
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "fsGFp4NW4JlrCdF0HR3uA",
|
||||
"type": "arrow",
|
||||
"x": 356.6613630516658,
|
||||
"y": 354.95416259765625,
|
||||
"width": 278.74351860741064,
|
||||
"height": 106.90020751953125,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 712384692,
|
||||
"version": 334,
|
||||
"versionNonce": 266493324,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355102070,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
278.74351860741064,
|
||||
106.90020751953125
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": 0.7721210277890177,
|
||||
"gap": 7.228485107421875
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "gNHiZJKo0ap69ALDobtZ-",
|
||||
"focus": 0.4493598001028791,
|
||||
"gap": 9.599578857421875
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "OTKoeHmKtqCxFArDbY-sP",
|
||||
"type": "arrow",
|
||||
"x": 361.02563221226757,
|
||||
"y": 356.2252197265625,
|
||||
"width": 95.3594006000182,
|
||||
"height": 105.52130126953125,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 1992252596,
|
||||
"version": 466,
|
||||
"versionNonce": 472553100,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355221749,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-95.3594006000182,
|
||||
105.52130126953125
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": -0.39325294425055324,
|
||||
"gap": 8.499542236328125
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"focus": -0.400636314282374,
|
||||
"gap": 8.763519287109375
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "XhPgLRBk-YhWAFcSQi9TJ",
|
||||
"type": "arrow",
|
||||
"x": 561.7189961327852,
|
||||
"y": 349.238037109375,
|
||||
"width": 266.271510370216,
|
||||
"height": 112.383544921875,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 588869260,
|
||||
"version": 453,
|
||||
"versionNonce": 584723212,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355150131,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-266.271510370216,
|
||||
112.383544921875
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "mIu-d0lmShCxzMLD5iA_p",
|
||||
"focus": -0.7084007026732048,
|
||||
"gap": 4.8007049560546875
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"focus": -0.4180430111580123,
|
||||
"gap": 8.888458251953125
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "Ar-hcDLlzVSoTPs2MaywO",
|
||||
"type": "arrow",
|
||||
"x": 557.5999880535809,
|
||||
"y": 352.71795654296875,
|
||||
"width": 112.87813113656136,
|
||||
"height": 106.3255615234375,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 1508782476,
|
||||
"version": 331,
|
||||
"versionNonce": 1137561908,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355153683,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
112.87813113656136,
|
||||
106.3255615234375
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "mIu-d0lmShCxzMLD5iA_p",
|
||||
"focus": 0.4358123206211808,
|
||||
"gap": 8.280624389648438
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "gNHiZJKo0ap69ALDobtZ-",
|
||||
"focus": 0.4783744286829653,
|
||||
"gap": 12.410430908203125
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "WqHnv9_g3SdkLZuy4WSHa",
|
||||
"type": "text",
|
||||
"x": 186.5891876220703,
|
||||
"y": 523.145263671875,
|
||||
"width": 272,
|
||||
"height": 19,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"seed": 1310147468,
|
||||
"version": 281,
|
||||
"versionNonce": 949157044,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355040939,
|
||||
"text": "Alertmanagers in cluster mode",
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top",
|
||||
"baseline": 15
|
||||
},
|
||||
{
|
||||
"id": "yA0kgvdF71wZbHJ7Cg2p8",
|
||||
"type": "rectangle",
|
||||
"x": 159.41685485839844,
|
||||
"y": 440.7666015625,
|
||||
"width": 625.9590148925781,
|
||||
"height": 110.13494873046878,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"seed": 1465986996,
|
||||
"version": 366,
|
||||
"versionNonce": 196040844,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355040939
|
||||
},
|
||||
{
|
||||
"id": "D6fkQH1E_MFbCuL697ArO",
|
||||
"type": "arrow",
|
||||
"x": 422.4853057861328,
|
||||
"y": 494.8779132311229,
|
||||
"width": 90.5517578125,
|
||||
"height": 0.12427004064892344,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 2025208204,
|
||||
"version": 316,
|
||||
"versionNonce": 2144102156,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638355040939,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
90.5517578125,
|
||||
0.12427004064892344
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"focus": 0.08064204025505255,
|
||||
"gap": 8.706283569335938
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "gNHiZJKo0ap69ALDobtZ-",
|
||||
"focus": -0.05976220061952895,
|
||||
"gap": 5.5225830078125
|
||||
},
|
||||
"startArrowhead": "arrow",
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 463,
|
||||
"versionNonce": 2118914484,
|
||||
"isDeleted": false,
|
||||
"id": "p4EDvSKPqZzfM_SReybeq",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 274.6086883544922,
|
||||
"y": 56.77886962890625,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 394,
|
||||
"height": 19,
|
||||
"seed": 1214462348,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355074872,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "VictoriaMetrics with deduplication enabled",
|
||||
"baseline": 15,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 712,
|
||||
"versionNonce": 737833612,
|
||||
"isDeleted": false,
|
||||
"id": "YfNFeOucMo8BJk2P2JRex",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 229.77923583984375,
|
||||
"y": 227.84378051757812,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 448.5542297363281,
|
||||
"height": 134.06329345703122,
|
||||
"seed": 932207756,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355133244
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 427,
|
||||
"versionNonce": 1458527796,
|
||||
"isDeleted": false,
|
||||
"id": "oldM7Q_aJtpWd2jXQ0iKf",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 238.9442901611328,
|
||||
"y": 230.862548828125,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 225,
|
||||
"height": 38,
|
||||
"seed": 2131899572,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638355259358,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalerts with identical \nconfigurations",
|
||||
"baseline": 34,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
}
|
||||
],
|
||||
"appState": {
|
||||
"gridSize": null,
|
||||
"viewBackgroundColor": "#ffffff"
|
||||
},
|
||||
"files": {}
|
||||
}
|
||||
BIN
app/vmalert/vmalert_ha.png
Normal file
BIN
app/vmalert/vmalert_ha.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
915
app/vmalert/vmalert_multicluster.excalidraw
Normal file
915
app/vmalert/vmalert_multicluster.excalidraw
Normal file
|
|
@ -0,0 +1,915 @@
|
|||
{
|
||||
"type": "excalidraw",
|
||||
"version": 2,
|
||||
"source": "https://excalidraw.com",
|
||||
"elements": [
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 791,
|
||||
"versionNonce": 48874036,
|
||||
"isDeleted": false,
|
||||
"id": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 289.6802978515625,
|
||||
"y": 399.3895568847656,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 123.7601318359375,
|
||||
"height": 72.13211059570312,
|
||||
"seed": 1194011660,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"miEbzHxOPXe4PEYvXiJp5",
|
||||
"rcmiQfIWtfbTTlwxqr1sl",
|
||||
"P-dpWlSTtnsux-zr5oqgF",
|
||||
"oAToSPttH7aWoD_AqXGFX",
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"sxEhnxlbT7ldlSsmHDUHp",
|
||||
"pD9DcILMxa6GaR1U5YyMO",
|
||||
"HPEwr85wL4IedW0AgdArp",
|
||||
"EyecK0YM9Cc8T6ju-nTOc"
|
||||
],
|
||||
"updated": 1638347812431
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 658,
|
||||
"versionNonce": 1653816076,
|
||||
"isDeleted": false,
|
||||
"id": "e9TDm09y-GhPm84XWt0Jv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 311.22686767578125,
|
||||
"y": 420.4738006591797,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 82,
|
||||
"height": 24,
|
||||
"seed": 327273100,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347796775,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalert",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "middle"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 802,
|
||||
"versionNonce": 995326644,
|
||||
"isDeleted": false,
|
||||
"id": "Sa4OBd1ZjD6itohm7Ll8z",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 603.05322265625,
|
||||
"y": 228.65371704101562,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 126267060,
|
||||
"groupIds": [
|
||||
"ek-pq3umtz1yN-J_-preq"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"he-SpFjCxEQEWpWny2kKP",
|
||||
"-pjrKo16rOsasM8viZPJ-",
|
||||
"MGdu6GDIPNBAaEUr0Gt-a"
|
||||
],
|
||||
"updated": 1638347737174
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 624,
|
||||
"versionNonce": 707755700,
|
||||
"isDeleted": false,
|
||||
"id": "we766A079lfGYu2_aC4Pl",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 640.7635803222656,
|
||||
"y": 241.02886962890625,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 478660236,
|
||||
"groupIds": [
|
||||
"ek-pq3umtz1yN-J_-preq"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347701075,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmselect",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 802,
|
||||
"versionNonce": 1974403340,
|
||||
"isDeleted": false,
|
||||
"id": "ia2QzZNl_tuvfY3ymLjyJ",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 401.241943359375,
|
||||
"y": 342.4627990722656,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 178,
|
||||
"height": 38,
|
||||
"seed": 157304972,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh"
|
||||
],
|
||||
"updated": 1638347863144,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "execute aggregating\nrecording rules",
|
||||
"baseline": 34,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"id": "jrFZeFNsxlss7IsEZpA-J",
|
||||
"type": "rectangle",
|
||||
"x": 594.1509857177734,
|
||||
"y": 192.09194946289062,
|
||||
"width": 397.1286010742188,
|
||||
"height": 209.62742614746097,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"seed": 48379060,
|
||||
"version": 665,
|
||||
"versionNonce": 617456652,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": [
|
||||
"pD9DcILMxa6GaR1U5YyMO"
|
||||
],
|
||||
"updated": 1638347763716
|
||||
},
|
||||
{
|
||||
"id": "6ibhLp94HJFIdfP5HCEv8",
|
||||
"type": "text",
|
||||
"x": 609.7205657958984,
|
||||
"y": 199.81723022460938,
|
||||
"width": 225,
|
||||
"height": 19,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"seed": 1053164852,
|
||||
"version": 256,
|
||||
"versionNonce": 538595124,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638347761092,
|
||||
"text": "VM cluster with raw data",
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top",
|
||||
"baseline": 15
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 914,
|
||||
"versionNonce": 1576666164,
|
||||
"isDeleted": false,
|
||||
"id": "R5v-yZCVJ97BkJqr0Qb7H",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 807.5664215087891,
|
||||
"y": 287.8628387451172,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1794212620,
|
||||
"groupIds": [
|
||||
"BJNOAY1MY3Evr9B3qQtHf"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"he-SpFjCxEQEWpWny2kKP",
|
||||
"-pjrKo16rOsasM8viZPJ-",
|
||||
"bZXA8PH9gYu-clotqJ4f7",
|
||||
"MGdu6GDIPNBAaEUr0Gt-a"
|
||||
],
|
||||
"updated": 1638347737174
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 725,
|
||||
"versionNonce": 267455500,
|
||||
"isDeleted": false,
|
||||
"id": "pWRC_smX7TuOI8_8UrA4H",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 840.0209197998047,
|
||||
"y": 300.2379913330078,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 105,
|
||||
"height": 24,
|
||||
"seed": 421856180,
|
||||
"groupIds": [
|
||||
"BJNOAY1MY3Evr9B3qQtHf"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347701076,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmstorage",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 843,
|
||||
"versionNonce": 1244193972,
|
||||
"isDeleted": false,
|
||||
"id": "EqROOfYulSPsZm7ovxfQN",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 604.9091339111328,
|
||||
"y": 345.2847442626953,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 2043521972,
|
||||
"groupIds": [
|
||||
"ls6uq-W9bbVBM_UxAuyba"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"bZXA8PH9gYu-clotqJ4f7"
|
||||
],
|
||||
"updated": 1638347718537
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 676,
|
||||
"versionNonce": 143370892,
|
||||
"isDeleted": false,
|
||||
"id": "ddQH1nnmT7HbKW7Xmv4zx",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 642.6199798583984,
|
||||
"y": 357.90354919433594,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 335223180,
|
||||
"groupIds": [
|
||||
"ls6uq-W9bbVBM_UxAuyba"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"bZXA8PH9gYu-clotqJ4f7"
|
||||
],
|
||||
"updated": 1638347701076,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vminsert",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"id": "bZXA8PH9gYu-clotqJ4f7",
|
||||
"type": "arrow",
|
||||
"x": 785.2667708463345,
|
||||
"y": 367.2342882272049,
|
||||
"width": 99.87819315552736,
|
||||
"height": 24.681162491468,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 1153029388,
|
||||
"version": 420,
|
||||
"versionNonce": 1726025228,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638347704644,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
99.87819315552736,
|
||||
-24.681162491468
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "EqROOfYulSPsZm7ovxfQN",
|
||||
"focus": 0.5247890891198189,
|
||||
"gap": 8.36404562660789
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "R5v-yZCVJ97BkJqr0Qb7H",
|
||||
"focus": -0.6931056940383433,
|
||||
"gap": 9.943033572650961
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 528,
|
||||
"versionNonce": 1908259468,
|
||||
"isDeleted": false,
|
||||
"id": "MGdu6GDIPNBAaEUr0Gt-a",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 781.4456641707259,
|
||||
"y": 248.777857603323,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 104.0940537386266,
|
||||
"height": 27.60697400233846,
|
||||
"seed": 1695061516,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347737174,
|
||||
"startBinding": {
|
||||
"elementId": "Sa4OBd1ZjD6itohm7Ll8z",
|
||||
"focus": -0.5921495247360469,
|
||||
"gap": 6.398850205882127
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "R5v-yZCVJ97BkJqr0Qb7H",
|
||||
"focus": 0.7021471697457312,
|
||||
"gap": 11.478007139455713
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
104.0940537386266,
|
||||
27.60697400233846
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 883,
|
||||
"versionNonce": 845352076,
|
||||
"isDeleted": false,
|
||||
"id": "4pW8hvBu3bo1eMtFvg_gS",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 601.6520690917969,
|
||||
"y": 473.7677536010742,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1678097076,
|
||||
"groupIds": [
|
||||
"3kSpFrIN3kg4jwjDKpNWw"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"he-SpFjCxEQEWpWny2kKP",
|
||||
"-pjrKo16rOsasM8viZPJ-",
|
||||
"5W90eDBjtfZvkSuQTG0Iw"
|
||||
],
|
||||
"updated": 1638347777173
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 703,
|
||||
"versionNonce": 519371956,
|
||||
"isDeleted": false,
|
||||
"id": "w_i2hO06oLa0bWbyAfFzU",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 639.3624267578125,
|
||||
"y": 486.14290618896484,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 340753036,
|
||||
"groupIds": [
|
||||
"3kSpFrIN3kg4jwjDKpNWw"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347776748,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmselect",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 745,
|
||||
"versionNonce": 879581324,
|
||||
"isDeleted": false,
|
||||
"id": "U5U-67wL5fPwvzlxOAF5A",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 592.7498321533203,
|
||||
"y": 437.2059860229492,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 397.1286010742188,
|
||||
"height": 209.62742614746097,
|
||||
"seed": 912540724,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"pD9DcILMxa6GaR1U5YyMO"
|
||||
],
|
||||
"updated": 1638347776748
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 345,
|
||||
"versionNonce": 1628116148,
|
||||
"isDeleted": false,
|
||||
"id": "0IGVrvICMVeZp2RCaqTeP",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "dotted",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 608.3194122314453,
|
||||
"y": 444.93126678466797,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 291,
|
||||
"height": 19,
|
||||
"seed": 68700428,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347784373,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "VM cluster with aggregated data",
|
||||
"baseline": 15,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 996,
|
||||
"versionNonce": 2112461580,
|
||||
"isDeleted": false,
|
||||
"id": "9QMf0HXPE1W4M9S-zMJVO",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 806.1652679443359,
|
||||
"y": 532.9768753051758,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 523607476,
|
||||
"groupIds": [
|
||||
"Eb2kWfz3ZWBu8cNul0h_c"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"he-SpFjCxEQEWpWny2kKP",
|
||||
"-pjrKo16rOsasM8viZPJ-",
|
||||
"9WBH34em4CdU2OwzqYIl5",
|
||||
"5W90eDBjtfZvkSuQTG0Iw"
|
||||
],
|
||||
"updated": 1638347777173
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 804,
|
||||
"versionNonce": 1660832052,
|
||||
"isDeleted": false,
|
||||
"id": "WOrOD5vn6EtMUQRJomt3-",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 838.6197662353516,
|
||||
"y": 545.3520278930664,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 105,
|
||||
"height": 24,
|
||||
"seed": 1321420684,
|
||||
"groupIds": [
|
||||
"Eb2kWfz3ZWBu8cNul0h_c"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347776748,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmstorage",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 925,
|
||||
"versionNonce": 2052807604,
|
||||
"isDeleted": false,
|
||||
"id": "4dtJZpXEUxSK3biwFG7vd",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 603.5079803466797,
|
||||
"y": 590.3987808227539,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 171.99359130859375,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1738524468,
|
||||
"groupIds": [
|
||||
"punXEDFtHkSpcd9seAkZj"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"9WBH34em4CdU2OwzqYIl5",
|
||||
"EyecK0YM9Cc8T6ju-nTOc"
|
||||
],
|
||||
"updated": 1638347812431
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 756,
|
||||
"versionNonce": 2040967820,
|
||||
"isDeleted": false,
|
||||
"id": "rAbyooo-X08-86qjoK0WR",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 641.2188262939453,
|
||||
"y": 603.0175857543945,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 94,
|
||||
"height": 24,
|
||||
"seed": 948598284,
|
||||
"groupIds": [
|
||||
"punXEDFtHkSpcd9seAkZj"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"9WBH34em4CdU2OwzqYIl5"
|
||||
],
|
||||
"updated": 1638347776748,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vminsert",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 660,
|
||||
"versionNonce": 1560678196,
|
||||
"isDeleted": false,
|
||||
"id": "9WBH34em4CdU2OwzqYIl5",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 783.8656172818814,
|
||||
"y": 612.3483247872634,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 99.87819315552736,
|
||||
"height": 24.681162491468,
|
||||
"seed": 1141424308,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347777173,
|
||||
"startBinding": {
|
||||
"elementId": "4dtJZpXEUxSK3biwFG7vd",
|
||||
"focus": 0.5247890891198189,
|
||||
"gap": 8.364045626608004
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "9QMf0HXPE1W4M9S-zMJVO",
|
||||
"focus": -0.6931056940383404,
|
||||
"gap": 9.943033572650847
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
99.87819315552736,
|
||||
-24.681162491468
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 768,
|
||||
"versionNonce": 1653264948,
|
||||
"isDeleted": false,
|
||||
"id": "5W90eDBjtfZvkSuQTG0Iw",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 780.0445106062728,
|
||||
"y": 493.8918941633816,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 104.0940537386266,
|
||||
"height": 27.60697400233846,
|
||||
"seed": 1852298380,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638347777173,
|
||||
"startBinding": {
|
||||
"elementId": "4pW8hvBu3bo1eMtFvg_gS",
|
||||
"focus": -0.5921495247360469,
|
||||
"gap": 6.398850205882127
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "9QMf0HXPE1W4M9S-zMJVO",
|
||||
"focus": 0.7021471697457312,
|
||||
"gap": 11.478007139455713
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
104.0940537386266,
|
||||
27.60697400233846
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "HPEwr85wL4IedW0AgdArp",
|
||||
"type": "arrow",
|
||||
"x": 423.70701599121094,
|
||||
"y": 428.34434509277344,
|
||||
"width": 179.54901123046875,
|
||||
"height": 182.9937744140625,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 389863732,
|
||||
"version": 47,
|
||||
"versionNonce": 1364399028,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638347805500,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
179.54901123046875,
|
||||
-182.9937744140625
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": 0.6700023593531782,
|
||||
"gap": 10.266586303710938
|
||||
},
|
||||
"endBinding": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"id": "EyecK0YM9Cc8T6ju-nTOc",
|
||||
"type": "arrow",
|
||||
"x": 424.7585906982422,
|
||||
"y": 441.12806701660156,
|
||||
"width": 174.29449462890625,
|
||||
"height": 170.56607055664062,
|
||||
"angle": 0,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"seed": 981082124,
|
||||
"version": 92,
|
||||
"versionNonce": 1261546252,
|
||||
"isDeleted": false,
|
||||
"boundElementIds": null,
|
||||
"updated": 1638347812431,
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
174.29449462890625,
|
||||
170.56607055664062
|
||||
]
|
||||
],
|
||||
"lastCommittedPoint": null,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"focus": -0.6826568395144794,
|
||||
"gap": 11.318161010742188
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "4dtJZpXEUxSK3biwFG7vd",
|
||||
"focus": -0.8207814548026305,
|
||||
"gap": 4.45489501953125
|
||||
},
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow"
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 775,
|
||||
"versionNonce": 95530764,
|
||||
"isDeleted": false,
|
||||
"id": "o-yIJ_WZzVubhbVODvhcC",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 407.27012634277344,
|
||||
"y": 489.33091735839844,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 141,
|
||||
"height": 19,
|
||||
"seed": 775116812,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"wRO0q9xKPHc8e8XPPsQWh"
|
||||
],
|
||||
"updated": 1638347824876,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "persist results",
|
||||
"baseline": 15,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
}
|
||||
],
|
||||
"appState": {
|
||||
"gridSize": null,
|
||||
"viewBackgroundColor": "#ffffff"
|
||||
},
|
||||
"files": {}
|
||||
}
|
||||
BIN
app/vmalert/vmalert_multicluster.png
Normal file
BIN
app/vmalert/vmalert_multicluster.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
354
app/vmalert/vmalert_single.excalidraw
Normal file
354
app/vmalert/vmalert_single.excalidraw
Normal file
|
|
@ -0,0 +1,354 @@
|
|||
{
|
||||
"type": "excalidraw",
|
||||
"version": 2,
|
||||
"source": "https://excalidraw.com",
|
||||
"elements": [
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 795,
|
||||
"versionNonce": 1195460364,
|
||||
"isDeleted": false,
|
||||
"id": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 422.3502197265625,
|
||||
"y": 215.55953979492188,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 123.7601318359375,
|
||||
"height": 72.13211059570312,
|
||||
"seed": 1194011660,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"miEbzHxOPXe4PEYvXiJp5",
|
||||
"rcmiQfIWtfbTTlwxqr1sl",
|
||||
"P-dpWlSTtnsux-zr5oqgF",
|
||||
"oAToSPttH7aWoD_AqXGFX",
|
||||
"Bpy5by47XGKB4yS99ZkuA",
|
||||
"wRO0q9xKPHc8e8XPPsQWh",
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348116633
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 660,
|
||||
"versionNonce": 1034125236,
|
||||
"isDeleted": false,
|
||||
"id": "e9TDm09y-GhPm84XWt0Jv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 443.89678955078125,
|
||||
"y": 236.64378356933594,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 82,
|
||||
"height": 24,
|
||||
"seed": 327273100,
|
||||
"groupIds": [
|
||||
"iBaXgbpyifSwPplm_GO5b"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348116633,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "vmalert",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "middle"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1690,
|
||||
"versionNonce": 236788620,
|
||||
"isDeleted": false,
|
||||
"id": "dd52BjHfPMPRji9Tws7U-",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 344.9837951660156,
|
||||
"y": 64.64248657226562,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 269.336669921875,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 1779959692,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"Bpy5by47XGKB4yS99ZkuA"
|
||||
],
|
||||
"updated": 1638348176044
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 1331,
|
||||
"versionNonce": 945335476,
|
||||
"isDeleted": false,
|
||||
"id": "9TEzv0sVCHAkc46ou0oNF",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 364.35296630859375,
|
||||
"y": 76.77398681640625,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 234,
|
||||
"height": 24,
|
||||
"seed": 1617178804,
|
||||
"groupIds": [
|
||||
"2Lijjn3PwPQW_8KrcDmdu"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348176045,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "victoriametrics:8428",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "rectangle",
|
||||
"version": 1099,
|
||||
"versionNonce": 671872012,
|
||||
"isDeleted": false,
|
||||
"id": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 360.6495666503906,
|
||||
"y": 382.2536315917969,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 240.10644531249997,
|
||||
"height": 44.74725341796875,
|
||||
"seed": 99322124,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348116633
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 865,
|
||||
"versionNonce": 635839156,
|
||||
"isDeleted": false,
|
||||
"id": "GUs816aggGqUSdoEsSmea",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 382.64422607421875,
|
||||
"y": 394.3433837890625,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 199,
|
||||
"height": 24,
|
||||
"seed": 1194745268,
|
||||
"groupIds": [
|
||||
"6obQBPHIfExBKfejeLLVO"
|
||||
],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348116633,
|
||||
"fontSize": 20,
|
||||
"fontFamily": 3,
|
||||
"text": "alertmanager:9093",
|
||||
"baseline": 19,
|
||||
"textAlign": "center",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 2444,
|
||||
"versionNonce": 361351692,
|
||||
"isDeleted": false,
|
||||
"id": "Bpy5by47XGKB4yS99ZkuA",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 486.0465703465111,
|
||||
"y": 204.98379516601562,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 0.7962088410123442,
|
||||
"height": 86.86798095703125,
|
||||
"seed": 357577356,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348176045,
|
||||
"startBinding": {
|
||||
"elementId": "VgBUzo0blGR-Ijd2mQEEf",
|
||||
"gap": 10.57574462890625,
|
||||
"focus": 0.0344528515859526
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "dd52BjHfPMPRji9Tws7U-",
|
||||
"gap": 8.72607421875,
|
||||
"focus": -0.039393828258510157
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-0.7962088410123442,
|
||||
-86.86798095703125
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 640,
|
||||
"versionNonce": 1892951180,
|
||||
"isDeleted": false,
|
||||
"id": "RbVSa4PnOgAMtzoKb-DhW",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 500.609619140625,
|
||||
"y": 134.11544799804688,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 188,
|
||||
"height": 95,
|
||||
"seed": 1989838604,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348163424,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n",
|
||||
"baseline": 91,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
},
|
||||
{
|
||||
"type": "arrow",
|
||||
"version": 1472,
|
||||
"versionNonce": 768565516,
|
||||
"isDeleted": false,
|
||||
"id": "sxEhnxlbT7ldlSsmHDUHp",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 486.5245361328125,
|
||||
"y": 300.5478957313172,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 0.3521007656264601,
|
||||
"height": 70.26802223743277,
|
||||
"seed": 1818348300,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "round",
|
||||
"boundElementIds": [],
|
||||
"updated": 1638348116633,
|
||||
"startBinding": {
|
||||
"elementId": "E9Run6wCm2chQ6JHrmc_y",
|
||||
"focus": 1.1925203824459496,
|
||||
"gap": 11.77154541015625
|
||||
},
|
||||
"endBinding": {
|
||||
"elementId": "8-XFSbd6Zw96EUSJbJXZv",
|
||||
"focus": 0.0441077573536454,
|
||||
"gap": 11.437713623046875
|
||||
},
|
||||
"lastCommittedPoint": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"points": [
|
||||
[
|
||||
0,
|
||||
0
|
||||
],
|
||||
[
|
||||
-0.3521007656264601,
|
||||
70.26802223743277
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"version": 577,
|
||||
"versionNonce": 1646003636,
|
||||
"isDeleted": false,
|
||||
"id": "E9Run6wCm2chQ6JHrmc_y",
|
||||
"fillStyle": "hachure",
|
||||
"strokeWidth": 1,
|
||||
"strokeStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"x": 498.29608154296875,
|
||||
"y": 298.6573791503906,
|
||||
"strokeColor": "#000000",
|
||||
"backgroundColor": "transparent",
|
||||
"width": 122,
|
||||
"height": 38,
|
||||
"seed": 1836541708,
|
||||
"groupIds": [],
|
||||
"strokeSharpness": "sharp",
|
||||
"boundElementIds": [
|
||||
"sxEhnxlbT7ldlSsmHDUHp"
|
||||
],
|
||||
"updated": 1638348116633,
|
||||
"fontSize": 16,
|
||||
"fontFamily": 3,
|
||||
"text": "send alert \nnotifications",
|
||||
"baseline": 34,
|
||||
"textAlign": "left",
|
||||
"verticalAlign": "top"
|
||||
}
|
||||
],
|
||||
"appState": {
|
||||
"gridSize": null,
|
||||
"viewBackgroundColor": "#ffffff"
|
||||
},
|
||||
"files": {}
|
||||
}
|
||||
BIN
app/vmalert/vmalert_single.png
Normal file
BIN
app/vmalert/vmalert_single.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
144
docs/vmalert.md
144
docs/vmalert.md
|
|
@ -6,7 +6,11 @@ sort: 4
|
|||
|
||||
`vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
|
||||
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
|
||||
rules against configured address. It is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
|
||||
rules against configured `-datasource.url`. For sending alerting notifications
|
||||
vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag.
|
||||
Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
|
||||
protocol and require `-remoteWrite.url` to be configured.
|
||||
Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
|
||||
implementation and aims to be compatible with its syntax.
|
||||
|
||||
## Features
|
||||
|
|
@ -22,12 +26,12 @@ implementation and aims to be compatible with its syntax.
|
|||
* Lightweight without extra dependencies.
|
||||
|
||||
## Limitations
|
||||
* `vmalert` execute queries against remote datasource which has reliability risks because of network.
|
||||
It is recommended to configure alerts thresholds and rules expressions with understanding that network request
|
||||
may fail;
|
||||
* `vmalert` execute queries against remote datasource which has reliability risks because of the network.
|
||||
It is recommended to configure alerts thresholds and rules expressions with the understanding that network
|
||||
requests may fail;
|
||||
* by default, rules execution is sequential within one group, but persistence of execution results to remote
|
||||
storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous
|
||||
recording rule is reused in next one;
|
||||
recording rule is reused in the next one;
|
||||
|
||||
## QuickStart
|
||||
|
||||
|
|
@ -37,13 +41,13 @@ git clone https://github.com/VictoriaMetrics/VictoriaMetrics
|
|||
cd VictoriaMetrics
|
||||
make vmalert
|
||||
```
|
||||
The build binary will be placed to `VictoriaMetrics/bin` folder.
|
||||
The build binary will be placed in `VictoriaMetrics/bin` folder.
|
||||
|
||||
To start using `vmalert` you will need the following things:
|
||||
* list of rules - PromQL/MetricsQL expressions to execute;
|
||||
* datasource address - reachable MetricsQL endpoint to run queries against;
|
||||
* notifier address [optional] - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing,
|
||||
aggregating alerts and sending notifications.
|
||||
aggregating alerts, and sending notifications.
|
||||
* remote write address [optional] - [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
|
||||
compatible storage to persist rules and alerts state info;
|
||||
* remote read address [optional] - MetricsQL compatible datasource to restore alerts state from.
|
||||
|
|
@ -62,7 +66,7 @@ Then configure `vmalert` accordingly:
|
|||
|
||||
Note there's a separate `remoteRead.url` to allow writing results of
|
||||
alerting/recording rules into a different storage than the initial data that's
|
||||
queried. This allows using `vmalert` to aggregate data from a short-term,
|
||||
queried. This allows using `vmalert` to aggregate data from a short-term,
|
||||
high-frequency, high-cardinality storage into a long-term storage with
|
||||
decreased cardinality and a bigger interval between samples.
|
||||
|
||||
|
|
@ -124,14 +128,14 @@ expression and then act according to the Rule type.
|
|||
|
||||
There are two types of Rules:
|
||||
* [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) -
|
||||
Alerting rules allow to define alert conditions via `expr` field and to send notifications to
|
||||
Alerting rules allow defining alert conditions via `expr` field and to send notifications to
|
||||
[Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty.
|
||||
* [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) -
|
||||
Recording rules allow to define `expr` which result will be then backfilled to configured
|
||||
Recording rules allow defining `expr` which result will be then backfilled to configured
|
||||
`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally
|
||||
expensive expressions and save their result as a new set of time series.
|
||||
|
||||
`vmalert` forbids defining duplicates - rules with the same combination of name, expression and labels
|
||||
`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels
|
||||
within one group.
|
||||
|
||||
#### Alerting rules
|
||||
|
|
@ -147,7 +151,7 @@ alert: <string>
|
|||
expr: <string>
|
||||
|
||||
# Alerts are considered firing once they have been returned for this long.
|
||||
# Alerts which have not yet fired for long enough are considered pending.
|
||||
# Alerts which have not yet been fired for long enough are considered pending.
|
||||
# If param is omitted or set to 0 then alerts will be immediately considered
|
||||
# as firing once they return.
|
||||
[ for: <duration> | default = 0s ]
|
||||
|
|
@ -192,19 +196,19 @@ For recording rules to work `-remoteWrite.url` must be specified.
|
|||
the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags:
|
||||
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
|
||||
into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
|
||||
These are regular time series and may be queried from VM just as any other time series.
|
||||
These are regular time series and maybe queried from VM just as any other time series.
|
||||
The state is stored to the configured address on every rule evaluation.
|
||||
* `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
|
||||
from configured address by querying time series with name `ALERTS_FOR_STATE`.
|
||||
|
||||
Both flags are required for proper state restoring. Restore process may fail if time series are missing
|
||||
Both flags are required for proper state restoration. Restore process may fail if time series are missing
|
||||
in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`)
|
||||
or received state doesn't match current `vmalert` rules configuration.
|
||||
|
||||
|
||||
### Multitenancy
|
||||
|
||||
The are the following approaches exist for alerting and recording rules across
|
||||
There are the following approaches exist for alerting and recording rules across
|
||||
[multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy):
|
||||
|
||||
* To run a separate `vmalert` instance per each tenant.
|
||||
|
|
@ -244,6 +248,106 @@ The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz`
|
|||
at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise`
|
||||
tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags).
|
||||
|
||||
### Topology examples
|
||||
|
||||
The following sections are showing how `vmalert` may be used and configured
|
||||
for different scenarios.
|
||||
|
||||
Please note, not all flags in examples are required:
|
||||
* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if
|
||||
you have recording rules or want to store [alerts state](#alerts-state-on-restarts) on `vmalert` restarts;
|
||||
* `-notifier.url` is optional and is needed only if you have alerting rules.
|
||||
|
||||
#### Single-node VictoriaMetrics
|
||||
|
||||
<img alt="vmalert single" src="vmalert_single.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
|
||||
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
|
||||
```
|
||||
|
||||
The simplest configuration where one single-node VM server is used for
|
||||
rules execution, storing recording rules results and alerts state.
|
||||
|
||||
#### Cluster VictoriaMetrics
|
||||
|
||||
<img alt="vmalert cluster" src="vmalert_cluster.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
|
||||
-remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
|
||||
```
|
||||
|
||||
In [cluster mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html)
|
||||
VictoriaMetrics has separate components for writing and reading path:
|
||||
`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions
|
||||
and `vminsert` is used to persist recording rules results and alerts state.
|
||||
Cluster mode could have multiple `vminsert` and `vmselect` components. In case when you want
|
||||
to spread the load on these components - add balancers before them and configure
|
||||
`vmalert` with balancer's addresses. Please, see more about VM's cluster architecture
|
||||
[here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview).
|
||||
|
||||
#### vmalert high availability
|
||||
|
||||
<img alt="vmalert ha" src="vmalert_ha.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
|
||||
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
|
||||
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
|
||||
-notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger
|
||||
-notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers
|
||||
```
|
||||
|
||||
For HA user can run multiple identically configured `vmalert` instances.
|
||||
It means all of them will execute the same rules, write state and results to
|
||||
the same destinations, and send alert notifications to multiple configured
|
||||
Alertmanagers.
|
||||
|
||||
To avoid recording rules results and alerts state duplication in VictoriaMetrics server
|
||||
don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).
|
||||
|
||||
Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
|
||||
all `vmalert`s are having the same config.
|
||||
|
||||
Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
for Alertmanagers for better reliability.
|
||||
|
||||
This example uses single-node VM server for the sake of simplicity.
|
||||
Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics) if needed.
|
||||
|
||||
#### Downsampling and aggregation via vmalert
|
||||
|
||||
<img alt="vmalert multi cluster" src="vmalert_multicluster.png">
|
||||
|
||||
`vmalert` configuration flags:
|
||||
```
|
||||
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
|
||||
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions
|
||||
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results
|
||||
```
|
||||
|
||||
Example shows how to build a topology where `vmalert` will process data from one cluster
|
||||
and write results into another. Such clusters may be called as "hot" (low retention,
|
||||
high-speed disks, used for operative monitoring) and "cold" (long term retention,
|
||||
slower/cheaper disks, low resolution data). With help of `vmalert`, user can setup
|
||||
recording rules to process raw data from "hot" cluster (by applying additional transformations
|
||||
or reducing resolution) and push results to "cold" cluster.
|
||||
|
||||
Please note, [replay](#rules-backfilling) feature may be used for transforming historical data.
|
||||
|
||||
Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used.
|
||||
|
||||
### Web
|
||||
|
||||
|
|
@ -263,7 +367,7 @@ vmalert sends requests to `<-datasource.url>/render?format=json` during evaluati
|
|||
if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render`
|
||||
implements [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) for `format=json`.
|
||||
When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget
|
||||
to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on query type.
|
||||
to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on the query type.
|
||||
|
||||
## Rules backfilling
|
||||
|
||||
|
|
@ -322,11 +426,11 @@ to prevent cache pollution and unwanted time range boundaries adjustment during
|
|||
|
||||
#### Recording rules
|
||||
|
||||
Result of recording rules `replay` should match with results of normal rules evaluation.
|
||||
The result of recording rules `replay` should match with results of normal rules evaluation.
|
||||
|
||||
#### Alerting rules
|
||||
|
||||
Result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
|
||||
The result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
|
||||
To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression:
|
||||
```
|
||||
ALERTS{alertname="your_alertname", alertstate="firing"}
|
||||
|
|
@ -339,7 +443,7 @@ There are following non-required `replay` flags:
|
|||
|
||||
* `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request.
|
||||
In two words, it affects the max time range for every `/query_range` request. The higher the value,
|
||||
the less requests will be issued during `replay`.
|
||||
the fewer requests will be issued during `replay`.
|
||||
* `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries
|
||||
per rule before giving up.
|
||||
* `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining
|
||||
|
|
@ -361,7 +465,7 @@ See full description for these flags in `./vmalert --help`.
|
|||
We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported
|
||||
metrics may be analyzed later.
|
||||
|
||||
Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
|
||||
Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
|
||||
If you have suggestions for improvements or have found a bug - please open an issue on github or add
|
||||
a review to the dashboard.
|
||||
|
||||
|
|
|
|||
BIN
docs/vmalert_cluster.png
Normal file
BIN
docs/vmalert_cluster.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
BIN
docs/vmalert_ha.png
Normal file
BIN
docs/vmalert_ha.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
BIN
docs/vmalert_multicluster.png
Normal file
BIN
docs/vmalert_multicluster.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
BIN
docs/vmalert_single.png
Normal file
BIN
docs/vmalert_single.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Loading…
Reference in a new issue