mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2025-01-20 15:16:42 +00:00
add vmalert managed vm integration guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
This commit is contained in:
Artem Navoiev
parent
8bab50dc29
commit
5872b5711d
8 changed files with 298 additions and 3 deletions
Binary file not shown.
|
After Width: | Height: | Size: 248 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 345 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 387 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 268 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 248 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 340 KiB |
|
|
@ -0,0 +1,295 @@
|
|||
---
|
||||
sort: 4
|
||||
weight: 4
|
||||
title: Kubernetes Monitoring with Managed VictoriaMetrics
|
||||
menu:
|
||||
docs:
|
||||
parent: "managed"
|
||||
weight: 4
|
||||
---
|
||||
# Alerting with vmalert and Managed VictoriaMetrics
|
||||
|
||||
This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics
|
||||
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-setup.png" width="800">
|
||||
|
||||
## Preconditions
|
||||
|
||||
* [vmalert](https://docs.victoriametrics.com/vmalert.html) is installed. You can obtain it by building it from [source](https://docs.victoriametrics.com/vmalert.html#quickstart), downloading it from the [GitHub releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), or using the [docker image](https://hub.docker.com/r/victoriametrics/vmalert) for the container ecosystem (such as docker, k8s, etc.).
|
||||
* [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is installed.
|
||||
* You have a [single or cluster](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#creating-instance) deployment in [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/overview.html).
|
||||
* If you are using helm, add the [VictoriaMetrics helm chart](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-alert#how-to-install) repository to your helm repositories. This step is optional.
|
||||
* If you are using [vmoperator](https://docs.victoriametrics.com/operator/quick-start.html#quick-start), make sure that it and its CRDs are installed. This step is also optional.
|
||||
|
||||
## Setup
|
||||
|
||||
### Alerting and recording rules file(s)
|
||||
|
||||
You need to prepare file(s) with alerting or recording rules.
|
||||
|
||||
An example file with one alerting rule.
|
||||
|
||||
alerts.yml
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: common
|
||||
rules:
|
||||
- alert: instanceIsDown
|
||||
for: 1m
|
||||
expr: up == 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up"
|
||||
description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute"
|
||||
```
|
||||
|
||||
### Managed VictoriaMetrics access token and deployment endpoint
|
||||
|
||||
To use vmalert with Managed VictoriaMetrics, you must create a read/write token, or use an existing one. The token must have write access to ingest recording rules, ALERTS and ALERTS_FOR_STATE metrics, and read access for rules evaluation.
|
||||
|
||||
For instructions on how to create tokens, please refer to this section of the [documentation](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#access).
|
||||
|
||||
#### Single-Node
|
||||
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-single-1.png" width="800">
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-single-2.png" width="800">
|
||||
|
||||
#### Cluster
|
||||
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-cluster-1.png" width="800">
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-cluster-2.png" width="800">
|
||||
<img src="alerting-vmalert-managed-victoria-metrics-cluster-3.png" width="800">
|
||||
|
||||
### vmalert configuration
|
||||
|
||||
#### Single-Node
|
||||
|
||||
##### Binary
|
||||
|
||||
```console
|
||||
export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf
|
||||
export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com
|
||||
export ALERTMANAGER_URL=http://localhost:9093
|
||||
./vmalert -rule=alerts.yml -datasource.url=$MANAGED_VM_URL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -remoteWrite.url=$MANAGED_VM_URL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_URL -remoteRead.bearerToken=$TOKEN
|
||||
```
|
||||
|
||||
##### Docker
|
||||
|
||||
```console
|
||||
export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf
|
||||
export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com
|
||||
export ALERTMANAGER_URL=http://alertmanager:9093
|
||||
docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGED_VM_URL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_URL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGED_VM_URL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -rule="/etc/alerts/*.yml"
|
||||
```
|
||||
|
||||
##### Helm Chart
|
||||
|
||||
```console
|
||||
export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf
|
||||
export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com
|
||||
export ALERTMANAGER=http://alertmanager:9093
|
||||
cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f -
|
||||
server:
|
||||
datasource:
|
||||
url: $MANAGED_VM_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
remote:
|
||||
write:
|
||||
url: $MANAGED_VM_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
read:
|
||||
url: $MANAGED_VM_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
notifier:
|
||||
alertmanager:
|
||||
url: $ALERTMANAGER
|
||||
config:
|
||||
alerts:
|
||||
groups:
|
||||
- name: common
|
||||
rules:
|
||||
- alert: instanceIsDown
|
||||
for: 1m
|
||||
expr: up == 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up"
|
||||
description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute"
|
||||
EOF
|
||||
```
|
||||
|
||||
##### VMalert CRD for vmoperator
|
||||
|
||||
```console
|
||||
export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf
|
||||
export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com
|
||||
export ALERTMANAGER=http://alertmanager:9093
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: vmalert-managed-vm
|
||||
spec:
|
||||
replicaCount: 1
|
||||
datasource:
|
||||
url: $MANAGED_VM_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
remoteWrite:
|
||||
url: $MANAGED_VM_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
remoteRead:
|
||||
url: $MANAGED_VM_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
notifier:
|
||||
url: $ALERTMANAGER
|
||||
ruleSelector:
|
||||
matchLabels:
|
||||
type: managed
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: managed-token
|
||||
stringData:
|
||||
token: $TOKEN
|
||||
EOF
|
||||
```
|
||||
|
||||
##### Testing
|
||||
|
||||
You can ingest metric that will raise an alert
|
||||
|
||||
```console
|
||||
export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf
|
||||
export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com/
|
||||
curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGED_VM_URLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0'
|
||||
```
|
||||
|
||||
#### Cluster
|
||||
|
||||
##### Binary
|
||||
|
||||
```console
|
||||
export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4
|
||||
export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/
|
||||
export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/
|
||||
export ALERTMANAGER_URL=http://localhost:9093
|
||||
./vmalert -rule=alerts.yml -datasource.url=$MANAGED_VM_READ_URL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -remoteWrite.url=$MANAGED_VM_WRITE_URL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_READ_URL -remoteRead.bearerToken=$TOKEN
|
||||
```
|
||||
|
||||
##### Docker
|
||||
|
||||
```console
|
||||
export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4
|
||||
export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/
|
||||
export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/
|
||||
export ALERTMANAGER_URL=http://alertmanager:9093
|
||||
docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGED_VM_READ_URL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_READ_URL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGED_VM_WRITE_URL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -rule="/etc/alerts/*.yml"
|
||||
```
|
||||
|
||||
##### Helm Chart
|
||||
|
||||
```console
|
||||
export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4
|
||||
export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/
|
||||
export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/
|
||||
export ALERTMANAGER=http://alertmanager:9093
|
||||
cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f -
|
||||
server:
|
||||
datasource:
|
||||
url: $MANAGED_VM_READ_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
remote:
|
||||
write:
|
||||
url: $MANAGED_VM_WRITE_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
read:
|
||||
url: $MANAGED_VM_READ_URL
|
||||
bearer:
|
||||
token: $TOKEN
|
||||
notifier:
|
||||
alertmanager:
|
||||
url: $ALERTMANAGER
|
||||
config:
|
||||
alerts:
|
||||
groups:
|
||||
- name: common
|
||||
rules:
|
||||
- alert: instanceIsDown
|
||||
for: 1m
|
||||
expr: up == 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up"
|
||||
description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute"
|
||||
EOF
|
||||
```
|
||||
|
||||
##### VMalert CRD for vmoperator
|
||||
|
||||
```console
|
||||
export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4
|
||||
export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/
|
||||
export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/
|
||||
export ALERTMANAGER=http://alertmanager:9093
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: vmalert-managed-vm
|
||||
spec:
|
||||
replicaCount: 1
|
||||
datasource:
|
||||
url: $MANAGED_VM_READ_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
remoteWrite:
|
||||
url: $MANAGED_VM_WRITE_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
remoteRead:
|
||||
url: $MANAGED_VM_READ_URL
|
||||
bearerTokenSecret:
|
||||
name: managed-token
|
||||
key: token
|
||||
notifier:
|
||||
url: $ALERTMANAGER
|
||||
ruleSelector:
|
||||
matchLabels:
|
||||
type: managed
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: managed-token
|
||||
stringData:
|
||||
token: $TOKEN
|
||||
EOF
|
||||
```
|
||||
|
||||
##### Testing
|
||||
|
||||
You can ingest metric that will raise an alert
|
||||
|
||||
```console
|
||||
export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4
|
||||
export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/
|
||||
curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGED_VM_WRITE_URLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0'
|
||||
```
|
||||
|
|
@ -1,11 +1,11 @@
|
|||
---
|
||||
sort: 4
|
||||
weight: 4
|
||||
sort: 5
|
||||
weight: 5
|
||||
title: User Management in Managed VictoriaMetrics
|
||||
menu:
|
||||
docs:
|
||||
parent: "managed"
|
||||
weight: 4
|
||||
weight: 5
|
||||
---
|
||||
# User Management in Managed VictoriaMetrics
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue