diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-1.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-1.png new file mode 100644 index 0000000000..de40f9cf40 Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-1.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-2.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-2.png new file mode 100644 index 0000000000..da0c34edfd Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-2.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-3.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-3.png new file mode 100644 index 0000000000..d6a1b84dc6 Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-cluster-3.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-setup.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-setup.png new file mode 100644 index 0000000000..11c00dc566 Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-setup.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-1.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-1.png new file mode 100644 index 0000000000..f63ac7bbe0 Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-1.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-2.png b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-2.png new file mode 100644 index 0000000000..3b36fb9b86 Binary files /dev/null and b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics-single-2.png differ diff --git a/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md new file mode 100644 index 0000000000..f5e0f84f03 --- /dev/null +++ b/docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md @@ -0,0 +1,295 @@ +--- +sort: 4 +weight: 4 +title: Kubernetes Monitoring with Managed VictoriaMetrics +menu: + docs: + parent: "managed" + weight: 4 +--- +# Alerting with vmalert and Managed VictoriaMetrics + +This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics + +<img src="alerting-vmalert-managed-victoria-metrics-setup.png" width="800"> + +## Preconditions + +* [vmalert](https://docs.victoriametrics.com/vmalert.html) is installed. You can obtain it by building it from [source](https://docs.victoriametrics.com/vmalert.html#quickstart), downloading it from the [GitHub releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), or using the [docker image](https://hub.docker.com/r/victoriametrics/vmalert) for the container ecosystem (such as docker, k8s, etc.). +* [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is installed. +* You have a [single or cluster](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#creating-instance) deployment in [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/overview.html). +* If you are using helm, add the [VictoriaMetrics helm chart](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-alert#how-to-install) repository to your helm repositories. This step is optional. +* If you are using [vmoperator](https://docs.victoriametrics.com/operator/quick-start.html#quick-start), make sure that it and its CRDs are installed. This step is also optional. + +## Setup + +### Alerting and recording rules file(s) + +You need to prepare file(s) with alerting or recording rules. + +An example file with one alerting rule. + +alerts.yml + +```yaml +groups: + - name: common + rules: + - alert: instanceIsDown + for: 1m + expr: up == 0 + labels: + severity: critical + annotations: + summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up" + description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" +``` + +### Managed VictoriaMetrics access token and deployment endpoint + +To use vmalert with Managed VictoriaMetrics, you must create a read/write token, or use an existing one. The token must have write access to ingest recording rules, ALERTS and ALERTS_FOR_STATE metrics, and read access for rules evaluation. + +For instructions on how to create tokens, please refer to this section of the [documentation](https://docs.victoriametrics.com/managed-victoriametrics/quickstart.html#access). + +#### Single-Node + +<img src="alerting-vmalert-managed-victoria-metrics-single-1.png" width="800"> +<img src="alerting-vmalert-managed-victoria-metrics-single-2.png" width="800"> + +#### Cluster + +<img src="alerting-vmalert-managed-victoria-metrics-cluster-1.png" width="800"> +<img src="alerting-vmalert-managed-victoria-metrics-cluster-2.png" width="800"> +<img src="alerting-vmalert-managed-victoria-metrics-cluster-3.png" width="800"> + +### vmalert configuration + +#### Single-Node + +##### Binary + +```console +export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf +export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com +export ALERTMANAGER_URL=http://localhost:9093 +./vmalert -rule=alerts.yml -datasource.url=$MANAGED_VM_URL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -remoteWrite.url=$MANAGED_VM_URL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_URL -remoteRead.bearerToken=$TOKEN +``` + +##### Docker + +```console +export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf +export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com +export ALERTMANAGER_URL=http://alertmanager:9093 +docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGED_VM_URL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_URL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGED_VM_URL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -rule="/etc/alerts/*.yml" +``` + +##### Helm Chart + +```console +export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf +export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com +export ALERTMANAGER=http://alertmanager:9093 +cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f - +server: + datasource: + url: $MANAGED_VM_URL + bearer: + token: $TOKEN + remote: + write: + url: $MANAGED_VM_URL + bearer: + token: $TOKEN + read: + url: $MANAGED_VM_URL + bearer: + token: $TOKEN + notifier: + alertmanager: + url: $ALERTMANAGER + config: + alerts: + groups: + - name: common + rules: + - alert: instanceIsDown + for: 1m + expr: up == 0 + labels: + severity: critical + annotations: + summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up" + description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" +EOF +``` + +##### VMalert CRD for vmoperator + +```console +export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf +export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com +export ALERTMANAGER=http://alertmanager:9093 +cat << EOF | kubectl apply -f - +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMAlert +metadata: + name: vmalert-managed-vm +spec: + replicaCount: 1 + datasource: + url: $MANAGED_VM_URL + bearerTokenSecret: + name: managed-token + key: token + remoteWrite: + url: $MANAGED_VM_URL + bearerTokenSecret: + name: managed-token + key: token + remoteRead: + url: $MANAGED_VM_URL + bearerTokenSecret: + name: managed-token + key: token + notifier: + url: $ALERTMANAGER + ruleSelector: + matchLabels: + type: managed +--- +apiVersion: v1 +kind: Secret +metadata: + name: managed-token +stringData: + token: $TOKEN +EOF +``` + +##### Testing + +You can ingest metric that will raise an alert + +```console +export TOKEN=81e8226e-4e25-480d-9397-31ba4007f0bf +export MANAGED_VM_URL=https://gw-c15-1c.cloud.victoriametrics.com/ +curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGED_VM_URLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0' +``` + +#### Cluster + +##### Binary + +```console +export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4 +export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ +export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ +export ALERTMANAGER_URL=http://localhost:9093 +./vmalert -rule=alerts.yml -datasource.url=$MANAGED_VM_READ_URL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -remoteWrite.url=$MANAGED_VM_WRITE_URL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_READ_URL -remoteRead.bearerToken=$TOKEN +``` + +##### Docker + +```console +export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4 +export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ +export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ +export ALERTMANAGER_URL=http://alertmanager:9093 +docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGED_VM_READ_URL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGED_VM_READ_URL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGED_VM_WRITE_URL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -rule="/etc/alerts/*.yml" +``` + +##### Helm Chart + +```console +export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4 +export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ +export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ +export ALERTMANAGER=http://alertmanager:9093 +cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f - +server: + datasource: + url: $MANAGED_VM_READ_URL + bearer: + token: $TOKEN + remote: + write: + url: $MANAGED_VM_WRITE_URL + bearer: + token: $TOKEN + read: + url: $MANAGED_VM_READ_URL + bearer: + token: $TOKEN + notifier: + alertmanager: + url: $ALERTMANAGER + config: + alerts: + groups: + - name: common + rules: + - alert: instanceIsDown + for: 1m + expr: up == 0 + labels: + severity: critical + annotations: + summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up" + description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" +EOF +``` + +##### VMalert CRD for vmoperator + +```console +export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4 +export MANAGED_VM_READ_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ +export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ +export ALERTMANAGER=http://alertmanager:9093 +cat << EOF | kubectl apply -f - +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMAlert +metadata: + name: vmalert-managed-vm +spec: + replicaCount: 1 + datasource: + url: $MANAGED_VM_READ_URL + bearerTokenSecret: + name: managed-token + key: token + remoteWrite: + url: $MANAGED_VM_WRITE_URL + bearerTokenSecret: + name: managed-token + key: token + remoteRead: + url: $MANAGED_VM_READ_URL + bearerTokenSecret: + name: managed-token + key: token + notifier: + url: $ALERTMANAGER + ruleSelector: + matchLabels: + type: managed +--- +apiVersion: v1 +kind: Secret +metadata: + name: managed-token +stringData: + token: $TOKEN +EOF +``` + +##### Testing + +You can ingest metric that will raise an alert + +```console +export TOKEN=76bc5470-d340-4e5e-9574-49ed30911cc4 +export MANAGED_VM_WRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ +curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGED_VM_WRITE_URLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0' +``` \ No newline at end of file diff --git a/docs/managed-victoriametrics/user-managment.md b/docs/managed-victoriametrics/user-managment.md index 94e3dabc17..c4001e0d00 100644 --- a/docs/managed-victoriametrics/user-managment.md +++ b/docs/managed-victoriametrics/user-managment.md @@ -1,11 +1,11 @@ --- -sort: 4 -weight: 4 +sort: 5 +weight: 5 title: User Management in Managed VictoriaMetrics menu: docs: parent: "managed" - weight: 4 + weight: 5 --- # User Management in Managed VictoriaMetrics