From c429bbf88902abf08df60d97ba133cf27c5bbaeb Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Fri, 5 Jul 2024 08:47:59 +0200 Subject: [PATCH] app/vmalert: add examples for `source` override (#6561) The change adds a new docs section with examples on how source can be overridden. It should address questions like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6536 While there, fix the example in `external.alert.source` cmd-line flag and docker-compose examples. ### Checklist The following checks are **mandatory**: - [x] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: hagen1778 --- app/vmalert/main.go | 2 +- deployment/docker/docker-compose-cluster.yml | 3 +- deployment/docker/docker-compose.yml | 3 +- docs/vmalert.md | 34 +++++++++++++++++++- 4 files changed, 36 insertions(+), 6 deletions(-) diff --git a/app/vmalert/main.go b/app/vmalert/main.go index f6e05a00f..57c1cbb40 100644 --- a/app/vmalert/main.go +++ b/app/vmalert/main.go @@ -72,7 +72,7 @@ absolute path to all .tpl files in root. externalAlertSource = flag.String("external.alert.source", "", `External Alert Source allows to override the Source link for alerts sent to AlertManager `+ `for cases where you want to build a custom link to Grafana, Prometheus or any other service. `+ `Supports templating - see https://docs.victoriametrics.com/vmalert/#templating . `+ - `For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{$expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. `+ + `For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. `+ `Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. `+ `If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.`) externalLabels = flagutil.NewArrayString("external.label", "Optional label in the form 'Name=value' to add to all generated recording rules and alerts. "+ diff --git a/deployment/docker/docker-compose-cluster.yml b/deployment/docker/docker-compose-cluster.yml index 3f55a272c..6821850b6 100644 --- a/deployment/docker/docker-compose-cluster.yml +++ b/deployment/docker/docker-compose-cluster.yml @@ -145,8 +145,7 @@ services: - '--rule=/etc/alerts/*.yml' # display source of alerts in grafana - '-external.url=http://127.0.0.1:3000' #grafana outside container - # when copypaste the line below be aware of '$$' for escaping in '$expr' - - '--external.alert.source=explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{$$expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}' + - '--external.alert.source=explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"{{ .ActiveAt.UnixMilli }}","to":"now"}}' restart: always # alertmanager receives alerting notifications from vmalert diff --git a/deployment/docker/docker-compose.yml b/deployment/docker/docker-compose.yml index 8be7e6501..812a0f7be 100644 --- a/deployment/docker/docker-compose.yml +++ b/deployment/docker/docker-compose.yml @@ -84,8 +84,7 @@ services: - "--rule=/etc/alerts/*.yml" # display source of alerts in grafana - "--external.url=http://127.0.0.1:3000" #grafana outside container - # when copypaste the line be aware of '$$' for escaping in '$expr' - - '--external.alert.source=explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{$$expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}' + - '--external.alert.source=explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"{{ .ActiveAt.UnixMilli }}","to":"now"}}' networks: - vm_net restart: always diff --git a/docs/vmalert.md b/docs/vmalert.md index 19fbe9e5d..7aceaeee5 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -418,6 +418,38 @@ in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules with `restored` label in [web UI](#web). +### Link to alert source + +Alerting notifications sent by vmalert always contain a `source` link. By default, the link format +is the following `http:///vmalert/alert?group_id=&alert_id=`. On click, it opens +vmalert [web UI](https://docs.victoriametrics.com/vmalert/#web) to show the alert status and its fields. + +It is possible to override the link format. For example, to make the link to [vmui](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) +specify the following cmd-line flags: +``` +./bin/vmalert \ + -external.url=http:// \ # the hostname and port for datasource vmui + -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}' # the path built using alert expr +``` + +Now, all `source` links will lead to `http:///vmui/#/?g0.expr=$expr`, where $expr is an alerting rule +expression. + +The `-external.alert.source` cmd-line flag supports [templating](https://docs.victoriametrics.com/vmalert/#templating) +and allows using labels and extra data related to the alert. For example, see the following link to Grafana: +``` +./bin/vmalert \ + -external.url=http:// \ # the hostname and port for Grafana + -external.alert.source='explore?left={"datasource":"VictoriaMetrics","queries":[{"expr":{{ .Expr|jsonEscape|queryEscape }},"refId":"A"}],"range":{"from":"{{ .ActiveAt.UnixMilli }}","to":"now"}}' +``` + +In this example, `-external.alert.source` will lead to Grafana's Explore page with `expr` field equal to alert expression, +and time range will be selected starting from `"from":"{{ .ActiveAt.UnixMilli }}"` when alert became active. + +In addition to `source` link, some extra links could be added to alert's [annotations](https://docs.victoriametrics.com/vmalert/#alerting-rules) +field. See [how we use them](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/839596c00df123c639d1244b28ee8137dfc9609c/deployment/docker/alerts-cluster.yml#L43) +to link alerting rule and the corresponding panel on Grafana dashboard. + ### Multitenancy There are the following approaches exist for alerting and recording rules across @@ -1064,7 +1096,7 @@ The shortlist of configuration flags is the following: -evaluationInterval duration How often to evaluate the rules (default 1m0s) -external.alert.source string - External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert/#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{$expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used. + External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert/#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used. -external.label array Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets. Supports an array of values separated by comma or specified via multiple flags.