2021-04-07 10:39:16 +00:00
---
2021-05-26 09:32:16 +00:00
sort: 4
2023-05-25 19:26:36 +00:00
weight: 4
menu:
docs:
parent: 'victoriametrics'
weight: 4
title: vmalert
aliases:
- /vmalert.html
2021-04-07 10:39:16 +00:00
---
# vmalert
2020-04-29 14:42:01 +00:00
2021-06-14 08:37:26 +00:00
`vmalert` executes a list of the given [alerting ](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ )
2020-06-02 16:12:41 +00:00
or [recording ](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ )
2022-08-08 07:45:21 +00:00
rules against configured `-datasource.url` compatible with Prometheus HTTP API. For sending alerting notifications
2022-03-13 13:56:01 +00:00
vmalert relies on [Alertmanager ](https://github.com/prometheus/alertmanager ) configured via `-notifier.url` flag.
2021-12-01 15:33:06 +00:00
Recording rules results are persisted via [remote write ](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations )
protocol and require `-remoteWrite.url` to be configured.
Vmalert is heavily inspired by [Prometheus ](https://prometheus.io/docs/alerting/latest/overview/ )
2021-06-14 08:37:26 +00:00
implementation and aims to be compatible with its syntax.
2020-04-29 14:42:01 +00:00
2023-01-30 15:28:33 +00:00
A [single-node ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmalert )
or [cluster version ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmalert )
of VictoriaMetrics are capable of proxying requests to vmalert via `-vmalert.proxyURL` command-line flag.
Use this feature for the following cases:
* for proxying requests from [Grafana Alerting UI ](https://grafana.com/docs/grafana/latest/alerting/ );
2023-05-10 07:50:41 +00:00
* for accessing vmalerts UI through VictoriaMetrics Web interface.
2023-01-30 15:28:33 +00:00
2021-04-07 10:39:16 +00:00
## Features
2022-03-22 11:40:55 +00:00
2020-04-29 14:42:01 +00:00
* Integration with [VictoriaMetrics ](https://github.com/VictoriaMetrics/VictoriaMetrics ) TSDB;
2021-04-20 17:16:17 +00:00
* VictoriaMetrics [MetricsQL ](https://docs.victoriametrics.com/MetricsQL.html )
2022-05-14 16:56:31 +00:00
support and expressions validation;
2020-04-29 14:42:01 +00:00
* Prometheus [alerting rules definition format ](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#defining-alerting-rules )
2022-05-14 16:56:31 +00:00
support;
2023-05-10 07:50:41 +00:00
* Integration with [Alertmanager ](https://github.com/prometheus/alertmanager ) starting from [Alertmanager v0.16.0-alpha ](https://github.com/prometheus/alertmanager/releases/tag/v0.16.0-alpha.0 );
2021-04-20 17:16:17 +00:00
* Keeps the alerts [state on restarts ](#alerts-state-on-restarts );
2021-06-09 09:27:31 +00:00
* Graphite datasource can be used for alerting and recording rules. See [these docs ](#graphite );
* Recording and Alerting rules backfilling (aka `replay` ). See [these docs ](#rules-backfilling );
2022-08-08 07:45:21 +00:00
* Lightweight and without extra dependencies.
2023-02-20 15:39:31 +00:00
* Supports [reusable templates ](#reusable-templates ) for annotations;
2023-05-08 11:31:54 +00:00
* Load of recording and alerting rules from local filesystem, URL, GCS and S3;
2023-05-08 07:36:39 +00:00
* Detect alerting rules which [don't match any series ](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039 ).
2020-04-29 14:42:01 +00:00
2021-04-07 10:39:16 +00:00
## Limitations
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
* `vmalert` execute queries against remote datasource which has reliability risks because of the network.
2022-05-14 16:56:31 +00:00
It is recommended to configure alerts thresholds and rules expressions with the understanding that network
requests may fail;
2021-11-05 18:44:02 +00:00
* by default, rules execution is sequential within one group, but persistence of execution results to remote
2022-05-14 16:56:31 +00:00
storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous
recording rule is reused in the next one;
2020-06-10 16:37:43 +00:00
2021-04-07 10:39:16 +00:00
## QuickStart
2020-04-29 14:42:01 +00:00
To build `vmalert` from sources:
2022-03-22 11:40:55 +00:00
2022-06-19 19:57:53 +00:00
```console
2020-04-29 14:42:01 +00:00
git clone https://github.com/VictoriaMetrics/VictoriaMetrics
cd VictoriaMetrics
make vmalert
```
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
The build binary will be placed in `VictoriaMetrics/bin` folder.
2020-04-29 14:42:01 +00:00
To start using `vmalert` you will need the following things:
2022-03-22 11:40:55 +00:00
2020-06-02 16:12:41 +00:00
* list of rules - PromQL/MetricsQL expressions to execute;
2022-08-08 07:45:21 +00:00
* datasource address - reachable endpoint with [Prometheus HTTP API ](https://prometheus.io/docs/prometheus/latest/querying/api/#http-api ) support for running queries against;
2021-11-29 23:22:37 +00:00
* notifier address [optional] - reachable [Alert Manager ](https://github.com/prometheus/alertmanager ) instance for processing,
2022-05-14 16:56:31 +00:00
aggregating alerts, and sending notifications. Please note, notifier address also supports Consul and DNS Service Discovery via
[config file ](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/notifier/config.go ).
2021-06-14 08:37:26 +00:00
* remote write address [optional] - [remote write ](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations )
2022-06-16 11:30:28 +00:00
compatible storage to persist rules and alerts state info. To persist results to multiple destinations use vmagent
configured with multiple remote writes as a proxy;
2021-11-16 22:54:06 +00:00
* remote read address [optional] - MetricsQL compatible datasource to restore alerts state from.
2020-04-29 14:42:01 +00:00
Then configure `vmalert` accordingly:
2022-03-22 11:40:55 +00:00
2022-06-19 19:57:53 +00:00
```console
2021-06-14 08:37:26 +00:00
./bin/vmalert -rule=alert.rules \ # Path to the file with rules configuration. Supports wildcard
2022-08-08 07:45:21 +00:00
-datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource
2021-11-29 23:22:37 +00:00
-notifier.url=http://localhost:9093 \ # AlertManager URL (required if alerting rules are used)
2020-07-30 18:56:48 +00:00
-notifier.url=http://127.0.0.1:9093 \ # AlertManager replica URL
2021-11-29 23:22:37 +00:00
-remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used)
2022-08-08 07:45:21 +00:00
-remoteRead.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource to restore alerts state from
2020-07-30 18:56:48 +00:00
-external.label=cluster=east-1 \ # External label to be applied for each rule
2021-03-25 13:29:04 +00:00
-external.label=replica=a # Multiple external labels may be set
2020-04-29 14:42:01 +00:00
```
2023-01-04 06:19:18 +00:00
Note there's a separate `-remoteWrite.url` command-line flag to allow writing results of
2021-11-16 22:54:06 +00:00
alerting/recording rules into a different storage than the initial data that's
2021-12-01 15:33:06 +00:00
queried. This allows using `vmalert` to aggregate data from a short-term,
2021-11-16 22:54:06 +00:00
high-frequency, high-cardinality storage into a long-term storage with
decreased cardinality and a bigger interval between samples.
2023-01-04 06:19:18 +00:00
See also [stream aggregation ](https://docs.victoriametrics.com/stream-aggregation.html ).
2021-11-16 22:54:06 +00:00
2021-11-05 18:44:02 +00:00
See the full list of configuration flags in [configuration ](#configuration ) section.
2021-06-14 08:37:26 +00:00
2020-07-30 18:56:48 +00:00
If you run multiple `vmalert` services for the same datastore or AlertManager - do not forget
2023-01-04 06:19:18 +00:00
to specify different `-external.label` command-line flags in order to define which `vmalert` generated rules or alerts.
2020-07-30 18:56:48 +00:00
2021-03-09 20:49:50 +00:00
Configuration for [recording ](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ )
and [alerting ](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ ) rules is very
similar to Prometheus rules and configured using YAML. Configuration examples may be found
2020-06-10 16:37:43 +00:00
in [testdata ](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/config/testdata ) folder.
2021-06-14 08:37:26 +00:00
Every `rule` belongs to a `group` and every configuration file may contain arbitrary number of groups:
2022-03-22 11:40:55 +00:00
2020-06-10 16:37:43 +00:00
```yaml
groups:
[ - < rule_group > ]
```
2020-06-02 16:12:41 +00:00
2021-04-07 10:39:16 +00:00
### Groups
2020-04-29 14:42:01 +00:00
2021-06-14 08:37:26 +00:00
Each group has the following attributes:
2022-03-22 11:40:55 +00:00
2020-06-10 16:37:43 +00:00
```yaml
# The name of the group. Must be unique within a file.
name: < string >
2020-04-29 14:42:01 +00:00
2020-06-10 16:37:43 +00:00
# How often rules in the group are evaluated.
2021-06-14 08:37:26 +00:00
[ interval: < duration > | default = -evaluationInterval flag ]
2020-06-10 16:37:43 +00:00
2023-09-06 14:29:59 +00:00
# Optional
# Group will be evaluated at the exact offset in the range of [0...interval].
# E.g. for Group with `interval: 1h` and `eval_offset: 5m` the evaluation will
# start at 5th minute of the hour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409
# `eval_offset` can't be bigger than `interval`.
[ eval_offset: < duration > ]
2023-10-25 09:54:18 +00:00
# Optional
2023-10-25 11:07:13 +00:00
# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource.
# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details.
# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`.
# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyConcepts.html#query-latency.
2023-10-25 09:54:18 +00:00
[ eval_delay: < duration > ]
2022-06-09 06:21:30 +00:00
# Limit the number of alerts an alerting rule and series a recording
# rule can produce. 0 is no limit.
[ limit: < int > | default = 0 ]
2021-06-14 08:37:26 +00:00
# How many rules execute at once within a group. Increasing concurrency may speed
2023-08-09 20:02:32 +00:00
# up group's evaluation duration (exposed via `vmalert_iteration_duration_seconds` metric).
2020-06-10 16:37:43 +00:00
[ concurrency: < integer > | default = 1 ]
2021-02-03 19:12:14 +00:00
# Optional type for expressions inside the rules. Supported values: "graphite" and "prometheus".
2023-08-09 20:02:32 +00:00
# By default, "prometheus" type is used.
2021-02-03 19:12:14 +00:00
[ type: < string > ]
2023-10-10 10:41:19 +00:00
# Optional
# The evaluation timestamp will be aligned with group's interval,
# instead of using the actual timestamp that evaluation happens at.
# By default, it's enabled to get more predictable results
# and to visually align with results plotted via Grafana or vmui.
# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049
# Available starting from v1.95
[ eval_alignment: < bool > | default true]
2021-12-02 22:08:42 +00:00
# Optional list of HTTP URL parameters
# applied for all rules requests within a group
# For example:
# params:
# nocache: ["1"] # disable caching for vmselect
# denyPartialResponse: ["true"] # fail if one or more vmstorage nodes returned an error
2022-09-28 07:15:05 +00:00
# extra_label: ["env=dev"] # apply additional label filter "env=dev" for all requests
2021-12-02 22:08:42 +00:00
# see more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements
params:
[ < string > : [< string > , ...]]
2022-07-21 13:59:55 +00:00
# Optional list of HTTP headers in form `header-name: value`
# applied for all rules requests within a group
# For example:
# headers:
# - "CustomHeader: foo"
# - "CustomHeader2: bar"
2022-09-28 07:15:05 +00:00
# Headers set via this param have priority over headers set via `-datasource.headers` flag.
2022-07-21 13:59:55 +00:00
headers:
[ < string > , ...]
2023-04-27 11:02:21 +00:00
# Optional list of HTTP headers in form `header-name: value`
# applied for all alert notifications sent to notifiers
# generated by rules of this group.
# For example:
# notifier_headers:
# - "TenantID: foo"
notifier_headers:
[ < string > , ...]
2021-08-31 11:52:34 +00:00
# Optional list of labels added to every rule within a group.
# It has priority over the external labels.
# Labels are commonly used for adding environment
# or tenant-specific tag.
labels:
[ < labelname > : < labelvalue > ... ]
2020-06-10 16:37:43 +00:00
rules:
[ - < rule > ... ]
```
2021-04-07 10:39:16 +00:00
### Rules
2020-06-10 16:37:43 +00:00
2021-06-14 08:37:26 +00:00
Every rule contains `expr` field for [PromQL ](https://prometheus.io/docs/prometheus/latest/querying/basics/ )
2022-08-08 07:45:21 +00:00
or [MetricsQL ](https://docs.victoriametrics.com/MetricsQL.html ) expression. `vmalert` will execute the configured
2021-06-14 08:37:26 +00:00
expression and then act according to the Rule type.
2020-06-10 16:37:43 +00:00
There are two types of Rules:
2022-03-22 11:40:55 +00:00
2021-03-09 20:49:50 +00:00
* [alerting ](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ ) -
2022-05-14 16:56:31 +00:00
Alerting rules allow defining alert conditions via `expr` field and to send notifications to
[Alertmanager ](https://github.com/prometheus/alertmanager ) if execution result is not empty.
2021-03-09 20:49:50 +00:00
* [recording ](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ ) -
2022-05-14 16:56:31 +00:00
Recording rules allow defining `expr` which result will be then backfilled to configured
`-remoteWrite.url` . Recording rules are used to precompute frequently needed or computationally
expensive expressions and save their result as a new set of time series.
2020-06-10 16:37:43 +00:00
2021-12-01 15:33:06 +00:00
`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels
2021-03-09 20:49:50 +00:00
within one group.
2020-06-18 20:54:44 +00:00
2021-04-07 10:39:16 +00:00
#### Alerting rules
2020-06-10 16:37:43 +00:00
2021-06-14 08:37:26 +00:00
The syntax for alerting rule is the following:
2022-03-22 11:40:55 +00:00
2020-06-10 16:37:43 +00:00
```yaml
# The name of the alert. Must be a valid metric name.
alert: < string >
2021-02-01 13:26:33 +00:00
# The expression to evaluate. The expression language depends on the type value.
2022-08-08 07:45:21 +00:00
# By default, PromQL/MetricsQL expression is used. If group.type="graphite", then the expression
2021-02-01 13:26:33 +00:00
# must contain valid Graphite expression.
2020-06-10 16:37:43 +00:00
expr: < string >
# Alerts are considered firing once they have been returned for this long.
2021-12-01 15:33:06 +00:00
# Alerts which have not yet been fired for long enough are considered pending.
2021-06-14 08:37:26 +00:00
# If param is omitted or set to 0 then alerts will be immediately considered
# as firing once they return.
2020-06-10 16:37:43 +00:00
[ for: < duration > | default = 0s ]
2020-06-02 16:12:41 +00:00
2023-07-27 13:13:13 +00:00
# Alert will continue firing for this long even when the alerting expression no longer has results.
# This allows you to delay alert resolution.
[ keep_firing_for: < duration > | default = 0s ]
2022-09-13 13:25:43 +00:00
# Whether to print debug information into logs.
2022-09-28 07:15:05 +00:00
# Information includes alerts state changes and requests sent to the datasource.
# Please note, that if rule's query params contain sensitive
2022-09-13 13:25:43 +00:00
# information - it will be printed to logs.
# Is applicable to alerting rules only.
2023-06-21 07:44:00 +00:00
# Available starting from https://docs.victoriametrics.com/CHANGELOG.html#v1820
2022-09-13 13:25:43 +00:00
[ debug: < bool > | default = false ]
2022-12-29 11:36:44 +00:00
# Defines the number of rule's updates entries stored in memory
# and available for view on rule's Details page.
# Overrides `rule.updateEntriesLimit` value for this specific rule.
2023-06-21 07:44:00 +00:00
# Available starting from https://docs.victoriametrics.com/CHANGELOG.html#v1860
2022-12-29 11:36:44 +00:00
[ update_entries_limit: < integer > | default 0 ]
2020-06-10 16:37:43 +00:00
# Labels to add or overwrite for each alert.
labels:
[ < labelname > : < tmpl_string > ]
# Annotations to add to each alert.
annotations:
[ < labelname > : < tmpl_string > ]
2021-03-09 20:49:50 +00:00
```
2020-06-10 16:37:43 +00:00
2022-09-28 07:15:05 +00:00
#### Templating
2022-08-16 06:08:27 +00:00
2022-09-28 07:15:05 +00:00
It is allowed to use [Go templating ](https://golang.org/pkg/text/template/ ) in annotations to format data, iterate over
2022-08-16 06:08:27 +00:00
or execute expressions.
The following variables are available in templating:
2022-08-22 11:49:17 +00:00
| Variable | Description | Example |
|------------------------------------|-----------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| $value or .Value | The current alert's value. Avoid using value in labels, it may cause unexpected issues. | {% raw %}Number of connections is {{ $value }}{% endraw %} |
| $activeAt or .ActiveAt | The moment of [time ](https://pkg.go.dev/time ) when alert became active (`pending` or `firing` ). | {% raw %}http://vm-grafana.com/panelId=xx?from={{($activeAt.Add (parseDurationTime \"1h\")).Unix}}& to={{($activeAt.Add (parseDurationTime \"-1h\")).Unix}}{% endraw %} |
| $labels or .Labels | The list of labels of the current alert. Use as ".Labels.< label_name > ". | {% raw %}Too high number of connections for {{ .Labels.instance }}{% endraw %} |
| $alertID or .AlertID | The current alert's ID generated by vmalert. | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}& alert_id={{.AlertID}}{% endraw %} |
| $groupID or .GroupID | The current alert's group ID generated by vmalert. | {% raw %}Link: vmalert/alert?group_id={{.GroupID}}& alert_id={{.AlertID}}{% endraw %} |
2022-12-12 19:16:10 +00:00
| $expr or .Expr | Alert's expression. Can be used for generating links to Grafana or other systems. | {% raw %}/api/v1/query?query={{ $expr| queryEscape }}{% endraw %} |
| $for or .For | Alert's configured for param. | {% raw %}Number of connections is too high for more than {{ .For }}{% endraw %} |
2022-08-22 11:49:17 +00:00
| $externalLabels or .ExternalLabels | List of labels configured via `-external.label` command-line flag. | {% raw %}Issues with {{ $labels.instance }} (datacenter-{{ $externalLabels.dc }}){% endraw %} |
| $externalURL or .ExternalURL | URL configured via `-external.url` command-line flag. Used for cases when vmalert is hidden behind proxy. | {% raw %}Visit {{ $externalURL }} for more details{% endraw %} |
2022-08-16 06:08:27 +00:00
app/vmalert/templates: properly escape all the special chars in `quotesEscape` function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
complications in user templates, while not fixing various corner cases
such as backslash chars in the input string.
See https://github.com/VictoriaMetrics/VictoriaMetrics/commit/1de15ad490dbde84ad2a657f3b65a6311991f372
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
was "fixed" by urlencoding the whole string passed to -external.alert.source
command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
See https://github.com/VictoriaMetrics/VictoriaMetrics/commit/00c838353d1246495fd7c7546f3d71095e855eab
and https://github.com/VictoriaMetrics/VictoriaMetrics/commit/4bd024459931a0671dee4abae4bc3556795ee398
This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.
This commit deprecates crlfEscape template function and adds the following new template functions:
- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
and for html-escaping the input string, so it could be safely embedded as a plaintext
into html.
This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-27 20:38:19 +00:00
Additionally, `vmalert` provides some extra templating functions listed [here ](#template-functions ) and [reusable templates ](#reusable-templates ).
#### Template functions
`vmalert` provides the following template functions, which can be used during [templating ](#templating ):
- `args arg0 ... argN` - converts the input args into a map with `arg0` , ..., `argN` keys.
- `externalURL` - returns the value of `-external.url` command-line flag.
- `first` - returns the first result from the input query results returned by `query` function.
- `htmlEscape` - escapes special chars in input string, so it can be safely embedded as a plaintext into HTML.
- `humanize` - converts the input number into human-readable format by adding [metric prefixes ](https://en.wikipedia.org/wiki/Metric_prefix ).
For example, `100000` is converted into `100K` .
- `humanize1024` - converts the input number into human-readable format with 1024 base.
For example, `1024` is converted into 1ki`.
- `humanizeDuration` - converts the input number in seconds into human-readable duration.
- `humanizePercentage` - converts the input number to percentage. For example, `0.123` is converted into `12.3%` .
- `humanizeTimestamp` - converts the input unix timestamp into human-readable time.
- `jsonEscape` - JSON-encodes the input string.
- `label name` - returns the value of the label with the given `name` from the input query result.
- `match regex` - matches the input string against the provided `regex` .
- `parseDuration` - parses the input string into duration in seconds. For example, `1h` is parsed into `3600` .
- `parseDurationTime` - parses the input string into [time.Duration ](https://pkg.go.dev/time#Duration ).
- `pathEscape` - escapes the input string, so it can be safely put inside path part of URL.
- `pathPrefix` - returns the path part of the `-external.url` command-line flag.
- `query` - executes the [MetricsQL ](https://docs.victoriametrics.com/MetricsQL.html ) query against `-datasource.url` and returns the query result.
For example, {% raw %}`{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}`{% endraw %} executes the `sort_desc(process_resident_memory_bytes)`
query at `-datasource.url` and returns the first result.
- `queryEscape` - escapes the input string, so it can be safely put inside [query arg ](https://en.wikipedia.org/wiki/Percent-encoding ) part of URL.
- `quotesEscape` - escapes the input string, so it can be safely embedded into JSON string.
2023-05-10 07:50:41 +00:00
- `reReplaceAll regex repl` - replaces all the occurrences of the `regex` in input string with the `repl` .
app/vmalert/templates: properly escape all the special chars in `quotesEscape` function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
complications in user templates, while not fixing various corner cases
such as backslash chars in the input string.
See https://github.com/VictoriaMetrics/VictoriaMetrics/commit/1de15ad490dbde84ad2a657f3b65a6311991f372
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
was "fixed" by urlencoding the whole string passed to -external.alert.source
command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
See https://github.com/VictoriaMetrics/VictoriaMetrics/commit/00c838353d1246495fd7c7546f3d71095e855eab
and https://github.com/VictoriaMetrics/VictoriaMetrics/commit/4bd024459931a0671dee4abae4bc3556795ee398
This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.
This commit deprecates crlfEscape template function and adds the following new template functions:
- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
and for html-escaping the input string, so it could be safely embedded as a plaintext
into html.
This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-27 20:38:19 +00:00
- `safeHtml` - marks the input string as safe to use in HTML context without the need to html-escape it.
- `sortByLabel name` - sorts the input query results by the label with the given `name` .
- `stripDomain` - leaves the first part of the domain. For example, `foo.bar.baz` is converted to `foo` .
The port part is left in the output string. E.g. `foo.bar:1234` is converted into `foo:1234` .
- `stripPort` - strips `port` part from `host:port` input string.
- `strvalue` - returns the metric name from the input query result.
- `title` - converts the first letters of every input word to uppercase.
- `toLower` - converts all the chars in the input string to lowercase.
- `toTime` - converts the input unix timestamp to [time.Time ](https://pkg.go.dev/time#Time ).
- `toUpper` - converts all the chars in the input string to uppercase.
- `value` - returns the numeric value from the input query result.
2022-05-14 09:38:44 +00:00
#### Reusable templates
2022-05-14 16:56:31 +00:00
Like in Alertmanager you can define [reusable templates ](https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#defining-reusable-templates )
2022-09-28 07:15:05 +00:00
to share same templates across annotations. Just define the templates in a file and
2022-05-14 16:56:31 +00:00
set the path via `-rule.templates` flag.
2022-05-14 17:29:09 +00:00
For example, template `grafana.filter` can be defined as following:
2022-05-14 09:38:44 +00:00
2022-05-16 07:27:19 +00:00
{% raw %}
2022-05-14 09:38:44 +00:00
```
{{ define "grafana.filter" -}}
{{- $labels := .arg0 -}}
{{- range $name, $label := . -}}
{{- if (ne $name "arg0") -}}
{{- ( or (index $labels $label) "All" ) | printf "& var-%s=%s" $label -}}
{{- end -}}
{{- end -}}
{{- end -}}
```
2022-05-16 07:27:19 +00:00
{% endraw %}
2022-05-14 09:38:44 +00:00
2022-05-14 17:29:09 +00:00
And then used in annotations:
2022-05-14 09:38:44 +00:00
2022-05-16 07:27:19 +00:00
{% raw %}
2022-05-14 09:38:44 +00:00
```yaml
groups:
- name: AlertGroupName
rules:
- alert: AlertName
expr: any_metric > 100
for: 30s
labels:
alertname: 'Any metric is too high'
severity: 'warning'
annotations:
dashboard: '{{ $externalURL }}/d/dashboard?orgId=1{{ template "grafana.filter" (args .CommonLabels "account_id" "any_label") }}'
```
2022-05-16 07:27:19 +00:00
{% endraw %}
2022-05-14 09:38:44 +00:00
2022-05-14 16:56:31 +00:00
The `-rule.templates` flag supports wildcards so multiple files with templates can be loaded.
The content of `-rule.templates` can be also [hot reloaded ](#hot-config-reload ).
2022-05-14 09:38:44 +00:00
2021-04-10 12:46:08 +00:00
2021-04-07 10:39:16 +00:00
#### Recording rules
2020-06-10 16:37:43 +00:00
The syntax for recording rules is following:
2022-03-22 11:40:55 +00:00
2020-06-10 16:37:43 +00:00
```yaml
# The name of the time series to output to. Must be a valid metric name.
record: < string >
2021-02-01 13:26:33 +00:00
# The expression to evaluate. The expression language depends on the type value.
2022-08-08 07:45:21 +00:00
# By default, MetricsQL expression is used. If group.type="graphite", then the expression
2021-02-01 13:26:33 +00:00
# must contain valid Graphite expression.
2020-06-10 16:37:43 +00:00
expr: < string >
# Labels to add or overwrite before storing the result.
labels:
[ < labelname > : < labelvalue > ]
2022-12-29 11:36:44 +00:00
# Defines the number of rule's updates entries stored in memory
# and available for view on rule's Details page.
# Overrides `rule.updateEntriesLimit` value for this specific rule.
[ update_entries_limit: < integer > | default 0 ]
2020-06-10 16:37:43 +00:00
```
2021-06-14 08:37:26 +00:00
For recording rules to work `-remoteWrite.url` must be specified.
2020-06-10 16:37:43 +00:00
2021-04-07 10:39:16 +00:00
### Alerts state on restarts
2020-10-13 15:34:25 +00:00
2023-07-31 14:47:50 +00:00
`vmalert` holds alerts state in the memory. Restart of the `vmalert` process will reset the state of all active alerts
in the memory. To prevent `vmalert` from losing the state on restarts configure it to persist the state
to the remote database via the following flags:
2022-03-22 11:40:55 +00:00
2021-04-20 08:42:17 +00:00
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
2023-07-27 09:29:17 +00:00
to the configured address in the form of [time series ](https://docs.victoriametrics.com/keyConcepts.html#time-series )
`ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
These time series can be queried from VictoriaMetrics just as any other time series.
The state will be persisted to the configured address on each evaluation.
2021-04-20 08:42:17 +00:00
* `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
2023-07-27 09:29:17 +00:00
from the configured address by querying time series with name `ALERTS_FOR_STATE` . The restore happens only once when
`vmalert` process starts, and only for the configured rules. Config [hot reload ](#hot-config-reload ) doesn't trigger
state restore.
2020-10-13 15:34:25 +00:00
2021-12-01 15:33:06 +00:00
Both flags are required for proper state restoration. Restore process may fail if time series are missing
2021-08-16 11:20:57 +00:00
in configured `-remoteRead.url` , weren't updated in the last `1h` (controlled by `-remoteRead.lookback` )
2023-07-27 09:29:17 +00:00
or received state doesn't match current `vmalert` rules configuration. `vmalert` marks successfully restored rules
2023-07-28 01:05:01 +00:00
with `restored` label in [web UI ](#web ).
2020-10-13 15:34:25 +00:00
2021-05-18 13:22:08 +00:00
### Multitenancy
2021-12-01 15:33:06 +00:00
There are the following approaches exist for alerting and recording rules across
2021-06-14 08:37:26 +00:00
[multiple tenants ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy ):
2021-05-18 13:22:08 +00:00
2021-08-16 11:20:57 +00:00
* To run a separate `vmalert` instance per each tenant.
The corresponding tenant must be specified in `-datasource.url` command-line flag
according to [these docs ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format ).
2021-06-14 08:37:26 +00:00
For example, `/path/to/vmalert -datasource.url=http://vmselect:8481/select/123/prometheus`
2021-08-16 11:20:57 +00:00
would run alerts against `AccountID=123` . For recording rules the `-remoteWrite.url` command-line
flag must contain the url for the specific tenant as well.
For example, `-remoteWrite.url=http://vminsert:8480/insert/123/prometheus` would write recording
2021-06-14 08:37:26 +00:00
rules to `AccountID=123` .
2021-05-18 13:22:08 +00:00
2021-08-16 11:20:57 +00:00
* To specify `tenant` parameter per each alerting and recording group if
2022-10-24 15:00:25 +00:00
[enterprise version of vmalert ](https://docs.victoriametrics.com/enterprise.html ) is used
2021-06-14 08:37:26 +00:00
with `-clusterMode` command-line flag. For example:
2021-05-18 13:22:08 +00:00
```yaml
groups:
- name: rules_for_tenant_123
tenant: "123"
rules:
# Rules for accountID=123
- name: rules_for_tenant_456:789
tenant: "456:789"
rules:
# Rules for accountID=456, projectID=789
```
2022-10-05 20:57:29 +00:00
The results of alerting and recording rules contain `vm_account_id` and `vm_project_id` labels
2022-10-06 12:03:45 +00:00
if `-clusterMode` is enabled. These labels can be used during [templating ](https://docs.victoriametrics.com/vmalert.html#templating ),
and help to identify to which account or project the triggered alert or produced recording belongs.
2022-10-05 20:57:29 +00:00
2021-08-16 11:20:57 +00:00
If `-clusterMode` is enabled, then `-datasource.url` , `-remoteRead.url` and `-remoteWrite.url` must
contain only the hostname without tenant id. For example: `-datasource.url=http://vmselect:8481` .
2021-09-10 14:57:14 +00:00
`vmalert` automatically adds the specified tenant to urls per each recording rule in this case.
2021-05-18 13:22:08 +00:00
2021-11-05 18:04:09 +00:00
If `-clusterMode` is enabled and the `tenant` in a particular group is missing, then the tenant value
is obtained from `-defaultTenant.prometheus` or `-defaultTenant.graphite` depending on the `type` of the group.
2021-08-16 11:20:57 +00:00
The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` files
2023-10-18 18:06:23 +00:00
at [release page ](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest ) and in `*-enterprise`
2021-06-14 08:37:26 +00:00
tags at [Docker Hub ](https://hub.docker.com/r/victoriametrics/vmalert/tags ).
2021-05-18 13:22:08 +00:00
2023-02-20 15:39:31 +00:00
### Reading rules from object storage
[Enterprise version ](https://docs.victoriametrics.com/enterprise.html ) of `vmalert` may read alerting and recording rules
from object storage:
- `./bin/vmalert -rule=s3://bucket/dir/alert.rules` would read rules from the given path at S3 bucket
- `./bin/vmalert -rule=gs://bucket/bir/alert.rules` would read rules from the given path at GCS bucket
S3 and GCS paths support only matching by prefix, e.g. `s3://bucket/dir/rule_` matches
all files with prefix `rule_` in the folder `dir` .
The following [command-line flags ](#flags ) can be used for fine-tuning access to S3 and GCS:
- `-s3.credsFilePath` - path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
- `-s3.configFilePath` - path to file with S3 configs. Configs are loaded from default location if not set.
- `-s3.configProfile` - profile name for S3 configs. If no set, the value of the environment variable will be loaded (`AWS_PROFILE` or `AWS_DEFAULT_PROFILE` ).
- `-s3.customEndpoint` - custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set.
- `-s3.forcePathStyle` - prefixing endpoint with bucket name when set false, true by default.
2021-12-01 15:33:06 +00:00
### Topology examples
2022-03-22 11:40:55 +00:00
The following sections are showing how `vmalert` may be used and configured
for different scenarios.
Please note, not all flags in examples are required:
2021-12-01 15:33:06 +00:00
* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if
2022-05-14 16:56:31 +00:00
you have recording rules or want to store [alerts state ](#alerts-state-on-restarts ) on `vmalert` restarts;
2021-12-01 15:33:06 +00:00
* `-notifier.url` is optional and is needed only if you have alerting rules.
#### Single-node VictoriaMetrics
2021-12-02 08:27:46 +00:00
The simplest configuration where one single-node VM server is used for
rules execution, storing recording rules results and alerts state.
2021-12-01 15:33:06 +00:00
`vmalert` configuration flags:
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
```
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
```
2021-12-02 08:27:46 +00:00
< img alt = "vmalert single" width = "500" src = "vmalert_single.png" >
2021-12-01 15:33:06 +00:00
#### Cluster VictoriaMetrics
2021-12-02 08:27:46 +00:00
In [cluster mode ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html )
VictoriaMetrics has separate components for writing and reading path:
`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions
and `vminsert` is used to persist recording rules results and alerts state.
2022-03-22 11:40:55 +00:00
Cluster mode could have multiple `vminsert` and `vmselect` components.
2021-12-01 15:33:06 +00:00
`vmalert` configuration flags:
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
```
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
2022-03-05 20:35:10 +00:00
-remoteWrite.url=http://vminsert:8480/insert/0/prometheus # vminsert addr to persist alerts state and recording rules results
2021-12-01 15:33:06 +00:00
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
```
2021-12-02 08:27:46 +00:00
< img alt = "vmalert cluster" src = "vmalert_cluster.png" >
In case when you want to spread the load on these components - add balancers before them and configure
2023-05-10 07:50:41 +00:00
`vmalert` with balancer addresses. Please, see more about VM's cluster architecture
2021-12-01 15:33:06 +00:00
[here ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview ).
2021-12-02 08:27:46 +00:00
#### HA vmalert
2021-12-01 15:33:06 +00:00
2021-12-02 08:27:46 +00:00
For HA user can run multiple identically configured `vmalert` instances.
It means all of them will execute the same rules, write state and results to
the same destinations, and send alert notifications to multiple configured
Alertmanagers.
2021-12-01 15:33:06 +00:00
`vmalert` configuration flags:
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
```
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://victoriametrics:8428 \ # VM-single addr for executing rules expressions
-remoteWrite.url=http://victoriametrics:8428 \ # VM-single addr to persist alerts state and recording rules results
-remoteRead.url=http://victoriametrics:8428 \ # VM-single addr for restoring alerts state after restart
-notifier.url=http://alertmanager1:9093 \ # Multiple AlertManager addresses to send alerts when they trigger
-notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers
```
2021-12-02 08:27:46 +00:00
< img alt = "vmalert ha" width = "800px" src = "vmalert_ha.png" >
2021-12-01 15:33:06 +00:00
To avoid recording rules results and alerts state duplication in VictoriaMetrics server
don't forget to configure [deduplication ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication ).
2023-10-01 18:32:53 +00:00
The recommended value for `-dedup.minScrapeInterval` must be multiple of vmalert's `-evaluationInterval` .
2021-12-01 15:33:06 +00:00
Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
all `vmalert` s are having the same config.
Don't forget to configure [cluster mode ](https://prometheus.io/docs/alerting/latest/alertmanager/ )
2023-05-10 07:50:41 +00:00
for Alertmanagers for better reliability. List all Alertmanager URLs in vmalert `-notifier.url`
2022-12-30 08:54:03 +00:00
to ensure [high availability ](https://github.com/prometheus/alertmanager#high-availability ).
2021-12-01 15:33:06 +00:00
2022-03-22 11:40:55 +00:00
This example uses single-node VM server for the sake of simplicity.
Check how to replace it with [cluster VictoriaMetrics ](#cluster-victoriametrics ) if needed.
2021-12-02 08:27:46 +00:00
2021-12-01 15:33:06 +00:00
#### Downsampling and aggregation via vmalert
2023-06-09 11:47:54 +00:00
_Please note, [stream aggregation ](https://docs.victoriametrics.com/stream-aggregation.html ) might be more efficient
for cases when downsampling or aggregation need to be applied **before data gets into the TSDB.**_
2022-06-22 06:53:54 +00:00
`vmalert` can't modify existing data. But it can run arbitrary PromQL/MetricsQL queries
via [recording rules ](#recording-rules ) and backfill results to the configured `-remoteWrite.url` .
This ability allows to aggregate data. For example, the following rule will calculate the average value for
metric `http_requests` on the `5m` interval:
```yaml
2022-09-28 07:15:05 +00:00
- record: http_requests:avg5m
expr: avg_over_time(http_requests[5m])
2022-06-22 06:53:54 +00:00
```
Every time this rule will be evaluated, `vmalert` will backfill its results as a new time series `http_requests:avg5m`
to the configured `-remoteWrite.url` .
`vmalert` executes rules with specified interval (configured via flag `-evaluationInterval`
or as [group's ](#groups ) `interval` param). The interval helps to control "resolution" of the produced series.
This ability allows to downsample data. For example, the following config will execute the rule only once every `5m` :
```yaml
groups:
- name: my_group
interval: 5m
2022-09-28 07:15:05 +00:00
rules:
2022-06-22 06:53:54 +00:00
- record: http_requests:avg5m
2022-09-28 07:15:05 +00:00
expr: avg_over_time(http_requests[5m])
2022-06-22 06:53:54 +00:00
```
2023-01-04 06:19:18 +00:00
Ability of `vmalert` to be configured with different `-datasource.url` and `-remoteWrite.url` command-line flags
allows reading data from one data source and backfilling results to another. This helps to build a system
2022-06-22 06:53:54 +00:00
for aggregating and downsampling the data.
2022-02-04 09:56:52 +00:00
The following example shows how to build a topology where `vmalert` will process data from one cluster
2021-12-02 08:27:46 +00:00
and write results into another. Such clusters may be called as "hot" (low retention,
high-speed disks, used for operative monitoring) and "cold" (long term retention,
slower/cheaper disks, low resolution data). With help of `vmalert` , user can setup
recording rules to process raw data from "hot" cluster (by applying additional transformations
or reducing resolution) and push results to "cold" cluster.
2021-12-01 15:33:06 +00:00
`vmalert` configuration flags:
2022-03-22 11:40:55 +00:00
2021-12-01 15:33:06 +00:00
```
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
2022-09-28 07:15:05 +00:00
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recording rules expressions
2022-03-05 20:35:10 +00:00
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results
2021-12-01 15:33:06 +00:00
```
2021-12-02 08:27:46 +00:00
< img alt = "vmalert multi cluster" src = "vmalert_multicluster.png" >
2021-12-01 15:33:06 +00:00
Please note, [replay ](#rules-backfilling ) feature may be used for transforming historical data.
Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used.
2021-05-18 13:22:08 +00:00
2023-01-04 06:19:18 +00:00
See also [stream aggregation ](https://docs.victoriametrics.com/stream-aggregation.html ) and [downsampling ](https://docs.victoriametrics.com/#downsampling ).
2022-02-04 09:56:52 +00:00
2022-06-16 11:30:28 +00:00
#### Multiple remote writes
For persisting recording or alerting rule results `vmalert` requires `-remoteWrite.url` to be set.
But this flag supports only one destination. To persist rule results to multiple destinations
we recommend using [vmagent ](https://docs.victoriametrics.com/vmagent.html ) as fan-out proxy:
< img alt = "vmalert multiple remote write destinations" src = "vmalert_multiple_rw.png" >
In this topology, `vmalert` is configured to persist rule results to `vmagent` . And `vmagent`
is configured to fan-out received data to two or more destinations.
2022-09-28 07:15:05 +00:00
Using `vmagent` as a proxy provides additional benefits such as
2022-06-16 11:30:28 +00:00
[data persisting when storage is unreachable ](https://docs.victoriametrics.com/vmagent.html#replication-and-high-availability ),
or time series modification via [relabeling ](https://docs.victoriametrics.com/vmagent.html#relabeling ).
2021-11-05 18:44:02 +00:00
### Web
2020-06-10 16:37:43 +00:00
`vmalert` runs a web-server (`-httpListenAddr`) for serving metrics and alerts endpoints:
2022-03-22 11:40:55 +00:00
2021-09-09 13:16:25 +00:00
* `http://<vmalert-addr>` - UI;
2022-03-28 10:07:02 +00:00
* `http://<vmalert-addr>/api/v1/rules` - list of all loaded groups and rules;
2020-04-29 14:42:01 +00:00
* `http://<vmalert-addr>/api/v1/alerts` - list of all active alerts;
2022-08-17 18:24:00 +00:00
* `http://<vmalert-addr>/vmalert/api/v1/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in JSON format.
2022-05-14 16:56:31 +00:00
Used as alert source in AlertManager.
2022-08-17 18:24:00 +00:00
* `http://<vmalert-addr>/vmalert/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in web UI.
2022-09-14 12:04:24 +00:00
* `http://<vmalert-addr>/vmalert/rule?group_id=<group_id>&rule_id=<rule_id>` - get rule status in web UI.
2020-04-29 14:42:01 +00:00
* `http://<vmalert-addr>/metrics` - application metrics.
2020-05-13 19:55:35 +00:00
* `http://<vmalert-addr>/-/reload` - hot configuration reload.
2020-04-29 14:42:01 +00:00
2022-08-04 14:56:07 +00:00
`vmalert` web UI can be accessed from [single-node version of VictoriaMetrics ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html )
and from [cluster version of VictoriaMetrics ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html ).
2023-05-10 07:50:41 +00:00
This may be used for better integration with Grafana unified alerting system. See the following docs for details:
2022-08-04 14:56:07 +00:00
* [How to query vmalert from single-node VictoriaMetrics ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmalert )
* [How to query vmalert from VictoriaMetrics cluster ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmalert )
2021-04-07 10:39:16 +00:00
## Graphite
2021-02-01 13:26:33 +00:00
vmalert sends requests to `<-datasource.url>/render?format=json` during evaluation of alerting and recording rules
2021-02-03 19:12:14 +00:00
if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render`
2021-02-01 13:26:33 +00:00
implements [Graphite Render API ](https://graphite.readthedocs.io/en/stable/render_api.html ) for `format=json` .
2021-02-03 21:44:37 +00:00
When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget
2021-12-01 15:33:06 +00:00
to set `-datasource.appendTypePrefix` flag to `true` , so vmalert can adjust URL prefix automatically based on the query type.
2021-02-01 13:26:33 +00:00
2021-06-09 09:27:31 +00:00
## Rules backfilling
vmalert supports alerting and recording rules backfilling (aka `replay` ). In replay mode vmalert
2021-11-05 18:44:02 +00:00
can read the same rules configuration as normal, evaluate them on the given time range and backfill
2021-06-09 09:27:31 +00:00
results via remote write to the configured storage. vmalert supports any PromQL/MetricsQL compatible
data source for backfilling.
2023-03-10 12:45:11 +00:00
Please note, that response caching may lead to unexpected results during and after backfilling process.
In order to avoid this you need to reset cache contents or disable caching when using backfilling
as described in [backfilling docs ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#backfilling ).
2023-02-21 21:16:15 +00:00
See a blogpost about [Rules backfilling via vmalert ](https://victoriametrics.com/blog/rules-replay/ ).
2021-06-09 09:27:31 +00:00
### How it works
In `replay` mode vmalert works as a cli-tool and exits immediately after work is done.
To run vmalert in `replay` mode:
2022-03-22 11:40:55 +00:00
2021-06-09 09:27:31 +00:00
```
./bin/vmalert -rule=path/to/your.rules \ # path to files with rules you usually use with vmalert
2022-08-08 07:45:21 +00:00
-datasource.url=http://localhost:8428 \ # Prometheus HTTP API compatible datasource
2021-06-09 09:27:31 +00:00
-remoteWrite.url=http://localhost:8428 \ # remote write compatible storage to persist results
-replay.timeFrom=2021-05-11T07:21:43Z \ # time from begin replay
-replay.timeTo=2021-05-29T18:40:43Z # time to finish replay
```
The output of the command will look like the following:
2022-03-22 11:40:55 +00:00
2021-06-09 09:27:31 +00:00
```
Replay mode:
from: 2021-05-11 07:21:43 +0000 UTC # set by -replay.timeFrom
to: 2021-05-29 18:40:43 +0000 UTC # set by -replay.timeTo
max data points per request: 1000 # set by -replay.maxDatapointsPerQuery
Group "ReplayGroup"
interval: 1m0s
requests to make: 27
max range per request: 16h40m0s
> Rule "type:vm_cache_entries:rate5m" (ID: 1792509946081842725)
27 / 27 [----------------------------------------------------------------------------------------------------] 100.00% 78 p/s
> Rule "go_cgo_calls_count:rate5m" (ID: 17958425467471411582)
27 / 27 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
Group "vmsingleReplay"
interval: 30s
requests to make: 54
max range per request: 8h20m0s
> Rule "RequestErrorsToAPI" (ID: 17645863024999990222)
54 / 54 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
> Rule "TooManyLogs" (ID: 9042195394653477652)
54 / 54 [-----------------------------------------------------------------------------------------------------] 100.00% ? p/s
2021-06-07T09:59:12.098Z info app/vmalert/replay.go:68 replay finished! Imported 511734 samples
```
In `replay` mode all groups are executed sequentially one-by-one. Rules within the group are
executed sequentially as well (`concurrency` setting is ignored). Vmalert sends rule's expression
2022-06-19 20:12:10 +00:00
to [/query_range ](https://docs.victoriametrics.com/keyConcepts.html#range-query ) endpoint
2021-11-05 18:44:02 +00:00
of the configured `-datasource.url` . Returned data is then processed according to the rule type and
backfilled to `-remoteWrite.url` via [remote Write protocol ](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations ).
2021-06-09 09:27:31 +00:00
Vmalert respects `evaluationInterval` value set by flag or per-group during the replay.
2021-09-01 09:02:28 +00:00
Vmalert automatically disables caching on VictoriaMetrics side by sending `nocache=1` param. It allows
to prevent cache pollution and unwanted time range boundaries adjustment during backfilling.
2021-06-09 09:27:31 +00:00
#### Recording rules
2021-12-01 15:33:06 +00:00
The result of recording rules `replay` should match with results of normal rules evaluation.
2021-06-09 09:27:31 +00:00
#### Alerting rules
2021-12-01 15:33:06 +00:00
The result of alerting rules `replay` is time series reflecting [alert's state ](#alerts-state-on-restarts ).
2021-06-09 09:27:31 +00:00
To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression:
2022-03-22 11:40:55 +00:00
2021-06-09 09:27:31 +00:00
```
ALERTS{alertname="your_alertname", alertstate="firing"}
```
2022-03-22 11:40:55 +00:00
2021-06-09 09:27:31 +00:00
Execute the query against storage which was used for `-remoteWrite.url` during the `replay` .
### Additional configuration
There are following non-required `replay` flags:
* `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request.
2022-05-14 16:56:31 +00:00
In two words, it affects the max time range for every `/query_range` request. The higher the value,
the fewer requests will be issued during `replay` .
2021-06-09 09:27:31 +00:00
* `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries
2022-05-14 16:56:31 +00:00
per rule before giving up.
2021-06-09 09:27:31 +00:00
* `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining
2022-05-14 16:56:31 +00:00
(rules which depend on each other) rules. It is expected, that remote storage will be able to persist
previously accepted data during the delay, so data will be available for the subsequent queries.
Keep it equal or bigger than `-remoteWrite.flushInterval` .
2022-05-30 06:16:07 +00:00
* `-replay.disableProgressBar` - whether to disable progress bar which shows progress work.
2022-05-14 16:56:31 +00:00
Progress bar may generate a lot of log records, which is not formatted as standard VictoriaMetrics logger.
It could break logs parsing by external system and generate additional load on it.
2021-06-09 09:27:31 +00:00
2022-09-26 05:36:22 +00:00
See full description for these flags in `./vmalert -help` .
2021-06-09 09:27:31 +00:00
### Limitations
* Graphite engine isn't supported yet;
* `query` template function is disabled for performance reasons (might be changed in future);
2022-06-09 06:58:25 +00:00
* `limit` group's param has no effect during replay (might be changed in future);
2023-07-27 13:13:13 +00:00
* `keep_firing_for` alerting rule param has no effect during replay (might be changed in future).
2021-06-09 09:27:31 +00:00
2023-10-13 11:54:33 +00:00
## Unit Testing for Rules
You can use `vmalert-tool` to test your alerting and recording rules like [promtool does ](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/ ).
See more details [here ](https://docs.victoriametrics.com/vmalert-tool.html#Unit-testing-for-rules ).
2021-09-01 09:02:28 +00:00
## Monitoring
2021-10-18 07:24:52 +00:00
`vmalert` exports various metrics in Prometheus exposition format at `http://vmalert-host:8880/metrics` page.
2022-11-21 13:45:45 +00:00
The default list of alerting rules for these metric can be found [here ](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker ).
2021-10-18 07:24:52 +00:00
We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported
2021-09-01 09:02:28 +00:00
metrics may be analyzed later.
2022-12-25 16:36:20 +00:00
Use the official [Grafana dashboard ](https://grafana.com/grafana/dashboards/14950-victoriametrics-vmalert/ ) for `vmalert` overview.
2022-11-21 13:45:45 +00:00
Graphs on this dashboard contain useful hints - hover the `i` icon in the top left corner of each graph in order to read it.
2021-10-18 07:24:52 +00:00
If you have suggestions for improvements or have found a bug - please open an issue on github or add
2021-09-01 09:02:28 +00:00
a review to the dashboard.
2022-09-15 14:15:39 +00:00
## Troubleshooting
2023-02-13 16:29:30 +00:00
### Data delay
Data delay is one of the most common issues with rules execution.
vmalert executes configured rules within certain intervals at specifics timestamps.
It expects that the data is already present in configured `-datasource.url` at the moment of time when rule is executed:
2022-09-15 14:15:39 +00:00
< img alt = "vmalert expected evaluation" src = "vmalert_ts_normal.gif" >
Usually, troubles start to appear when data in `-datasource.url` is delayed or absent. In such cases, evaluations
2023-02-13 16:29:30 +00:00
may get empty response from the datasource and produce empty recording rules or reset alerts state:
2022-09-15 14:15:39 +00:00
< img alt = "vmalert evaluation when data is delayed" src = "vmalert_ts_data_delay.gif" >
2023-02-13 16:29:30 +00:00
Try the following recommendations to reduce the chance of hitting the data delay issue:
2022-09-15 14:15:39 +00:00
2023-10-01 18:32:53 +00:00
* Always configure group's `-evaluationInterval` to be bigger or at least equal to
2023-02-13 16:29:30 +00:00
[time series resolution ](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution );
* Ensure that `[duration]` value is at least twice bigger than
[time series resolution ](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution ). For example,
if expression is `rate(my_metric[2m]) > 0` then ensure that `my_metric` resolution is at least `1m` or better `30s` .
If you use VictoriaMetrics as datasource, `[duration]` can be omitted and VictoriaMetrics will adjust it automatically.
2023-10-25 09:54:18 +00:00
* Extend `[duration]` in expr to help tolerate the delay. For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m` .
2023-02-13 16:29:30 +00:00
* If [time series resolution ](https://docs.victoriametrics.com/keyConcepts.html#time-series-resolution )
2023-05-10 07:50:41 +00:00
in datasource is inconsistent or `>=5min` - try changing vmalerts `-datasource.queryStep` command-line flag to specify
2023-02-13 16:29:30 +00:00
how far search query can lookback for the recent datapoint. The recommendation is to have the step
at least two times bigger than the resolution.
> Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring.
2023-10-25 11:07:13 +00:00
By default, recently written samples to VictoriaMetrics [aren't visible for queries ](https://docs.victoriametrics.com/keyConcepts.html#query-latency )
for up to 30s (see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of
incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time.
2023-10-25 09:54:18 +00:00
To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to 30s by default.
2023-10-25 11:07:13 +00:00
If you changed the `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect) value
or specified custom `latency_offset` param via [Group ](#groups ) and observed a delay in timestamps for produced
evaluation results - try changing `-rule.evalDelay` equal to `-search.latencyOffset` .
2023-02-13 16:29:30 +00:00
### Alerts state
2022-09-15 14:15:39 +00:00
Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
alerts with `for: 0` fire immediately when their expression becomes true. And alerts with `for > 0` will fire only
after multiple consecutive evaluations, and at each evaluation their expression must be true. If at least one evaluation
becomes false, then alert's state resets to the initial state.
If `-remoteWrite.url` command-line flag is configured, vmalert will persist alert's state in form of time series
2022-09-28 07:15:05 +00:00
`ALERTS` and `ALERTS_FOR_STATE` to the specified destination. Such time series can be then queried via
2022-09-15 14:15:39 +00:00
[vmui ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui ) or Grafana to track how alerts state
changed in time.
2023-01-04 06:40:00 +00:00
vmalert stores last `-rule.updateEntriesLimit` (or `update_entries_limit` [per-rule config ](https://docs.victoriametrics.com/vmalert.html#alerting-rules ))
2023-06-21 07:44:00 +00:00
state updates for each rule starting from [v1.86 ](https://docs.victoriametrics.com/CHANGELOG.html#v1860 ).
To check updates, click on `Details` link next to rule's name on `/vmalert/groups` page
2022-12-29 11:36:44 +00:00
and check the `Last updates` section:
2022-09-15 14:15:39 +00:00
< img alt = "vmalert state" src = "vmalert_state.png" >
2022-09-28 07:15:05 +00:00
Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of
2022-09-15 14:15:39 +00:00
HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were
2022-09-28 07:15:05 +00:00
no samples returned and curl command returns data - then it is very likely there was no data in datasource on the
2023-10-10 09:40:27 +00:00
moment when rule was evaluated. Sensitive info is stripped from the `curl` examples - see [security ](#security ) section
for more details.
2022-09-15 14:15:39 +00:00
2023-02-13 16:29:30 +00:00
### Debug mode
2023-06-21 07:44:00 +00:00
vmalert allows configuring more detailed logging for specific alerting rule starting from [v1.82 ](https://docs.victoriametrics.com/CHANGELOG.html#v1820 ).
Just set `debug: true` in rule's configuration and vmalert will start printing additional log messages:
2022-09-15 14:15:39 +00:00
```terminal
2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms)
2022-09-15T13:35:56.149Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true& query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29+by%28instance%29+%3E+0& step=15s& time=1663248945"
2022-09-15T13:35:56.178Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: query returned 1 samples (elapsed: 28.368208ms)
2022-09-15T13:35:56.178Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true& query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29& step=15s& time=1663248945"
2022-09-15T13:35:56.179Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} created in state PENDING
...
2022-09-15T13:36:56.153Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
```
2023-10-10 09:40:27 +00:00
Sensitive info is stripped from the `curl` examples - see [security ](#security ) section for more details.
2023-05-08 07:36:39 +00:00
### Never-firing alerts
2023-06-21 07:44:00 +00:00
vmalert can detect if alert's expression doesn't match any time series in runtime
starting from [v1.91 ](https://docs.victoriametrics.com/CHANGELOG.html#v1910 ). This problem usually happens
2023-05-08 07:36:39 +00:00
when alerting expression selects time series which aren't present in the datasource (i.e. wrong `job` label)
or there is a typo in the series selector (i.e. `env=rpod` ). Such alerting rules will be marked with special icon in
2023-05-10 07:50:41 +00:00
vmalerts UI and exposed via `vmalert_alerting_rules_last_evaluation_series_fetched` metric. The metric value will
show how many time series were matched before the filtering by rule's expression. If metric value is `-1` , then
2023-05-08 07:36:39 +00:00
this feature is not supported by the datasource (old versions of VictoriaMetrics). The following expression can be
used to detect rules matching no series:
```
max(vmalert_alerting_rules_last_evaluation_series_fetched) by(group, alertname) == 0
```
See more details [here ](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039 ).
This feature is available only if vmalert is using VictoriaMetrics v1.90 or higher as a datasource.
2023-06-13 15:03:53 +00:00
### Series with the same labelset
vmalert can produce the following error message during rules evaluation:
```
result contains metrics with the same labelset after applying rule labels
```
The error means there is a collision between [time series ](https://docs.victoriametrics.com/keyConcepts.html#time-series )
after applying extra labels to result.
For example, a rule with `expr: foo > 0` returns two distinct time series in response:
```
foo{bar="baz"} 1
foo{bar="qux"} 2
```
If user configures `-external.label=bar=baz` cmd-line flag to enforce
adding `bar="baz"` label-value pair, then time series won't be distinct anymore:
```
foo{bar="baz"} 1
foo{bar="baz"} 2 # 'bar' label was overriden by `-external.label=bar=baz
```
The same issue can be caused by collision of configured `labels` on [Group ](#groups ) or [Rule ](#rules ) levels.
To fix it one should avoid collisions by carefully picking label overrides in configuration.
2022-09-15 14:15:39 +00:00
2023-10-10 09:40:27 +00:00
## Security
See general recommendations regarding security [here ](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#security ).
vmalert [web UI ](#web ) exposes configuration details such as list of [Groups ](#groups ), active alerts,
[alerts state ](#alerts-state ), [notifiers ](#notifier-configuration-file ). Notifier addresses (sanitized) are attached
as labels to metrics `vmalert_alerts_sent_.*` on `http://<vmalert>/metrics` page. Consider limiting user's access
to the web UI or `/metrics` page if this information is sensitive.
[Alerts state ](#alerts-state ) page or [debug mode ](#debug-mode ) could emit additional information about configured
datasource URL, GET params and headers. Sensitive information such as passwords or auth tokens is stripped by default.
To disable stripping of such info pass `-datasource.showURL` cmd-line flag to vmalert.
2022-08-24 11:07:33 +00:00
## Profiling
`vmalert` provides handlers for collecting the following [Go profiles ](https://blog.golang.org/profiling-go-programs ):
* Memory profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed):
< div class = "with-copy" markdown = "1" >
```console
curl http://0.0.0.0:8880/debug/pprof/heap > mem.pprof
```
< / div >
* CPU profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed):
< div class = "with-copy" markdown = "1" >
```console
curl http://0.0.0.0:8880/debug/pprof/profile > cpu.pprof
```
< / div >
The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with [go tool pprof ](https://github.com/google/pprof ).
It is safe sharing the collected profiles from security point of view, since they do not contain sensitive information.
2021-04-07 10:39:16 +00:00
## Configuration
2020-04-29 14:42:01 +00:00
2021-12-02 22:08:42 +00:00
### Flags
2021-06-14 08:37:26 +00:00
Pass `-help` to `vmalert` in order to see the full list of supported
command-line flags with their descriptions.
2020-04-29 14:42:01 +00:00
The shortlist of configuration flags is the following:
2022-08-16 08:44:08 +00:00
{% raw %}
2023-07-19 08:43:52 +00:00
```console
2022-08-08 11:02:03 +00:00
-clusterMode
2023-10-13 11:07:16 +00:00
If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2022-02-08 13:37:38 +00:00
-configCheckInterval duration
2023-05-10 07:50:41 +00:00
Interval for checking for changes in '-rule' or '-notifier.config' files. By default, the checking is disabled. Send SIGHUP signal in order to force config check for changes.
2021-02-03 21:44:37 +00:00
-datasource.appendTypePrefix
2022-08-08 11:02:03 +00:00
Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.
2020-04-29 14:42:01 +00:00
-datasource.basicAuth.password string
2022-08-08 11:02:03 +00:00
Optional basic auth password for -datasource.url
2021-09-14 11:47:40 +00:00
-datasource.basicAuth.passwordFile string
2022-08-08 11:02:03 +00:00
Optional path to basic auth password to use for -datasource.url
2020-04-29 14:42:01 +00:00
-datasource.basicAuth.username string
2022-08-08 11:02:03 +00:00
Optional basic auth username for -datasource.url
2021-09-14 11:47:40 +00:00
-datasource.bearerToken string
2022-08-08 11:02:03 +00:00
Optional bearer auth token to use for -datasource.url.
2021-09-14 11:47:40 +00:00
-datasource.bearerTokenFile string
2022-08-08 11:02:03 +00:00
Optional path to bearer token file to use for -datasource.url.
2022-04-04 09:59:04 +00:00
-datasource.disableKeepAlive
2022-08-08 11:02:03 +00:00
Whether to disable long-lived connections to the datasource. If true, disables HTTP keep-alives and will only use the connection to the server for a single HTTP request.
2023-07-07 05:44:34 +00:00
-datasource.disableStepParam
2023-07-07 09:50:26 +00:00
Whether to disable adding 'step' param to the issued instant queries. This might be useful when using vmalert with datasources that do not support 'step' param for instant queries, like Google Managed Prometheus. It is not recommended to enable this flag if you use vmalert with VictoriaMetrics.
2022-07-21 11:57:53 +00:00
-datasource.headers string
2022-08-08 11:02:03 +00:00
Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2'
2020-09-30 06:50:29 +00:00
-datasource.lookback duration
2023-10-25 11:07:13 +00:00
Will be deprecated soon, please adjust "-search.latencyOffset" at datasource side or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
2020-09-30 06:50:29 +00:00
-datasource.maxIdleConnections int
2022-08-08 11:02:03 +00:00
Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
2022-03-10 11:09:12 +00:00
-datasource.oauth2.clientID string
2022-11-21 13:34:13 +00:00
Optional OAuth2 clientID to use for -datasource.url.
2022-03-10 11:09:12 +00:00
-datasource.oauth2.clientSecret string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecret to use for -datasource.url.
2022-03-10 11:09:12 +00:00
-datasource.oauth2.clientSecretFile string
2022-11-21 13:34:13 +00:00
Optional OAuth2 clientSecretFile to use for -datasource.url.
2022-03-10 11:09:12 +00:00
-datasource.oauth2.scopes string
2022-08-08 11:02:03 +00:00
Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
2022-03-10 11:09:12 +00:00
-datasource.oauth2.tokenUrl string
2022-08-08 11:02:03 +00:00
Optional OAuth2 tokenURL to use for -datasource.url.
2021-03-01 15:01:27 +00:00
-datasource.queryStep duration
2022-09-26 05:46:21 +00:00
How far a value can fallback to when evaluating queries. For example, if -datasource.queryStep=15s then param "step" with value "15s" will be added to every query. If set to 0, rule's evaluation interval will be used instead. (default 5m0s)
2022-03-16 10:30:02 +00:00
-datasource.queryTimeAlignment
2023-10-13 11:07:16 +00:00
Deprecated: please use "eval_alignment" in rule group instead. Whether to align "time" parameter with evaluation interval. Alignment supposed to produce deterministic results despite number of vmalert replicas or time they were started. See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257 (default true)
2021-05-10 08:18:26 +00:00
-datasource.roundDigits int
2022-08-08 11:02:03 +00:00
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
2022-08-11 11:30:32 +00:00
-datasource.showURL
2023-10-13 11:07:16 +00:00
Whether to avoid stripping sensitive information such as auth headers or passwords from URLs in log messages or UI and exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
2020-08-14 16:13:24 +00:00
-datasource.tlsCAFile string
2022-08-08 11:02:03 +00:00
Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
2020-08-14 16:13:24 +00:00
-datasource.tlsCertFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate file to use when connecting to -datasource.url
2020-06-23 19:48:25 +00:00
-datasource.tlsInsecureSkipVerify
2022-08-08 11:02:03 +00:00
Whether to skip tls verification when connecting to -datasource.url
2020-08-14 16:13:24 +00:00
-datasource.tlsKeyFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate key to use when connecting to -datasource.url
2020-08-14 16:13:24 +00:00
-datasource.tlsServerName string
2022-08-08 11:02:03 +00:00
Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used
2020-05-05 04:50:57 +00:00
-datasource.url string
2022-11-30 04:36:38 +00:00
Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL
2022-08-08 11:02:03 +00:00
-defaultTenant.graphite string
2023-10-13 11:07:16 +00:00
Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2022-08-08 11:02:03 +00:00
-defaultTenant.prometheus string
2023-10-13 11:07:16 +00:00
Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2021-08-21 17:24:56 +00:00
-disableAlertgroupLabel
2022-08-08 11:02:03 +00:00
Whether to disable adding group's Name as label to generated alerts and time series.
2022-09-15 10:00:36 +00:00
-dryRun
2023-10-13 11:07:16 +00:00
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
2020-08-14 16:13:24 +00:00
-enableTCP6
2023-07-19 08:10:51 +00:00
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
2020-08-14 16:13:24 +00:00
-envflag.enable
2023-07-19 08:10:51 +00:00
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
2020-08-14 16:13:24 +00:00
-envflag.prefix string
2022-08-08 11:02:03 +00:00
Prefix for environment variables if -envflag.enable is set
-eula
2023-10-13 11:07:16 +00:00
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2020-04-29 14:42:01 +00:00
-evaluationInterval duration
2022-08-08 11:02:03 +00:00
How often to evaluate the rules (default 1m0s)
2020-08-14 16:13:24 +00:00
-external.alert.source string
2023-04-04 11:25:29 +00:00
External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert.html#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1& left={"datasource":"VictoriaMetrics","queries":[{"expr":{{$expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. If empty 'vmalert/alert?group_id={{.GroupID}}& alert_id={{.AlertID}}' is used.
2020-07-30 18:56:48 +00:00
-external.label array
2022-08-08 11:02:03 +00:00
Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-external.url string
2023-05-30 10:33:17 +00:00
External URL is used as alert's source for sent alerts to the notifier. By default, hostname is used as address.
2023-10-13 11:07:16 +00:00
-filestream.disableFadvise
Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
2022-08-08 11:02:03 +00:00
-flagsAuthKey string
Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
-fs.disableMmap
2023-05-10 07:50:41 +00:00
Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
2022-08-08 11:02:03 +00:00
-http.connTimeout duration
Incoming http connections are closed after the configured timeout. This may help to spread the incoming load among a cluster of services behind a load balancer. Please note that the real timeout may be bigger by up to 10% as a protection against the thundering herd problem (default 2m0s)
-http.disableResponseCompression
2023-05-10 07:50:41 +00:00
Disable compression of HTTP responses to save CPU resources. By default, compression is enabled to save network bandwidth
2023-10-30 10:33:38 +00:00
-http.header.csp string
Value for 'Content-Security-Policy' header
-http.header.frameOptions string
Value for 'X-Frame-Options' header
-http.header.hsts string
Value for 'Strict-Transport-Security' header
2022-08-08 11:02:03 +00:00
-http.idleConnTimeout duration
Timeout for incoming idle http connections (default 1m0s)
-http.maxGracefulShutdownDuration duration
The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
-http.pathPrefix string
An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
-http.shutdownDelay duration
Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
2020-08-14 16:13:24 +00:00
-httpAuth.password string
2023-07-07 11:50:13 +00:00
Password for HTTP server's Basic Auth. The authentication is disabled if -httpAuth.username is empty
2020-08-14 16:13:24 +00:00
-httpAuth.username string
2023-07-07 11:50:13 +00:00
Username for HTTP server's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
2020-04-29 14:42:01 +00:00
-httpListenAddr string
2023-01-27 08:00:30 +00:00
Address to listen for http connections. See also -httpListenAddr.useProxyProtocol (default ":8880")
-httpListenAddr.useProxyProtocol
2023-07-07 05:50:33 +00:00
Whether to use proxy protocol for connections accepted at -httpListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . With enabled proxy protocol http server cannot serve regular /metrics endpoint. Use -pushmetrics.url for metrics pushing
-internStringCacheExpireDuration duration
The expiry duration for caches for interned strings. See https://en.wikipedia.org/wiki/String_interning . See also -internStringMaxLen and -internStringDisableCache (default 6m0s)
-internStringDisableCache
Whether to disable caches for interned strings. This may reduce memory usage at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringCacheExpireDuration and -internStringMaxLen
2023-02-10 01:51:00 +00:00
-internStringMaxLen int
2023-07-19 08:10:51 +00:00
The maximum length for strings to intern. A lower limit may save memory at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringDisableCache and -internStringCacheExpireDuration (default 500)
2023-09-08 13:09:34 +00:00
-license string
2023-10-13 11:07:16 +00:00
Lisense key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/ . Trial Enterprise license can be obtained from https://victoriametrics.com/products/enterprise/trial/ . This flag is available only in Enterprise binaries. The license key can be also passed via file specified by -licenseFile command-line flag
2023-09-08 13:09:34 +00:00
-license.forceOffline
2023-10-13 11:07:16 +00:00
Whether to enable offline verification for VictoriaMetrics Enterprise license key, which has been passed either via -license or via -licenseFile command-line flag. The issued license key must support offline verification feature. Contact info@victoriametrics.com if you need offline license verification. This flag is avilable only in Enterprise binaries
2023-09-08 13:09:34 +00:00
-licenseFile string
2023-10-13 11:07:16 +00:00
Path to file with license key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/ . Trial Enterprise license can be obtained from https://victoriametrics.com/products/enterprise/trial/ . This flag is available only in Enterprise binaries. The license key can be also passed inline via -license command-line flag
2022-08-08 11:02:03 +00:00
-loggerDisableTimestamps
Whether to disable writing timestamps in logs
-loggerErrorsPerSecondLimit int
Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, the remaining errors are suppressed. Zero values disable the rate limit
-loggerFormat string
Format for logs. Possible values: default, json (default "default")
2022-12-29 11:36:44 +00:00
-loggerJSONFields string
Allows renaming fields in JSON formatted logs. Example: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message". Supported fields: ts, level, caller, msg
2022-08-08 11:02:03 +00:00
-loggerLevel string
Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")
-loggerOutput string
Output for the logs. Supported values: stderr, stdout (default "stderr")
-loggerTimezone string
Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC")
-loggerWarnsPerSecondLimit int
Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit
2023-02-10 01:51:00 +00:00
-memory.allowedBytes size
2023-07-19 08:10:51 +00:00
Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage
2023-02-10 01:51:00 +00:00
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
-memory.allowedPercent float
2023-07-19 08:10:51 +00:00
Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache which will result in higher disk IO usage (default 60)
2020-06-02 16:12:41 +00:00
-metricsAuthKey string
2022-08-08 11:02:03 +00:00
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings
2020-08-14 16:13:24 +00:00
-notifier.basicAuth.password array
2022-08-08 11:02:03 +00:00
Optional basic auth password for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-02-02 21:26:16 +00:00
-notifier.basicAuth.passwordFile array
2022-08-08 11:02:03 +00:00
Optional path to basic auth password file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-notifier.basicAuth.username array
2022-08-08 11:02:03 +00:00
Optional basic auth username for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.bearerToken array
2022-08-08 11:02:03 +00:00
Optional bearer token for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.bearerTokenFile array
2022-08-08 11:02:03 +00:00
Optional path to bearer token file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2023-10-13 11:07:16 +00:00
-notifier.blackhole -notifier.url
Whether to blackhole alerting notifications. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). -notifier.url, `-notifier.config` and `-notifier.blackhole` are mutually exclusive.
2022-02-08 13:37:38 +00:00
-notifier.config string
2022-08-08 11:02:03 +00:00
Path to configuration file for notifiers
2022-03-10 11:09:12 +00:00
-notifier.oauth2.clientID array
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientID to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.oauth2.clientSecret array
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecret to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.oauth2.clientSecretFile array
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecretFile to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.oauth2.scopes array
2022-08-08 11:02:03 +00:00
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2022-03-10 11:09:12 +00:00
-notifier.oauth2.tokenUrl array
2022-08-08 11:02:03 +00:00
Optional OAuth2 tokenURL to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2023-10-13 11:07:16 +00:00
-notifier.showURL
Whether to avoid stripping sensitive information such as passwords from URL in log messages or UI for -notifier.url. It is hidden by default, since it can contain sensitive info such as auth key
2022-02-08 13:37:38 +00:00
-notifier.suppressDuplicateTargetErrors
2022-08-08 11:02:03 +00:00
Whether to suppress 'duplicate target' errors during discovery
2020-08-14 16:13:24 +00:00
-notifier.tlsCAFile array
2023-05-10 07:50:41 +00:00
Optional path to TLS CA file to use for verifying connections to -notifier.url. By default, system CA is used
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-notifier.tlsCertFile array
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate file to use when connecting to -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2020-12-22 20:32:10 +00:00
-notifier.tlsInsecureSkipVerify array
2022-08-08 11:02:03 +00:00
Whether to skip tls verification when connecting to -notifier.url
Supports array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-notifier.tlsKeyFile array
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate key to use when connecting to -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-notifier.tlsServerName array
2023-05-10 07:50:41 +00:00
Optional TLS server name to use for connections to -notifier.url. By default, the server name from -notifier.url is used
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-notifier.url array
2023-01-04 06:40:00 +00:00
Prometheus Alertmanager URL, e.g. http://127.0.0.1:9093. List all Alertmanager URLs if it runs in the cluster mode to ensure high availability.
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-pprofAuthKey string
2022-08-08 11:02:03 +00:00
Auth key for /debug/pprof/* endpoints. It must be passed via authKey query arg. It overrides httpAuth.* settings
-promscrape.consul.waitTime duration
Wait time used by Consul service discovery. Default value is used if not set
-promscrape.consulSDCheckInterval duration
2022-09-15 10:00:36 +00:00
Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#consul_sd_configs for details (default 30s)
2022-08-08 11:02:03 +00:00
-promscrape.discovery.concurrency int
The maximum number of concurrent requests to Prometheus autodiscovery API (Consul, Kubernetes, etc.) (default 100)
-promscrape.discovery.concurrentWaitTime duration
The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s)
-promscrape.dnsSDCheckInterval duration
2022-09-15 10:00:36 +00:00
Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#dns_sd_configs for details (default 30s)
2022-08-08 11:02:03 +00:00
-pushmetrics.extraLabel array
Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags.
-pushmetrics.interval duration
Interval for pushing metrics to -pushmetrics.url (default 10s)
-pushmetrics.url array
2023-05-10 07:50:41 +00:00
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2020-05-13 19:55:35 +00:00
-remoteRead.basicAuth.password string
2022-08-08 11:02:03 +00:00
Optional basic auth password for -remoteRead.url
2021-09-14 11:47:40 +00:00
-remoteRead.basicAuth.passwordFile string
2022-08-08 11:02:03 +00:00
Optional path to basic auth password to use for -remoteRead.url
2020-05-13 19:55:35 +00:00
-remoteRead.basicAuth.username string
2022-08-08 11:02:03 +00:00
Optional basic auth username for -remoteRead.url
2021-09-14 11:47:40 +00:00
-remoteRead.bearerToken string
2022-08-08 11:02:03 +00:00
Optional bearer auth token to use for -remoteRead.url.
2021-09-14 11:47:40 +00:00
-remoteRead.bearerTokenFile string
2022-08-08 11:02:03 +00:00
Optional path to bearer token file to use for -remoteRead.url.
2021-10-18 07:24:52 +00:00
-remoteRead.disablePathAppend
2022-08-08 11:02:03 +00:00
Whether to disable automatic appending of '/api/v1/query' path to the configured -datasource.url and -remoteRead.url
2022-07-21 11:57:53 +00:00
-remoteRead.headers string
2022-08-08 11:02:03 +00:00
Optional HTTP headers to send with each request to the corresponding -remoteRead.url. For example, -remoteRead.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteRead.url. Multiple headers must be delimited by '^^': -remoteRead.headers='header1:value1^^header2:value2'
2021-05-10 08:18:26 +00:00
-remoteRead.ignoreRestoreErrors
2023-02-06 08:51:30 +00:00
Whether to ignore errors from remote storage when restoring alerts state on startup. DEPRECATED - this flag has no effect and will be removed in the next releases. (default true)
2020-05-13 19:55:35 +00:00
-remoteRead.lookback duration
2022-08-08 11:02:03 +00:00
Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
2022-03-10 11:09:12 +00:00
-remoteRead.oauth2.clientID string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientID to use for -remoteRead.url.
2022-03-10 11:09:12 +00:00
-remoteRead.oauth2.clientSecret string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecret to use for -remoteRead.url.
2022-03-10 11:09:12 +00:00
-remoteRead.oauth2.clientSecretFile string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecretFile to use for -remoteRead.url.
2022-03-10 11:09:12 +00:00
-remoteRead.oauth2.scopes string
2022-08-08 11:02:03 +00:00
Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.
2022-03-10 11:09:12 +00:00
-remoteRead.oauth2.tokenUrl string
2022-11-21 13:34:13 +00:00
Optional OAuth2 tokenURL to use for -remoteRead.url.
2022-08-11 11:30:32 +00:00
-remoteRead.showURL
Whether to show -remoteRead.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
2020-08-14 16:13:24 +00:00
-remoteRead.tlsCAFile string
2023-05-10 07:50:41 +00:00
Optional path to TLS CA file to use for verifying connections to -remoteRead.url. By default, system CA is used
2020-08-14 16:13:24 +00:00
-remoteRead.tlsCertFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate file to use when connecting to -remoteRead.url
2020-06-23 19:48:25 +00:00
-remoteRead.tlsInsecureSkipVerify
2022-08-08 11:02:03 +00:00
Whether to skip tls verification when connecting to -remoteRead.url
2020-08-14 16:13:24 +00:00
-remoteRead.tlsKeyFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate key to use when connecting to -remoteRead.url
2020-08-14 16:13:24 +00:00
-remoteRead.tlsServerName string
2023-05-10 07:50:41 +00:00
Optional TLS server name to use for connections to -remoteRead.url. By default, the server name from -remoteRead.url is used
2020-05-13 19:55:35 +00:00
-remoteRead.url vmalert
2022-08-11 11:30:32 +00:00
Optional URL to datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect.Remote read is used to restore alerts state.This configuration makes sense only if vmalert was configured with `remoteWrite.url` before and has been successfully persisted its state. E.g. http://127.0.0.1:8428. See also '-remoteRead.disablePathAppend', '-remoteRead.showURL'.
2020-05-13 19:55:35 +00:00
-remoteWrite.basicAuth.password string
2022-08-08 11:02:03 +00:00
Optional basic auth password for -remoteWrite.url
2021-09-14 11:47:40 +00:00
-remoteWrite.basicAuth.passwordFile string
2022-08-08 11:02:03 +00:00
Optional path to basic auth password to use for -remoteWrite.url
2020-05-13 19:55:35 +00:00
-remoteWrite.basicAuth.username string
2022-08-08 11:02:03 +00:00
Optional basic auth username for -remoteWrite.url
2021-09-14 11:47:40 +00:00
-remoteWrite.bearerToken string
2022-08-08 11:02:03 +00:00
Optional bearer auth token to use for -remoteWrite.url.
2021-09-14 11:47:40 +00:00
-remoteWrite.bearerTokenFile string
2022-08-08 11:02:03 +00:00
Optional path to bearer token file to use for -remoteWrite.url.
2020-06-02 16:12:41 +00:00
-remoteWrite.concurrency int
2022-08-08 11:02:03 +00:00
Defines number of writers for concurrent writing into remote querier (default 1)
2021-08-16 12:20:22 +00:00
-remoteWrite.disablePathAppend
2022-08-08 11:02:03 +00:00
Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.
2020-06-29 19:17:56 +00:00
-remoteWrite.flushInterval duration
2022-08-08 11:02:03 +00:00
Defines interval of flushes to remote write endpoint (default 5s)
2022-07-21 11:57:53 +00:00
-remoteWrite.headers string
2022-08-08 11:02:03 +00:00
Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'
2020-06-02 16:12:41 +00:00
-remoteWrite.maxBatchSize int
2023-05-10 07:50:41 +00:00
Defines max number of timeseries to be flushed at once (default 1000)
2020-06-02 16:12:41 +00:00
-remoteWrite.maxQueueSize int
2022-08-08 11:02:03 +00:00
Defines the max number of pending datapoints to remote write endpoint (default 100000)
2022-03-10 11:09:12 +00:00
-remoteWrite.oauth2.clientID string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientID to use for -remoteWrite.url.
2022-03-10 11:09:12 +00:00
-remoteWrite.oauth2.clientSecret string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecret to use for -remoteWrite.url.
2022-03-10 11:09:12 +00:00
-remoteWrite.oauth2.clientSecretFile string
2022-08-08 11:02:03 +00:00
Optional OAuth2 clientSecretFile to use for -remoteWrite.url.
2022-03-10 11:09:12 +00:00
-remoteWrite.oauth2.scopes string
2022-08-08 11:02:03 +00:00
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
2022-03-10 11:09:12 +00:00
-remoteWrite.oauth2.tokenUrl string
2022-08-08 11:02:03 +00:00
Optional OAuth2 tokenURL to use for -notifier.url.
2023-06-22 13:14:23 +00:00
-remoteWrite.retryMaxTime duration
The max time spent on retry attempts for the failed remote-write request. Change this value if it is expected for remoteWrite.url to be unreachable for more than -remoteWrite.retryMaxTime. See also -remoteWrite.retryMinInterval (default 30s)
-remoteWrite.retryMinInterval duration
The minimum delay between retry attempts. Every next retry attempt will double the delay to prevent hammering of remote database. See also -remoteWrite.retryMaxInterval (default 1s)
2022-12-03 03:01:46 +00:00
-remoteWrite.sendTimeout duration
Timeout for sending data to the configured -remoteWrite.url. (default 30s)
2022-08-11 11:30:32 +00:00
-remoteWrite.showURL
Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
2020-08-14 16:13:24 +00:00
-remoteWrite.tlsCAFile string
2023-05-10 07:50:41 +00:00
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default, system CA is used
2020-08-14 16:13:24 +00:00
-remoteWrite.tlsCertFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate file to use when connecting to -remoteWrite.url
2020-06-23 19:48:25 +00:00
-remoteWrite.tlsInsecureSkipVerify
2022-08-08 11:02:03 +00:00
Whether to skip tls verification when connecting to -remoteWrite.url
2020-08-14 16:13:24 +00:00
-remoteWrite.tlsKeyFile string
2022-08-08 11:02:03 +00:00
Optional path to client-side TLS certificate key to use when connecting to -remoteWrite.url
2020-08-14 16:13:24 +00:00
-remoteWrite.tlsServerName string
2023-05-10 07:50:41 +00:00
Optional TLS server name to use for connections to -remoteWrite.url. By default, the server name from -remoteWrite.url is used
2020-05-13 19:55:35 +00:00
-remoteWrite.url string
2022-08-11 11:30:32 +00:00
Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend, '-remoteWrite.showURL'.
2022-05-20 12:07:16 +00:00
-replay.disableProgressBar
2022-08-08 11:02:03 +00:00
Whether to disable rendering progress bars during the replay. Progress bar rendering might be verbose or break the logs parsing, so it is recommended to be disabled when not used in interactive mode.
2023-02-10 01:51:00 +00:00
-replay.maxDatapointsPerQuery /query_range
Max number of data points expected in one request. It affects the max time range for every /query_range request during the replay. The higher the value, the less requests will be made during replay. (default 1000)
2021-06-09 09:27:31 +00:00
-replay.ruleRetryAttempts int
2022-08-08 11:02:03 +00:00
Defines how many retries to make before giving up on rule if request for it returns an error. (default 5)
2021-06-09 09:27:31 +00:00
-replay.rulesDelay duration
2023-05-10 07:50:41 +00:00
Delay between rules evaluation within the group. Could be important if there are chained rules inside the group and processing need to wait for previous rule results to be persisted by remote storage before evaluating the next rule.Keep it equal or bigger than -remoteWrite.flushInterval. (default 1s)
2021-06-09 09:27:31 +00:00
-replay.timeFrom string
2022-08-08 11:02:03 +00:00
The time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z'
2021-06-09 09:27:31 +00:00
-replay.timeTo string
2022-08-08 11:02:03 +00:00
The time filter in RFC3339 format to select timeseries with timestamp equal or lower than provided value. E.g. '2020-01-01T20:07:00Z'
2020-08-14 16:13:24 +00:00
-rule array
2023-07-07 05:50:33 +00:00
Path to the files or http url with alerting and/or recording rules.
2023-02-10 01:51:00 +00:00
Supports hierarchical patterns and regexpes.
2022-08-08 11:02:03 +00:00
Examples:
2023-05-08 11:31:54 +00:00
-rule="/path/to/file". Path to a single file with alerting rules.
-rule="http://< some-server-addr > /path/to/rules". HTTP URL to a page with alerting rules.
2023-07-07 05:50:33 +00:00
-rule="dir/*.yaml" -rule="/*.yaml" -rule="gcs://vmalert-rules/tenant_%{TENANT_ID}/prod".
2023-05-08 23:21:42 +00:00
-rule="dir/**/*.yaml". Includes all the .yaml files in "dir" subfolders recursively.
2022-08-08 11:02:03 +00:00
Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.
2023-02-10 01:51:00 +00:00
Enterprise version of vmalert supports S3 and GCS paths to rules.
For example: gs://bucket/path/to/rules, s3://bucket/path/to/rules
S3 and GCS paths support only matching by prefix, e.g. s3://bucket/dir/rule_ matches
all files with prefix rule_ in folder dir.
See https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2023-10-25 11:07:13 +00:00
-rule.evalDelay time
Adjustment of the time parameter for rule evaluation requests to compensate intentional data delay from the datasource.Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s)
2023-11-02 19:23:32 +00:00
-rule.maxResolveDuration duration
Limits the maxiMum duration for automatic alert expiration, which by default is 4 times evaluationInterval of the parent group
-rule.resendDelay duration
MiniMum amount of time to wait before resending an alert to notifier
2022-05-20 12:07:16 +00:00
-rule.templates array
2023-11-02 19:23:32 +00:00
Path or glob pattern to location with go template definitions for rules annotations templating. Flag can be specified multiple times.
2022-08-08 11:02:03 +00:00
Examples:
-rule.templates="/path/to/file". Path to a single file with go templates
-rule.templates="dir/*.tpl" -rule.templates="/*.tpl". Relative path to all .tpl files in "dir" folder,
absolute path to all .tpl files in root.
2023-05-08 23:21:42 +00:00
-rule.templates="dir/**/*.tpl". Includes all the .tpl files in "dir" subfolders recursively.
2023-07-07 05:50:33 +00:00
2022-08-08 11:02:03 +00:00
Supports an array of values separated by comma or specified via multiple flags.
2023-01-04 06:40:00 +00:00
-rule.updateEntriesLimit int
2023-05-10 07:50:41 +00:00
Defines the max number of rule's state updates stored in-memory. Rule's updates are available on rule's Details page and are used for debugging purposes. The number of stored updates can be overridden per rule via update_entries_limit param. (default 20)
2020-06-10 16:37:43 +00:00
-rule.validateExpressions
2022-08-08 11:02:03 +00:00
Whether to validate rules expressions via MetricsQL engine (default true)
2020-04-29 14:42:01 +00:00
-rule.validateTemplates
2022-08-08 11:02:03 +00:00
Whether to validate annotation and label templates (default true)
2023-02-10 01:51:00 +00:00
-s3.configFilePath string
Path to file with S3 configs. Configs are loaded from default location if not set.
2023-10-13 11:07:16 +00:00
See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2023-02-10 01:51:00 +00:00
-s3.configProfile string
2023-10-13 11:07:16 +00:00
Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2023-02-10 01:51:00 +00:00
-s3.credsFilePath string
Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
2023-10-13 11:07:16 +00:00
See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2023-02-10 01:51:00 +00:00
-s3.customEndpoint string
2023-10-13 11:07:16 +00:00
Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html
2023-02-10 01:51:00 +00:00
-s3.forcePathStyle
2023-10-13 11:07:16 +00:00
Prefixing endpoint with bucket name when set false, true by default. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise.html (default true)
2020-08-14 16:13:24 +00:00
-tls
2022-08-08 11:02:03 +00:00
Whether to enable TLS for incoming HTTP requests at -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set
2020-08-14 16:13:24 +00:00
-tlsCertFile string
2022-08-08 11:02:03 +00:00
Path to file with TLS certificate if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated
2022-04-16 12:59:45 +00:00
-tlsCipherSuites array
2022-08-08 11:02:03 +00:00
Optional list of TLS cipher suites for incoming requests over HTTPS if -tls is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants
Supports an array of values separated by comma or specified via multiple flags.
2020-08-14 16:13:24 +00:00
-tlsKeyFile string
2022-08-08 11:02:03 +00:00
Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
2022-09-26 14:35:45 +00:00
-tlsMinVersion string
Optional minimum TLS version to use for incoming requests over HTTPS if -tls is set. Supported values: TLS10, TLS11, TLS12, TLS13
2020-08-14 16:13:24 +00:00
-version
2022-08-08 11:02:03 +00:00
Show VictoriaMetrics version
2020-04-29 14:42:01 +00:00
```
2022-08-16 08:44:08 +00:00
{% endraw %}
2020-04-29 14:42:01 +00:00
2021-12-02 22:08:42 +00:00
### Hot config reload
2022-03-22 11:40:55 +00:00
2021-05-26 09:32:16 +00:00
`vmalert` supports "hot" config reload via the following methods:
2022-03-22 11:40:55 +00:00
2021-05-26 09:32:16 +00:00
* send SIGHUP signal to `vmalert` process;
* send GET request to `/-/reload` endpoint;
2022-02-02 21:26:16 +00:00
* configure `-configCheckInterval` flag for periodic reload
2022-05-14 16:56:31 +00:00
on config change.
2020-05-13 19:55:35 +00:00
2021-12-02 22:08:42 +00:00
### URL params
To set additional URL params for `datasource.url` , `remoteWrite.url` or `remoteRead.url`
just add them in address: `-datasource.url=http://localhost:8428?nocache=1` .
To set additional URL params for specific [group of rules ](#Groups ) modify
the `params` group:
2022-03-22 11:40:55 +00:00
2021-12-02 22:08:42 +00:00
```yaml
groups:
- name: TestGroup
params:
denyPartialResponse: ["true"]
extra_label: ["env=dev"]
```
2022-03-22 11:40:55 +00:00
2021-12-02 22:08:42 +00:00
Please note, `params` are used only for executing rules expressions (requests to `datasource.url` ).
If there would be a conflict between URL params set in `datasource.url` flag and params in group definition
the latter will have higher priority.
2022-02-02 21:26:16 +00:00
### Notifier configuration file
Notifier also supports configuration via file specified with flag `notifier.config` :
2022-03-22 11:40:55 +00:00
2022-02-02 21:26:16 +00:00
```
./bin/vmalert -rule=app/vmalert/config/testdata/rules.good.rules \
2022-03-22 11:40:55 +00:00
-datasource.url=http://localhost:8428 \
-notifier.config=app/vmalert/notifier/testdata/consul.good.yaml
2022-02-02 21:26:16 +00:00
```
2022-04-13 08:50:26 +00:00
The configuration file allows to configure static notifiers, discover notifiers via
[Consul ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config )
and [DNS ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config ):
2022-02-02 21:26:16 +00:00
For example:
2022-03-22 11:40:55 +00:00
2023-07-19 08:43:52 +00:00
```yaml
2022-09-28 07:15:05 +00:00
static_configs:
2022-02-02 21:26:16 +00:00
- targets:
2023-11-01 11:53:50 +00:00
# support using full url
- 'http://alertmanager:9093/test/api/v2/alerts'
- 'https://alertmanager:9093/api/v2/alerts'
# the following target with only host:port will be used as < scheme > ://localhost:9093/< path_prefix > /api/v2/alerts
2022-02-02 21:26:16 +00:00
- localhost:9093
consul_sd_configs:
- server: localhost:8500
services:
- alertmanager
2022-09-28 07:15:05 +00:00
2022-04-13 08:50:26 +00:00
dns_sd_configs:
- names:
- my.domain.com
type: 'A'
port: 9093
2022-02-02 21:26:16 +00:00
```
2023-07-28 01:05:01 +00:00
The list of configured or discovered Notifiers can be explored via [UI ](#web ).
2022-12-30 08:54:03 +00:00
If Alertmanager runs in cluster mode then all its URLs needs to be available during discovery
to ensure [high availability ](https://github.com/prometheus/alertmanager#high-availability ).
2022-02-02 21:26:16 +00:00
The configuration file [specification ](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/notifier/config.go )
is the following:
2022-03-22 11:40:55 +00:00
2023-07-19 08:43:52 +00:00
```yaml
2022-02-02 21:26:16 +00:00
# Per-target Notifier timeout when pushing alerts.
[ timeout: < duration > | default = 10s ]
# Prefix for the HTTP path alerts are pushed to.
[ path_prefix: < path > | default = / ]
# Configures the protocol scheme used for requests.
[ scheme: < scheme > | default = http ]
# Sets the `Authorization` header on every request with the
# configured username and password.
# password and password_file are mutually exclusive.
basic_auth:
[ username: < string > ]
2022-09-30 15:10:17 +00:00
[ password: < string > ]
2022-02-02 21:26:16 +00:00
[ password_file: < string > ]
# Optional `Authorization` header configuration.
authorization:
# Sets the authentication type.
[ type: < string > | default: Bearer ]
# Sets the credentials. It is mutually exclusive with
# `credentials_file` .
[ credentials: < secret > ]
# Sets the credentials to the credentials read from the configured file.
# It is mutually exclusive with `credentials` .
[ credentials_file: < filename > ]
# Configures the scrape request's TLS settings.
# see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config
tls_config:
[ < tls_config > ]
2022-09-30 15:10:17 +00:00
# Configures Bearer authentication token via string
bearer_token: < string >
# or by passing path to the file with token.
bearer_token_file: < string >
# Configures OAuth 2.0 authentication
# see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#oauth2
oauth2:
[ < oauth2_config > ]
# Optional list of HTTP headers in form `header-name: value`
# applied for all requests to notifiers
# For example:
# headers:
# - "CustomHeader: foo"
# - "CustomHeader2: bar"
headers:
[ < string > , ...]
2022-02-02 21:26:16 +00:00
# List of labeled statically configured Notifiers.
2022-09-30 15:10:17 +00:00
#
# Each list of targets may be additionally instructed with
# authorization params. Target's authorization params will
# inherit params from global authorization params if there
# are no conflicts.
2022-02-02 21:26:16 +00:00
static_configs:
2022-09-30 15:10:17 +00:00
[ - targets: ]
[ - '< host > ' ]
[ oauth2 ]
[ basic_auth ]
[ authorization ]
[ tls_config ]
[ bearer_token ]
[ bearer_token_file ]
[ headers ]
2022-02-02 21:26:16 +00:00
# List of Consul service discovery configurations.
# See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
consul_sd_configs:
[ - < consul_sd_config > ... ]
2022-04-13 08:50:26 +00:00
# List of DNS service discovery configurations.
# See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config
dns_sd_configs:
[ - < dns_sd_config > ... ]
2022-04-11 09:11:32 +00:00
# List of relabel configurations for entities discovered via service discovery.
2022-02-02 21:26:16 +00:00
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent.html#relabeling
relabel_configs:
[ - < relabel_config > ... ]
2022-04-11 09:11:32 +00:00
# List of relabel configurations for alert labels sent via Notifier.
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent.html#relabeling
alert_relabel_configs:
[ - < relabel_config > ... ]
2022-02-02 21:26:16 +00:00
```
The configuration file can be [hot-reloaded ](#hot-config-reload ).
2021-04-07 10:39:16 +00:00
## Contributing
2020-04-29 14:42:01 +00:00
`vmalert` is mostly designed and built by VictoriaMetrics community.
2021-03-09 20:49:50 +00:00
Feel free to share your experience and ideas for improving this
2020-04-29 14:42:01 +00:00
software. Please keep simplicity as the main priority.
2020-05-15 10:26:51 +00:00
2021-04-07 10:39:16 +00:00
## How to build from sources
2020-05-15 10:26:51 +00:00
2021-03-09 20:49:50 +00:00
It is recommended using
2023-10-18 18:06:23 +00:00
[binary releases ](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest )
2020-05-15 10:26:51 +00:00
2022-03-22 11:40:55 +00:00
* `vmalert` is located in `vmutils-*` archives there.
2020-05-15 10:26:51 +00:00
2022-05-22 22:59:34 +00:00
### Docker image
You can build `vmalert` docker image from source and push it to your own docker repository.
Run the following commands from the root folder of [the repository ](https://github.com/VictoriaMetrics/VictoriaMetrics ):
2022-06-19 19:57:53 +00:00
```console
2022-05-22 22:59:34 +00:00
make package-vmalert
docker tag victoria-metrics/vmalert:version my-repo:my-version-name
docker push my-repo:my-version-name
```
To run the built image in `victoria-metrics-k8s-stack` or `VMAlert` CR object apply the following config change:
2023-07-19 08:43:52 +00:00
2022-05-22 22:59:34 +00:00
```yaml
kind: VMAlert
spec:
image:
repository: my-repo
tag: my-version-name
```
2021-04-07 10:39:16 +00:00
### Development build
2020-05-15 10:26:51 +00:00
2023-09-07 10:13:03 +00:00
1. [Install Go ](https://golang.org/doc/install ). The minimum supported version is Go 1.20.
2023-07-26 21:39:35 +00:00
1. Run `make vmalert` from the root folder of [the repository ](https://github.com/VictoriaMetrics/VictoriaMetrics ).
2020-05-15 10:26:51 +00:00
It builds `vmalert` binary and puts it into the `bin` folder.
2021-04-07 10:39:16 +00:00
### Production build
2020-05-15 10:26:51 +00:00
1. [Install docker ](https://docs.docker.com/install/ ).
2023-07-26 21:39:35 +00:00
1. Run `make vmalert-prod` from the root folder of [the repository ](https://github.com/VictoriaMetrics/VictoriaMetrics ).
2020-05-15 10:26:51 +00:00
It builds `vmalert-prod` binary and puts it into the `bin` folder.
2020-07-31 06:25:15 +00:00
2021-04-07 10:39:16 +00:00
### ARM build
2020-07-31 06:25:15 +00:00
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers ](https://blog.cloudflare.com/arm-takes-wing/ ).
2021-04-07 10:39:16 +00:00
### Development ARM build
2020-07-31 06:25:15 +00:00
2023-09-07 10:13:03 +00:00
1. [Install Go ](https://golang.org/doc/install ). The minimum supported version is Go 1.20.
2023-07-26 21:39:35 +00:00
1. Run `make vmalert-linux-arm` or `make vmalert-linux-arm64` from the root folder of [the repository ](https://github.com/VictoriaMetrics/VictoriaMetrics ).
2022-07-13 13:44:39 +00:00
It builds `vmalert-linux-arm` or `vmalert-linux-arm64` binary respectively and puts it into the `bin` folder.
2020-07-31 06:25:15 +00:00
2021-04-07 10:39:16 +00:00
### Production ARM build
2020-07-31 06:25:15 +00:00
1. [Install docker ](https://docs.docker.com/install/ ).
2023-07-26 21:39:35 +00:00
1. Run `make vmalert-linux-arm-prod` or `make vmalert-linux-arm64-prod` from the root folder of [the repository ](https://github.com/VictoriaMetrics/VictoriaMetrics ).
2022-07-13 13:44:39 +00:00
It builds `vmalert-linux-arm-prod` or `vmalert-linux-arm64-prod` binary respectively and puts it into the `bin` folder.