+
+Usually, troubles start to appear when data in `-datasource.url` is delayed or absent. In such cases, evaluations
+may get empty response from datasource and produce empty recording rules or reset alerts state:
+
+
+
+Try the following recommendations in such cases:
+
+* Always configure group's `evaluationInterval` to be bigger or equal to `scrape_interval` at which metrics
+are delivered to the datasource;
+* If you know in advance, that data in datasource is delayed - try changing vmalert's `-datasource.lookback`
+command-line flag to add a time shift for evaluations;
+* If time intervals between datapoints in datasource are irregular - try changing vmalert's `-datasource.queryStep`
+command-line flag to specify how far search query can lookback for the recent datapoint. By default, this value
+is equal to group's `evaluationInterval`.
+
+Sometimes, it is not clear why some specific alert fired or didn't fire. It is very important to remember, that
+alerts with `for: 0` fire immediately when their expression becomes true. And alerts with `for > 0` will fire only
+after multiple consecutive evaluations, and at each evaluation their expression must be true. If at least one evaluation
+becomes false, then alert's state resets to the initial state.
+
+If `-remoteWrite.url` command-line flag is configured, vmalert will persist alert's state in form of time series
+`ALERTS` and `ALERTS_FOR_STATE` to the specified destination. Such time series can be then queried via
+[vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) or Grafana to track how alerts state
+changed in time.
+
+vmalert also stores last N state updates for each rule. To check updates, click on `Details` link next to rule's name
+on `/vmalert/groups` page and check the `Last updates` section:
+
+
+
+Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of
+HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were
+no samples returned and curl command returns data - then it is very likely there was no data in datasource on the
+moment when rule was evaluated.
+
+vmalert also alows configuring more detailed logging for specific rule. Just set `debug: true` in rule's configuration
+and vmalert will start printing additional log messages:
+```terminal
+2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms)
+2022-09-15T13:35:56.149Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29+by%28instance%29+%3E+0&step=15s&time=1663248945"
+2022-09-15T13:35:56.178Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: query returned 1 samples (elapsed: 28.368208ms)
+2022-09-15T13:35:56.178Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29&step=15s&time=1663248945"
+2022-09-15T13:35:56.179Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} created in state PENDING
+...
+2022-09-15T13:36:56.153Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
+```
+
+
## Profiling
`vmalert` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@@ -678,7 +733,7 @@ The shortlist of configuration flags is the following:
{% raw %}
```
-clusterMode
- If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy
+ If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in enterprise version of VictoriaMetrics
-configCheckInterval duration
Interval for checking for changes in '-rule' or '-notifier.config' files. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes.
-datasource.appendTypePrefix
@@ -732,12 +787,12 @@ The shortlist of configuration flags is the following:
-datasource.url string
Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. E.g. http://127.0.0.1:8428 . See also '-datasource.disablePathAppend', '-datasource.showURL'.
-defaultTenant.graphite string
- Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy
+ Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy .This flag is available only in enterprise version of VictoriaMetrics
-defaultTenant.prometheus string
- Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy
+ Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert.html#multitenancy . This flag is available only in enterprise version of VictoriaMetrics
-disableAlertgroupLabel
Whether to disable adding group's Name as label to generated alerts and time series.
- -dryRun -rule
+ -dryRun
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP and UDP is used
@@ -746,12 +801,13 @@ The shortlist of configuration flags is the following:
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
- By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
+ By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf . This flag is available only in enterprise version of VictoriaMetrics
-evaluationInterval duration
How often to evaluate the rules (default 1m0s)
-external.alert.source string
- External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service.
- Supports templating. For example, link to Grafana: 'explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]'. (default "{{.ExternalURL}}/vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}")
+ External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service.
+ Supports templating. For example, link to Grafana: 'explore?orgId=1&left=[\"now-1h\",\"now\",\"VictoriaMetrics\",{\"expr\": \"{{$expr|quotesEscape|crlfEscape|queryEscape}}\"},{\"mode\":\"Metrics\"},{\"ui\":[true,true,true,\"none\"]}]'.
+ If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
-external.label array
Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
Supports an array of values separated by comma or specified via multiple flags.
@@ -857,13 +913,13 @@ The shortlist of configuration flags is the following:
-promscrape.consul.waitTime duration
Wait time used by Consul service discovery. Default value is used if not set
-promscrape.consulSDCheckInterval duration
- Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config for details (default 30s)
+ Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#consul_sd_configs for details (default 30s)
-promscrape.discovery.concurrency int
The maximum number of concurrent requests to Prometheus autodiscovery API (Consul, Kubernetes, etc.) (default 100)
-promscrape.discovery.concurrentWaitTime duration
The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s)
-promscrape.dnsSDCheckInterval duration
- Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config for details (default 30s)
+ Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#dns_sd_configs for details (default 30s)
-pushmetrics.extraLabel array
Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags.
diff --git a/app/vmalert/alerting.go b/app/vmalert/alerting.go
index 8c9d98060..89b35deb3 100644
--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@@ -264,13 +264,14 @@ const resolvedRetention = 15 * time.Minute
// Based on the Querier results AlertingRule maintains notifier.Alerts
func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
start := time.Now()
- qMetrics, err := ar.q.Query(ctx, ar.Expr, ts)
+ qMetrics, req, err := ar.q.Query(ctx, ar.Expr, ts)
curState := ruleStateEntry{
time: start,
at: ts,
duration: time.Since(start),
samples: len(qMetrics),
err: err,
+ req: req,
}
defer func() {
@@ -294,7 +295,10 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
}
}
- qFn := func(query string) ([]datasource.Metric, error) { return ar.q.Query(ctx, query, ts) }
+ qFn := func(query string) ([]datasource.Metric, error) {
+ res, _, err := ar.q.Query(ctx, query, ts)
+ return res, err
+ }
updated := make(map[uint64]struct{})
// update list of active alerts
for _, m := range qMetrics {
@@ -320,16 +324,11 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
a.ActiveAt = ts
ar.logDebugf(ts, a, "INACTIVE => PENDING")
}
- if a.Value != m.Values[0] {
- // update Value field with the latest value
- a.Value = m.Values[0]
- // and re-exec template since Value can be used
- // in annotations
- a.Annotations, err = a.ExecTemplate(qFn, ls.origin, ar.Annotations)
- if err != nil {
- curState.err = err
- return nil, curState.err
- }
+ a.Value = m.Values[0]
+ // re-exec template since Value or query can be used in annotations
+ a.Annotations, err = a.ExecTemplate(qFn, ls.origin, ar.Annotations)
+ if err != nil {
+ return nil, err
}
continue
}
@@ -595,7 +594,10 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, lookb
}
ts := time.Now()
- qFn := func(query string) ([]datasource.Metric, error) { return ar.q.Query(ctx, query, ts) }
+ qFn := func(query string) ([]datasource.Metric, error) {
+ res, _, err := ar.q.Query(ctx, query, ts)
+ return res, err
+ }
// account for external labels in filter
var labelsFilter string
@@ -605,7 +607,7 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, lookb
expr := fmt.Sprintf("last_over_time(%s{alertname=%q%s}[%ds])",
alertForStateMetricName, ar.Name, labelsFilter, int(lookback.Seconds()))
- qMetrics, err := q.Query(ctx, expr, ts)
+ qMetrics, _, err := q.Query(ctx, expr, ts)
if err != nil {
return err
}
diff --git a/app/vmalert/datasource/datasource.go b/app/vmalert/datasource/datasource.go
index 8f8662125..898061fe7 100644
--- a/app/vmalert/datasource/datasource.go
+++ b/app/vmalert/datasource/datasource.go
@@ -2,18 +2,27 @@ package datasource
import (
"context"
+ "net/http"
"net/url"
"time"
)
// Querier interface wraps Query and QueryRange methods
type Querier interface {
- Query(ctx context.Context, query string, ts time.Time) ([]Metric, error)
+ // Query executes instant request with the given query at the given ts.
+ // It returns list of Metric in response, the http.Request used for sending query
+ // and error if any. Returned http.Request can't be reused and its body is already read.
+ // Query should stop once ctx is cancelled.
+ Query(ctx context.Context, query string, ts time.Time) ([]Metric, *http.Request, error)
+ // QueryRange executes range request with the given query on the given time range.
+ // It returns list of Metric in response and error if any.
+ // QueryRange should stop once ctx is cancelled.
QueryRange(ctx context.Context, query string, from, to time.Time) ([]Metric, error)
}
// QuerierBuilder builds Querier with given params.
type QuerierBuilder interface {
+ // BuildWithParams creates a new Querier object with the given params
BuildWithParams(params QuerierParams) Querier
}
diff --git a/app/vmalert/datasource/vm.go b/app/vmalert/datasource/vm.go
index 331a5827a..83fb5060d 100644
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@@ -98,10 +98,10 @@ func NewVMStorage(baseURL string, authCfg *promauth.Config, lookBack time.Durati
}
// Query executes the given query and returns parsed response
-func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Metric, error) {
+func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Metric, *http.Request, error) {
req, err := s.newRequestPOST()
if err != nil {
- return nil, err
+ return nil, nil, err
}
switch s.dataSourceType {
@@ -110,12 +110,12 @@ func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Me
case datasourceGraphite:
s.setGraphiteReqParams(req, query, ts)
default:
- return nil, fmt.Errorf("engine not found: %q", s.dataSourceType)
+ return nil, nil, fmt.Errorf("engine not found: %q", s.dataSourceType)
}
resp, err := s.do(ctx, req)
if err != nil {
- return nil, err
+ return nil, req, err
}
defer func() {
_ = resp.Body.Close()
@@ -125,7 +125,8 @@ func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Me
if s.dataSourceType != datasourcePrometheus {
parseFn = parseGraphiteResponse
}
- return parseFn(req, resp)
+ result, err := parseFn(req, resp)
+ return result, req, err
}
// QueryRange executes the given query on the given time range.
diff --git a/app/vmalert/datasource/vm_test.go b/app/vmalert/datasource/vm_test.go
index a63e7220d..c01c021e6 100644
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@@ -94,7 +94,7 @@ func TestVMInstantQuery(t *testing.T) {
ts := time.Now()
expErr := func(err string) {
- if _, err := pq.Query(ctx, query, ts); err == nil {
+ if _, _, err := pq.Query(ctx, query, ts); err == nil {
t.Fatalf("expected %q got nil", err)
}
}
@@ -106,7 +106,7 @@ func TestVMInstantQuery(t *testing.T) {
expErr("unknown status") // 4
expErr("non-vector resultType error") // 5
- m, err := pq.Query(ctx, query, ts) // 6 - vector
+ m, _, err := pq.Query(ctx, query, ts) // 6 - vector
if err != nil {
t.Fatalf("unexpected %s", err)
}
@@ -129,10 +129,13 @@ func TestVMInstantQuery(t *testing.T) {
t.Fatalf("unexpected metric %+v want %+v", m, expected)
}
- m, err = pq.Query(ctx, query, ts) // 7 - scalar
+ m, req, err := pq.Query(ctx, query, ts) // 7 - scalar
if err != nil {
t.Fatalf("unexpected %s", err)
}
+ if req == nil {
+ t.Fatalf("expected request to be non-nil")
+ }
if len(m) != 1 {
t.Fatalf("expected 1 metrics got %d in %+v", len(m), m)
}
@@ -148,7 +151,7 @@ func TestVMInstantQuery(t *testing.T) {
gq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourceGraphite)})
- m, err = gq.Query(ctx, queryRender, ts) // 8 - graphite
+ m, _, err = gq.Query(ctx, queryRender, ts) // 8 - graphite
if err != nil {
t.Fatalf("unexpected %s", err)
}
diff --git a/app/vmalert/helpers_test.go b/app/vmalert/helpers_test.go
index 1a7978cc1..5ee9bb945 100644
--- a/app/vmalert/helpers_test.go
+++ b/app/vmalert/helpers_test.go
@@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
+ "net/http"
"reflect"
"sort"
"sync"
@@ -44,18 +45,20 @@ func (fq *fakeQuerier) BuildWithParams(_ datasource.QuerierParams) datasource.Qu
}
func (fq *fakeQuerier) QueryRange(ctx context.Context, q string, _, _ time.Time) ([]datasource.Metric, error) {
- return fq.Query(ctx, q, time.Now())
+ req, _, err := fq.Query(ctx, q, time.Now())
+ return req, err
}
-func (fq *fakeQuerier) Query(_ context.Context, _ string, _ time.Time) ([]datasource.Metric, error) {
+func (fq *fakeQuerier) Query(_ context.Context, _ string, _ time.Time) ([]datasource.Metric, *http.Request, error) {
fq.Lock()
defer fq.Unlock()
if fq.err != nil {
- return nil, fq.err
+ return nil, nil, fq.err
}
cp := make([]datasource.Metric, len(fq.metrics))
copy(cp, fq.metrics)
- return cp, nil
+ req, _ := http.NewRequest(http.MethodPost, "foo.com", nil)
+ return cp, req, nil
}
type fakeNotifier struct {
diff --git a/app/vmalert/main.go b/app/vmalert/main.go
index 2b8d7e8f1..01adb9972 100644
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@@ -71,7 +71,7 @@ If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.`)
disableAlertGroupLabel = flag.Bool("disableAlertgroupLabel", false, "Whether to disable adding group's Name as label to generated alerts and time series.")
- dryRun = flag.Bool("dryRun", false, "Whether to check only config files without running vmalert. The rules file are validated. The `-rule` flag must be specified.")
+ dryRun = flag.Bool("dryRun", false, "Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.")
)
var alertURLGeneratorFn notifier.AlertURLGenerator
diff --git a/app/vmalert/recording.go b/app/vmalert/recording.go
index b15059395..f7bfcb09b 100644
--- a/app/vmalert/recording.go
+++ b/app/vmalert/recording.go
@@ -115,12 +115,13 @@ func (rr *RecordingRule) ExecRange(ctx context.Context, start, end time.Time) ([
// Exec executes RecordingRule expression via the given Querier.
func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
start := time.Now()
- qMetrics, err := rr.q.Query(ctx, rr.Expr, ts)
+ qMetrics, req, err := rr.q.Query(ctx, rr.Expr, ts)
curState := ruleStateEntry{
time: start,
at: ts,
duration: time.Since(start),
samples: len(qMetrics),
+ req: req,
}
defer func() {
diff --git a/app/vmalert/rule.go b/app/vmalert/rule.go
index 55e6f7070..18734e5cf 100644
--- a/app/vmalert/rule.go
+++ b/app/vmalert/rule.go
@@ -3,6 +3,7 @@ package main
import (
"context"
"errors"
+ "net/http"
"sync"
"time"
@@ -53,6 +54,8 @@ type ruleStateEntry struct {
// stores the number of samples returned during
// the last evaluation
samples int
+ // stores the HTTP request used by datasource during rule.Exec
+ req *http.Request
}
const defaultStateEntriesLimit = 20
diff --git a/app/vmalert/tpl/header.qtpl b/app/vmalert/tpl/header.qtpl
index acc98573d..42390fa1f 100644
--- a/app/vmalert/tpl/header.qtpl
+++ b/app/vmalert/tpl/header.qtpl
@@ -59,6 +59,15 @@
background-color: rgba(0,0,0,.5);
-webkit-box-shadow: 0 0 1px rgba(255,255,255,.5);
}
+ textarea.curl-area{
+ width: 100%;
+ line-height: 1;
+ font-size: 12px;
+ border: none;
+ margin: 0;
+ padding: 0;
+ overflow: scroll;
+ }
diff --git a/app/vmalert/tpl/header.qtpl.go b/app/vmalert/tpl/header.qtpl.go
index 2caee4cf8..56c9002a1 100644
--- a/app/vmalert/tpl/header.qtpl.go
+++ b/app/vmalert/tpl/header.qtpl.go
@@ -101,143 +101,152 @@ func StreamHeader(qw422016 *qt422016.Writer, r *http.Request, navItems []NavItem
background-color: rgba(0,0,0,.5);
-webkit-box-shadow: 0 0 1px rgba(255,255,255,.5);
}
+ textarea.curl-area{
+ width: 100%;
+ line-height: 1;
+ font-size: 12px;
+ border: none;
+ margin: 0;
+ padding: 0;
+ overflow: scroll;
+ }
`)
-//line app/vmalert/tpl/header.qtpl:65
+//line app/vmalert/tpl/header.qtpl:74
streamprintNavItems(qw422016, r, title, navItems)
-//line app/vmalert/tpl/header.qtpl:65
+//line app/vmalert/tpl/header.qtpl:74
qw422016.N().S(`