diff --git a/app/vmalert/README.md b/app/vmalert/README.md index 2c79e58fa6..30416928b2 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -120,9 +120,10 @@ name: [ eval_offset: ] # Optional -# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from datasource. -# By default, use flag `-rule.evalDelay` equal to `-search.latencyOffset` (a cmd-line flag configured for VictoriaMetrics single-node or vmselect). But if group has `latency_offset` param which value differs from `-search.latencyOffset`, set `eval_delay` equal to `latency_offset`. -# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155. +# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource. +# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details. +# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`. +# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyConcepts.html#query-latency. [ eval_delay: ] # Limit the number of alerts an alerting rule and series a recording @@ -808,11 +809,13 @@ at least two times bigger than the resolution. > Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring. -By default, recently written samples to VictoriaMetrics aren't visible for queries for up to 30s -(see `-search.latencyOffset` command-line flag at vmselect, and it can be overridden by adding `latency_offset` to group's params). Such delay is needed to eliminate risk of incomplete -data on the moment of querying, since metrics collectors won't be able to deliver the data in time. +By default, recently written samples to VictoriaMetrics [aren't visible for queries](https://docs.victoriametrics.com/keyConcepts.html#query-latency) +for up to 30s (see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of +incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time. To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to 30s by default. -If you changed the `-search.latencyOffset`(cmd-line flag configured for VictoriaMetrics single-node or vmselect) value and observed a delay in timestamps for produced evaluation results, try changing `-rule.evalDelay` equal to `-search.latencyOffset`. +If you changed the `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect) value +or specified custom `latency_offset` param via [Group](#groups) and observed a delay in timestamps for produced +evaluation results - try changing `-rule.evalDelay` equal to `-search.latencyOffset`. ### Alerts state @@ -975,7 +978,7 @@ The shortlist of configuration flags is the following: -datasource.headers string Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2' -datasource.lookback duration - Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query. + Will be deprecated soon, please adjust "-search.latencyOffset" at datasource side or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query. -datasource.maxIdleConnections int Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100) -datasource.oauth2.clientID string @@ -1297,9 +1300,8 @@ The shortlist of configuration flags is the following: Limits the maximum duration for automatic alert expiration, which by default is 4 times evaluationInterval of the parent group. -rule.resendDelay duration Minimum amount of time to wait before resending an alert to notifier - -rule.evalDelay duration - Adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource. Normally should equal to `-search.latencyOffset`(a cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s) - See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). + -rule.evalDelay time + Adjustment of the time parameter for rule evaluation requests to compensate intentional data delay from the datasource.Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s) -rule.templates array Path or glob pattern to location with go template definitions for rules annotations templating. Flag can be specified multiple times. diff --git a/app/vmalert/config/testdata/rules/rules3-good.rules b/app/vmalert/config/testdata/rules/rules3-good.rules index eb2a78d4f1..ea65a9486e 100644 --- a/app/vmalert/config/testdata/rules/rules3-good.rules +++ b/app/vmalert/config/testdata/rules/rules3-good.rules @@ -15,6 +15,7 @@ groups: interval: 2s concurrency: 2 type: prometheus + eval_delay: 30s rules: - alert: Conns expr: sum(vm_tcplistener_conns) by (instance) > 1 diff --git a/app/vmalert/datasource/init.go b/app/vmalert/datasource/init.go index 391c9e2bb4..b8bb570f84 100644 --- a/app/vmalert/datasource/init.go +++ b/app/vmalert/datasource/init.go @@ -84,10 +84,10 @@ func Init(extraParams url.Values) (QuerierBuilder, error) { return nil, fmt.Errorf("datasource.url is empty") } if !*queryTimeAlignment { - logger.Warnf("flag `datasource.queryTimeAlignment` is deprecated and will be removed in next releases, please use `eval_alignment` in rule group instead") + logger.Warnf("flag `-datasource.queryTimeAlignment` is deprecated and will be removed in next releases. Please use `eval_alignment` in rule group instead.") } if *lookBack != 0 { - logger.Warnf("flag `datasource.lookback` will be deprecated soon, please adjust `-search.latencyOffset`(a cmd-line flag configured for VictoriaMetrics single-node or vmselect) or specify `latency_offset` in rule group's params. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155.") + logger.Warnf("flag `-datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 for details.") } tr, err := utils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify) diff --git a/app/vmalert/rule/group.go b/app/vmalert/rule/group.go index 0d9b9814e2..9369b43f13 100644 --- a/app/vmalert/rule/group.go +++ b/app/vmalert/rule/group.go @@ -32,8 +32,8 @@ var ( resendDelay = flag.Duration("rule.resendDelay", 0, "MiniMum amount of time to wait before resending an alert to notifier") maxResolveDuration = flag.Duration("rule.maxResolveDuration", 0, "Limits the maxiMum duration for automatic alert expiration, "+ "which by default is 4 times evaluationInterval of the parent ") - evalDelay = flag.Duration("rule.evalDelay", 30*time.Second, "Adjust the `time` parameter of rule evaluation requests to compensate intentional data query delay from datasource."+ - "Normally should equal to `-search.latencyOffset`(a cmd-line flag configured for VictoriaMetrics single-node or vmselect)") + evalDelay = flag.Duration("rule.evalDelay", 30*time.Second, "Adjustment of the `time` parameter for rule evaluation requests to compensate intentional data delay from the datasource."+ + "Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect).") disableAlertGroupLabel = flag.Bool("disableAlertgroupLabel", false, "Whether to disable adding group's Name as label to generated alerts and time series.") remoteReadLookBack = flag.Duration("remoteRead.lookback", time.Hour, "Lookback defines how far to look into past for alerts timeseries."+ " For example, if lookback=1h then range from now() to now()-1h will be scanned.") @@ -48,7 +48,7 @@ type Group struct { Type config.Type Interval time.Duration EvalOffset *time.Duration - // EvalDelay will adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource. + // EvalDelay will adjust timestamp for rule evaluation requests to compensate intentional query delay from datasource. // see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 EvalDelay *time.Duration Limit int @@ -339,7 +339,7 @@ func (g *Group) Start(ctx context.Context, nts func() []notifier.Notifier, rw re Rw: rw, Notifiers: nts, notifierHeaders: g.NotifierHeaders, - PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), + previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), } g.infof("started") @@ -528,7 +528,7 @@ func (g *Group) ExecOnce(ctx context.Context, nts func() []notifier.Notifier, rw Rw: rw, Notifiers: nts, notifierHeaders: g.NotifierHeaders, - PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), + previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), } if len(g.Rules) < 1 { return nil @@ -593,13 +593,18 @@ func (g *Group) adjustReqTimestamp(timestamp time.Time) time.Time { // since it should be always aligned. return ts } + + // account for delay before adjusting the timestamp. + // otherwise, the alignment may be off if `delay!=g.Interval` + timestamp = timestamp.Add(-g.getEvalDelay()) + if g.evalAlignment == nil || *g.evalAlignment { // align query time with interval to get similar result with grafana when plotting time series. // see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049 // and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232 - return timestamp.Truncate(g.Interval).Add(-g.getEvalDelay()) + return timestamp.Truncate(g.Interval) } - return timestamp.Add(-g.getEvalDelay()) + return timestamp } func (g *Group) getEvalDelay() time.Duration { @@ -617,11 +622,11 @@ type executor struct { Rw remotewrite.RWClient previouslySentSeriesToRWMu sync.Mutex - // PreviouslySentSeriesToRW stores series sent to RW on previous iteration + // previouslySentSeriesToRW stores series sent to RW on previous iteration // map[ruleID]map[ruleLabels][]prompb.Label // where `ruleID` is ID of the Rule within a Group // and `ruleLabels` is []prompb.Label marshalled to a string - PreviouslySentSeriesToRW map[uint64]map[string][]prompbmarshal.Label + previouslySentSeriesToRW map[uint64]map[string][]prompbmarshal.Label } // execConcurrently executes rules concurrently if concurrency>1 @@ -738,7 +743,7 @@ func (e *executor) getStaleSeries(r Rule, tss []prompbmarshal.TimeSeries, timest var staleS []prompbmarshal.TimeSeries // check whether there are series which disappeared and need to be marked as stale e.previouslySentSeriesToRWMu.Lock() - for key, labels := range e.PreviouslySentSeriesToRW[rID] { + for key, labels := range e.previouslySentSeriesToRW[rID] { if _, ok := ruleLabels[key]; ok { continue } @@ -747,7 +752,7 @@ func (e *executor) getStaleSeries(r Rule, tss []prompbmarshal.TimeSeries, timest staleS = append(staleS, ss) } // set previous series to current - e.PreviouslySentSeriesToRW[rID] = ruleLabels + e.previouslySentSeriesToRW[rID] = ruleLabels e.previouslySentSeriesToRWMu.Unlock() return staleS @@ -765,14 +770,14 @@ func (e *executor) purgeStaleSeries(activeRules []Rule) { for _, rule := range activeRules { id := rule.ID() - prev, ok := e.PreviouslySentSeriesToRW[id] + prev, ok := e.previouslySentSeriesToRW[id] if ok { // keep previous series for staleness detection newPreviouslySentSeriesToRW[id] = prev } } - e.PreviouslySentSeriesToRW = nil - e.PreviouslySentSeriesToRW = newPreviouslySentSeriesToRW + e.previouslySentSeriesToRW = nil + e.previouslySentSeriesToRW = newPreviouslySentSeriesToRW e.previouslySentSeriesToRWMu.Unlock() } diff --git a/app/vmalert/rule/group_test.go b/app/vmalert/rule/group_test.go index c3dde435d7..65bdd061d3 100644 --- a/app/vmalert/rule/group_test.go +++ b/app/vmalert/rule/group_test.go @@ -321,7 +321,7 @@ func TestResolveDuration(t *testing.T) { func TestGetStaleSeries(t *testing.T) { ts := time.Now() e := &executor{ - PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), + previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), } f := func(r Rule, labels, expLabels [][]prompbmarshal.Label) { t.Helper() @@ -414,7 +414,7 @@ func TestPurgeStaleSeries(t *testing.T) { f := func(curRules, newRules, expStaleRules []Rule) { t.Helper() e := &executor{ - PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), + previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), } // seed executor with series for // current rules @@ -424,13 +424,13 @@ func TestPurgeStaleSeries(t *testing.T) { e.purgeStaleSeries(newRules) - if len(e.PreviouslySentSeriesToRW) != len(expStaleRules) { + if len(e.previouslySentSeriesToRW) != len(expStaleRules) { t.Fatalf("expected to get %d stale series, got %d", - len(expStaleRules), len(e.PreviouslySentSeriesToRW)) + len(expStaleRules), len(e.previouslySentSeriesToRW)) } for _, exp := range expStaleRules { - if _, ok := e.PreviouslySentSeriesToRW[exp.ID()]; !ok { + if _, ok := e.previouslySentSeriesToRW[exp.ID()]; !ok { t.Fatalf("expected to have rule %d; got nil instead", exp.ID()) } } @@ -515,7 +515,7 @@ func TestFaultyRW(t *testing.T) { e := &executor{ Rw: &remotewrite.Client{}, - PreviouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), + previouslySentSeriesToRW: make(map[uint64]map[string][]prompbmarshal.Label), } err := e.exec(context.Background(), r, time.Now(), 0, 10) diff --git a/app/vmalert/web_types.go b/app/vmalert/web_types.go index 7099c53e05..62d17f379b 100644 --- a/app/vmalert/web_types.go +++ b/app/vmalert/web_types.go @@ -94,6 +94,10 @@ type apiGroup struct { NotifierHeaders []string `json:"notifier_headers,omitempty"` // Labels is a set of label value pairs, that will be added to every rule. Labels map[string]string `json:"labels,omitempty"` + // EvalOffset Group will be evaluated at the exact time offset on the range of [0...evaluationInterval] + EvalOffset float64 `json:"eval_offset,omitempty"` + // EvalDelay will adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource. + EvalDelay float64 `json:"eval_delay,omitempty"` } // groupAlerts represents a group of alerts for WEB view @@ -309,6 +313,12 @@ func groupToAPI(g *rule.Group) apiGroup { Labels: g.Labels, } + if g.EvalOffset != nil { + ag.EvalOffset = g.EvalOffset.Seconds() + } + if g.EvalDelay != nil { + ag.EvalDelay = g.EvalDelay.Seconds() + } ag.Rules = make([]apiRule, 0) for _, r := range g.Rules { ag.Rules = append(ag.Rules, ruleToAPI(r)) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index b357abb0b7..3b60a11430 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -29,7 +29,7 @@ The sandbox cluster installation is running under the constant load generated by ## tip **vmalert's cmd-line flag `datasource.queryTimeAlignment` was deprecated and will have no effect anymore. It will be completely removed in next releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049) and more detailed changes below.** -**vmalert's cmd-line flag `datasource.lookback` will be deprecated soon, please use `-search.latencyOffset` in datasource or override it by adding `latency_offset` in rule group's params. It will have no effect in next release and be removed in future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).** +**vmalert's cmd-line flag `datasource.lookback` will be deprecated soon. Please use `-rule.evalDelay` command-line flag instead. It will have no effect in next release and be removed in future releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155).** * SECURITY: upgrade Go builder from Go1.21.1 to Go1.21.3. See [the list of issues addressed in Go1.21.2](https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved) and [the list of issues addressed in Go1.21.3](https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved). @@ -38,7 +38,7 @@ The sandbox cluster installation is running under the constant load generated by * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `eval_alignment` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups), it will align group query requests timestamp with interval like `datasource.queryTimeAlignment` did. This also means that `datasource.queryTimeAlignment` command-line flag becomes deprecated now and will have no effect if configured. If `datasource.queryTimeAlignment` was set to `false` before, then `eval_alignment` has to be set to `false` explicitly under group. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049). -* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `-rule.evalDelay` flag and `eval_delay` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups). The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match intentional query delay from the datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). +* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `-rule.evalDelay` flag and `eval_delay` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups). The new flag and param can be used to adjust the `time` parameter for rule evaluation requests to match [intentional query delay](https://docs.victoriametrics.com/keyConcepts.html#query-latency) from the datasource. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): support data ingestion from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-newrelic-agent), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712). * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): add `-filestream.disableFadvise` command-line flag, which can be used for disabling `fadvise` syscall during backup upload to the remote storage. By default `vmbackup` uses `fadvise` syscall in order to prevent from eviction of recently accessed data from the [OS page cache](https://en.wikipedia.org/wiki/Page_cache) when backing up large files. Sometimes the `fadvise` syscall may take significant amounts of CPU when the backup is performed with large value of `-concurrency` command-line flag on systems with big number of CPU cores. In this case it is better to manually disable `fadvise` syscall by passing `-filestream.disableFadvise` command-line flag to `vmbackup`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120) for details. * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): add `-deleteAllObjectVersions` command-line flag, which can be used for forcing removal of all object versions in remote object storage. See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121) issue and [these docs](https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages) for the details. diff --git a/docs/vmalert.md b/docs/vmalert.md index 5c3834a4b5..7da9e888cf 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -131,9 +131,10 @@ name: [ eval_offset: ] # Optional -# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from datasource. -# By default, use flag `-rule.evalDelay` equal to `-search.latencyOffset` (a cmd-line flag configured for VictoriaMetrics single-node or vmselect). But if group has `latency_offset` param which value differs from `-search.latencyOffset`, set `eval_delay` equal to `latency_offset`. -# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155. +# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource. +# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details. +# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`. +# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyConcepts.html#query-latency. [ eval_delay: ] # Limit the number of alerts an alerting rule and series a recording @@ -819,11 +820,13 @@ at least two times bigger than the resolution. > Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring. -By default, recently written samples to VictoriaMetrics aren't visible for queries for up to 30s -(see `-search.latencyOffset` command-line flag at vmselect, and it can be overridden by adding `latency_offset` to group's params). Such delay is needed to eliminate risk of incomplete -data on the moment of querying, since metrics collectors won't be able to deliver the data in time. +By default, recently written samples to VictoriaMetrics [aren't visible for queries](https://docs.victoriametrics.com/keyConcepts.html#query-latency) +for up to 30s (see `-search.latencyOffset` command-line flag at vmselect). Such delay is needed to eliminate risk of +incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time. To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to 30s by default. -If you changed the `-search.latencyOffset`(cmd-line flag configured for VictoriaMetrics single-node or vmselect) value and observed a delay in timestamps for produced evaluation results, try changing `-rule.evalDelay` equal to `-search.latencyOffset`. +If you changed the `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect) value +or specified custom `latency_offset` param via [Group](#groups) and observed a delay in timestamps for produced +evaluation results - try changing `-rule.evalDelay` equal to `-search.latencyOffset`. ### Alerts state @@ -986,7 +989,7 @@ The shortlist of configuration flags is the following: -datasource.headers string Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. For example, -datasource.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -datasource.url. Multiple headers must be delimited by '^^': -datasource.headers='header1:value1^^header2:value2' -datasource.lookback duration - Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query. + Will be deprecated soon, please adjust "-search.latencyOffset" at datasource side or specify "latency_offset" in rule group's params. Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query. -datasource.maxIdleConnections int Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100) -datasource.oauth2.clientID string @@ -1308,9 +1311,8 @@ The shortlist of configuration flags is the following: Limits the maximum duration for automatic alert expiration, which by default is 4 times evaluationInterval of the parent group. -rule.resendDelay duration Minimum amount of time to wait before resending an alert to notifier - -rule.evalDelay duration - Adjust the `time` parameter of rule evaluation requests to compensate intentional query delay from datasource. Normally should equal to `-search.latencyOffset`(a cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s) - See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155). + -rule.evalDelay time + Adjustment of the time parameter for rule evaluation requests to compensate intentional data delay from the datasource.Normally, should be equal to `-search.latencyOffset` (cmd-line flag configured for VictoriaMetrics single-node or vmselect). (default 30s) -rule.templates array Path or glob pattern to location with go template definitions for rules annotations templating. Flag can be specified multiple times.