github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2025-03-11 15:34:56 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

Browse source
This commit is contained in:
Aliaksandr Valialkin 2022-03-16 12:59:06 +02:00
commit e636cab272
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
31 changed files with 990 additions and 629 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
README.md
View file

@ -1455,6 +1455,27 @@ See the full description of flags [here](#list-of-command-line-flags).
## Data migration
### From VictoriaMetrics
The simplest way to migrate data from one single-node (source) to another (destination), or from one vmstorage node
to another do the following:
1. Stop the VictoriaMetrics (source) with `kill -INT`;
2. Copy (via [rsync](https://en.wikipedia.org/wiki/Rsync) or any other tool) the entire folder specified
via `-storageDataPath` from the source node to the empty folder at the destination node.
3. Once copy is done, stop the VictoriaMetrics (destination) with `kill -INT` and verify that
its `-storageDataPath` points to the copied folder from p.2;
4. Start the VictoriaMetrics (destination). The copied data should be now available.
Things to consider when copying data:
1. Data formats between single-node and vmstorage node aren't compatible and can't be copied.
2. Copying data folder means complete replacement of the previous data on destination VictoriaMetrics.
For more complex scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction
of data - see [vmctl. Migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl.html#migrating-data-from-victoriametrics).
### From other systems
Use [vmctl](https://docs.victoriametrics.com/vmctl.html) for data migration. It supports the following data migration types:
* From Prometheus to VictoriaMetrics

59
app/vmalert/README.md
View file

@ -3,7 +3,7 @@
`vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
rules against configured `-datasource.url`. For sending alerting notifications
vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag.
vmalert relies on [Alertmanager](https://github.com/prometheus/alertmanager) configured via `-notifier.url` flag.
Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
protocol and require `-remoteWrite.url` to be configured.
Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
@ -296,7 +296,7 @@ Cluster mode could have multiple `vminsert` and `vmselect` components.
```
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
-remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results
-remoteWrite.url=http://vminsert:8480/insert/0/prometheus # vminsert addr to persist alerts state and recording rules results
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
```
@ -352,7 +352,7 @@ or reducing resolution) and push results to "cold" cluster.
```
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results
```
<img alt="vmalert multi cluster" src="vmalert_multicluster.png">
@ -514,8 +514,20 @@ The shortlist of configuration flags is the following:
Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
-datasource.maxIdleConnections int
Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
-datasource.oauth2.clientID string
Optional OAuth2 clientID to use for -datasource.url.
-datasource.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -datasource.url.
-datasource.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -datasource.url.
-datasource.oauth2.scopes string
Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
-datasource.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -datasource.url.
-datasource.queryStep duration
queryStep defines how far a value can fallback to when evaluating queries. For example, if datasource.queryStep=15s then param "step" with value "15s" will be added to every query.If queryStep isn't specified, rule's evaluationInterval will be used instead.
-datasource.queryTimeAlignment
Whether to align "time" parameter with evaluation interval.Alignment supposed to produce deterministic results despite of number of vmalert replicas or time they were started. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257 (default true)
-datasource.roundDigits int
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
-datasource.tlsCAFile string
@ -606,8 +618,29 @@ The shortlist of configuration flags is the following:
-notifier.basicAuth.username array
Optional basic auth username for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerToken array
Optional bearer token for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerTokenFile array
Optional path to bearer token file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.config string
Path to configuration file for notifiers
-notifier.oauth2.clientID array
Optional OAuth2 clientID to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecret array
Optional OAuth2 clientSecret to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecretFile array
Optional OAuth2 clientSecretFile to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.scopes array
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.tokenUrl array
Optional OAuth2 tokenURL to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.suppressDuplicateTargetErrors
Whether to suppress 'duplicate target' errors during discovery
-notifier.tlsCAFile array
@ -654,6 +687,16 @@ The shortlist of configuration flags is the following:
Whether to ignore errors from remote storage when restoring alerts state on startup. (default true)
-remoteRead.lookback duration
Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
-remoteRead.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteRead.url.
-remoteRead.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteRead.url.
-remoteRead.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteRead.url.
-remoteRead.oauth2.scopes string
Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.
-remoteRead.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -remoteRead.url.
-remoteRead.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteRead.url. By default system CA is used
-remoteRead.tlsCertFile string
@ -686,6 +729,16 @@ The shortlist of configuration flags is the following:
Defines defines max number of timeseries to be flushed at once (default 1000)
-remoteWrite.maxQueueSize int
Defines the max number of pending datapoints to remote write endpoint (default 100000)
-remoteWrite.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteWrite.url.
-remoteWrite.oauth2.scopes string
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
-remoteWrite.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -notifier.url.
-remoteWrite.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default system CA is used
-remoteWrite.tlsCertFile string

63
app/vmalert/alerting.go
View file

@ -38,6 +38,8 @@ type AlertingRule struct {
alerts map[uint64]*notifier.Alert
// stores last moment of time Exec was called
lastExecTime time.Time
// stores the duration of the last Exec call
lastExecDuration time.Duration
// stores last error that happened in Exec func
// resets on every successful Exec
// may be used as Health state
@ -203,12 +205,14 @@ func (ar *AlertingRule) ExecRange(ctx context.Context, start, end time.Time) ([]
// Exec executes AlertingRule expression via the given Querier.
// Based on the Querier results AlertingRule maintains notifier.Alerts
func (ar *AlertingRule) Exec(ctx context.Context) ([]prompbmarshal.TimeSeries, error) {
start := time.Now()
qMetrics, err := ar.q.Query(ctx, ar.Expr)
ar.mu.Lock()
defer ar.mu.Unlock()
ar.lastExecTime = start
ar.lastExecDuration = time.Since(start)
ar.lastExecError = err
ar.lastExecTime = time.Now()
ar.lastExecSamples = len(qMetrics)
if err != nil {
return nil, fmt.Errorf("failed to execute query %q: %w", ar.Expr, err)
@ -386,31 +390,48 @@ func (ar *AlertingRule) AlertAPI(id uint64) *APIAlert {
return ar.newAlertAPI(*a)
}
// RuleAPI returns Rule representation in form
// of APIAlertingRule
func (ar *AlertingRule) RuleAPI() APIAlertingRule {
var lastErr string
// ToAPI returns Rule representation in form
// of APIRule
func (ar *AlertingRule) ToAPI() APIRule {
r := APIRule{
Type: "alerting",
DatasourceType: ar.Type.String(),
Name: ar.Name,
Query: ar.Expr,
Duration: ar.For.Seconds(),
Labels: ar.Labels,
Annotations: ar.Annotations,
LastEvaluation: ar.lastExecTime,
EvaluationTime: ar.lastExecDuration.Seconds(),
Health: "ok",
State: "inactive",
Alerts: ar.AlertsToAPI(),
LastSamples: ar.lastExecSamples,
// encode as strings to avoid rounding in JSON
ID: fmt.Sprintf("%d", ar.ID()),
GroupID: fmt.Sprintf("%d", ar.GroupID),
}
if ar.lastExecError != nil {
lastErr = ar.lastExecError.Error()
r.LastError = ar.lastExecError.Error()
r.Health = "err"
}
return APIAlertingRule{
// encode as strings to avoid rounding
ID: fmt.Sprintf("%d", ar.ID()),
GroupID: fmt.Sprintf("%d", ar.GroupID),
Type: ar.Type.String(),
Name: ar.Name,
Expression: ar.Expr,
For: ar.For.String(),
LastError: lastErr,
LastSamples: ar.lastExecSamples,
LastExec: ar.lastExecTime,
Labels: ar.Labels,
Annotations: ar.Annotations,
// satisfy APIRule.State logic
if len(r.Alerts) > 0 {
r.State = notifier.StatePending.String()
stateFiring := notifier.StateFiring.String()
for _, a := range r.Alerts {
if a.State == stateFiring {
r.State = stateFiring
break
}
}
}
return r
}
// AlertsAPI generates list of APIAlert objects from existing alerts
func (ar *AlertingRule) AlertsAPI() []*APIAlert {
// AlertsToAPI generates list of APIAlert objects from existing alerts
func (ar *AlertingRule) AlertsToAPI() []*APIAlert {
var alerts []*APIAlert
ar.mu.RLock()
for _, a := range ar.alerts {

21
app/vmalert/datasource/init.go
View file

@ -13,12 +13,14 @@ import (
var (
addr = flag.String("datasource.url", "", "VictoriaMetrics or vmselect url. Required parameter. "+
"E.g. http://127.0.0.1:8428")
appendTypePrefix = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
appendTypePrefix = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
basicAuthUsername = flag.String("datasource.basicAuth.username", "", "Optional basic auth username for -datasource.url")
basicAuthPassword = flag.String("datasource.basicAuth.password", "", "Optional basic auth password for -datasource.url")
basicAuthPasswordFile = flag.String("datasource.basicAuth.passwordFile", "", "Optional path to basic auth password to use for -datasource.url")
bearerToken = flag.String("datasource.bearerToken", "", "Optional bearer auth token to use for -datasource.url.")
bearerTokenFile = flag.String("datasource.bearerTokenFile", "", "Optional path to bearer token file to use for -datasource.url.")
bearerToken = flag.String("datasource.bearerToken", "", "Optional bearer auth token to use for -datasource.url.")
bearerTokenFile = flag.String("datasource.bearerTokenFile", "", "Optional path to bearer token file to use for -datasource.url.")
tlsInsecureSkipVerify = flag.Bool("datasource.tlsInsecureSkipVerify", false, "Whether to skip tls verification when connecting to -datasource.url")
tlsCertFile = flag.String("datasource.tlsCertFile", "", "Optional path to client-side TLS certificate file to use when connecting to -datasource.url")
@ -26,10 +28,18 @@ var (
tlsCAFile = flag.String("datasource.tlsCAFile", "", `Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used`)
tlsServerName = flag.String("datasource.tlsServerName", "", `Optional TLS server name to use for connections to -datasource.url. By default, the server name from -datasource.url is used`)
oauth2ClientID = flag.String("datasource.oauth2.clientID", "", "Optional OAuth2 clientID to use for -datasource.url. ")
oauth2ClientSecret = flag.String("datasource.oauth2.clientSecret", "", "Optional OAuth2 clientSecret to use for -datasource.url.")
oauth2ClientSecretFile = flag.String("datasource.oauth2.clientSecretFile", "", "Optional OAuth2 clientSecretFile to use for -datasource.url. ")
oauth2TokenURL = flag.String("datasource.oauth2.tokenUrl", "", "Optional OAuth2 tokenURL to use for -datasource.url.")
oauth2Scopes = flag.String("datasource.oauth2.scopes", "", "Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'")
lookBack = flag.Duration("datasource.lookback", 0, `Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.`)
queryStep = flag.Duration("datasource.queryStep", 0, "queryStep defines how far a value can fallback to when evaluating queries. "+
"For example, if datasource.queryStep=15s then param \"step\" with value \"15s\" will be added to every query."+
"If queryStep isn't specified, rule's evaluationInterval will be used instead.")
queryTimeAlignment = flag.Bool("datasource.queryTimeAlignment", true, `Whether to align "time" parameter with evaluation interval.`+
"Alignment supposed to produce deterministic results despite of number of vmalert replicas or time they were started. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257")
maxIdleConnections = flag.Int("datasource.maxIdleConnections", 100, `Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state.`)
roundDigits = flag.Int("datasource.roundDigits", 0, `Adds "round_digits" GET param to datasource requests. `+
`In VM "round_digits" limits the number of digits after the decimal point in response values.`)
@ -64,7 +74,10 @@ func Init(extraParams url.Values) (QuerierBuilder, error) {
extraParams.Set("round_digits", fmt.Sprintf("%d", *roundDigits))
}
authCfg, err := utils.AuthConfig(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile, *bearerToken, *bearerTokenFile)
authCfg, err := utils.AuthConfig(
utils.WithBasicAuth(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile),
utils.WithBearer(*bearerToken, *bearerTokenFile),
utils.WithOAuth(*oauth2ClientID, *oauth2ClientSecret, *oauth2ClientSecretFile, *oauth2TokenURL, *oauth2Scopes))
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}

2
app/vmalert/datasource/vm_prom_api.go
View file

@ -125,7 +125,7 @@ func (s *VMStorage) setPrometheusInstantReqParams(r *http.Request, query string,
if s.lookBack > 0 {
timestamp = timestamp.Add(-s.lookBack)
}
if s.evaluationInterval > 0 {
if *queryTimeAlignment && s.evaluationInterval > 0 {
// see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232
timestamp = timestamp.Truncate(s.evaluationInterval)
}

54
app/vmalert/datasource/vm_test.go
View file

@ -12,6 +12,7 @@ import (
"testing"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
)
@ -517,6 +518,59 @@ func TestRequestParams(t *testing.T) {
}
}
func TestAuthConfig(t *testing.T) {
var testCases = []struct {
name string
vmFn func() *VMStorage
checkFn func(t *testing.T, r *http.Request)
}{
{
name: "basic auth",
vmFn: func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBasicAuth("foo", "bar", ""))
if err != nil {
t.Errorf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
},
checkFn: func(t *testing.T, r *http.Request) {
u, p, _ := r.BasicAuth()
checkEqualString(t, "foo", u)
checkEqualString(t, "bar", p)
},
},
{
name: "bearer auth",
vmFn: func() *VMStorage {
cfg, err := utils.AuthConfig(utils.WithBearer("foo", ""))
if err != nil {
t.Errorf("Error get auth config: %s", err)
}
return &VMStorage{authCfg: cfg}
},
checkFn: func(t *testing.T, r *http.Request) {
reqToken := r.Header.Get("Authorization")
splitToken := strings.Split(reqToken, "Bearer ")
if len(splitToken) != 2 {
t.Errorf("expected two items got %d", len(splitToken))
}
token := splitToken[1]
checkEqualString(t, "foo", token)
},
},
}
for _, tt := range testCases {
t.Run(tt.name, func(t *testing.T) {
vm := tt.vmFn()
req, err := vm.newRequestPOST()
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
tt.checkFn(t, req)
})
}
}
func checkEqualString(t *testing.T, exp, got string) {
t.Helper()
if got != exp {

23
app/vmalert/group.go
View file

@ -19,14 +19,15 @@ import (
// Group is an entity for grouping rules
type Group struct {
mu sync.RWMutex
Name string
File string
Rules []Rule
Type datasource.Type
Interval time.Duration
Concurrency int
Checksum string
mu sync.RWMutex
Name string
File string
Rules []Rule
Type datasource.Type
Interval time.Duration
Concurrency int
Checksum string
LastEvaluation time.Time
Labels map[string]string
Params url.Values
@ -283,6 +284,7 @@ func (g *Group) start(ctx context.Context, nts func() []notifier.Notifier, rw *r
logger.Errorf("group %q: %s", g.Name, err)
}
}
g.LastEvaluation = iterationStart
}
g.metrics.iterationDuration.UpdateDuration(iterationStart)
}
@ -347,6 +349,7 @@ var (
func (e *executor) exec(ctx context.Context, rule Rule, resolveDuration time.Duration) error {
execTotal.Inc()
now := time.Now()
tss, err := rule.Exec(ctx)
if err != nil {
execErrors.Inc()
@ -371,12 +374,12 @@ func (e *executor) exec(ctx context.Context, rule Rule, resolveDuration time.Dur
for _, a := range ar.alerts {
switch a.State {
case notifier.StateFiring:
a.End = time.Now().Add(resolveDuration)
a.End = now.Add(resolveDuration)
alerts = append(alerts, *a)
case notifier.StateInactive:
// set End to execStart to notify
// that it was just resolved
a.End = time.Now()
a.End = now
alerts = append(alerts, *a)
}
}

22
app/vmalert/manager.go
View file

@ -163,21 +163,17 @@ func (g *Group) toAPI() APIGroup {
// encode as string to avoid rounding
ID: fmt.Sprintf("%d", g.ID()),
Name: g.Name,
Type: g.Type.String(),
File: g.File,
Interval: g.Interval.String(),
Concurrency: g.Concurrency,
Params: urlValuesToStrings(g.Params),
Labels: g.Labels,
Name: g.Name,
Type: g.Type.String(),
File: g.File,
Interval: g.Interval.Seconds(),
LastEvaluation: g.LastEvaluation,
Concurrency: g.Concurrency,
Params: urlValuesToStrings(g.Params),
Labels: g.Labels,
}
for _, r := range g.Rules {
switch v := r.(type) {
case *AlertingRule:
ag.AlertingRules = append(ag.AlertingRules, v.RuleAPI())
case *RecordingRule:
ag.RecordingRules = append(ag.RecordingRules, v.RuleAPI())
}
ag.Rules = append(ag.Rules, r.ToAPI())
}
return ag
}

13
app/vmalert/notifier/alertmanager.go
View file

@ -6,6 +6,7 @@ import (
"fmt"
"io/ioutil"
"net/http"
"strings"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
@ -112,11 +113,19 @@ func NewAlertManager(alertManagerURL string, fn AlertURLGenerator, authCfg proma
return nil, fmt.Errorf("failed to create transport: %w", err)
}
ba := &promauth.BasicAuthConfig{}
ba := new(promauth.BasicAuthConfig)
oauth := new(promauth.OAuth2Config)
if authCfg.BasicAuth != nil {
ba = authCfg.BasicAuth
}
aCfg, err := utils.AuthConfig(ba.Username, ba.Password.String(), ba.PasswordFile, authCfg.BearerToken.String(), authCfg.BearerTokenFile)
if authCfg.OAuth2 != nil {
oauth = authCfg.OAuth2
}
aCfg, err := utils.AuthConfig(
utils.WithBasicAuth(ba.Username, ba.Password.String(), ba.PasswordFile),
utils.WithBearer(authCfg.BearerToken.String(), authCfg.BearerTokenFile),
utils.WithOAuth(oauth.ClientID, oauth.ClientSecretFile, oauth.ClientSecretFile, oauth.TokenURL, strings.Join(oauth.Scopes, ";")))
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}

27
app/vmalert/notifier/init.go
View file

@ -15,11 +15,15 @@ var (
configPath = flag.String("notifier.config", "", "Path to configuration file for notifiers")
suppressDuplicateTargetErrors = flag.Bool("notifier.suppressDuplicateTargetErrors", false, "Whether to suppress 'duplicate target' errors during discovery")
addrs = flagutil.NewArray("notifier.url", "Prometheus alertmanager URL, e.g. http://127.0.0.1:9093")
addrs = flagutil.NewArray("notifier.url", "Prometheus alertmanager URL, e.g. http://127.0.0.1:9093")
basicAuthUsername = flagutil.NewArray("notifier.basicAuth.username", "Optional basic auth username for -notifier.url")
basicAuthPassword = flagutil.NewArray("notifier.basicAuth.password", "Optional basic auth password for -notifier.url")
basicAuthPasswordFile = flagutil.NewArray("notifier.basicAuth.passwordFile", "Optional path to basic auth password file for -notifier.url")
bearerToken = flagutil.NewArray("notifier.bearerToken", "Optional bearer token for -notifier.url")
bearerTokenFile = flagutil.NewArray("notifier.bearerTokenFile", "Optional path to bearer token file for -notifier.url")
tlsInsecureSkipVerify = flagutil.NewArrayBool("notifier.tlsInsecureSkipVerify", "Whether to skip tls verification when connecting to -notifier.url")
tlsCertFile = flagutil.NewArray("notifier.tlsCertFile", "Optional path to client-side TLS certificate file to use when connecting to -notifier.url")
tlsKeyFile = flagutil.NewArray("notifier.tlsKeyFile", "Optional path to client-side TLS certificate key to use when connecting to -notifier.url")
@ -27,6 +31,17 @@ var (
"By default system CA is used")
tlsServerName = flagutil.NewArray("notifier.tlsServerName", "Optional TLS server name to use for connections to -notifier.url. "+
"By default the server name from -notifier.url is used")
oauth2ClientID = flagutil.NewArray("notifier.oauth2.clientID", "Optional OAuth2 clientID to use for -notifier.url. "+
"If multiple args are set, then they are applied independently for the corresponding -notifier.url")
oauth2ClientSecret = flagutil.NewArray("notifier.oauth2.clientSecret", "Optional OAuth2 clientSecret to use for -notifier.url. "+
"If multiple args are set, then they are applied independently for the corresponding -notifier.url")
oauth2ClientSecretFile = flagutil.NewArray("notifier.oauth2.clientSecretFile", "Optional OAuth2 clientSecretFile to use for -notifier.url. "+
"If multiple args are set, then they are applied independently for the corresponding -notifier.url")
oauth2TokenURL = flagutil.NewArray("notifier.oauth2.tokenUrl", "Optional OAuth2 tokenURL to use for -notifier.url. "+
"If multiple args are set, then they are applied independently for the corresponding -notifier.url")
oauth2Scopes = flagutil.NewArray("notifier.oauth2.scopes", "Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'. "+
"If multiple args are set, then they are applied independently for the corresponding -notifier.url")
)
// cw holds a configWatcher for configPath configuration file
@ -111,7 +126,17 @@ func notifiersFromFlags(gen AlertURLGenerator) ([]Notifier, error) {
Password: promauth.NewSecret(basicAuthPassword.GetOptionalArg(i)),
PasswordFile: basicAuthPasswordFile.GetOptionalArg(i),
},
BearerToken: promauth.NewSecret(bearerToken.GetOptionalArg(i)),
BearerTokenFile: bearerTokenFile.GetOptionalArg(i),
OAuth2: &promauth.OAuth2Config{
ClientID: oauth2ClientID.GetOptionalArg(i),
ClientSecret: promauth.NewSecret(oauth2ClientSecret.GetOptionalArg(i)),
ClientSecretFile: oauth2ClientSecretFile.GetOptionalArg(i),
Scopes: strings.Split(oauth2Scopes.GetOptionalArg(i), ";"),
TokenURL: oauth2TokenURL.GetOptionalArg(i),
},
}
addr = strings.TrimSuffix(addr, "/")
am, err := NewAlertManager(addr+alertManagerPath, gen, authCfg, time.Minute)
if err != nil {

9
app/vmalert/notifier/template_func.go
View file

@ -20,6 +20,7 @@ import (
"net"
"net/url"
"regexp"
"sort"
"strings"
"time"
@ -282,6 +283,14 @@ func InitTemplateFunc(externalURL *url.URL) {
return m.Labels[label]
},
// sortByLabel sorts the given metrics by provided label key
"sortByLabel": func(label string, metrics []metric) []metric {
sort.SliceStable(metrics, func(i, j int) bool {
return metrics[i].Labels[label] < metrics[j].Labels[label]
})
return metrics
},
// value returns the value of the given metric.
// usually used alongside with `query` template function.
"value": func(m metric) float64 {

44
app/vmalert/recording.go
View file

@ -31,6 +31,8 @@ type RecordingRule struct {
mu sync.RWMutex
// stores last moment of time Exec was called
lastExecTime time.Time
// stores the duration of the last Exec call
lastExecDuration time.Duration
// stores last error that happened in Exec func
// resets on every successful Exec
// may be used as Health state
@ -123,11 +125,13 @@ func (rr *RecordingRule) ExecRange(ctx context.Context, start, end time.Time) ([
// Exec executes RecordingRule expression via the given Querier.
func (rr *RecordingRule) Exec(ctx context.Context) ([]prompbmarshal.TimeSeries, error) {
start := time.Now()
qMetrics, err := rr.q.Query(ctx, rr.Expr)
rr.mu.Lock()
defer rr.mu.Unlock()
rr.lastExecTime = time.Now()
rr.lastExecTime = start
rr.lastExecDuration = time.Since(start)
rr.lastExecError = err
rr.lastExecSamples = len(qMetrics)
if err != nil {
@ -193,23 +197,27 @@ func (rr *RecordingRule) UpdateWith(r Rule) error {
return nil
}
// RuleAPI returns Rule representation in form
// of APIRecordingRule
func (rr *RecordingRule) RuleAPI() APIRecordingRule {
var lastErr string
if rr.lastExecError != nil {
lastErr = rr.lastExecError.Error()
}
return APIRecordingRule{
// ToAPI returns Rule's representation in form
// of APIRule
func (rr *RecordingRule) ToAPI() APIRule {
r := APIRule{
Type: "recording",
DatasourceType: rr.Type.String(),
Name: rr.Name,
Query: rr.Expr,
Labels: rr.Labels,
LastEvaluation: rr.lastExecTime,
EvaluationTime: rr.lastExecDuration.Seconds(),
Health: "ok",
LastSamples: rr.lastExecSamples,
// encode as strings to avoid rounding
ID: fmt.Sprintf("%d", rr.ID()),
GroupID: fmt.Sprintf("%d", rr.GroupID),
Name: rr.Name,
Type: rr.Type.String(),
Expression: rr.Expr,
LastError: lastErr,
LastSamples: rr.lastExecSamples,
LastExec: rr.lastExecTime,
Labels: rr.Labels,
ID: fmt.Sprintf("%d", rr.ID()),
GroupID: fmt.Sprintf("%d", rr.GroupID),
}
if rr.lastExecError != nil {
r.LastError = rr.lastExecError.Error()
r.Health = "err"
}
return r
}

19
app/vmalert/remoteread/init.go
View file

@ -13,11 +13,13 @@ var (
addr = flag.String("remoteRead.url", "", "Optional URL to VictoriaMetrics or vmselect that will be used to restore alerts "+
"state. This configuration makes sense only if `vmalert` was configured with `remoteWrite.url` before and has been successfully persisted its state. "+
"E.g. http://127.0.0.1:8428. See also -remoteRead.disablePathAppend")
basicAuthUsername = flag.String("remoteRead.basicAuth.username", "", "Optional basic auth username for -remoteRead.url")
basicAuthPassword = flag.String("remoteRead.basicAuth.password", "", "Optional basic auth password for -remoteRead.url")
basicAuthPasswordFile = flag.String("remoteRead.basicAuth.passwordFile", "", "Optional path to basic auth password to use for -remoteRead.url")
bearerToken = flag.String("remoteRead.bearerToken", "", "Optional bearer auth token to use for -remoteRead.url.")
bearerTokenFile = flag.String("remoteRead.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteRead.url.")
bearerToken = flag.String("remoteRead.bearerToken", "", "Optional bearer auth token to use for -remoteRead.url.")
bearerTokenFile = flag.String("remoteRead.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteRead.url.")
tlsInsecureSkipVerify = flag.Bool("remoteRead.tlsInsecureSkipVerify", false, "Whether to skip tls verification when connecting to -remoteRead.url")
tlsCertFile = flag.String("remoteRead.tlsCertFile", "", "Optional path to client-side TLS certificate file to use when connecting to -remoteRead.url")
@ -26,6 +28,13 @@ var (
"By default system CA is used")
tlsServerName = flag.String("remoteRead.tlsServerName", "", "Optional TLS server name to use for connections to -remoteRead.url. "+
"By default the server name from -remoteRead.url is used")
oauth2ClientID = flag.String("remoteRead.oauth2.clientID", "", "Optional OAuth2 clientID to use for -remoteRead.url.")
oauth2ClientSecret = flag.String("remoteRead.oauth2.clientSecret", "", "Optional OAuth2 clientSecret to use for -remoteRead.url.")
oauth2ClientSecretFile = flag.String("remoteRead.oauth2.clientSecretFile", "", "Optional OAuth2 clientSecretFile to use for -remoteRead.url.")
oauth2TokenURL = flag.String("remoteRead.oauth2.tokenUrl", "", "Optional OAuth2 tokenURL to use for -remoteRead.url. ")
oauth2Scopes = flag.String("remoteRead.oauth2.scopes", "", "Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.")
disablePathAppend = flag.Bool("remoteRead.disablePathAppend", false, "Whether to disable automatic appending of '/api/v1/query' path to the configured -remoteRead.url.")
)
@ -39,7 +48,11 @@ func Init() (datasource.QuerierBuilder, error) {
if err != nil {
return nil, fmt.Errorf("failed to create transport: %w", err)
}
authCfg, err := utils.AuthConfig(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile, *bearerToken, *bearerTokenFile)
authCfg, err := utils.AuthConfig(
utils.WithBasicAuth(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile),
utils.WithBearer(*bearerToken, *bearerTokenFile),
utils.WithOAuth(*oauth2ClientID, *oauth2ClientSecret, *oauth2ClientSecretFile, *oauth2TokenURL, *oauth2Scopes))
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}

18
app/vmalert/remotewrite/init.go
View file

@ -13,11 +13,13 @@ var (
addr = flag.String("remoteWrite.url", "", "Optional URL to VictoriaMetrics or vminsert where to persist alerts state "+
"and recording rules results in form of timeseries. For example, if -remoteWrite.url=http://127.0.0.1:8428 is specified, "+
"then the alerts state will be written to http://127.0.0.1:8428/api/v1/write . See also -remoteWrite.disablePathAppend")
basicAuthUsername = flag.String("remoteWrite.basicAuth.username", "", "Optional basic auth username for -remoteWrite.url")
basicAuthPassword = flag.String("remoteWrite.basicAuth.password", "", "Optional basic auth password for -remoteWrite.url")
basicAuthPasswordFile = flag.String("remoteWrite.basicAuth.passwordFile", "", "Optional path to basic auth password to use for -remoteWrite.url")
bearerToken = flag.String("remoteWrite.bearerToken", "", "Optional bearer auth token to use for -remoteWrite.url.")
bearerTokenFile = flag.String("remoteWrite.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteWrite.url.")
bearerToken = flag.String("remoteWrite.bearerToken", "", "Optional bearer auth token to use for -remoteWrite.url.")
bearerTokenFile = flag.String("remoteWrite.bearerTokenFile", "", "Optional path to bearer token file to use for -remoteWrite.url.")
maxQueueSize = flag.Int("remoteWrite.maxQueueSize", 1e5, "Defines the max number of pending datapoints to remote write endpoint")
maxBatchSize = flag.Int("remoteWrite.maxBatchSize", 1e3, "Defines defines max number of timeseries to be flushed at once")
@ -31,6 +33,13 @@ var (
"By default system CA is used")
tlsServerName = flag.String("remoteWrite.tlsServerName", "", "Optional TLS server name to use for connections to -remoteWrite.url. "+
"By default the server name from -remoteWrite.url is used")
oauth2ClientID = flag.String("remoteWrite.oauth2.clientID", "", "Optional OAuth2 clientID to use for -remoteWrite.url.")
oauth2ClientSecret = flag.String("remoteWrite.oauth2.clientSecret", "", "Optional OAuth2 clientSecret to use for -remoteWrite.url.")
oauth2ClientSecretFile = flag.String("remoteWrite.oauth2.clientSecretFile", "", "Optional OAuth2 clientSecretFile to use for -remoteWrite.url.")
oauth2TokenURL = flag.String("remoteWrite.oauth2.tokenUrl", "", "Optional OAuth2 tokenURL to use for -notifier.url.")
oauth2Scopes = flag.String("remoteWrite.oauth2.scopes", "", "Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.")
disablePathAppend = flag.Bool("remoteWrite.disablePathAppend", false, "Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.")
)
@ -46,7 +55,10 @@ func Init(ctx context.Context) (*Client, error) {
return nil, fmt.Errorf("failed to create transport: %w", err)
}
authCfg, err := utils.AuthConfig(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile, *bearerToken, *bearerTokenFile)
authCfg, err := utils.AuthConfig(
utils.WithBasicAuth(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile),
utils.WithBearer(*bearerToken, *bearerTokenFile),
utils.WithOAuth(*oauth2ClientID, *oauth2ClientSecret, *oauth2ClientSecretFile, *oauth2TokenURL, *oauth2Scopes))
if err != nil {
return nil, fmt.Errorf("failed to configure auth: %w", err)
}

2
app/vmalert/rule.go
View file

@ -21,6 +21,8 @@ type Rule interface {
// UpdateWith performs modification of current Rule
// with fields of the given Rule.
UpdateWith(Rule) error
// ToAPI converts Rule into APIRule
ToAPI() APIRule
// Close performs the shutdown procedures for rule
// such as metrics unregister
Close()

58
app/vmalert/utils/auth.go
View file

@ -1,18 +1,60 @@
package utils
import (
"strings"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
)
// AuthConfigOptions options which helps build promauth.Config
type AuthConfigOptions func(config *promauth.HTTPClientConfig)
// AuthConfig returns promauth.Config based on the given params
func AuthConfig(baUser, baPass, baFile, bearerToken, bearerTokenFile string) (*promauth.Config, error) {
var baCfg *promauth.BasicAuthConfig
if baUser != "" || baPass != "" || baFile != "" {
baCfg = &promauth.BasicAuthConfig{
Username: baUser,
Password: promauth.NewSecret(baPass),
PasswordFile: baFile,
func AuthConfig(filterOptions ...AuthConfigOptions) (*promauth.Config, error) {
authCfg := &promauth.HTTPClientConfig{}
for _, option := range filterOptions {
option(authCfg)
}
return authCfg.NewConfig(".")
}
// WithBasicAuth returns AuthConfigOptions and initialized promauth.BasicAuthConfig based on given params
func WithBasicAuth(username, password, passwordFile string) AuthConfigOptions {
return func(config *promauth.HTTPClientConfig) {
if username != "" || password != "" || passwordFile != "" {
config.BasicAuth = &promauth.BasicAuthConfig{
Username: username,
Password: promauth.NewSecret(password),
PasswordFile: passwordFile,
}
}
}
}
// WithBearer returns AuthConfigOptions and set BearerToken or BearerTokenFile based on given params
func WithBearer(token, tokenFile string) AuthConfigOptions {
return func(config *promauth.HTTPClientConfig) {
if token != "" {
config.BearerToken = promauth.NewSecret(token)
}
if tokenFile != "" {
config.BearerTokenFile = tokenFile
}
}
}
// WithOAuth returns AuthConfigOptions and set OAuth params based on given params
func WithOAuth(clientID, clientSecret, clientSecretFile, tokenURL, scopes string) AuthConfigOptions {
return func(config *promauth.HTTPClientConfig) {
if clientSecretFile != "" || clientSecret != "" {
config.OAuth2 = &promauth.OAuth2Config{
ClientID: clientID,
ClientSecret: promauth.NewSecret(clientSecret),
ClientSecretFile: clientSecretFile,
TokenURL: tokenURL,
Scopes: strings.Split(scopes, ";"),
}
}
}
return promauth.NewConfig(".", nil, baCfg, bearerToken, bearerTokenFile, nil, nil)
}

17
app/vmalert/web.go
View file

@ -29,7 +29,7 @@ func initLinks() {
pathPrefix = "/"
}
apiLinks = [][2]string{
{path.Join(pathPrefix, "api/v1/groups"), "list all loaded groups and rules"},
{path.Join(pathPrefix, "api/v1/rules"), "list all loaded groups and rules"},
{path.Join(pathPrefix, "api/v1/alerts"), "list all active alerts"},
{path.Join(pathPrefix, "api/v1/groupID/alertID/status"), "get alert status by ID"},
{path.Join(pathPrefix, "flags"), "command-line flags"},
@ -75,7 +75,7 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
case "/notifiers":
WriteListTargets(w, notifier.GetTargets())
return true
case "/api/v1/groups":
case "/api/v1/rules":
data, err := rh.listGroups()
if err != nil {
httpserver.Errorf(w, r, "%s", err)
@ -127,10 +127,10 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
}
type listGroupsResponse struct {
Data struct {
Status string `json:"status"`
Data struct {
Groups []APIGroup `json:"groups"`
} `json:"data"`
Status string `json:"status"`
}
func (rh *requestHandler) groups() []APIGroup {
@ -149,6 +149,7 @@ func (rh *requestHandler) groups() []APIGroup {
return groups
}
func (rh *requestHandler) listGroups() ([]byte, error) {
lr := listGroupsResponse{Status: "success"}
lr.Data.Groups = rh.groups()
@ -163,10 +164,10 @@ func (rh *requestHandler) listGroups() ([]byte, error) {
}
type listAlertsResponse struct {
Data struct {
Status string `json:"status"`
Data struct {
Alerts []*APIAlert `json:"alerts"`
} `json:"data"`
Status string `json:"status"`
}
func (rh *requestHandler) groupAlerts() []GroupAlerts {
@ -181,7 +182,7 @@ func (rh *requestHandler) groupAlerts() []GroupAlerts {
if !ok {
continue
}
alerts = append(alerts, a.AlertsAPI()...)
alerts = append(alerts, a.AlertsToAPI()...)
}
if len(alerts) > 0 {
groupAlerts = append(groupAlerts, GroupAlerts{
@ -204,7 +205,7 @@ func (rh *requestHandler) listAlerts() ([]byte, error) {
if !ok {
continue
}
lr.Data.Alerts = append(lr.Data.Alerts, a.AlertsAPI()...)
lr.Data.Alerts = append(lr.Data.Alerts, a.AlertsToAPI()...)
}
}

49
app/vmalert/web.qtpl
View file

@ -31,14 +31,7 @@
rOk := make(map[string]int)
rNotOk := make(map[string]int)
for _, g := range groups {
for _, r := range g.AlertingRules{
if r.LastError != "" {
rNotOk[g.Name]++
} else {
rOk[g.Name]++
}
}
for _, r := range g.RecordingRules{
for _, r := range g.Rules {
if r.LastError != "" {
rNotOk[g.Name]++
} else {
@ -52,7 +45,7 @@
{% for _, g := range groups %}
<div class="group-heading{% if rNotOk[g.Name] > 0 %} alert-danger{% endif %}" data-bs-target="rules-{%s g.ID %}">
<span class="anchor" id="group-{%s g.ID %}"></span>
<a href="#group-{%s g.ID %}">{%s g.Name %}{% if g.Type != "prometheus" %} ({%s g.Type %}){% endif %} (every {%s g.Interval %})</a>
<a href="#group-{%s g.ID %}">{%s g.Name %}{% if g.Type != "prometheus" %} ({%s g.Type %}){% endif %} (every {%f.0 g.Interval %}s)</a>
{% if rNotOk[g.Name] > 0 %}<span class="badge bg-danger" title="Number of rules with status Error">{%d rNotOk[g.Name] %}</span> {% endif %}
<span class="badge bg-success" title="Number of rules withs status Ok">{%d rOk[g.Name] %}</span>
<p class="fs-6 fw-lighter">{%s g.File %}</p>
@ -75,34 +68,24 @@
</tr>
</thead>
<tbody>
{% for _, ar := range g.AlertingRules %}
<tr{% if ar.LastError != "" %} class="alert-danger"{% endif %}>
{% for _, r := range g.Rules %}
<tr{% if r.LastError != "" %} class="alert-danger"{% endif %}>
<td>
<b>alert:</b> {%s ar.Name %} (for: {%v ar.For %})<br>
<code><pre>{%s ar.Expression %}</pre></code><br>
{% if len(ar.Labels) > 0 %} <b>Labels:</b>{% endif %}
{% for k, v := range ar.Labels %}
{% if r.Type == "alerting" %}
<b>alert:</b> (for: {%v r.Duration %})
{% else %}
<b>record:</b> {%s r.Name %}
{% endif %}
<br>
<code><pre>{%s r.Query %}</pre></code><br>
{% if len(r.Labels) > 0 %} <b>Labels:</b>{% endif %}
{% for k, v := range r.Labels %}
<span class="ms-1 badge bg-primary">{%s k %}={%s v %}</span>
{% endfor %}
</td>
<td><div class="error-cell">{%s ar.LastError %}</div></td>
<td>{%d ar.LastSamples %}</td>
<td>{%f.3 time.Since(ar.LastExec).Seconds() %}s ago</td>
</tr>
{% endfor %}
{% for _, rr := range g.RecordingRules %}
<tr>
<td>
<b>record:</b> {%s rr.Name %}<br>
<code><pre>{%s rr.Expression %}</pre></code>
{% if len(rr.Labels) > 0 %} <b>Labels:</b>{% endif %}
{% for k, v := range rr.Labels %}
<span class="ms-1 badge bg-primary">{%s k %}={%s v %}</span>
{% endfor %}
</td>
<td><div class="error-cell">{%s rr.LastError %}</div></td>
<td>{%d rr.LastSamples %}</td>
<td>{%f.3 time.Since(rr.LastExec).Seconds() %}s ago</td>
<td><div class="error-cell">{%s r.LastError %}</div></td>
<td>{%d r.LastSamples %}</td>
<td>{%f.3 time.Since(r.LastEvaluation).Seconds() %}s ago</td>
</tr>
{% endfor %}
</tbody>

805
app/vmalert/web.qtpl.go
View file

File diff suppressed because it is too large Load diff

4
app/vmalert/web_test.go
View file

@ -54,9 +54,9 @@ func TestHandler(t *testing.T) {
t.Errorf("expected 1 alert got %d", length)
}
})
t.Run("/api/v1/groups", func(t *testing.T) {
t.Run("/api/v1/rules", func(t *testing.T) {
lr := listGroupsResponse{}
getResp(ts.URL+"/api/v1/groups", &lr, 200)
getResp(ts.URL+"/api/v1/rules", &lr, 200)
if length := len(lr.Data.Groups); length != 1 {
t.Errorf("expected 1 group got %d", length)
}

128
app/vmalert/web_types.go
View file

@ -4,63 +4,61 @@ import (
"time"
)
// APIAlert represents an notifier.AlertingRule state
// APIAlert represents a notifier.AlertingRule state
// for WEB view
// https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
type APIAlert struct {
ID string `json:"id"`
Name string `json:"name"`
RuleID string `json:"rule_id"`
GroupID string `json:"group_id"`
Expression string `json:"expression"`
State string `json:"state"`
Name string `json:"name"`
Value string `json:"value"`
Labels map[string]string `json:"labels"`
Labels map[string]string `json:"labels,omitempty"`
Annotations map[string]string `json:"annotations"`
ActiveAt time.Time `json:"activeAt"`
SourceLink string `json:"source"`
Restored bool `json:"restored"`
// Additional fields
// ID is an unique Alert's ID within a group
ID string `json:"id"`
// RuleID is an unique Rule's ID within a group
RuleID string `json:"rule_id"`
// GroupID is an unique Group's ID
GroupID string `json:"group_id"`
// Expression contains the PromQL/MetricsQL expression
// for Rule's evaluation
Expression string `json:"expression"`
// SourceLink contains a link to a system which should show
// why Alert was generated
SourceLink string `json:"source"`
// Restored shows whether Alert's state was restored on restart
Restored bool `json:"restored"`
}
// APIGroup represents Group for WEB view
// https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
type APIGroup struct {
Name string `json:"name"`
Type string `json:"type"`
ID string `json:"id"`
File string `json:"file"`
Interval string `json:"interval"`
Concurrency int `json:"concurrency"`
Params []string `json:"params"`
Labels map[string]string `json:"labels,omitempty"`
AlertingRules []APIAlertingRule `json:"alerting_rules"`
RecordingRules []APIRecordingRule `json:"recording_rules"`
}
// Name is the group name as present in the config
Name string `json:"name"`
// Rules contains both recording and alerting rules
Rules []APIRule `json:"rules"`
// Interval is the Group's evaluation interval in float seconds as present in the file.
Interval float64 `json:"interval"`
// LastEvaluation is the timestamp of the last time the Group was executed
LastEvaluation time.Time `json:"lastEvaluation"`
// APIAlertingRule represents AlertingRule for WEB view
type APIAlertingRule struct {
ID string `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
GroupID string `json:"group_id"`
Expression string `json:"expression"`
For string `json:"for"`
LastError string `json:"last_error"`
LastSamples int `json:"last_samples"`
LastExec time.Time `json:"last_exec"`
Labels map[string]string `json:"labels"`
Annotations map[string]string `json:"annotations"`
}
// Additional fields
// APIRecordingRule represents RecordingRule for WEB view
type APIRecordingRule struct {
ID string `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
GroupID string `json:"group_id"`
Expression string `json:"expression"`
LastError string `json:"last_error"`
LastSamples int `json:"last_samples"`
LastExec time.Time `json:"last_exec"`
Labels map[string]string `json:"labels"`
// Type shows the datasource type (prometheus or graphite) of the Group
Type string `json:"type"`
// ID is a unique Group ID
ID string `json:"id"`
// File contains a path to the file with Group's config
File string `json:"file"`
// Concurrency shows how many rules may be evaluated simultaneously
Concurrency int `json:"concurrency"`
// Params contains HTTP URL parameters added to each Rule's request
Params []string `json:"params,omitempty"`
// Labels is a set of label value pairs, that will be added to every rule.
Labels map[string]string `json:"labels,omitempty"`
}
// GroupAlerts represents a group of alerts for WEB view
@ -68,3 +66,43 @@ type GroupAlerts struct {
Group APIGroup
Alerts []*APIAlert
}
// APIRule represents a Rule for WEB view
// see https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
type APIRule struct {
// State must be one of these under following scenarios
// "pending": at least 1 alert in the rule in pending state and no other alert in firing state.
// "firing": at least 1 alert in the rule in firing state.
// "inactive": no alert in the rule in firing or pending state.
State string `json:"state"`
Name string `json:"name"`
// Query represents Rule's `expression` field
Query string `json:"query"`
// Duration represents Rule's `for` field
Duration float64 `json:"duration"`
Labels map[string]string `json:"labels,omitempty"`
Annotations map[string]string `json:"annotations,omitempty"`
// LastError contains the error faced while executing the rule.
LastError string `json:"lastError"`
// EvaluationTime is the time taken to completely evaluate the rule in float seconds.
EvaluationTime float64 `json:"evaluationTime"`
// LastEvaluation is the timestamp of the last time the rule was executed
LastEvaluation time.Time `json:"lastEvaluation"`
// Alerts is the list of all the alerts in this rule that are currently pending or firing
Alerts []*APIAlert `json:"alerts,omitempty"`
// Health is the health of rule evaluation.
// It MUST be one of "ok", "err", "unknown"
Health string `json:"health"`
// Type of the rule: recording or alerting
Type string `json:"type"`
// Additional fields
// Type of the rule: recording or alerting
DatasourceType string `json:"datasourceType"`
LastSamples int `json:"lastSamples"`
// ID is an unique Alert's ID within a group
ID string `json:"id"`
// GroupID is an unique Group's ID
GroupID string `json:"group_id"`
}

23
docs/CHANGELOG.md
View file

@ -14,6 +14,9 @@ The following tip changes can be tested by building VictoriaMetrics components f
## tip
* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add ability to use OAuth2 for `-datasource.url`, `-notifier.url` and `-remoteRead.url`. See the corresponding command-line flags containing `oauth2` in their names [here](https://docs.victoriametrics.com/vmalert.html#flags).
* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add ability to use Bearer Token for `-notifier.url` via `-notifier.bearerToken` and `-notifier.bearerTokenFile` command-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1824).
## [v1.74.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.74.0)
@ -69,6 +72,17 @@ Released at 22-02-2022
Released at 14-02-2022
**Update notes:** changes introduced in [#1401](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401)
are heavily related to the TSID cache, which is mostly utilized during the ingestion. The fix implies the cache
internal structure change which would cause the cache reset on startup. After the update, users need to expect higher
CPU and memory usage during the ingestion while the cache is being populated. Normally, we expect this process to take
less than hour.
We recommend updating in "off-peak" time when load on the VictoriaMetrics is on its minimum. We also recommend
skipping [v1.73.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.0) in favour of
[v1.73.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.1) which contains the fix for reducing
the memory usage after the update.
* FEATURE: publish VictoriaMetrics binaries for MacOS amd64 and MacOS arm64 (aka MacBook M1) at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1896) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1851).
* FEATURE: reduce CPU and disk IO usage during `indexdb` rotation once per `-retentionPeriod`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401).
* FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): add `-dropSamplesOnOverload` command-line flag for `vminsert`. If this flag is set, then `vminsert` drops incoming data if the destination `vmstorage` is temporarily unavailable or cannot keep up with the ingestion rate. The number of dropped rows can be [monitored](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#monitoring) via `vm_rpc_rows_dropped_on_overload_total` metric at `vminsert`.
@ -151,6 +165,13 @@ Released at 18-01-2022
Released at 20-12-2021
**Update notes:** deduplication logic was slightly changed on the release, which may cause extra
[background merges](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
for already existing data parts. This process is intentionally limited by one CPU core, but still can result
into increase of CPU usage until merges are finished.
We recommend updating in "off-peak" time when load on the VictoriaMetrics is on its minimum.
* FEATURE: [VictoriaMetrics enterprise](https://victoriametrics.com/products/enterprise/): add multi-level downsampling support. See [these docs](https://docs.victoriametrics.com/#downsampling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/36).
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to analyze the correlation between two queries on a single graph. Just click `+Query` button, enter the second query in the newly appeared input field and press `Ctrl+Enter`. Results for both queries should be displayed simultaneously on the same graph. Every query has its own vertical scale, which is displayed on the left and the right side of the graph. Lines for the second query are dashed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1916).
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add ability to override the interval between returned datapoints. By default it is automatically calculated depending on the selected time range and horizontal resolution of the graph. Now it is possible to override it with custom values. This may be useful during data exploration and debugging.
@ -439,7 +460,7 @@ Released at 25-06-2021
* FEATURE: vmstorage: reduce memory usage on a system with many CPU cores under high ingestion rate.
* BUGFIX: prevent from adding new samples to deleted time series after the rotation of the inverted index (the rotation is performed once per `-retentionPeriod`). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136) for details.
* BUGFIX: vmstorage: reduce high disk write IO usage on systems with big number of CPU cores. The issue has been introduced in the release [v1.59.0](#v1590). See [this commit](aa9b56a046b6ae8083fa659df35dd5e994bf9115) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999) for details.
* BUGFIX: vmstorage: reduce high disk write IO usage on systems with big number of CPU cores. The issue has been introduced in the release [v1.59.0](#v1590). See [this commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/aa9b56a046b6ae8083fa659df35dd5e994bf9115) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999) for details.
* BUGFIX: vmstorage: prevent from incorrect stats collection when multiple concurrent queries execute the same tag filter. This may help reducing CPU usage under certain workloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338).
* BUGFIX: vmselect: return the last timestamp for the max / min value from `tmax_over_time(m[d])` and `tmin_over_time(m[d])` [MetricsQL functions](https://docs.victoriametrics.com/MetricsQL.html) as most users expect. See also [this issue](https://github.com/prometheus/prometheus/issues/8966).
* BUGFIX: vmselect: return the expected value for `increase_pure()` [MetricsQL function](https://docs.victoriametrics.com/MetricsQL.html) after a gap in a time series. Previously incorrect too big value could be returned after the gap from `increase_pure()`.

2
docs/Cluster-VictoriaMetrics.md
View file

@ -840,7 +840,7 @@ Below is the output for `/path/to/vmstorage -help`:
## VictoriaMetrics Logo
[Zip](VM_logo.zip) contains three folders with different image orientation (main color and inverted version).
[Zip](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/VM_logo.zip) contains three folders with different image orientation (main color and inverted version).
Files included in each folder:

2
docs/FAQ.md
View file

@ -175,7 +175,7 @@ The main differences between Cortex and VictoriaMetrics:
- VictoriaMetrics requires [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and it [works faster](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
- VictoriaMetrics needs lower amounts of storage space than InfluxDB for production data.
- VictoriaMetrics provides a better query language – [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) – than InfluxQL or Flux. See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics accepts data in multiple popular data ingestion protocols in addition to InfluxDB – Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-time-series-data) for details.
- VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).

21
docs/README.md
View file

@ -1455,6 +1455,27 @@ See the full description of flags [here](#list-of-command-line-flags).
## Data migration
### From VictoriaMetrics
The simplest way to migrate data from one single-node (source) to another (destination), or from one vmstorage node
to another do the following:
1. Stop the VictoriaMetrics (source) with `kill -INT`;
2. Copy (via [rsync](https://en.wikipedia.org/wiki/Rsync) or any other tool) the entire folder specified
via `-storageDataPath` from the source node to the empty folder at the destination node.
3. Once copy is done, stop the VictoriaMetrics (destination) with `kill -INT` and verify that
its `-storageDataPath` points to the copied folder from p.2;
4. Start the VictoriaMetrics (destination). The copied data should be now available.
Things to consider when copying data:
1. Data formats between single-node and vmstorage node aren't compatible and can't be copied.
2. Copying data folder means complete replacement of the previous data on destination VictoriaMetrics.
For more complex scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction
of data - see [vmctl. Migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl.html#migrating-data-from-victoriametrics).
### From other systems
Use [vmctl](https://docs.victoriametrics.com/vmctl.html) for data migration. It supports the following data migration types:
* From Prometheus to VictoriaMetrics

21
docs/Single-server-VictoriaMetrics.md
View file

@ -1459,6 +1459,27 @@ See the full description of flags [here](#list-of-command-line-flags).
## Data migration
### From VictoriaMetrics
The simplest way to migrate data from one single-node (source) to another (destination), or from one vmstorage node
to another do the following:
1. Stop the VictoriaMetrics (source) with `kill -INT`;
2. Copy (via [rsync](https://en.wikipedia.org/wiki/Rsync) or any other tool) the entire folder specified
via `-storageDataPath` from the source node to the empty folder at the destination node.
3. Once copy is done, stop the VictoriaMetrics (destination) with `kill -INT` and verify that
its `-storageDataPath` points to the copied folder from p.2;
4. Start the VictoriaMetrics (destination). The copied data should be now available.
Things to consider when copying data:
1. Data formats between single-node and vmstorage node aren't compatible and can't be copied.
2. Copying data folder means complete replacement of the previous data on destination VictoriaMetrics.
For more complex scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction
of data - see [vmctl. Migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl.html#migrating-data-from-victoriametrics).
### From other systems
Use [vmctl](https://docs.victoriametrics.com/vmctl.html) for data migration. It supports the following data migration types:
* From Prometheus to VictoriaMetrics

27
docs/operator/VictoriaMetrics-Operator.md
View file

@ -6,25 +6,25 @@ sort: 1
## Overview
Design and implementation inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator). It's great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.
Design and implementation inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator). It's great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.
So you can use familiar CRD objects: `ServiceMonitor`, `PodMonitor`, `PrometheusRule` and `Probe`. Or you can use VictoriaMetrics CRDs:
- `VMServiceScrape` - defines scraping metrics configuration from pods backed by services.
- `VMPodScrape` - defines scraping metrics configuration from pods.
- `VMRule` - defines alerting or recording rules.
- `VMProbe` - defines a probing configuration for targets with blackbox exporter.
Besides, operator allows you to manage VictoriaMetrics applications inside kubernetes cluster and simplifies this process [quick-start](/Operator/quick-start.html)
With CRD (Custom Resource Definition) you can define application configuration and apply it to your cluster [crd-objects](/Operator/api.html).
Besides, operator allows you to manage VictoriaMetrics applications inside kubernetes cluster and simplifies this process [quick-start](https://docs.victoriametrics.com/operator/quick-start.html)
With CRD (Custom Resource Definition) you can define application configuration and apply it to your cluster [crd-objects](https://docs.victoriametrics.com/operator/api.html).
Operator simplifies VictoriaMetrics cluster installation, upgrading and managing.
Operator simplifies VictoriaMetrics cluster installation, upgrading and managing.
It has integration with VictoriaMetrics `vmbackupmanager` - advanced tools for making backups. Check backup [docs](/Operator/backups.html)
It has integration with VictoriaMetrics `vmbackupmanager` - advanced tools for making backups. Check backup [docs](https://docs.victoriametrics.com/operator/backups.html)
## Use cases
For kubernetes-cluster administrators, it simplifies installation, configuration and management for `VictoriaMetrics` application. The main feature of operator is its ability to delegate the configuration of applications monitoring to the end-users.
For kubernetes-cluster administrators, it simplifies installation, configuration and management for `VictoriaMetrics` application. The main feature of operator is its ability to delegate the configuration of applications monitoring to the end-users.
For applications developers, its great possibility for managing observability of applications. You can define metrics scraping and alerting configuration for your application and manage it with an application deployment process. Just define app_deployment.yaml, app_vmpodscrape.yaml and app_vmrule.yaml. That's it, you can apply it to a kubernetes cluster. Check [quick-start](/Operator/quick-start.html) for an example.
For applications developers, its great possibility for managing observability of applications. You can define metrics scraping and alerting configuration for your application and manage it with an application deployment process. Just define app_deployment.yaml, app_vmpodscrape.yaml and app_vmrule.yaml. That's it, you can apply it to a kubernetes cluster. Check [quick-start](/Operator/quick-start.html) for an example.
## Operator vs helm-chart
@ -33,18 +33,17 @@ VictoriaMetrics provides [helm charts](https://github.com/VictoriaMetrics/helm-c
## Configuration
Operator configured by env variables, list of it can be found at [link](/vars.html)
Operator configured by env variables, list of it can be found at [link](https://docs.victoriametrics.com/operator/vars.html)
It defines default configuration options, like images for components, timeouts, features.
It defines default configuration options, like images for components, timeouts, features.
## Kubernetes' compatibility versions
operator tested at kubernetes versions
from 1.16 to 1.22
Operator tested at kubernetes versions from 1.16 to 1.22.
For clusters version below 1.16 you must use legacy CRDs from [path](config/crd/legacy)
and disable CRD controller with flag: `--controller.disableCRDOwnership=true`
For clusters version below 1.16 you must use legacy CRDs from [path](https://github.com/VictoriaMetrics/operator/tree/master/config/crd/legacy)
and disable CRD controller with flag: `--controller.disableCRDOwnership=true`
## Troubleshooting
@ -58,7 +57,7 @@ Error from server (Invalid): error when creating "release/crds/crd.yaml": Custom
## Development
- operator-sdk verson v1.0.0 + [https://github.com/operator-framework/operator-sdk]
- [operator-sdk](https://github.com/operator-framework/operator-sdk) version v1.0.0+
- golang 1.15 +
- minikube or kind

2
docs/operator/auth.MD
View file

@ -142,7 +142,7 @@ EOF
## Basic auth for targets
To authenticate a `VMServiceScrape`s over a metrics endpoint use [`basicAuth`](../api.html#basicauth)
To authenticate a `VMServiceScrape`s over a metrics endpoint use [`basicAuth`](https://docs.victoriametrics.com/operator/api.html#basicauth)
```yaml
cat <<EOF | kubectl apply -f -

2
docs/operator/backups.MD
View file

@ -71,7 +71,7 @@ It's needed to make consistent backups for each storage node.
You can read more about backup configuration options and mechanics [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmbackup)
Possible configuration options for backup crd can be found at [link](/docs/api.html#vmbackup)
Possible configuration options for backup crd can be found at [link](https://docs.victoriametrics.com/operator/api.html#vmbackup)
## Restoring backups

59
docs/vmalert.md
View file

@ -7,7 +7,7 @@ sort: 4
`vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
rules against configured `-datasource.url`. For sending alerting notifications
vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag.
vmalert relies on [Alertmanager](https://github.com/prometheus/alertmanager) configured via `-notifier.url` flag.
Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
protocol and require `-remoteWrite.url` to be configured.
Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
@ -300,7 +300,7 @@ Cluster mode could have multiple `vminsert` and `vmselect` components.
```
./bin/vmalert -rule=rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://vmselect:8481/select/0/prometheus # vmselect addr for executing rules expressions
-remoteWrite.url=http://vminsert:8480/insert/0/prometheuss # vminsert addr to persist alerts state and recording rules results
-remoteWrite.url=http://vminsert:8480/insert/0/prometheus # vminsert addr to persist alerts state and recording rules results
-remoteRead.url=http://vmselect:8481/select/0/prometheus # vmselect addr for restoring alerts state after restart
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
```
@ -356,7 +356,7 @@ or reducing resolution) and push results to "cold" cluster.
```
./bin/vmalert -rule=downsampling-rules.yml \ # Path to the file with rules configuration. Supports wildcard
-datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus # vmselect addr for executing recordi ng rules expressions
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss # vminsert addr to persist recording rules results
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results
```
<img alt="vmalert multi cluster" src="vmalert_multicluster.png">
@ -518,8 +518,20 @@ The shortlist of configuration flags is the following:
Lookback defines how far into the past to look when evaluating queries. For example, if the datasource.lookback=5m then param "time" with value now()-5m will be added to every query.
-datasource.maxIdleConnections int
Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
-datasource.oauth2.clientID string
Optional OAuth2 clientID to use for -datasource.url.
-datasource.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -datasource.url.
-datasource.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -datasource.url.
-datasource.oauth2.scopes string
Optional OAuth2 scopes to use for -datasource.url. Scopes must be delimited by ';'
-datasource.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -datasource.url.
-datasource.queryStep duration
queryStep defines how far a value can fallback to when evaluating queries. For example, if datasource.queryStep=15s then param "step" with value "15s" will be added to every query.If queryStep isn't specified, rule's evaluationInterval will be used instead.
-datasource.queryTimeAlignment
Whether to align "time" parameter with evaluation interval.Alignment supposed to produce deterministic results despite of number of vmalert replicas or time they were started. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1257 (default true)
-datasource.roundDigits int
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
-datasource.tlsCAFile string
@ -610,8 +622,29 @@ The shortlist of configuration flags is the following:
-notifier.basicAuth.username array
Optional basic auth username for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerToken array
Optional bearer token for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.bearerTokenFile array
Optional path to bearer token file for -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.config string
Path to configuration file for notifiers
-notifier.oauth2.clientID array
Optional OAuth2 clientID to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecret array
Optional OAuth2 clientSecret to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.clientSecretFile array
Optional OAuth2 clientSecretFile to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.scopes array
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.oauth2.tokenUrl array
Optional OAuth2 tokenURL to use for -notifier.url. If multiple args are set, then they are applied independently for the corresponding -notifier.url
Supports an array of values separated by comma or specified via multiple flags.
-notifier.suppressDuplicateTargetErrors
Whether to suppress 'duplicate target' errors during discovery
-notifier.tlsCAFile array
@ -658,6 +691,16 @@ The shortlist of configuration flags is the following:
Whether to ignore errors from remote storage when restoring alerts state on startup. (default true)
-remoteRead.lookback duration
Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
-remoteRead.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteRead.url.
-remoteRead.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteRead.url.
-remoteRead.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteRead.url.
-remoteRead.oauth2.scopes string
Optional OAuth2 scopes to use for -remoteRead.url. Scopes must be delimited by ';'.
-remoteRead.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -remoteRead.url.
-remoteRead.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteRead.url. By default system CA is used
-remoteRead.tlsCertFile string
@ -690,6 +733,16 @@ The shortlist of configuration flags is the following:
Defines defines max number of timeseries to be flushed at once (default 1000)
-remoteWrite.maxQueueSize int
Defines the max number of pending datapoints to remote write endpoint (default 100000)
-remoteWrite.oauth2.clientID string
Optional OAuth2 clientID to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecret string
Optional OAuth2 clientSecret to use for -remoteWrite.url.
-remoteWrite.oauth2.clientSecretFile string
Optional OAuth2 clientSecretFile to use for -remoteWrite.url.
-remoteWrite.oauth2.scopes string
Optional OAuth2 scopes to use for -notifier.url. Scopes must be delimited by ';'.
-remoteWrite.oauth2.tokenUrl string
Optional OAuth2 tokenURL to use for -notifier.url.
-remoteWrite.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default system CA is used
-remoteWrite.tlsCertFile string

2
lib/writeconcurrencylimiter/concurrencylimiter.go
View file

@ -31,7 +31,7 @@ func Init() {
// Do calls f with the limited concurrency.
func Do(f func() error) error {
// Limit the number of conurrent f calls in order to prevent from excess
// memory usage and CPU trashing.
// memory usage and CPU thrashing.
select {
case ch <- struct{}{}:
err := f()