diff --git a/app/vmalert/README.md b/app/vmalert/README.md index 1e1f1b1b3d..441005c6ee 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -107,11 +107,11 @@ expression and then act according to the Rule type. There are two types of Rules: * [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - -Alerting rules allows to define alert conditions via `expr` field and to send notifications +Alerting rules allows to define alert conditions via `expr` field and to send notifications [Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty. * [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) - Recording rules allows to define `expr` which result will be than backfilled to configured -`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally +`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. `vmalert` forbids to define duplicates - rules with the same combination of name, expression and labels @@ -189,26 +189,26 @@ The state stored to the configured address on every rule evaluation. from configured address by querying time series with name `ALERTS_FOR_STATE`. Both flags are required for the proper state restoring. Restore process may fail if time series are missing -in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) +in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) or received state doesn't match current `vmalert` rules configuration. ### Multitenancy -There are the following approaches for alerting and recording rules across +There are the following approaches for alerting and recording rules across [multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy): -* To run a separate `vmalert` instance per each tenant. - The corresponding tenant must be specified in `-datasource.url` command-line flag - according to [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format). +* To run a separate `vmalert` instance per each tenant. + The corresponding tenant must be specified in `-datasource.url` command-line flag + according to [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format). For example, `/path/to/vmalert -datasource.url=http://vmselect:8481/select/123/prometheus` - would run alerts against `AccountID=123`. For recording rules the `-remoteWrite.url` command-line - flag must contain the url for the specific tenant as well. - For example, `-remoteWrite.url=http://vminsert:8480/insert/123/prometheus` would write recording + would run alerts against `AccountID=123`. For recording rules the `-remoteWrite.url` command-line + flag must contain the url for the specific tenant as well. + For example, `-remoteWrite.url=http://vminsert:8480/insert/123/prometheus` would write recording rules to `AccountID=123`. -* To specify `tenant` parameter per each alerting and recording group if - [enterprise version of vmalert](https://victoriametrics.com/enterprise.html) is used +* To specify `tenant` parameter per each alerting and recording group if + [enterprise version of vmalert](https://victoriametrics.com/enterprise.html) is used with `-clusterMode` command-line flag. For example: ```yaml @@ -224,12 +224,12 @@ groups: # Rules for accountID=456, projectID=789 ``` -If `-clusterMode` is enabled, then `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url` must -contain only the hostname without tenant id. For example: `-datasource.url=http://vmselect:8481`. +If `-clusterMode` is enabled, then `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url` must +contain only the hostname without tenant id. For example: `-datasource.url=http://vmselect:8481`. `vmselect` automatically adds the specified tenant to urls per each recording rule in this case. -The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` files -at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` +The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` files +at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags). @@ -502,6 +502,8 @@ The shortlist of configuration flags is the following: Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used -remoteWrite.url string Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. E.g. http://127.0.0.1:8428 + -remoteWrite.disablePathAppend + Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url. -replay.maxDatapointsPerQuery int Max number of data points expected in one request. The higher the value, the less requests will be made during replay. (default 1000) -replay.ruleRetryAttempts int diff --git a/app/vmalert/remotewrite/init.go b/app/vmalert/remotewrite/init.go index 50614fa0da..7d68da9b90 100644 --- a/app/vmalert/remotewrite/init.go +++ b/app/vmalert/remotewrite/init.go @@ -27,6 +27,7 @@ var ( "By default system CA is used") tlsServerName = flag.String("remoteWrite.tlsServerName", "", "Optional TLS server name to use for connections to -remoteWrite.url. "+ "By default the server name from -remoteWrite.url is used") + disablePathAppend = flag.Bool("remoteWrite.disablePathAppend", false, "Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.") ) // Init creates Client object from given flags. @@ -42,13 +43,14 @@ func Init(ctx context.Context) (*Client, error) { } return NewClient(ctx, Config{ - Addr: *addr, - Concurrency: *concurrency, - MaxQueueSize: *maxQueueSize, - MaxBatchSize: *maxBatchSize, - FlushInterval: *flushInterval, - BasicAuthUser: *basicAuthUsername, - BasicAuthPass: *basicAuthPassword, - Transport: t, + Addr: *addr, + Concurrency: *concurrency, + MaxQueueSize: *maxQueueSize, + MaxBatchSize: *maxBatchSize, + FlushInterval: *flushInterval, + BasicAuthUser: *basicAuthUsername, + BasicAuthPass: *basicAuthPassword, + DisablePathAppend: *disablePathAppend, + Transport: t, }) } diff --git a/app/vmalert/remotewrite/remotewrite.go b/app/vmalert/remotewrite/remotewrite.go index 492a7eac65..3647f4f32d 100644 --- a/app/vmalert/remotewrite/remotewrite.go +++ b/app/vmalert/remotewrite/remotewrite.go @@ -19,13 +19,14 @@ import ( // Client is an asynchronous HTTP client for writing // timeseries via remote write protocol. type Client struct { - addr string - c *http.Client - input chan prompbmarshal.TimeSeries - baUser, baPass string - flushInterval time.Duration - maxBatchSize int - maxQueueSize int + addr string + c *http.Client + input chan prompbmarshal.TimeSeries + baUser, baPass string + flushInterval time.Duration + maxBatchSize int + maxQueueSize int + disablePathAppend bool wg sync.WaitGroup doneCh chan struct{} @@ -56,6 +57,8 @@ type Config struct { WriteTimeout time.Duration // Transport will be used by the underlying http.Client Transport *http.Transport + // DisablePathAppend can be used to not automatically append '/api/v1/write' to the remote write url + DisablePathAppend bool } const ( @@ -94,14 +97,15 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) { Timeout: cfg.WriteTimeout, Transport: cfg.Transport, }, - addr: strings.TrimSuffix(cfg.Addr, "/"), - baUser: cfg.BasicAuthUser, - baPass: cfg.BasicAuthPass, - flushInterval: cfg.FlushInterval, - maxBatchSize: cfg.MaxBatchSize, - maxQueueSize: cfg.MaxQueueSize, - doneCh: make(chan struct{}), - input: make(chan prompbmarshal.TimeSeries, cfg.MaxQueueSize), + addr: strings.TrimSuffix(cfg.Addr, "/"), + baUser: cfg.BasicAuthUser, + baPass: cfg.BasicAuthPass, + flushInterval: cfg.FlushInterval, + maxBatchSize: cfg.MaxBatchSize, + maxQueueSize: cfg.MaxQueueSize, + doneCh: make(chan struct{}), + input: make(chan prompbmarshal.TimeSeries, cfg.MaxQueueSize), + disablePathAppend: cfg.DisablePathAppend, } cc := defaultConcurrency if cfg.Concurrency > 0 { @@ -231,7 +235,9 @@ func (c *Client) send(ctx context.Context, data []byte) error { if c.baPass != "" { req.SetBasicAuth(c.baUser, c.baPass) } - req.URL.Path += writePath + if !c.disablePathAppend { + req.URL.Path += writePath + } resp, err := c.c.Do(req.WithContext(ctx)) if err != nil { return fmt.Errorf("error while sending request to %s: %w; Data len %d(%d)", diff --git a/docs/vmalert.md b/docs/vmalert.md index ba5a6c55f9..015aee911f 100644 --- a/docs/vmalert.md +++ b/docs/vmalert.md @@ -111,11 +111,11 @@ expression and then act according to the Rule type. There are two types of Rules: * [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) - -Alerting rules allows to define alert conditions via `expr` field and to send notifications +Alerting rules allows to define alert conditions via `expr` field and to send notifications [Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty. * [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) - Recording rules allows to define `expr` which result will be than backfilled to configured -`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally +`-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. `vmalert` forbids to define duplicates - rules with the same combination of name, expression and labels @@ -193,26 +193,26 @@ The state stored to the configured address on every rule evaluation. from configured address by querying time series with name `ALERTS_FOR_STATE`. Both flags are required for the proper state restoring. Restore process may fail if time series are missing -in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) +in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`) or received state doesn't match current `vmalert` rules configuration. ### Multitenancy -There are the following approaches for alerting and recording rules across +There are the following approaches for alerting and recording rules across [multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy): -* To run a separate `vmalert` instance per each tenant. - The corresponding tenant must be specified in `-datasource.url` command-line flag - according to [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format). +* To run a separate `vmalert` instance per each tenant. + The corresponding tenant must be specified in `-datasource.url` command-line flag + according to [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format). For example, `/path/to/vmalert -datasource.url=http://vmselect:8481/select/123/prometheus` - would run alerts against `AccountID=123`. For recording rules the `-remoteWrite.url` command-line - flag must contain the url for the specific tenant as well. - For example, `-remoteWrite.url=http://vminsert:8480/insert/123/prometheus` would write recording + would run alerts against `AccountID=123`. For recording rules the `-remoteWrite.url` command-line + flag must contain the url for the specific tenant as well. + For example, `-remoteWrite.url=http://vminsert:8480/insert/123/prometheus` would write recording rules to `AccountID=123`. -* To specify `tenant` parameter per each alerting and recording group if - [enterprise version of vmalert](https://victoriametrics.com/enterprise.html) is used +* To specify `tenant` parameter per each alerting and recording group if + [enterprise version of vmalert](https://victoriametrics.com/enterprise.html) is used with `-clusterMode` command-line flag. For example: ```yaml @@ -228,12 +228,12 @@ groups: # Rules for accountID=456, projectID=789 ``` -If `-clusterMode` is enabled, then `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url` must -contain only the hostname without tenant id. For example: `-datasource.url=http://vmselect:8481`. +If `-clusterMode` is enabled, then `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url` must +contain only the hostname without tenant id. For example: `-datasource.url=http://vmselect:8481`. `vmselect` automatically adds the specified tenant to urls per each recording rule in this case. -The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` files -at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` +The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz` files +at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise` tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags). @@ -506,6 +506,8 @@ The shortlist of configuration flags is the following: Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used -remoteWrite.url string Optional URL to VictoriaMetrics or vminsert where to persist alerts state and recording rules results in form of timeseries. E.g. http://127.0.0.1:8428 + -remoteWrite.disablePathAppend + Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url. -replay.maxDatapointsPerQuery int Max number of data points expected in one request. The higher the value, the less requests will be made during replay. (default 1000) -replay.ruleRetryAttempts int