github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

Browse source
This commit is contained in:
Aliaksandr Valialkin 2022-06-30 20:20:19 +03:00
commit 44dfb2ec0d
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
41 changed files with 756 additions and 453 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
README.md
View file

@ -244,7 +244,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`.
The UI allows exploring query results via graphs and tables.
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query tracec](#query-tracing).
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query traces](#query-tracing).
Graphs in vmui support scrolling and zooming:
@ -255,7 +255,7 @@ Query history can be navigated by holding `Ctrl` (or `Cmd` on MacOS) and pressin
Multi-line queries can be entered by pressing `Shift-Enter` in query input field.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Enable cache` checkbox.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Disable cache` checkbox.
VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
@ -1439,24 +1439,12 @@ Graphs on the dashboards contain useful hints - hover the `i` icon in the top le
We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml)
via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus.
The most interesting health metrics are the following:
* `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour
aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_new_timeseries_created_total[1h])` - time series [churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) during the previous hour.
* `sum(vm_rows{type=~"storage/.*"})` - total number of `(timestamp, value)` data points in the database.
* `sum(rate(vm_rows_inserted_total[5m]))` - ingestion rate, i.e. how many samples are inserted in the database per second.
* `vm_free_disk_space_bytes` - free space left at `-storageDataPath`.
* `sum(vm_data_size_bytes)` - the total size of data on disk.
* `increase(vm_slow_row_inserts_total[5m])` - the number of slow inserts during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_slow_metric_name_loads_total[5m])` - the number of slow loads of metric names during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page.
VictoriaMetrics exposes queries, which take the most time to execute, at `/api/v1/status/top_queries` page.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## TSDB stats
VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way similar to Prometheus - see [these Prometheus docs](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). VictoriaMetrics accepts the following optional query args at `/api/v1/status/tsdb` page:
@ -1621,6 +1609,8 @@ See also more advanced [cardinality limiter in vmagent](https://docs.victoriamet
* VictoriaMetrics ignores `NaN` values during data ingestion.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Cache removal
VictoriaMetrics uses various internal caches. These caches are stored to `<-storageDataPath>/cache` directory during graceful shutdown (e.g. when VictoriaMetrics is stopped by sending `SIGINT` signal). The caches are read on the next VictoriaMetrics startup. Sometimes it is needed to remove such caches on the next startup. This can be performed by placing `reset_cache_on_startup` file inside the `<-storageDataPath>/cache` directory before the restart of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447) for details.

53
app/vmagent/README.md
View file

@ -582,6 +582,8 @@ It may be useful to perform `vmagent` rolling update without any scrape loss.
regex: true
```
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Kafka integration
[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read and write metrics from / to Kafka:
@ -976,40 +978,43 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
-promscrape.suppressScrapeErrorsDelay duration
The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors
-remoteWrite.aws.accessKey array
Optional AWS AccessKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS AccessKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.region array
Optional AWS region to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.roleARN array
Optional AWS roleARN to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS roleARN to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.secretKey array
Optional AWS SecretKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS SecretKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.serice array
Optional AWS Service to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url. Defaults to "aps".
Optional AWS Service to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. Defaults to "aps"
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.useSigv4 array
Enables SigV4 request signing for -remoteWrite.url. It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Enables SigV4 request signing for the corresponding -remoteWrite.url. It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.password array
Optional basic auth password to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional basic auth password to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.passwordFile array
Optional path to basic auth password to use for -remoteWrite.url. The file is re-read every second. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to basic auth password to use for the corresponding -remoteWrite.url. The file is re-read every second
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.username array
Optional basic auth username to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional basic auth username to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.bearerToken array
Optional bearer auth token to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional bearer auth token to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.bearerTokenFile array
Optional path to bearer token file to use for -remoteWrite.url. The token is re-read from the file every second. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to bearer token file to use for the corresponding -remoteWrite.url. The token is re-read from the file every second
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.flushInterval duration
Interval for flushing the data to remote storage. This option takes effect only when less than 10K data points per second are pushed to -remoteWrite.url (default 1s)
-remoteWrite.headers array
Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.label array
Optional label in the form 'name=value' to add to all the metrics before sending them to -remoteWrite.url. Pass multiple -remoteWrite.label flags in order to add multiple labels to metrics before sending them to remote storage
Supports an array of values separated by comma or specified via multiple flags.
@ -1029,27 +1034,27 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
Base path for multitenant remote storage URL to write data to. See https://docs.victoriametrics.com/vmagent.html#multitenancy for details. Example url: http://<vminsert>:8480 . Pass multiple -remoteWrite.multitenantURL flags in order to replicate data to multiple remote storage systems. See also -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientID array
Optional OAuth2 clientID to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientID to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientSecret array
Optional OAuth2 clientSecret to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientSecret to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientSecretFile array
Optional OAuth2 clientSecretFile to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientSecretFile to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.scopes array
Optional OAuth2 scopes to use for -remoteWrite.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 scopes to use for the corresponding -remoteWrite.url. Scopes must be delimited by ';'
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.tokenUrl array
Optional OAuth2 tokenURL to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 tokenURL to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.proxyURL array
Optional proxy URL for writing data to -remoteWrite.url. Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234
Optional proxy URL for writing data to the corresponding -remoteWrite.url. Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.queues int
The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues isn't enough for sending high volume of collected data to remote storage. Default value is 2 * numberOfAvailableCPUs (default 8)
-remoteWrite.rateLimit array
Optional rate limit in bytes per second for data sent to -remoteWrite.url. By default the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data is sent after temporary unavailability of the remote storage
Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. By default the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data is sent after temporary unavailability of the remote storage
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.relabelConfig string
Optional path to file with relabel_config entries. The path can point either to local file or to http url. These entries are applied to all the metrics before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details
@ -1059,7 +1064,7 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
Round metric values to this number of decimal digits after the point before writing them to remote storage. Examples: -remoteWrite.roundDigits=2 would round 1.236 to 1.24, while -remoteWrite.roundDigits=-1 would round 126.78 to 130. By default digits rounding is disabled. Set it to 100 for disabling it for a particular remote storage. This option may be used for improving data compression for the stored metrics
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.sendTimeout array
Timeout for sending a single block of data to -remoteWrite.url
Timeout for sending a single block of data to the corresponding -remoteWrite.url
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.showURL
Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
@ -1067,19 +1072,19 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
The number of significant figures to leave in metric values before writing them to remote storage. See https://en.wikipedia.org/wiki/Significant_figures . Zero value saves all the significant figures. This option may be used for improving data compression for the stored metrics. See also -remoteWrite.roundDigits
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsCAFile array
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default system CA is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to TLS CA file to use for verifying connections to the corresponding -remoteWrite.url. By default system CA is used
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsCertFile array
Optional path to client-side TLS certificate file to use when connecting to -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to client-side TLS certificate file to use when connecting to the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsInsecureSkipVerify array
Whether to skip tls verification when connecting to -remoteWrite.url
Whether to skip tls verification when connecting to the corresponding -remoteWrite.url
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsKeyFile array
Optional path to client-side TLS certificate key to use when connecting to -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to client-side TLS certificate key to use when connecting to the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsServerName array
Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional TLS server name to use for connections to the corresponding -remoteWrite.url. By default the server name from -remoteWrite.url is used
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tmpDataPath string
Path to directory where temporary data for remote write component is stored. See also -remoteWrite.maxDiskUsagePerURL (default "vmagent-remotewrite-data")

93
app/vmagent/remotewrite/client.go
View file

@ -20,60 +20,48 @@ import (
)
var (
rateLimit = flagutil.NewArrayInt("remoteWrite.rateLimit", "Optional rate limit in bytes per second for data sent to -remoteWrite.url. "+
rateLimit = flagutil.NewArrayInt("remoteWrite.rateLimit", "Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. "+
"By default the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data "+
"is sent after temporary unavailability of the remote storage")
sendTimeout = flagutil.NewArrayDuration("remoteWrite.sendTimeout", "Timeout for sending a single block of data to -remoteWrite.url")
proxyURL = flagutil.NewArray("remoteWrite.proxyURL", "Optional proxy URL for writing data to -remoteWrite.url. Supported proxies: http, https, socks5. "+
"Example: -remoteWrite.proxyURL=socks5://proxy:1234")
sendTimeout = flagutil.NewArrayDuration("remoteWrite.sendTimeout", "Timeout for sending a single block of data to the corresponding -remoteWrite.url")
proxyURL = flagutil.NewArray("remoteWrite.proxyURL", "Optional proxy URL for writing data to the corresponding -remoteWrite.url. "+
"Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234")
tlsInsecureSkipVerify = flagutil.NewArrayBool("remoteWrite.tlsInsecureSkipVerify", "Whether to skip tls verification when connecting to -remoteWrite.url")
tlsCertFile = flagutil.NewArray("remoteWrite.tlsCertFile", "Optional path to client-side TLS certificate file to use when connecting to -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
tlsKeyFile = flagutil.NewArray("remoteWrite.tlsKeyFile", "Optional path to client-side TLS certificate key to use when connecting to -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
tlsCAFile = flagutil.NewArray("remoteWrite.tlsCAFile", "Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. "+
"By default system CA is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
tlsServerName = flagutil.NewArray("remoteWrite.tlsServerName", "Optional TLS server name to use for connections to -remoteWrite.url. "+
"By default the server name from -remoteWrite.url is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
tlsInsecureSkipVerify = flagutil.NewArrayBool("remoteWrite.tlsInsecureSkipVerify", "Whether to skip tls verification when connecting to the corresponding -remoteWrite.url")
tlsCertFile = flagutil.NewArray("remoteWrite.tlsCertFile", "Optional path to client-side TLS certificate file to use when connecting "+
"to the corresponding -remoteWrite.url")
tlsKeyFile = flagutil.NewArray("remoteWrite.tlsKeyFile", "Optional path to client-side TLS certificate key to use when connecting to the corresponding -remoteWrite.url")
tlsCAFile = flagutil.NewArray("remoteWrite.tlsCAFile", "Optional path to TLS CA file to use for verifying connections to the corresponding -remoteWrite.url. "+
"By default system CA is used")
tlsServerName = flagutil.NewArray("remoteWrite.tlsServerName", "Optional TLS server name to use for connections to the corresponding -remoteWrite.url. "+
"By default the server name from -remoteWrite.url is used")
basicAuthUsername = flagutil.NewArray("remoteWrite.basicAuth.username", "Optional basic auth username to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
basicAuthPassword = flagutil.NewArray("remoteWrite.basicAuth.password", "Optional basic auth password to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
basicAuthPasswordFile = flagutil.NewArray("remoteWrite.basicAuth.passwordFile", "Optional path to basic auth password to use for -remoteWrite.url. "+
"The file is re-read every second. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
bearerToken = flagutil.NewArray("remoteWrite.bearerToken", "Optional bearer auth token to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
bearerTokenFile = flagutil.NewArray("remoteWrite.bearerTokenFile", "Optional path to bearer token file to use for -remoteWrite.url. "+
"The token is re-read from the file every second. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
headers = flagutil.NewArray("remoteWrite.headers", "Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. "+
"For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. "+
"Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'")
oauth2ClientID = flagutil.NewArray("remoteWrite.oauth2.clientID", "Optional OAuth2 clientID to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
oauth2ClientSecret = flagutil.NewArray("remoteWrite.oauth2.clientSecret", "Optional OAuth2 clientSecret to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
oauth2ClientSecretFile = flagutil.NewArray("remoteWrite.oauth2.clientSecretFile", "Optional OAuth2 clientSecretFile to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
oauth2TokenURL = flagutil.NewArray("remoteWrite.oauth2.tokenUrl", "Optional OAuth2 tokenURL to use for -remoteWrite.url. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
oauth2Scopes = flagutil.NewArray("remoteWrite.oauth2.scopes", "Optional OAuth2 scopes to use for -remoteWrite.url. Scopes must be delimited by ';'. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
basicAuthUsername = flagutil.NewArray("remoteWrite.basicAuth.username", "Optional basic auth username to use for the corresponding -remoteWrite.url")
basicAuthPassword = flagutil.NewArray("remoteWrite.basicAuth.password", "Optional basic auth password to use for the corresponding -remoteWrite.url")
basicAuthPasswordFile = flagutil.NewArray("remoteWrite.basicAuth.passwordFile", "Optional path to basic auth password to use for the corresponding -remoteWrite.url. "+
"The file is re-read every second")
bearerToken = flagutil.NewArray("remoteWrite.bearerToken", "Optional bearer auth token to use for the corresponding -remoteWrite.url")
bearerTokenFile = flagutil.NewArray("remoteWrite.bearerTokenFile", "Optional path to bearer token file to use for the corresponding -remoteWrite.url. "+
"The token is re-read from the file every second")
awsUseSigv4 = flagutil.NewArrayBool("remoteWrite.aws.useSigv4", "Enables SigV4 request signing for -remoteWrite.url. "+
"It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
awsRegion = flagutil.NewArray("remoteWrite.aws.region", "Optional AWS region to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
awsRoleARN = flagutil.NewArray("remoteWrite.aws.roleARN", "Optional AWS roleARN to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
awsAccessKey = flagutil.NewArray("remoteWrite.aws.accessKey", "Optional AWS AccessKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
awsService = flagutil.NewArray("remoteWrite.aws.serice", "Optional AWS Service to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url. Defaults to \"aps\".")
awsSecretKey = flagutil.NewArray("remoteWrite.aws.secretKey", "Optional AWS SecretKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url")
oauth2ClientID = flagutil.NewArray("remoteWrite.oauth2.clientID", "Optional OAuth2 clientID to use for the corresponding -remoteWrite.url")
oauth2ClientSecret = flagutil.NewArray("remoteWrite.oauth2.clientSecret", "Optional OAuth2 clientSecret to use for the corresponding -remoteWrite.url")
oauth2ClientSecretFile = flagutil.NewArray("remoteWrite.oauth2.clientSecretFile", "Optional OAuth2 clientSecretFile to use for the corresponding -remoteWrite.url")
oauth2TokenURL = flagutil.NewArray("remoteWrite.oauth2.tokenUrl", "Optional OAuth2 tokenURL to use for the corresponding -remoteWrite.url")
oauth2Scopes = flagutil.NewArray("remoteWrite.oauth2.scopes", "Optional OAuth2 scopes to use for the corresponding -remoteWrite.url. Scopes must be delimited by ';'")
awsUseSigv4 = flagutil.NewArrayBool("remoteWrite.aws.useSigv4", "Enables SigV4 request signing for the corresponding -remoteWrite.url. "+
"It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled")
awsRegion = flagutil.NewArray("remoteWrite.aws.region", "Optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set")
awsRoleARN = flagutil.NewArray("remoteWrite.aws.roleARN", "Optional AWS roleARN to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set")
awsAccessKey = flagutil.NewArray("remoteWrite.aws.accessKey", "Optional AWS AccessKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set")
awsService = flagutil.NewArray("remoteWrite.aws.serice", "Optional AWS Service to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. "+
"Defaults to \"aps\"")
awsSecretKey = flagutil.NewArray("remoteWrite.aws.secretKey", "Optional AWS SecretKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set")
)
type client struct {
@ -183,6 +171,11 @@ func (c *client) MustStop() {
}
func getAuthConfig(argIdx int) (*promauth.Config, error) {
headersValue := headers.GetOptionalArg(argIdx)
var hdrs []string
if headersValue != "" {
hdrs = strings.Split(headersValue, "^^")
}
username := basicAuthUsername.GetOptionalArg(argIdx)
password := basicAuthPassword.GetOptionalArg(argIdx)
passwordFile := basicAuthPasswordFile.GetOptionalArg(argIdx)
@ -219,7 +212,7 @@ func getAuthConfig(argIdx int) (*promauth.Config, error) {
InsecureSkipVerify: tlsInsecureSkipVerify.GetOptionalArg(argIdx),
}
authCfg, err := promauth.NewConfig(".", nil, basicAuthCfg, token, tokenFile, oauth2Cfg, tlsCfg, nil)
authCfg, err := promauth.NewConfig(".", nil, basicAuthCfg, token, tokenFile, oauth2Cfg, tlsCfg, hdrs)
if err != nil {
return nil, fmt.Errorf("cannot populate OAuth2 config for remoteWrite idx: %d, err: %w", argIdx, err)
}
@ -301,12 +294,12 @@ again:
if err != nil {
logger.Panicf("BUG: unexpected error from http.NewRequest(%q): %s", c.sanitizedURL, err)
}
c.authCfg.SetHeaders(req, true)
h := req.Header
h.Set("User-Agent", "vmagent")
h.Set("Content-Type", "application/x-protobuf")
h.Set("Content-Encoding", "snappy")
h.Set("X-Prometheus-Remote-Write-Version", "0.1.0")
c.authCfg.SetHeaders(req, true)
if c.awsCfg != nil {
if err := c.awsCfg.SignRequest(req, sigv4Hash); err != nil {
// there is no need in retry, request will be rejected by client.Do and retried by code below

3
app/vmselect/promql/eval.go
View file

@ -201,7 +201,8 @@ func getTimestamps(start, end, step int64) []int64 {
func evalExpr(qt *querytracer.Tracer, ec *EvalConfig, e metricsql.Expr) ([]*timeseries, error) {
if qt.Enabled() {
query := e.AppendString(nil)
query := string(e.AppendString(nil))
query = bytesutil.LimitStringLen(query, 300)
mayCache := ec.mayCache()
qt = qt.NewChild("eval: query=%s, timeRange=%s, step=%d, mayCache=%v", query, ec.timeRangeString(), ec.Step, mayCache)
}

6
app/vmselect/promql/rollup_result_cache.go
View file

@ -201,7 +201,8 @@ func ResetRollupResultCache() {
func (rrc *rollupResultCache) Get(qt *querytracer.Tracer, ec *EvalConfig, expr metricsql.Expr, window int64) (tss []*timeseries, newStart int64) {
if qt.Enabled() {
query := expr.AppendString(nil)
query := string(expr.AppendString(nil))
query = bytesutil.LimitStringLen(query, 300)
qt = qt.NewChild("rollup cache get: query=%s, timeRange=%s, step=%d, window=%d", query, ec.timeRangeString(), ec.Step, window)
defer qt.Done()
}
@ -301,7 +302,8 @@ var resultBufPool bytesutil.ByteBufferPool
func (rrc *rollupResultCache) Put(qt *querytracer.Tracer, ec *EvalConfig, expr metricsql.Expr, window int64, tss []*timeseries) {
if qt.Enabled() {
query := expr.AppendString(nil)
query := string(expr.AppendString(nil))
query = bytesutil.LimitStringLen(query, 300)
qt = qt.NewChild("rollup cache put: query=%s, timeRange=%s, step=%d, window=%d, series=%d", query, ec.timeRangeString(), ec.Step, window, len(tss))
defer qt.Done()
}

4
app/vmselect/vmui/asset-manifest.json
View file

@ -1,12 +1,12 @@
{
"files": {
"main.css": "./static/css/main.7e6d0c89.css",
"main.js": "./static/js/main.645fe611.js",
"main.js": "./static/js/main.84a0d8f8.js",
"static/js/27.939f971b.chunk.js": "./static/js/27.939f971b.chunk.js",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.7e6d0c89.css",
"static/js/main.645fe611.js"
"static/js/main.84a0d8f8.js"
]
}

2
app/vmselect/vmui/index.html
View file

@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.645fe611.js"></script><link href="./static/css/main.7e6d0c89.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script src="./dashboards/index.js" type="module"></script><script defer="defer" src="./static/js/main.84a0d8f8.js"></script><link href="./static/css/main.7e6d0c89.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

2
app/vmselect/vmui/static/js/main.645fe611.js
View file

File diff suppressed because one or more lines are too long

2
app/vmselect/vmui/static/js/main.84a0d8f8.js Normal file
View file

File diff suppressed because one or more lines are too long

0
app/vmselect/vmui/static/js/main.645fe611.js.LICENSE.txt → app/vmselect/vmui/static/js/main.84a0d8f8.js.LICENSE.txt
View file

2
app/vmui/packages/vmui/src/components/CardinalityPanel/CardinalityConfigurator/CardinalityConfigurator.tsx
View file

@ -81,7 +81,7 @@ const CardinalityConfigurator: FC<CardinalityConfiguratorProps> = ({
onChange={onFocusLabelChange} />
</Box>
<Box>
<FormControlLabel label="Enable autocomplete"
<FormControlLabel label="Autocomplete"
control={<BasicSwitch checked={autocomplete} onChange={onChangeAutocomplete}/>}
/>
</Box>

8
app/vmui/packages/vmui/src/components/CustomPanel/Configurator/Query/AdditionalSettings.tsx
View file

@ -32,17 +32,17 @@ const AdditionalSettings: FC = () => {
return <Box display="flex" alignItems="center">
<Box>
<FormControlLabel label="Enable autocomplete"
<FormControlLabel label="Autocomplete"
control={<BasicSwitch checked={autocomplete} onChange={onChangeAutocomplete}/>}
/>
</Box>
<Box ml={2}>
<FormControlLabel label="Enable cache"
control={<BasicSwitch checked={!nocache} onChange={onChangeCache}/>}
<FormControlLabel label="Disable cache"
control={<BasicSwitch checked={nocache} onChange={onChangeCache}/>}
/>
</Box>
<Box ml={2}>
<FormControlLabel label="Enable query tracing"
<FormControlLabel label="Trace query"
control={<BasicSwitch checked={isTracingEnabled} onChange={onChangeQueryTracing} />}
/>
</Box>

22
app/vmui/packages/vmui/src/components/CustomPanel/CustomPanel.tsx
View file

@ -18,7 +18,7 @@ import Trace from "./Trace/Trace";
const CustomPanel: FC = () => {
const [tracingsData, setTracingData] = useState<Trace[]>([]);
const [tracesState, setTracesState] = useState<Trace[]>([]);
const {displayType, time: {period}, query, queryControls: {isTracingEnabled}} = useAppState();
const { customStep, yaxis } = useGraphState();
@ -38,24 +38,24 @@ const CustomPanel: FC = () => {
};
const {queryOptions} = useFetchQueryOptions();
const {isLoading, liveData, graphData, error, tracingData} = useFetchQuery({
const {isLoading, liveData, graphData, error, traces} = useFetchQuery({
visible: true,
customStep
});
const handleTraceDelete = (tracingData: Trace) => {
const updatedTracings = tracingsData.filter((data) => data.idValue !== tracingData.idValue);
setTracingData([...updatedTracings]);
const handleTraceDelete = (trace: Trace) => {
const updatedTraces = tracesState.filter((data) => data.idValue !== trace.idValue);
setTracesState([...updatedTraces]);
};
useEffect(() => {
if (tracingData) {
setTracingData([...tracingsData, tracingData]);
if (traces) {
setTracesState([...tracesState, ...traces]);
}
}, [tracingData]);
}, [traces]);
useEffect(() => {
setTracingData([]);
setTracesState([]);
}, [displayType]);
return (
@ -78,7 +78,7 @@ const CustomPanel: FC = () => {
{error && <Alert color="error" severity="error" sx={{whiteSpace: "pre-wrap", mt: 2}}>{error}</Alert>}
{graphData && period && (displayType === "chart") && <>
{isTracingEnabled && <TracingsView
tracingsData={tracingsData}
traces={tracesState}
onDeleteClick={handleTraceDelete}
/>}
<GraphView data={graphData} period={period} customStep={customStep} query={query} yaxis={yaxis}
@ -87,7 +87,7 @@ const CustomPanel: FC = () => {
{liveData && (displayType === "code") && <JsonView data={liveData}/>}
{liveData && (displayType === "table") && <>
{isTracingEnabled && <TracingsView
tracingsData={tracingsData}
traces={tracesState}
onDeleteClick={handleTraceDelete}
/>}
<TableView data={liveData}/>

26
app/vmui/packages/vmui/src/components/CustomPanel/NestedNav/NestedNav.tsx
View file

@ -1,4 +1,4 @@
import React, {FC} from "preact/compat";
import React, {FC, useState} from "preact/compat";
import Box from "@mui/material/Box";
import ListItem from "@mui/material/ListItem";
import ListItemText from "@mui/material/ListItemText";
@ -14,20 +14,28 @@ import Trace from "../Trace/Trace";
interface RecursiveProps {
trace: Trace;
totalMsec: number;
openLevels: Record<number, boolean>;
onChange: (level: number) => void;
}
const NestedNav: FC<RecursiveProps> = ({ trace, openLevels, totalMsec, onChange}) => {
const handleListClick = (traceID: number) => () => onChange(traceID);
interface OpenLevels {
[x: number]: boolean
}
const NestedNav: FC<RecursiveProps> = ({ trace, totalMsec}) => {
const [openLevels, setOpenLevels] = useState({} as OpenLevels);
const handleListClick = (level: number) => () => {
setOpenLevels((prevState:OpenLevels) => {
return {...prevState, [level]: !prevState[level]};
});
};
const hasChildren = trace.children && trace.children.length;
const progress = trace.duration / totalMsec * 100;
return (
<Box sx={{ bgcolor: "rgba(201, 227, 246, 0.4)" }}>
<ListItem onClick={handleListClick(trace.duration)} sx={!hasChildren ? {p:0, pl: 7} : {p:0}}>
<ListItem onClick={handleListClick(trace.idValue)} sx={!hasChildren ? {p:0, pl: 7} : {p:0}}>
<ListItemButton alignItems={"flex-start"} sx={{ pt: 0, pb: 0}}>
{hasChildren ? <ListItemIcon>
{openLevels[trace.duration] ?
{openLevels[trace.idValue] ?
<ExpandLess fontSize={"large"} color={"info"} /> :
<AddCircleRoundedIcon fontSize={"large"} color={"info"} />}
</ListItemIcon>: null}
@ -43,15 +51,13 @@ const NestedNav: FC<RecursiveProps> = ({ trace, openLevels, totalMsec, onChange}
</ListItemButton>
</ListItem>
<>
<Collapse in={openLevels[trace.duration]} timeout="auto" unmountOnExit>
<Collapse in={openLevels[trace.idValue]} timeout="auto" unmountOnExit>
<List component="div" disablePadding sx={{ pl: 4 }}>
{hasChildren ?
trace.children.map((trace) => <NestedNav
key={trace.duration}
trace={trace}
openLevels={openLevels}
totalMsec={totalMsec}
onChange={onChange}
/>) : null}
</List>
</Collapse>

30
app/vmui/packages/vmui/src/components/CustomPanel/Trace/Trace.ts
View file

@ -1,33 +1,18 @@
import {TracingData} from "../../../api/types";
let traceId = 0;
export default class Trace {
private readonly tracing: TracingData;
private readonly tracingChildren: Trace[];
private readonly query: string;
private readonly id: number;
constructor(tracingData: TracingData, query: string) {
this.tracing = tracingData;
this.query = query;
this.id = new Date().getTime();
}
recursiveMap(oldArray: TracingData[], callback: (tr: TracingData) => Trace, newArray: Trace[]): Trace[] {
if (!oldArray) return [];
//base case: check if there are any items left in the original array to process
if (oldArray && oldArray.length <= 0){
//if all items have been processed return the new array
return newArray;
} else {
//destructure the first item from old array and put remaining in a separate array
const [item, ...theRest] = oldArray;
// create an array of the current new array and the result of the current item and the callback function
const interimArray = [...newArray, callback(item)];
// return a recursive call to to map to process the next item.
return this.recursiveMap(theRest, callback, interimArray);
}
}
createTrace(traceData: TracingData) {
return new Trace(traceData, "");
this.id = traceId++;
const children = tracingData.children || [];
this.tracingChildren = children.map((x: TracingData) => new Trace(x, query));
}
get queryValue(): string {
@ -37,8 +22,7 @@ export default class Trace {
return this.id;
}
get children(): Trace[] {
const arr: Trace[] = [];
return this.recursiveMap(this.tracing.children, this.createTrace, arr);
return this.tracingChildren;
}
get message(): string {
return this.tracing.message;

17
app/vmui/packages/vmui/src/components/CustomPanel/Views/TraceView.tsx
View file

@ -1,4 +1,4 @@
import React, {FC, useState} from "preact/compat";
import React, {FC} from "preact/compat";
import List from "@mui/material/List";
import NestedNav from "../NestedNav/NestedNav";
import Trace from "../Trace/Trace";
@ -7,23 +7,10 @@ interface TraceViewProps {
trace: Trace;
}
interface OpenLevels {
[x: number]: boolean
}
const TraceView: FC<TraceViewProps> = ({trace}) => {
const [openLevels, setOpenLevels] = useState({} as OpenLevels);
const handleClick = (level: number) => {
setOpenLevels((prevState:OpenLevels) => ({...prevState, [level]: !prevState[level]}));
};
return (<List sx={{ width: "100%" }} component="nav">
<NestedNav
trace={trace}
openLevels={openLevels}
totalMsec={trace.duration}
onChange={handleClick}
/>
<NestedNav trace={trace} totalMsec={trace.duration} />
</List>);
};

24
app/vmui/packages/vmui/src/components/CustomPanel/Views/TracingsView.tsx
View file

@ -7,17 +7,15 @@ import Button from "@mui/material/Button";
import Trace from "../Trace/Trace";
interface TraceViewProps {
tracingsData: Trace[];
onDeleteClick: (tracingData: Trace) => void;
traces: Trace[];
onDeleteClick: (trace: Trace) => void;
}
const EMPTY_MESSAGE = "Please re-run the query to see results of the tracing";
const TracingsView: FC<TraceViewProps> = ({tracingsData, onDeleteClick}) => {
if (!tracingsData.length) {
const TracingsView: FC<TraceViewProps> = ({traces, onDeleteClick}) => {
if (!traces.length) {
return (
<Alert color={"info"} severity="info" sx={{whiteSpace: "pre-wrap", mt: 2}}>
{EMPTY_MESSAGE}
Please re-run the query to see results of the tracing
</Alert>
);
}
@ -26,14 +24,14 @@ const TracingsView: FC<TraceViewProps> = ({tracingsData, onDeleteClick}) => {
onDeleteClick(tracingData);
};
return <>{tracingsData.map((tracingData) => <>
<Typography variant="h4" gutterBottom component="div">
{"Tracing for"} {tracingData.queryValue}
<Button onClick={handleDeleteClick(tracingData)}>
<RemoveCircleIcon fontSize={"large"} color={"error"} />
return <>{traces.map((trace: Trace) => <>
<Typography variant="h5" component="div">
Trace for <b>{trace.queryValue}</b>
<Button onClick={handleDeleteClick(trace)}>
<RemoveCircleIcon fontSize={"medium"} color={"error"} />
</Button>
</Typography>
<TraceView trace={tracingData} />
<TraceView trace={trace} />
</>)}</>;
};

28
app/vmui/packages/vmui/src/hooks/useFetchQuery.ts
View file

@ -1,7 +1,7 @@
import {useCallback, useEffect, useMemo, useState} from "preact/compat";
import {getQueryRangeUrl, getQueryUrl} from "../api/query-range";
import {useAppState} from "../state/common/StateContext";
import {InstantMetricResult, MetricBase, MetricResult, TracingData} from "../api/types";
import {InstantMetricResult, MetricBase, MetricResult} from "../api/types";
import {isValidHttpUrl} from "../utils/url";
import {ErrorTypes} from "../types";
import {getAppModeEnable, getAppModeParams} from "../utils/app-mode";
@ -28,14 +28,14 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
graphData?: MetricResult[],
liveData?: InstantMetricResult[],
error?: ErrorTypes | string,
tracingData?: Trace,
traces?: Trace[],
} => {
const {query, displayType, serverUrl, time: {period}, queryControls: {nocache, isTracingEnabled}} = useAppState();
const [isLoading, setIsLoading] = useState(false);
const [graphData, setGraphData] = useState<MetricResult[]>();
const [liveData, setLiveData] = useState<InstantMetricResult[]>();
const [tracingData, setTracingData] = useState<Trace>();
const [traces, setTraces] = useState<Trace[]>();
const [error, setError] = useState<ErrorTypes | string>();
const [fetchQueue, setFetchQueue] = useState<AbortController[]>([]);
@ -43,33 +43,26 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
if (error) {
setGraphData(undefined);
setLiveData(undefined);
setTracingData(undefined);
setTraces(undefined);
}
}, [error]);
const updateTracingData = (tracing: TracingData, queries: string[]) => {
if (tracing) {
queries.forEach((query) => {
const {message} = tracing;
if (message.includes(query)) {
setTracingData(new Trace(tracing, query));
}
});
}
};
const fetchData = async (fetchUrl: string[], fetchQueue: AbortController[], displayType: DisplayType, query: string[]) => {
const controller = new AbortController();
setFetchQueue([...fetchQueue, controller]);
try {
const responses = await Promise.all(fetchUrl.map(url => fetch(url, {signal: controller.signal})));
const tempData = [];
const tempTraces: Trace[] = [];
let counter = 1;
for await (const response of responses) {
const resp = await response.json();
if (response.ok) {
setError(undefined);
updateTracingData(resp.trace, query);
if (resp.trace) {
const trace = new Trace(resp.trace, query[counter-1]);
tempTraces.push(trace);
}
tempData.push(...resp.data.result.map((d: MetricBase) => {
d.group = counter;
return d;
@ -80,6 +73,7 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
}
}
displayType === "chart" ? setGraphData(tempData) : setLiveData(tempData);
setTraces(tempTraces);
} catch (e) {
if (e instanceof Error && e.name !== "AbortError") {
setError(`${e.name}: ${e.message}`);
@ -127,5 +121,5 @@ export const useFetchQuery = ({predefinedQuery, visible, display, customStep}: F
setFetchQueue(fetchQueue.filter(f => !f.signal.aborted));
}, [fetchQueue]);
return {fetchUrl, isLoading, graphData, liveData, error, tracingData};
return {fetchUrl, isLoading, graphData, liveData, error, traces};
};

4
app/vmui/packages/vmui/src/state/common/reducer.ts
View file

@ -80,8 +80,8 @@ export const initialState: AppState = {
queryControls: {
autoRefresh: false,
autocomplete: getFromStorage("AUTOCOMPLETE") as boolean || false,
nocache: getFromStorage("NO_CACHE") as boolean || false,
isTracingEnabled: getFromStorage("QUERY_TRACING") as boolean || false,
nocache: false,
isTracingEnabled: false,
}
};

13
docs/CHANGELOG.md
View file

@ -20,6 +20,7 @@ If you use alerting rules or Grafana dashboards, which rely on this metric, then
* FEATURE: add `-search.setLookbackToStep` command-line flag, which enables InfluxDB-like gap filling during querying. See [these docs](https://docs.victoriametrics.com/guides/migrate-from-influx.html) for details.
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add an UI for [query tracing](https://docs.victoriametrics.com/#query-tracing). It can be enabled by clicking `enable query tracing` checkbox and re-running the query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2703).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-remoteWrite.headers` command-line option for specifying optional HTTP headers to send to the configured `-remoteWrite.url`. For example, `-remoteWrite.headers='Foo:Bar^^Baz:x'` would send `Foo: Bar` and `Baz: x` HTTP headers with every request to `-remoteWrite.url`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to specify additional HTTP headers to send to scrape targets via `headers` section in `scrape_configs`. This can be used when the scrape target requires custom authorization and authentication like in [this stackoverflow question](https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header). For example, the following config instructs sending `My-Auth: top-secret` and `TenantID: FooBar` headers with each request to `http://host123:8080/metrics`:
```yaml
@ -32,7 +33,7 @@ scrape_configs:
- targets: ["host123:8080"]
```
* FEATURE: [query tracing](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing): show timestamps in query traces in human-readable format (aka `RFC3339` in UTC timezone) instead of milliseconds since Unix epoch. For example, `2022-06-27T10:32:54.506Z` instead of `1656325974506`.
* FEATURE: [query tracing](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing): show timestamps in query traces in human-readable format (aka `RFC3339` in UTC timezone) instead of milliseconds since Unix epoch. For example, `2022-06-27T10:32:54.506Z` instead of `1656325974506`. This improves traces' readability.
* FEATURE: improve performance of [/api/v1/series](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers) requests, which return big number of time series.
* FEATURE: expose additional histogram metrics at `http://victoriametrics:8428/metrics`, which may help understanding query workload:
@ -41,9 +42,17 @@ scrape_configs:
* `vm_rows_read_per_series` - the number of raw samples read per queried series.
* `vm_series_read_per_query` - the number of series read per query.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow using `__name__` label (aka [metric name](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)) in alerting annotations. For example `{{ $labels.__name__ }}: Too high connection number for "{{ $labels.instance }}`.
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow using `__name__` label (aka [metric name](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)) in alerting annotations. For example:
{% raw %}
```console
{{ $labels.__name__ }}: Too high connection number for "{{ $labels.instance }}
```
{% endraw %}
* BUGFIX: limit max memory occupied by the cache, which stores parsed regular expressions. Previously too long regular expressions passed in [MetricsQL queries](https://docs.victoriametrics.com/MetricsQL.html) could result in big amounts of used memory (e.g. multiple of gigabytes). Now the max cache size for parsed regexps is limited to a a few megabytes.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): make sure that [stale markers](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) are generated with the actual timestamp when unsuccessful scrape occurs. This should prevent from possible time series overlap on scrape target restart in dynmaic envirnoments such as Kubernetes.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly reload changed `-promscrape.config` file when `-promscrape.configCheckInterval` option is set. The changed config file wasn't reloaded in this case since [v1.69.0](#v1690). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2786). Thanks to @ttyv for the fix.
* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): assume that the response is complete if `-search.denyPartialResponse` is enabled and up to `-replicationFactor - 1` `vmstorage` nodes are unavailable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767).
* BUGFIX: [vmselect](https://docs.victoriametrics.com/#vmselect): update `vm_partial_results_total` metric labels to be consistent with `vm_requests_total` labels.

4
docs/Cluster-VictoriaMetrics.md
View file

@ -191,6 +191,10 @@ or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/gr
It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/deployment/docker/alerts.yml).
## Troubleshooting
See [trobuleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Readonly mode
`vmstorage` nodes automatically switch to readonly mode when the directory pointed by `-storageDataPath` contains less than `-storage.minFreeDiskSpaceBytes` of free space. `vminsert` nodes stop sending data to such nodes and start re-routing the data to the remaining `vmstorage` nodes.

26
docs/README.md
View file

@ -244,7 +244,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`.
The UI allows exploring query results via graphs and tables.
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query tracec](#query-tracing).
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query traces](#query-tracing).
Graphs in vmui support scrolling and zooming:
@ -255,7 +255,7 @@ Query history can be navigated by holding `Ctrl` (or `Cmd` on MacOS) and pressin
Multi-line queries can be entered by pressing `Shift-Enter` in query input field.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Enable cache` checkbox.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Disable cache` checkbox.
VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
@ -1439,24 +1439,12 @@ Graphs on the dashboards contain useful hints - hover the `i` icon in the top le
We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml)
via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus.
The most interesting health metrics are the following:
* `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour
aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_new_timeseries_created_total[1h])` - time series [churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) during the previous hour.
* `sum(vm_rows{type=~"storage/.*"})` - total number of `(timestamp, value)` data points in the database.
* `sum(rate(vm_rows_inserted_total[5m]))` - ingestion rate, i.e. how many samples are inserted in the database per second.
* `vm_free_disk_space_bytes` - free space left at `-storageDataPath`.
* `sum(vm_data_size_bytes)` - the total size of data on disk.
* `increase(vm_slow_row_inserts_total[5m])` - the number of slow inserts during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_slow_metric_name_loads_total[5m])` - the number of slow loads of metric names during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page.
VictoriaMetrics exposes queries, which take the most time to execute, at `/api/v1/status/top_queries` page.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## TSDB stats
VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way similar to Prometheus - see [these Prometheus docs](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). VictoriaMetrics accepts the following optional query args at `/api/v1/status/tsdb` page:
@ -1621,6 +1609,8 @@ See also more advanced [cardinality limiter in vmagent](https://docs.victoriamet
* VictoriaMetrics ignores `NaN` values during data ingestion.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Cache removal
VictoriaMetrics uses various internal caches. These caches are stored to `<-storageDataPath>/cache` directory during graceful shutdown (e.g. when VictoriaMetrics is stopped by sending `SIGINT` signal). The caches are read on the next VictoriaMetrics startup. Sometimes it is needed to remove such caches on the next startup. This can be performed by placing `reset_cache_on_startup` file inside the `<-storageDataPath>/cache` directory before the restart of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447) for details.

26
docs/Single-server-VictoriaMetrics.md
View file

@ -248,7 +248,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui`.
The UI allows exploring query results via graphs and tables.
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query tracec](#query-tracing).
It also provides the ability to [explore cardinality](#cardinality-explorer) and to [investigate query traces](#query-tracing).
Graphs in vmui support scrolling and zooming:
@ -259,7 +259,7 @@ Query history can be navigated by holding `Ctrl` (or `Cmd` on MacOS) and pressin
Multi-line queries can be entered by pressing `Shift-Enter` in query input field.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Enable cache` checkbox.
When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling), it may be useful disabling response cache by clicking `Disable cache` checkbox.
VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by clickhing `Override step value` checkbox.
@ -1443,24 +1443,12 @@ Graphs on the dashboards contain useful hints - hover the `i` icon in the top le
We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml)
via [vmalert](https://docs.victoriametrics.com/vmalert.html) or via Prometheus.
The most interesting health metrics are the following:
* `vm_cache_entries{type="storage/hour_metric_ids"}` - the number of time series with new data points during the last hour
aka [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_new_timeseries_created_total[1h])` - time series [churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate) during the previous hour.
* `sum(vm_rows{type=~"storage/.*"})` - total number of `(timestamp, value)` data points in the database.
* `sum(rate(vm_rows_inserted_total[5m]))` - ingestion rate, i.e. how many samples are inserted in the database per second.
* `vm_free_disk_space_bytes` - free space left at `-storageDataPath`.
* `sum(vm_data_size_bytes)` - the total size of data on disk.
* `increase(vm_slow_row_inserts_total[5m])` - the number of slow inserts during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
* `increase(vm_slow_metric_name_loads_total[5m])` - the number of slow loads of metric names during the last 5 minutes.
If this number remains high during extended periods of time, then it is likely more RAM is needed for optimal handling
of the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
VictoriaMetrics exposes currently running queries and their execution times at `/api/v1/status/active_queries` page.
VictoriaMetrics exposes queries, which take the most time to execute, at `/api/v1/status/top_queries` page.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## TSDB stats
VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way similar to Prometheus - see [these Prometheus docs](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). VictoriaMetrics accepts the following optional query args at `/api/v1/status/tsdb` page:
@ -1625,6 +1613,8 @@ See also more advanced [cardinality limiter in vmagent](https://docs.victoriamet
* VictoriaMetrics ignores `NaN` values during data ingestion.
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Cache removal
VictoriaMetrics uses various internal caches. These caches are stored to `<-storageDataPath>/cache` directory during graceful shutdown (e.g. when VictoriaMetrics is stopped by sending `SIGINT` signal). The caches are read on the next VictoriaMetrics startup. Sometimes it is needed to remove such caches on the next startup. This can be performed by placing `reset_cache_on_startup` file inside the `<-storageDataPath>/cache` directory before the restart of VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447) for details.

275
docs/Troubleshooting.md Normal file
View file

@ -0,0 +1,275 @@
---
sort: 23
---
# Troubleshooting
This document contains troubleshooting guides for most common issues when working with VictoriaMetrics:
- [Unexpected query results](#unexpected-query-results)
- [Slow data ingestion](#slow-data-ingestion)
- [Slow queries](#slow-queries)
- [Out of memory errors](#out-of-memory-errors)
- [Cluster instability](#cluster-instability)
## Unexpected query results
If you see unexpected or unreliable query results from VictoriaMetrics, then try the following steps:
1. Check whether simplified queries return unexpected results. For example, if the query looks like
`sum(rate(http_requests_total[5m])) by (job)`, then check whether the following queries return
expected results:
- Remove the outer `sum`: `rate(http_requests_total[5m])`. If this query returns too many time series,
then try adding more specific label filters to it. For example, if you see that the original query
returns unexpected results for the `job="foo"`, then use `rate(http_requests_total{job="foo"}[5m])` query.
If this isn't enough, then continue adding more specific label filters, so the resulting query returns
manageable number of time series.
- Remove the outer `rate`: `http_requests_total`. Additional label filters may be added here in order
to reduce the number of returned series.
Sometimes the query may be improperly constructed, so it returns unexpected results.
It is recommended reading and understanding [MetricsQL docs](https://docs.victoriametrics.com/MetricsQL.html),
especially [subqueries](https://docs.victoriametrics.com/MetricsQL.html#subqueries)
and [rollup functions](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions) sections.
2. If the simplest query continues returning unexpected / unreliable results, then export raw samples
for this query via [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
on the given `[start..end]` time range and check whether they are expected:
```console
curl http://victoriametrics:8428/api/v1/export -d 'match[]=http_requests_total' -d 'start=...' -d 'end=...'
```
Note that responses returned from [/api/v1/query](https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries)
and from [/api/v1/query_range](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries) contain **evaluated** data
instead of raw samples stored in VictoriaMetrics. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness)
for details.
3. Sometimes response caching may lead to unexpected results when samples with older timestamps
are ingested into VictoriaMetrics (aka [backfilling](https://docs.victoriametrics.com/#backfilling)).
Try disabling response cache and see whether this helps. This can be done in the following ways:
- By passing `-search.disableCache` command-line flag to a single-node VictoriaMetrics
or to all the `vmselect` components if cluster version of VictoriaMetrics is used.
- By passing `nocache=1` query arg to every request to `/api/v1/query` and `/api/v1/query_range`.
If you use Grafana, then this query arg can be specified in `Custom Query Parameters` field
at Prometheus datasource settings - see [these docs](https://grafana.com/docs/grafana/latest/datasources/prometheus/) for details.
4. If you use cluster version of VictoriaMetrics, then it may return partial responses by default
when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-availability)
for details. If you want prioritizing query consistency over cluster availability,
then you can pass `-search.denyPartialResponse` command-line flag to all the `vmselect` nodes.
In this case VictoriaMetrics returns an error during querying if at least a single `vmstorage` node is unavailable.
Another option is to pass `deny_partial_response=1` query arg to `/api/v1/query` and `/api/v1/query_range`.
If you use Grafana, then this query arg can be specified in `Custom Query Parameters` field
at Prometheus datasource settings - see [these docs](https://grafana.com/docs/grafana/latest/datasources/prometheus/) for details.
5. If you pass `-replicationFactor` command-line flag to `vmselect`, then it is recommended removing this flag from `vmselect`,
since it may lead to incomplete responses when `vmstorage` nodes contain less than `-replicationFactor`
copies of the requested data.
6. Try upgrading to the [latest available version of VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
and verifying whether the issue is fixed there.
7. Try executing the query with `trace=1` query arg. This enables query tracing, which may contain
useful information on why the query returns unexpected data. See [query tracing docs](https://docs.victoriametrics.com/#query-tracing) for details.
8. Inspect command-line flags passed to VictoriaMetrics components. If you don't understand clearly the purpose
or the effect of some flags, then remove them from the list of flags passed to VictoriaMetrics components,
because some command-line flags may change query results in unexpected ways when set to improper values.
VictoriaMetrics is optimized for running with default flag values (e.g. when they aren't set explicitly).
9. If the steps above didn't help identifying the root cause of unexpected query results,
then [file a bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new) with details on how to reproduce the issue.
## Slow data ingestion
There are the following most commons reasons for slow data ingestion in VictoriaMetrics:
1. Memory shortage for the given amounts of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series).
VictoriaMetrics (or `vmstorage` in cluster version of VictoriaMetrics) maintains an in-memory cache
for quick search for internal series ids per each incoming metric.
This cache is named `storage/tsid`. VictoriaMetrics automatically determines the maximum size for this cache
depending on the available memory on the host where VictoriaMetrics (or `vmstorage`) runs. If the cache size isn't enough
for holding all the entries for active time series, then VictoriaMetrics locates the needed data on disk,
unpacks it, re-constructs the missing entry and puts it into the cache. This takes additional CPU time and disk read IO.
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
contain `Slow inserts` graph, which shows the cache miss percentage for `storage/tsid` cache
during data ingestion. If `slow inserts` graph shows values greater than 5% for more than 10 minutes,
then it is likely the current number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series)
cannot fit the `storage/tsid` cache.
There are the following solutions exist for this issue:
- To increase the available memory on the host where VictoriaMetrics runs until `slow inserts` percentage
will become lower than 5%. If you run VictoriaMetrics cluster, then you need increasing total available
memory at `vmstorage` nodes. This can be done in two ways: either to increase the available memory
per each existing `vmstorage` node or to add more `vmstorage` nodes to the cluster.
- To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
contain a graph showing the number of active time series. Recent versions of VictoriaMetrics
provide [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer),
which can help determining and fixing the source of [high cardinality](https://docs.victoriametrics.com/FAQ.html#what-is-high-cardinality).
2. [High churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate),
e.g. when old time series are substituted with new time series at a high rate.
When VictoriaMetrics encounters a sample for new time series, it needs to register the time series
in the internal index (aka `indexdb`), so it can be quickly located on subsequent select queries.
The process of registering new time series in the internal index is an order of magnitude slower
than the process of adding new sample to already registered time series.
So VictoriaMetrics may work slower than expected under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
provides `Churn rate` graph, which shows the average number of new time series registered
during the last 24 hours. If this number exceeds the number of [active time series](https://docs.victoriametrics.com/FAQ.html#what-is-an-active-time-series),
then you need to identify and fix the source of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
The most commons source of high churn rate is a label, which frequently changes its value. Try avoiding such labels.
The [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help identifying
such labels.
3. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
contain `resource usage` graphs, which show memory usage, CPU usage, disk IO usage and free disk size.
Make sure VictoriaMetrics has enough free resources for graceful handling of potential spikes in workload
according to the following recommendations:
- 50% of free CPU
- 50% of free memory
- 20% of free disk space
If VictoriaMetrics components have lower amounts of free resources, then this may lead
to **significant** performance degradation after workload increases slightly.
For example:
- If the percentage of free CPU is close to 0, then VictoriaMetrics
may experience arbitrary long delays during data ingestion when it cannot keep up
with the data ingestion rate.
- If the percentage of free memory reaches 0, then the Operating System where VictoriaMetrics components run,
may have no enough memory for [page cache](https://en.wikipedia.org/wiki/Page_cache).
VictoriaMetrics relies on page cache for quick queries over recently ingested data.
If the operating system has no enough free memory for page cache, then it needs
to re-read the requested data from disk. This may **significantly** increase disk read IO
and slow down both queries and data ingestion.
- If free disk space is lower than 20%, then VictoriaMetrics is unable to perform optimal
background merge of the incoming data. This leads to increased number of data files on disk,
which, in turn, slows down both data ingestion and querying. See [these docs](https://docs.victoriametrics.com/#storage) for details.
4. If you run cluster version of VictoriaMetrics, then make sure `vminsert` and `vmstorage` components
are located in the same network with short network latency between them.
`vminsert` packs incoming data into in-memory packets and sends them to `vmstorage` on-by-one.
It waits until `vmstorage` returns back `ack` response before sending the next packet.
If the network latency between `vminsert` and `vmstorage` is big (for example, if they run in different datacenters),
then this may become limiting factor for data ingestion speed.
The [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#monitoring)
contain `connection saturation` graph for `vminsert` components. If this graph reaches 100%,
then it is likely you have issues with network latency between `vminsert` and `vmstorage`.
Another possible issue for 100% connection saturation between `vminsert` and `vmstorage`
is resource shortage at `vmstorage` nodes. In this case you need to increase amounts
of available resources (CPU, RAM, disk IO) at `vmstorage` nodes or to add more `vmstorage` nodes to the cluster.
5. Noisy neighboor. Make sure VictoriaMetrics components run in envirnoments without other resource-hungry apps.
Such apps may steal RAM, CPU, disk IO and network bandwidth, which is needed for VictoriaMetrics components.
## Slow queries
Some queries may take more time and resources (CPU, RAM, network bandwidth) than others.
VictoriaMetrics logs slow queries if their execution time exceeds the duration passed
to `-search.logSlowQueryDuration` command-line flag.
VictoriaMetrics also provides `/api/v1/status/top_queries` endpoint, which returns
queries that took the most time to execute.
See [these docs](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) for details.
There are the following solutions exist for slow queries:
- Adding more CPU and memory to VictoriaMetrics, so it may perform the slow query faster.
If you use cluster version of VictoriaMetrics, then migration of `vmselect` nodes to machines
with more CPU and RAM should help improving speed for slow queries.
Sometimes adding more `vmstorage` nodes also can help improving the speed for slow queries.
- Rewriting slow queries, so they become faster. Unfortunately it is hard determining
whether the given query is slow by just looking at it.
VictoriaMetrics provides [query tracing](https://docs.victoriametrics.com/#query-tracing) feature,
which can help determine the source of slow query.
See also [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986),
which explains how to determine and optimize slow queries.
In practice many slow queries are generated because of improper use of [subqueries](https://docs.victoriametrics.com/MetricsQL.html#subqueries).
## Out of memory errors
There are the following most common sources of out of memory (aka OOM) crashes in VictoriaMetrics:
1. Improper command-line flag values. Inspect command-line flags passed to VictoriaMetrics components.
If you don't understand clearly the purpose or the effect of some flags, then remove them
from the list of flags passed to VictoriaMetrics components, because some command-line flags
may lead to increased memory usage and increased CPU usage. The increased memory usage increases chances for OOM crashes.
VictoriaMetrics is optimized for running with default flag values (e.g. when they aren't set explicitly).
For example, it isn't recommended tuning cache sizes in VictoriaMetrics, since it frequently leads to OOM.
[These docs](https://docs.victoriametrics.com/#cache-tuning) refer command-line flags, which aren't
recommended to tune. If you see that VictoriaMetrics needs increasing some cache sizes for the current workload,
then it is better migrating to a host with more memory instead of trying to tune cache sizes.
2. Unexpected heavy queries. The query is considered heavy if it needs to select and process millions of unique time series.
Such query may lead to OOM, since VictoriaMetrics needs to keep some per-series data in memory.
VictoriaMetrics provides various settings, which can help limiting resource usage in this case -
see [these docs](https://docs.victoriametrics.com/#resource-usage-limits).
See also [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986),
which explains how to detect and optimize heavy queries.
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing),
which may help identifying the source of heavy query.
3. Lack of free memory for processing workload spikes. If VictoriaMetrics components use almost all the available memory
under the current workload, then it is recommended migrating to a host with bigger amounts of memory
in order to protect from possible OOM crashes on workload spikes. It is recommended to have at least 50%
of free memory for graceful handling of possible workload spikes.
See [capacity planning for single-node VictoriaMetrics](https://docs.victoriametrics.com/#capacity-planning)
and [capacity planning for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#capacity-planning).
## Cluster instability
VictoriaMetrics cluster may become unstable if there is no enough free resources (CPU, RAM, disk IO, network bandwidth)
for processing the current workload.
The most common sources of cluster instability are:
- Workload spikes. For example, if the number of active time series increases by 2x while
the cluster has no enough free resources for processing the increased workload,
then it may become unstable.
VictoriaMetrics provides various configuration settings, which can be used for limiting unexpected workload spikes.
See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits) for details.
- Various maintenance tasks such as rolling upgrades or rolling restarts during configuration changes.
For example, if a cluster contains `N=3` `vmstorage` nodes and they are restarted one-by-one (aka rolling restart),
then the cluster will have only `N-1=2` healthy `vmstorage` nodes during the rolling restart.
This means that the load on healthy `vmstorage` nodes increases by at least `100%/(N-1)=50%`
comparing to the load before rolling restart. E.g. they need to process 50% more incoming
data and to return 50% more data during queries. In reality the load on the remaining `vmstorage`
nodes increases even more because they need to register new time series, which were re-routed
from temporarily unavailable `vmstorage` node. If `vmstorage` nodes had less than 50%
of free resources (CPU, RAM, disk IO) before the rolling restart, then the rolling restart
can lead to cluster overload and instability for both data ingestion and querying.
As you can see, the workload increase during rolling restart can be reduced by increasing
the number of `vmstorage` nodes in the cluster. For example, if VictoriaMetrics cluster contains
`N=11` `vmstorage` nodes, then the workload increase during rolling restart of `vmstorage` nodes
would be `100%/(N-1)=10%`. So it is recommended to have at least 8 `vmstorage` nodes in the cluster.
The recommended number of `vmstorage` nodes should be multiplied by `-replicationFactor` if replication is enabled -
see [replication and data safety docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#replication-and-data-safety)
for details.
The obvious solution against VictoriaMetrics cluster instability is to make sure cluster components
have enough free resources for graceful processing the increased workload.
See [capacity planning docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#capacity-planning)
and [cluster resizing and scalability docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-resizing-and-scalability)
for details.

2
docs/guides/README.md
View file

@ -1,5 +1,5 @@
---
sort: 22
sort: 24
---
# Guides

2
docs/managed_victoriametrics/README.md
View file

@ -1,5 +1,5 @@
---
sort: 22
sort: 26
---
# Managed VictoriaMetrics

2
docs/operator/README.md
View file

@ -1,5 +1,5 @@
---
sort: 23
sort: 25
---
# VictoriaMetrics Operator

341
docs/url-examples.md
View file

@ -8,11 +8,11 @@ sort: 21
**Deletes time series from VictoriaMetrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -v 'http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=vm_http_request_errors_total'
curl -v http://localhost:8428/api/v1/admin/tsdb/delete_series -d 'match[]=vm_http_request_errors_total'
```
</div>
@ -39,11 +39,11 @@ The expected output should return [HTTP Status 204](https://datatracker.ietf.org
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -v'http://<vmselect>:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=vm_http_request_errors_total'
curl -v http://<vmselect>:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series -d 'match[]=vm_http_request_errors_total'
```
</div>
@ -73,113 +73,137 @@ The expected output should return [HTTP Status 204](https://datatracker.ietf.org
Additional information:
* [How to delete time series](https://docs.victoriametrics.com/#how-to-delete-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/export/csv
## /api/v1/export
**Exports CSV data from VictoriaMetrics**
**Exports raw samples from VictoriaMetrics in JSON line format**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl 'http://<victoriametrics-addr>:8428/api/v1/export/csv?format=__name__,__value__,__timestamp__:unix_s&match=vm_http_request_errors_total' > filename.txt
curl http://localhost:8428/api/v1/export -d 'match[]=vm_http_request_errors_total' > filename.json
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/export/csv?format=__name__,__value__,__timestamp__:unix_s&match=vm_http_request_errors_total' > filename.txt
curl http://<vmselect>:8481/select/0/prometheus/api/v1/export -d 'match[]=vm_http_request_errors_total' > filename.json
```
</div>
Additional information:
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-csv-data)
* [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/export/native
**Exports data from VictoriaMetrics in native format**
## /api/v1/export/csv
Single:
**Exports raw samples from VictoriaMetrics in CSV format**
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=vm_http_request_errors_total' > filename.txt
curl http://localhost:8428/api/v1/export/csv -d 'format=__name__,__value__,__timestamp__:unix_s' -d 'match[]=vm_http_request_errors_total' > filename.csv
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/export/native?match=vm_http_request_errors_total' > filename.txt
curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/csv -d 'format=__name__,__value__,__timestamp__:unix_s' -d 'match[]=vm_http_request_errors_total' > filename.csv
```
</div>
Additional information:
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/export/native
**Exports raw samples from VictoriaMetrics in native format**
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl http://localhost:8428/api/v1/export/native -d 'match[]=vm_http_request_errors_total' > filename.bin
```
</div>
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/native -d 'match[]=vm_http_request_errors_total' > filename.bin
```
</div>
More information:
* [How to export data in native format](https://docs.victoriametrics.com/#how-to-export-data-in-native-format)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/import
**Imports custom data as well as data obtained via /api/v1/export**
**Imports data to VictoriaMetrics in JSON line format**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl --data-binary "@import.txt" -X POST 'http://destination-victoriametrics:8428/api/v1/import'
curl --data-binary "@filename.json" -X POST http://localhost:8428/api/v1/import
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl --data-binary "@import.txt" -X POST 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import'
curl --data-binary "@filename.json" -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import
```
</div>
<div class="with-copy" markdown="1">
More information:
```console
curl -d 'metric_name{foo="bar"} 123' -X POST 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/prometheus'
```
</div>
Additional information:
* [How to import time series data](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/import/csv
**Imports CSV data to VictoriaMetrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl --data-binary "@import.txt" -X POST 'http://localhost:8428/api/v1/import/prometheus'
curl -d "GOOG,1.23,4.56,NYSE" 'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl --data-binary "@import.txt" -X POST 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/csv'
curl -d "GOOG,1.23,4.56,NYSE" 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
```
@ -187,210 +211,229 @@ curl -d "GOOG,1.23,4.56,NYSE" 'http://<vminsert>:8480/insert/0/prometheus/api/v1
Additional information:
* [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
* [How to import CSV data](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-csv-data)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/import/native
**Imports data to VictoriaMetrics in native format**
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -X POST http://localhost:8428/api/v1/import/native -T filename.bin
```
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import/native -T filename.bin
```
</div>
Additional information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/import/prometheus
**Imports data to VictoriaMetrics in Prometheus text exposition format**
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -d 'metric_name{foo="bar"} 123' -X POST http://localhost:8428/api/v1/import/prometheus
```
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -d 'metric_name{foo="bar"} 123' -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import/prometheus
```
</div>
Additional information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/labels
**Get a list of label names**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/labels'
curl http://localhost:8428/prometheus/api/v1/labels
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/labels'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/labels
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Querying label values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/label/\<label_name\>/values
## /api/v1/label/.../values
**Querying label values**
**Get a list of values for a particular label**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/label/job/values'
curl http://localhost:8428/prometheus/api/v1/label/job/values
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/label/job/values'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/label/job/values
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Getting label names](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/query
**Performs PromQL/MetricsQL instant query**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/query?query=vm_http_request_errors_total&time=2021-02-22T19:10:30.781Z'
curl http://localhost:8428/prometheus/api/v1/query -d 'query=vm_http_request_errors_total'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/query?query=vm_http_request_errors_total&time=2021-02-22T19:10:30.781Z'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/query -d 'query=vm_http_request_errors_total'
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Instant queries](https://docs.victoriametrics.com/keyConcepts.html#instant-query)
* [Query language](https://docs.victoriametrics.com/keyConcepts.html#metricsql)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/query_range
**Performs PromQL/MetricsQL range_query**
**Performs PromQL/MetricsQL range query**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/query_range?query=vm_http_request_errors_total&start=2021-02-22T19:10:30.781Z&step=20m'
curl http://localhost:8428/prometheus/api/v1/query_range -d 'query=sum(increase(vm_http_request_errors_total{job="foo"}[5m]))' -d 'start=-1d' -d 'step=1h'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/query_range?query=vm_http_request_errors_total&start=2021-02-22T19:10:30.781Z&step=20m'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/query_range -d 'query=sum(increase(vm_http_request_errors_total{job="foo"}[5m]))' -d 'start=-1d' -d 'step=1h'
```
</div>
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/query_range?query=vm_http_request_errors_total&start=-1h&step=10m'
```
```console
curl -G http://<vmselect>:8481/select/0/prometheus/api/v1/query_range --data-urlencode 'query=sum(increase(vm_http_request_errors_total{status=""}[5m])) by (status)'
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Range queries](https://docs.victoriametrics.com/keyConcepts.html#range-query)
* [Query language](https://docs.victoriametrics.com/keyConcepts.html#metricsql)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/series
**Finding series by label matchers**
**Returns series names with their labels**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/series?match[]=vm_http_request_errors_total&start=-1h'
curl http://localhost:8428/prometheus/api/v1/series -d 'match[]=vm_http_request_errors_total'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/series?match[]=vm_http_request_errors_total&start=-1h'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/series -d 'match[]=vm_http_request_errors_total'
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Finding series by label matchers](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /api/v1/status/tsdb
**Cardinality statistics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/prometheus/api/v1/status/tsdb'
curl http://localhost:8428/prometheus/api/v1/status/tsdb
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/status/tsdb'
curl http://<vmselect>:8481/select/0/prometheus/api/v1/status/tsdb
```
</div>
Additional information:
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [TSDB Stats](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats)
## /api/v1/targets
**Checking targets**
Should be sent to vmagent/VMsingle
Single:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmsingle>:8428/api/v1/targets'
```
</div>
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmagent>:8429/api/v1/targets'
```
</div>
Additional information:
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Targets](https://prometheus.io/docs/prometheus/latest/querying/api/#targets)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /datadog/api/v1/series
**Sends data from DataDog agent to VM**
**Imports data in DataDog format into VictoriaMetrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -417,7 +460,7 @@ echo '
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -447,25 +490,26 @@ echo '
Additional information:
* [How to send data from datadog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /federate
**Returns federated metrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/federate?match[]=vm_http_request_errors_total&start=2021-02-22T19:10:30.781Z'
curl http://localhost:8428/federate -d 'match[]=vm_http_request_errors_total'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/prometheus/federate?match[]=vm_http_request_errors_total&start=2021-02-22T19:10:30.781Z'
curl http://<vmselect>:8481/select/0/prometheus/federate -d 'match[]=vm_http_request_errors_total'
```
</div>
@ -474,60 +518,63 @@ Additional information:
* [Federation](https://docs.victoriametrics.com/#federation)
* [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/#configuring-federation)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /graphite/metrics/find
**Searches Graphite metrics in VictoriaMetrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://localhost:8428/graphite/metrics/find?query=vm_http_request_errors_total'
curl http://localhost:8428/graphite/metrics/find -d 'query=vm_http_request_errors_total'
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -G 'http://<vmselect>:8481/select/0/graphite/metrics/find?query=vm_http_request_errors_total'
curl http://<vmselect>:8481/select/0/graphite/metrics/find -d 'query=vm_http_request_errors_total'
```
</div>
Additional information:
* [Metrics find](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
* [How to send data from graphite compatible agents such as statsd](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [Metrics find API in Graphite](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
* [Graphite API in VictoriaMetrics](https://docs.victoriametrics.com/#graphite-api-usage)
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## /influx/write
**Writes data with InfluxDB line protocol to VictoriaMetrics**
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/write'
curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST http://localhost:8428/write
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://<vminsert>:8480/insert/0/influx/write'
curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST http://<vminsert>:8480/insert/0/influx/write
```
</div>
Additional information:
* [How to send data from influxdb compatible agents such as telegraf](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
* [How to send Influx data to VictoriaMetrics](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
* [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
## TCP and UDP
@ -536,7 +583,7 @@ Additional information:
Turned off by default. Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command-line flag.
*If run from docker, '-opentsdbListenAddr' port should be exposed*
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -545,7 +592,7 @@ echo "put foo.bar.baz `date +%s` 123 tag1=value1 tag2=value2" | nc -N localhost
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -556,7 +603,7 @@ echo "put foo.bar.baz `date +%s` 123 tag1=value1 tag2=value2 VictoriaMetrics_Ac
Enable HTTP server for OpenTSDB /api/put requests by setting `-opentsdbHTTPListenAddr` command-line flag.
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -565,36 +612,34 @@ curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"m
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]'
'http://<vminsert>:8480/insert/42/opentsdb/api/put'
curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]' http://<vminsert>:8480/insert/42/opentsdb/api/put
```
</div>
Additional information:
* [Api http put](http://opentsdb.net/docs/build/html/api_http/put.html)
* [How to send data from opentsdb compatible agents](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-opentsdb-compatible-agents)
* [OpenTSDB http put API](http://opentsdb.net/docs/build/html/api_http/put.html)
* [How to send data OpenTSDB data to VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-opentsdb-compatible-agents)
**How to write data with Graphite plaintext protocol to VictoriaMetrics**
**How to send Graphite data to VictoriaMetrics**
Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command-line flag.
Single:
Single-node VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" |
nc -N localhost 2003
echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N localhost 2003
```
</div>
Cluster:
Cluster version of VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
@ -605,7 +650,7 @@ echo "foo.bar.baz;tag1=value1;tag2=value2;VictoriaMetrics_AccountID=42 123 `date
Additional information:
`VictoriaMetrics_AccountID=42` - tag that indicated tenant ID.
`VictoriaMetrics_AccountID=42` - [tenant ID](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy) in cluster version of VictoriaMetrics
* [Request handler](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/a3eafd2e7fc75776dfc19d3c68c85589454d9dce/app/vminsert/opentsdb/request_handler.go#L47)
* [How to send data from graphite compatible agents such as statsd](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [Multitenancy in cluster version of VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy)

53
docs/vmagent.md
View file

@ -586,6 +586,8 @@ It may be useful to perform `vmagent` rolling update without any scrape loss.
regex: true
```
See also [troubleshooting docs](https://docs.victoriametrics.com/Troubleshooting.html).
## Kafka integration
[Enterprise version](https://victoriametrics.com/products/enterprise/) of `vmagent` can read and write metrics from / to Kafka:
@ -980,40 +982,43 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
-promscrape.suppressScrapeErrorsDelay duration
The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors
-remoteWrite.aws.accessKey array
Optional AWS AccessKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS AccessKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.region array
Optional AWS region to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.roleARN array
Optional AWS roleARN to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS roleARN to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.secretKey array
Optional AWS SecretKey to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional AWS SecretKey to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.serice array
Optional AWS Service to use for -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url. Defaults to "aps".
Optional AWS Service to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set. Defaults to "aps"
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.aws.useSigv4 array
Enables SigV4 request signing for -remoteWrite.url. It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Enables SigV4 request signing for the corresponding -remoteWrite.url. It is expected that other -remoteWrite.aws.* command-line flags are set if sigv4 request signing is enabled
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.password array
Optional basic auth password to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional basic auth password to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.passwordFile array
Optional path to basic auth password to use for -remoteWrite.url. The file is re-read every second. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to basic auth password to use for the corresponding -remoteWrite.url. The file is re-read every second
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.basicAuth.username array
Optional basic auth username to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional basic auth username to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.bearerToken array
Optional bearer auth token to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional bearer auth token to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.bearerTokenFile array
Optional path to bearer token file to use for -remoteWrite.url. The token is re-read from the file every second. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to bearer token file to use for the corresponding -remoteWrite.url. The token is re-read from the file every second
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.flushInterval duration
Interval for flushing the data to remote storage. This option takes effect only when less than 10K data points per second are pushed to -remoteWrite.url (default 1s)
-remoteWrite.headers array
Optional HTTP headers to send with each request to the corresponding -remoteWrite.url. For example, -remoteWrite.headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding -remoteWrite.url. Multiple headers must be delimited by '^^': -remoteWrite.headers='header1:value1^^header2:value2'
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.label array
Optional label in the form 'name=value' to add to all the metrics before sending them to -remoteWrite.url. Pass multiple -remoteWrite.label flags in order to add multiple labels to metrics before sending them to remote storage
Supports an array of values separated by comma or specified via multiple flags.
@ -1033,27 +1038,27 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
Base path for multitenant remote storage URL to write data to. See https://docs.victoriametrics.com/vmagent.html#multitenancy for details. Example url: http://<vminsert>:8480 . Pass multiple -remoteWrite.multitenantURL flags in order to replicate data to multiple remote storage systems. See also -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientID array
Optional OAuth2 clientID to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientID to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientSecret array
Optional OAuth2 clientSecret to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientSecret to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.clientSecretFile array
Optional OAuth2 clientSecretFile to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 clientSecretFile to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.scopes array
Optional OAuth2 scopes to use for -remoteWrite.url. Scopes must be delimited by ';'. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 scopes to use for the corresponding -remoteWrite.url. Scopes must be delimited by ';'
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.oauth2.tokenUrl array
Optional OAuth2 tokenURL to use for -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional OAuth2 tokenURL to use for the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.proxyURL array
Optional proxy URL for writing data to -remoteWrite.url. Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234
Optional proxy URL for writing data to the corresponding -remoteWrite.url. Supported proxies: http, https, socks5. Example: -remoteWrite.proxyURL=socks5://proxy:1234
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.queues int
The number of concurrent queues to each -remoteWrite.url. Set more queues if default number of queues isn't enough for sending high volume of collected data to remote storage. Default value is 2 * numberOfAvailableCPUs (default 8)
-remoteWrite.rateLimit array
Optional rate limit in bytes per second for data sent to -remoteWrite.url. By default the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data is sent after temporary unavailability of the remote storage
Optional rate limit in bytes per second for data sent to the corresponding -remoteWrite.url. By default the rate limit is disabled. It can be useful for limiting load on remote storage when big amounts of buffered data is sent after temporary unavailability of the remote storage
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.relabelConfig string
Optional path to file with relabel_config entries. The path can point either to local file or to http url. These entries are applied to all the metrics before sending them to -remoteWrite.url. See https://docs.victoriametrics.com/vmagent.html#relabeling for details
@ -1063,7 +1068,7 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
Round metric values to this number of decimal digits after the point before writing them to remote storage. Examples: -remoteWrite.roundDigits=2 would round 1.236 to 1.24, while -remoteWrite.roundDigits=-1 would round 126.78 to 130. By default digits rounding is disabled. Set it to 100 for disabling it for a particular remote storage. This option may be used for improving data compression for the stored metrics
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.sendTimeout array
Timeout for sending a single block of data to -remoteWrite.url
Timeout for sending a single block of data to the corresponding -remoteWrite.url
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.showURL
Whether to show -remoteWrite.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
@ -1071,19 +1076,19 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
The number of significant figures to leave in metric values before writing them to remote storage. See https://en.wikipedia.org/wiki/Significant_figures . Zero value saves all the significant figures. This option may be used for improving data compression for the stored metrics. See also -remoteWrite.roundDigits
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsCAFile array
Optional path to TLS CA file to use for verifying connections to -remoteWrite.url. By default system CA is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to TLS CA file to use for verifying connections to the corresponding -remoteWrite.url. By default system CA is used
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsCertFile array
Optional path to client-side TLS certificate file to use when connecting to -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to client-side TLS certificate file to use when connecting to the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsInsecureSkipVerify array
Whether to skip tls verification when connecting to -remoteWrite.url
Whether to skip tls verification when connecting to the corresponding -remoteWrite.url
Supports array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsKeyFile array
Optional path to client-side TLS certificate key to use when connecting to -remoteWrite.url. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional path to client-side TLS certificate key to use when connecting to the corresponding -remoteWrite.url
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tlsServerName array
Optional TLS server name to use for connections to -remoteWrite.url. By default the server name from -remoteWrite.url is used. If multiple args are set, then they are applied independently for the corresponding -remoteWrite.url
Optional TLS server name to use for connections to the corresponding -remoteWrite.url. By default the server name from -remoteWrite.url is used
Supports an array of values separated by comma or specified via multiple flags.
-remoteWrite.tmpDataPath string
Path to directory where temporary data for remote write component is stored. See also -remoteWrite.maxDiskUsagePerURL (default "vmagent-remotewrite-data")

2
go.mod
View file

@ -10,7 +10,7 @@ require (
// like https://github.com/valyala/fasthttp/commit/996610f021ff45fdc98c2ce7884d5fa4e7f9199b
github.com/VictoriaMetrics/fasthttp v1.1.0
github.com/VictoriaMetrics/metrics v1.18.1
github.com/VictoriaMetrics/metricsql v0.44.0
github.com/VictoriaMetrics/metricsql v0.44.1
github.com/aws/aws-sdk-go v1.44.43
github.com/cespare/xxhash/v2 v2.1.2
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect

4
go.sum
View file

@ -108,8 +108,8 @@ github.com/VictoriaMetrics/fasthttp v1.1.0 h1:3crd4YWHsMwu60GUXRH6OstowiFvqrwS4a
github.com/VictoriaMetrics/fasthttp v1.1.0/go.mod h1:/7DMcogqd+aaD3G3Hg5kFgoFwlR2uydjiWvoLp5ZTqQ=
github.com/VictoriaMetrics/metrics v1.18.1 h1:OZ0+kTTto8oPfHnVAnTOoyl0XlRhRkoQrD2n2cOuRw0=
github.com/VictoriaMetrics/metrics v1.18.1/go.mod h1:ArjwVz7WpgpegX/JpB0zpNF2h2232kErkEnzH1sxMmA=
github.com/VictoriaMetrics/metricsql v0.44.0 h1:zxBVeg9tbm6zl2ft2Ica87ItUWYey02hy3MN2ti1ljg=
github.com/VictoriaMetrics/metricsql v0.44.0/go.mod h1:6pP1ZeLVJHqJrHlF6Ij3gmpQIznSsgktEcZgsAWYel0=
github.com/VictoriaMetrics/metricsql v0.44.1 h1:qGoRt0g84uMUscVjS7P3uDZKmjJubWKaIx9v0iHKgck=
github.com/VictoriaMetrics/metricsql v0.44.1/go.mod h1:6pP1ZeLVJHqJrHlF6Ij3gmpQIznSsgktEcZgsAWYel0=
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=

11
lib/bytesutil/bytesutil.go
View file

@ -79,3 +79,14 @@ func ToUnsafeBytes(s string) (b []byte) {
slh.Cap = sh.Len
return b
}
// LimitStringLen limits the length of s to maxLen.
//
// If len(s) > maxLen, then the function concatenates s prefix with s suffix.
func LimitStringLen(s string, maxLen int) string {
if maxLen <= 4 || len(s) <= maxLen {
return s
}
n := maxLen/2 - 1
return s[:n] + ".." + s[len(s)-n:]
}

2
lib/promauth/config.go
View file

@ -419,6 +419,8 @@ func (ba *BasicAuthConfig) NewConfig(baseDir string) (*Config, error) {
}
// NewConfig creates auth config from the given args.
//
// headers must be in the form 'HeaderName: header value'
func NewConfig(baseDir string, az *Authorization, basicAuth *BasicAuthConfig, bearerToken, bearerTokenFile string, o *OAuth2Config, tlsConfig *TLSConfig, headers []string) (*Config, error) {
var getAuthHeader func() string
authDigest := ""

1
lib/promscrape/scraper.go
View file

@ -163,6 +163,7 @@ func runScraper(configFile string, pushData func(wr *prompbmarshal.WriteRequest)
cfgNew.mustRestart(cfg)
cfg = cfgNew
data = dataNew
marshaledData = cfgNew.marshal()
configData.Store(&marshaledData)
case <-globalStopCh:
cfg.mustStop()

26
lib/storage/index_db.go
View file

@ -414,7 +414,7 @@ func (is *indexSearch) maybeCreateIndexes(tsid *TSID, metricNameRaw []byte, date
return false, fmt.Errorf("cannot create global indexes: %w", err)
}
if err := is.createPerDayIndexes(date, tsid.MetricID, mn); err != nil {
return false, fmt.Errorf("cannot create per-day indexes for date=%d: %w", date, err)
return false, fmt.Errorf("cannot create per-day indexes for date=%s: %w", dateToString(date), err)
}
PutMetricName(mn)
atomic.AddUint64(&is.db.timeseriesRepopulated, 1)
@ -595,7 +595,7 @@ func (is *indexSearch) createTSIDByName(dst *TSID, metricName, metricNameRaw []b
return fmt.Errorf("cannot create global indexes: %w", err)
}
if err := is.createPerDayIndexes(date, dst.MetricID, mn); err != nil {
return fmt.Errorf("cannot create per-day indexes for date=%d: %w", date, err)
return fmt.Errorf("cannot create per-day indexes for date=%s: %w", dateToString(date), err)
}
// There is no need in invalidating tag cache, since it is invalidated
@ -777,7 +777,7 @@ func (is *indexSearch) searchLabelNamesWithFiltersOnTimeRange(qt *querytracer.Tr
qt = qt.NewChild("parallel search for label names: filters=%s, timeRange=%s", tfss, &tr)
for date := minDate; date <= maxDate; date++ {
wg.Add(1)
qtChild := qt.NewChild("search for label names: filters=%s, date=%d", tfss, date)
qtChild := qt.NewChild("search for label names: filters=%s, date=%s", tfss, dateToString(date))
go func(date uint64) {
defer func() {
qtChild.Done()
@ -933,7 +933,7 @@ func (is *indexSearch) searchLabelValuesWithFiltersOnTimeRange(qt *querytracer.T
qt = qt.NewChild("parallel search for label values: labelName=%q, filters=%s, timeRange=%s", labelName, tfss, &tr)
for date := minDate; date <= maxDate; date++ {
wg.Add(1)
qtChild := qt.NewChild("search for label names: filters=%s, date=%d", tfss, date)
qtChild := qt.NewChild("search for label names: filters=%s, date=%s", tfss, dateToString(date))
go func(date uint64) {
defer func() {
qtChild.Done()
@ -2460,7 +2460,7 @@ func (is *indexSearch) tryUpdatingMetricIDsForDateRange(qt *querytracer.Tracer,
var errGlobal error
var mu sync.Mutex // protects metricIDs + errGlobal vars from concurrent access below
for minDate <= maxDate {
qtChild := qt.NewChild("parallel thread for date=%d", minDate)
qtChild := qt.NewChild("parallel thread for date=%s", dateToString(minDate))
wg.Add(1)
go func(date uint64) {
defer func() {
@ -2496,8 +2496,10 @@ func (is *indexSearch) tryUpdatingMetricIDsForDateRange(qt *querytracer.Tracer,
}
func (is *indexSearch) getMetricIDsForDateAndFilters(qt *querytracer.Tracer, date uint64, tfs *TagFilters, maxMetrics int) (*uint64set.Set, error) {
qt = qt.NewChild("search for metric ids on a particular day: filters=%s, date=%d, maxMetrics=%d", tfs, date, maxMetrics)
defer qt.Done()
if qt.Enabled() {
qt = qt.NewChild("search for metric ids on a particular day: filters=%s, date=%s, maxMetrics=%d", tfs, dateToString(date), maxMetrics)
defer qt.Done()
}
// Sort tfs by loopsCount needed for performing each filter.
// This stats is usually collected from the previous queries.
// This way we limit the amount of work below by applying fast filters at first.
@ -2842,18 +2844,20 @@ func (is *indexSearch) hasDateMetricID(date, metricID uint64) (bool, error) {
if err == io.EOF {
return false, nil
}
return false, fmt.Errorf("error when searching for (date=%d, metricID=%d) entry: %w", date, metricID, err)
return false, fmt.Errorf("error when searching for (date=%s, metricID=%d) entry: %w", dateToString(date), metricID, err)
}
if string(ts.Item) != string(kb.B) {
return false, fmt.Errorf("unexpected entry for (date=%d, metricID=%d); got %q; want %q", date, metricID, ts.Item, kb.B)
return false, fmt.Errorf("unexpected entry for (date=%s, metricID=%d); got %q; want %q", dateToString(date), metricID, ts.Item, kb.B)
}
return true, nil
}
func (is *indexSearch) getMetricIDsForDateTagFilter(qt *querytracer.Tracer, tf *tagFilter, date uint64, commonPrefix []byte,
maxMetrics int, maxLoopsCount int64) (*uint64set.Set, int64, error) {
qt = qt.NewChild("get metric ids for filter and date: filter={%s}, date=%d, maxMetrics=%d, maxLoopsCount=%d", tf, date, maxMetrics, maxLoopsCount)
defer qt.Done()
if qt.Enabled() {
qt = qt.NewChild("get metric ids for filter and date: filter={%s}, date=%s, maxMetrics=%d, maxLoopsCount=%d", tf, dateToString(date), maxMetrics, maxLoopsCount)
defer qt.Done()
}
if !bytes.HasPrefix(tf.prefix, commonPrefix) {
logger.Panicf("BUG: unexpected tf.prefix %q; must start with commonPrefix %q", tf.prefix, commonPrefix)
}

5
lib/storage/search.go
View file

@ -260,10 +260,11 @@ type TagFilter struct {
// String returns string representation of tf.
func (tf *TagFilter) String() string {
op := tf.getOp()
value := bytesutil.LimitStringLen(string(tf.Value), 60)
if len(tf.Key) == 0 {
return fmt.Sprintf("__name__%s%q", op, tf.Value)
return fmt.Sprintf("__name__%s%q", op, value)
}
return fmt.Sprintf("%s%s%q", tf.Key, op, tf.Value)
return fmt.Sprintf("%s%s%q", tf.Key, op, value)
}
func (tf *TagFilter) getOp() string {

4
lib/storage/storage.go
View file

@ -2110,7 +2110,7 @@ func (s *Storage) updatePerDateData(rows []rawRow, mrs []*MetricRow) error {
ok, err := is.hasDateMetricID(date, metricID)
if err != nil {
if firstError == nil {
firstError = fmt.Errorf("error when locating (date=%d, metricID=%d) in database: %w", date, metricID, err)
firstError = fmt.Errorf("error when locating (date=%s, metricID=%d) in database: %w", dateToString(date), metricID, err)
}
continue
}
@ -2127,7 +2127,7 @@ func (s *Storage) updatePerDateData(rows []rawRow, mrs []*MetricRow) error {
mn.sortTags()
if err := is.createPerDayIndexes(date, metricID, mn); err != nil {
if firstError == nil {
firstError = fmt.Errorf("error when storing per-date inverted index for (date=%d, metricID=%d): %w", date, metricID, err)
firstError = fmt.Errorf("error when storing per-date inverted index for (date=%s, metricID=%d): %w", dateToString(date), metricID, err)
}
continue
}

35
lib/storage/tag_filters.go
View file

@ -291,30 +291,35 @@ func (tf *tagFilter) Less(other *tagFilter) bool {
// String returns human-readable tf value.
func (tf *tagFilter) String() string {
op := "="
if tf.isNegative {
op = "!="
if tf.isRegexp {
op = "!~"
}
} else if tf.isRegexp {
op = "=~"
}
op := tf.getOp()
value := bytesutil.LimitStringLen(string(tf.value), 60)
if bytes.Equal(tf.key, graphiteReverseTagKey) {
return fmt.Sprintf("__graphite_reverse__%s%q", op, tf.value)
return fmt.Sprintf("__graphite_reverse__%s%q", op, value)
}
if tf.isComposite() {
metricName, key, err := unmarshalCompositeTagKey(tf.key)
if err != nil {
logger.Panicf("BUG: cannot unmarshal composite tag key: %s", err)
}
return fmt.Sprintf("composite(%s,%s)%s%q", metricName, key, op, tf.value)
return fmt.Sprintf("composite(%s,%s)%s%q", metricName, key, op, value)
}
key := tf.key
if len(key) == 0 {
key = []byte("__name__")
if len(tf.key) == 0 {
return fmt.Sprintf("__name__%s%q", op, value)
}
return fmt.Sprintf("%s%s%q", key, op, tf.value)
return fmt.Sprintf("%s%s%q", tf.key, op, value)
}
func (tf *tagFilter) getOp() string {
if tf.isNegative {
if tf.isRegexp {
return "!~"
}
return "!="
}
if tf.isRegexp {
return "=~"
}
return "="
}
// Marshal appends marshaled tf to dst

8
lib/storage/time.go
View file

@ -5,6 +5,14 @@ import (
"time"
)
func dateToString(date uint64) string {
if date == 0 {
return "1970-01-01"
}
t := time.Unix(int64(date*24*3600), 0).UTC()
return t.Format("2006-01-02")
}
// timestampToTime returns time representation of the given timestamp.
//
// The returned time is in UTC timezone.

13
vendor/github.com/VictoriaMetrics/metricsql/regexp_cache.go generated vendored
View file

@ -60,13 +60,6 @@ type regexpCacheValue struct {
err error
}
func (rcv *regexpCacheValue) RegexpLen() int {
if r := rcv.r; r != nil {
return len(r.String())
}
return len(rcv.err.Error())
}
type regexpCache struct {
// Move atomic counters to the top of struct for 8-byte alignment on 32-bit arch.
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
@ -131,10 +124,10 @@ func (rc *regexpCache) Put(regexp string, rcv *regexpCacheValue) {
if rc.charsCurrent > rc.charsLimit {
// Remove items accounting for 10% chars from the cache.
overflow := int(float64(rc.charsLimit) * 0.1)
for k, v := range rc.m {
for k := range rc.m {
delete(rc.m, k)
size := len(k) + v.RegexpLen()
size := len(k)
overflow -= size
rc.charsCurrent -= size
@ -144,6 +137,6 @@ func (rc *regexpCache) Put(regexp string, rcv *regexpCacheValue) {
}
}
rc.m[regexp] = rcv
rc.charsCurrent += len(regexp) + rcv.RegexpLen()
rc.charsCurrent += len(regexp)
rc.mu.Unlock()
}

2
vendor/modules.txt vendored
View file

@ -27,7 +27,7 @@ github.com/VictoriaMetrics/fasthttp/stackless
# github.com/VictoriaMetrics/metrics v1.18.1
## explicit; go 1.12
github.com/VictoriaMetrics/metrics
# github.com/VictoriaMetrics/metricsql v0.44.0
# github.com/VictoriaMetrics/metricsql v0.44.1
## explicit; go 1.13
github.com/VictoriaMetrics/metricsql
github.com/VictoriaMetrics/metricsql/binaryop