github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

replaced global http refs with relative markdown ones (#6692)

Browse source 
### Describe Your Changes

Replaced global http links in docs with relative markdown ones

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit is contained in:
Andrii Chubatiuk 2024-07-24 17:50:09 +03:00 committed by Aliaksandr Valialkin
parent 9a051fc80f
commit b8d80ddae5
No known key found for this signature in database
GPG key ID: 52C003EE2BCDB9EB
25 changed files with 1892 additions and 1892 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/Articles.md
View file

@ -9,7 +9,7 @@ menu:
aliases:
- /Articles.html
---
See also [case studies](https://docs.victoriametrics.com/casestudies/).
See also [case studies](./CaseStudies.md).
## Third-party articles and slides about VictoriaMetrics

4
docs/BestPractices.md
View file

@ -64,7 +64,7 @@ The following steps must be performed during the upgrade / downgrade procedure:
## Backup Recommendations
VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/), which simplifies backup automation.
VictoriaMetrics supports backups via [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md) tools. There is also [vmbackupmanager](./vmbackupmanager.md), which simplifies backup automation.
## Technical Support and Services
@ -74,4 +74,4 @@ There are the following channels for providing technical support for VictoriaMet
* [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)
* [Telegram channel](https://t.me/VictoriaMetrics_en)
We also provide [Enterprise support](https://docs.victoriametrics.com/enterprise/).
We also provide [Enterprise support](./enterprise.md).

78
docs/Cluster-VictoriaMetrics.md
View file

@ -112,9 +112,9 @@ The `vm_account_id` and `vm_project_id` labels are extracted after applying the
set via `-relabelConfig` command-line flag, so these labels can be set at this stage.
The `vm_account_id` and `vm_project_id` labels are also taken into account when ingesting data via non-http-based protocols
such as [Graphite](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd),
[InfluxDB line protocol via TCP and UDP](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and
[OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol).
such as [Graphite](./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd),
[InfluxDB line protocol via TCP and UDP](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and
[OpenTSDB telnet put protocol](./#sending-data-via-telnet-put-protocol).
**Security considerations:** it is recommended restricting access to `multitenant` endpoints only to trusted sources,
since untrusted source may break per-tenant data by writing unwanted samples to arbitrary tenants.
@ -278,7 +278,7 @@ General security recommendations:
- Prefer using lists of allowed [API endpoints](#url-format), while disallowing access to other endpoints when configuring auth proxy in front of `vminsert` and `vmselect`.
This minimizes attack surface.
See also [security recommendation for single-node VictoriaMetrics](./README.md#security)
See also [security recommendation for single-node VictoriaMetrics](./#security)
and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/).
@ -344,7 +344,7 @@ See more details in the article [VictoriaMetrics Monitoring](https://victoriamet
Note that these limits are set and applied individually per each `vmstorage` node in the cluster. So, if the cluster has `N` `vmstorage` nodes, then the cluster-level limits will be `N` times bigger than the per-`vmstorage` limits.
See more details about cardinality limiter in [these docs](./README.md#cardinality-limiter).
See more details about cardinality limiter in [these docs](./#cardinality-limiter).
## Troubleshooting
@ -387,7 +387,7 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md).
- `datadog/api/v2/series` - for ingesting data with [DataDog submit metrics API](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent) for details.
- `datadog/api/beta/sketches` - for ingesting data with [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/).
- `influx/write` and `influx/api/v2/write` - for ingesting data with [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/). TCP and UDP receiver is disabled by default. It is exposed on a distinct TCP address set via `-influxListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) for details.
- `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./README.md#how-to-send-data-from-newrelic-agent) for details.
- `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./#how-to-send-data-from-newrelic-agent) for details.
- `opentsdb/api/put` - for accepting [OpenTSDB HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html). This handler is disabled by default. It is exposed on a distinct TCP address set via `-opentsdbHTTPListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#sending-opentsdb-data-via-http-apiput-requests) for details.
- URLs for [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): `http://<vmselect>:8481/select/<accountID>/prometheus/<suffix>`, where:
@ -403,7 +403,7 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md).
- `api/v1/export/native` - exports raw data in native binary format. It may be imported into another VictoriaMetrics via `api/v1/import/native` (see above).
- `api/v1/export/csv` - exports data in CSV. It may be imported into another VictoriaMetrics via `api/v1/import/csv` (see above).
- `api/v1/series/count` - returns the total number of series.
- `api/v1/status/tsdb` - for time series stats. See [these docs](./README.md#tsdb-stats) for details.
- `api/v1/status/tsdb` - for time series stats. See [these docs](./#tsdb-stats) for details.
- `api/v1/status/active_queries` - for currently executed active queries. Note that every `vmselect` maintains an independent list of active queries,
which is returned in the response.
- `api/v1/status/top_queries` - for listing the most frequently executed queries and queries taking the most duration.
@ -440,7 +440,7 @@ The `start` and `end` query args are optional. If they are missing, then all the
This URL works only when `-vmalert.proxyURL` flag is set. See more about vmalert [here](#vmalert).
- `vmstorage` nodes provide the following HTTP endpoints on `8482` port:
- `/internal/force_merge` - initiate [forced compactions](./README.md#forced-merge) on the given `vmstorage` node.
- `/internal/force_merge` - initiate [forced compactions](./#forced-merge) on the given `vmstorage` node.
- `/snapshot/create` - create [instant snapshot](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282),
which can be used for backups in background. Snapshots are created in `<storageDataPath>/snapshots` folder, where `<storageDataPath>` is the corresponding
command-line flag value.
@ -620,7 +620,7 @@ The cluster works in the following way when some of `vmstorage` nodes are unavai
`vmselect` doesn't serve partial responses for API handlers returning [raw datapoints](./keyConcepts.md#raw-samples),
since users usually expect this data is always complete. The following handlers return raw samples:
- [`/api/v1/export*` endpoints](./README.md#how-to-export-time-series)
- [`/api/v1/export*` endpoints](./#how-to-export-time-series)
- [`/api/v1/query`](./url-examples.md#apiv1query) when the `query` contains [series selector](./keyConcepts.md#filtering)
ending with some duration in square brackets. For example, `/api/v1/query?query=up[1h]&time=2024-01-02T03:00:00Z`.
This query returns [raw samples](./keyConcepts.md#raw-samples) for [time series](./keyConcepts.md#time-series)
@ -633,7 +633,7 @@ Data replication can be used for increasing storage durability. See [these docs]
VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](./CaseStudies/README.md).
Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), [series churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](./README.md#troubleshooting).
Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), [series churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](./#troubleshooting).
The needed storage space for the given retention (the retention is set via `-retentionPeriod` command-line flag at `vmstorage`) can be extrapolated from disk space usage in a test run. For example, if the storage space usage is 10GB after a day-long test run on a production workload, then it will need at least `10GB*100=1TB` of disk space for `-retentionPeriod=100d` (100-days retention period). Storage space usage can be monitored with [the official Grafana dashboard for VictoriaMetrics cluster](#monitoring).
@ -685,7 +685,7 @@ Some workloads may need fine-grained resource usage limits. In these cases the f
See also `-search.maxMemoryPerQuery` command-line flag at `vmselect`.
- `-search.maxQueueDuration` at `vmselect` and `vmstorage` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests`
concurrent queries are executed.
- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./README.md#prometheus-querying-api-enhancements)
- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./#prometheus-querying-api-enhancements)
query args at [/api/v1/labels](./url-examples.md#apiv1labels) and
[/api/v1/label/.../values](./url-examples.md#apiv1labelvalues).
This may be useful for reducing the load on `vmstorage` if the provided extra filters match too many time series.
@ -880,7 +880,7 @@ HDD-based persistent disks should be enough for the majority of use cases. It is
## Deduplication
Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](./README.md#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects:
Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](./#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects:
By default, `vminsert` tries to route all the samples for a single time series to a single `vmstorage` node. But samples for a single time series can be spread among multiple `vmstorage` nodes under certain conditions:
* when adding/removing `vmstorage` nodes. Then new samples for a part of time series will be routed to another `vmstorage` nodes;
@ -911,7 +911,7 @@ Restoring from backup:
## Retention filters
[VictoriaMetrics enterprise](./enterprise.md) supports configuring multiple retentions for distinct sets of time series
by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](./README.md#retention-filters) for details on this feature.
by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](./#retention-filters) for details on this feature.
Additionally, enterprise version of VictoriaMetrics cluster supports multiple retentions for distinct sets of [tenants](#multitenancy)
by specifying filters on `vm_account_id` and/or `vm_project_id` pseudo-labels in `-retentionFilter` command-line flag.
@ -932,7 +932,7 @@ For example, the following config sets retention to 5 days for time series with
-retentionFilter='{vm_account_id="5",env="dev"}:5d'
```
See also [these docs](./README.md#retention-filters) for additional details on retention filters.
See also [these docs](./#retention-filters) for additional details on retention filters.
See also [downsampling](#downsampling).
Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
@ -941,7 +941,7 @@ See how to request a free trial license [here](https://victoriametrics.com/produ
## Downsampling
Downsampling is available in [enterprise version of VictoriaMetrics](./enterprise.md).
It is configured with `-downsampling.period` command-line flag according to [these docs](./README.md#downsampling).
It is configured with `-downsampling.period` command-line flag according to [these docs](./#downsampling).
It is possible to downsample series, which belong to a particular [tenant](#multitenancy) by using [filters](./keyConcepts.md#filtering)
on `vm_account_id` or `vm_project_id` pseudo-labels in `-downsampling.period` command-line flag. For example, the following config leaves the last sample per each minute for samples
@ -1064,7 +1064,7 @@ Below is the output for `/path/to/vminsert -help`:
-datadog.sanitizeMetricName
Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
-denyQueryTracing
Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}}
Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}}
-disableRerouting
Whether to disable re-routing when some of vmstorage nodes accept incoming data at slower speed compared to other storage nodes. Disabled re-routing limits the ingestion rate by the slowest vmstorage node. On the other side, disabled re-routing minimizes the number of active time series in the cluster during rolling restarts and during spikes in series churn rate. See also -disableReroutingOnUnavailable and -dropSamplesOnOverload (default true)
-disableReroutingOnUnavailable
@ -1074,7 +1074,7 @@ Below is the output for `/path/to/vminsert -help`:
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
@ -1087,7 +1087,7 @@ Below is the output for `/path/to/vminsert -help`:
-fs.disableMmap
Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
-graphite.sanitizeMetricName
Sanitize metric names for the ingested Graphite data. See {{% ref "./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}}
Sanitize metric names for the ingested Graphite data. See {{% ref "./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}}
-graphiteListenAddr string
TCP and UDP address to listen for Graphite plaintext data. Usually :2003 must be set. Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol
-graphiteListenAddr.useProxyProtocol
@ -1212,7 +1212,7 @@ Below is the output for `/path/to/vminsert -help`:
The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864)
-opentelemetry.usePrometheusNaming
Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./README.md#sending-data-via-opentelemetry" %}}
Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./#sending-data-via-opentelemetry" %}}
-opentsdbHTTPListenAddr string
TCP address to listen for OpenTSDB HTTP put requests. Usually :4242 must be set. Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol
-opentsdbHTTPListenAddr.useProxyProtocol
@ -1246,11 +1246,11 @@ Below is the output for `/path/to/vminsert -help`:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-relabelConfig string
Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./README.md#relabeling" %}} for details. The config is reloaded on SIGHUP signal
Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./#relabeling" %}} for details. The config is reloaded on SIGHUP signal
-relabelConfigCheckInterval duration
Interval for checking for changes in '-relabelConfig' file. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes
-replicationFactor int
@ -1355,20 +1355,20 @@ Below is the output for `/path/to/vmselect -help`:
-clusternativeListenAddr string
TCP address to listen for requests from other vmselect nodes in multi-level cluster setup. See {{% ref "./Cluster-VictoriaMetrics.md#multi-level-cluster-setup" %}}. Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty
-dedup.minScrapeInterval duration
Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details
Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./#deduplication" %}} for details
-deleteAuthKey value
authKey for metrics' deletion via /prometheus/api/v1/admin/tsdb/delete_series and /graphite/tags/delSeries
Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path
-denyQueryTracing
Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}}
Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}}
-downsampling.period array
Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
@ -1479,7 +1479,7 @@ Below is the output for `/path/to/vmselect -help`:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-replicationFactor array
@ -1490,7 +1490,7 @@ Below is the output for `/path/to/vmselect -help`:
-search.denyPartialResponse
Whether to deny partial responses if a part of -storageNode instances fail to perform queries; this trades availability over consistency; see also -search.maxQueryDuration
-search.disableCache
Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./README.md#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup
Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup
-search.disableImplicitConversion
Whether to return an error for queries that rely on implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. See also -search.logImplicitConversion
-search.graphiteMaxPointsPerSeries int
@ -1517,11 +1517,11 @@ Below is the output for `/path/to/vmselect -help`:
-search.maxFederateSeries int
The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000)
-search.maxGraphiteSeries int
The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./README.md#graphite-render-api-usage" %}} (default 300000)
The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./#graphite-render-api-usage" %}} (default 300000)
-search.maxGraphiteTagKeys int
The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000)
The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000)
-search.maxGraphiteTagValues int
The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000)
The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000)
-search.maxLabelsAPIDuration duration
The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s)
-search.maxLabelsAPISeries int
@ -1581,7 +1581,7 @@ Below is the output for `/path/to/vmselect -help`:
Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call
Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path
-search.resetRollupResultCacheOnStartup
Whether to reset rollup result cache on startup. See {{% ref "./README.md#rollup-result-cache" %}}. See also -search.disableCache
Whether to reset rollup result cache on startup. See {{% ref "./#rollup-result-cache" %}}. See also -search.disableCache
-search.setLookbackToStep
Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored
-search.skipSlowReplicas
@ -1668,19 +1668,19 @@ Below is the output for `/path/to/vmstorage -help`:
-cluster.tlsKeyFile string
Path to server-side TLS key file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
-dedup.minScrapeInterval duration
Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details
Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./#deduplication" %}} for details
-denyQueriesOutsideRetention
Whether to deny queries outside of the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee
-denyQueryTracing
Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}}
Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}}
-downsampling.period array
Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
@ -1807,11 +1807,11 @@ Below is the output for `/path/to/vmstorage -help`:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-retentionFilter array
Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./README.md#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-retentionPeriod value
@ -1856,9 +1856,9 @@ Below is the output for `/path/to/vmstorage -help`:
Overrides max size for storage/tsid cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}}
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0)
-storage.maxDailySeries int
The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxHourlySeries
The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxHourlySeries
-storage.maxHourlySeries int
The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxDailySeries
The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxDailySeries
-storage.minFreeDiskSpaceBytes size
The minimum free disk space at -storageDataPath after which the storage stops accepting new data
Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 10000000)

2
docs/ExtendedPromQL.md
View file

@ -5,4 +5,4 @@ weight: 100
# MetricsQL old page
The page has been moved to [MetricsQL](https://docs.victoriametrics.com/metricsql/).
The page has been moved to [MetricsQL](./MetricsQL.md).

214
docs/FAQ.md
View file

@ -16,40 +16,40 @@ To provide the best monitoring solution.
## Who uses VictoriaMetrics?
See [case studies](https://docs.victoriametrics.com/casestudies/).
See [case studies](./CaseStudies.md).
## Which features does VictoriaMetrics have?
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#prominent-features).
See [these docs](./Single-Server-VictoriaMetrics.md#prominent-features).
## Are there performance comparisons with other solutions?
Yes. See [these benchmarks](https://docs.victoriametrics.com/articles/#benchmarks).
Yes. See [these benchmarks](./Articles.md#benchmarks).
## How to start using VictoriaMetrics?
See [these docs](https://docs.victoriametrics.com/quick-start/).
See [these docs](./Quick-Start.md).
## Hot to contribute to VictoriaMetrics?
See [these docs](https://docs.victoriametrics.com/contributing/).
See [these docs](./CONTRIBUTING.md).
## Does VictoriaMetrics support replication?
Yes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details.
Yes. See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details.
## Can I use VictoriaMetrics instead of Prometheus?
Yes in most cases. VictoriaMetrics can substitute Prometheus in the following aspects:
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/vmagent/) and with single-node VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](https://docs.victoriametrics.com/vmalert/).
* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#grafana-setup).
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](./vmagent.md) and with single-node VictoriaMetrics. See [these docs](./#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](./vmalert.md).
* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](./#grafana-setup).
## What is the difference between vmagent and Prometheus?
While both [vmagent](https://docs.victoriametrics.com/vmagent/) and Prometheus may scrape Prometheus targets (aka `/metrics` pages)
according to the provided Prometheus-compatible [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
While both [vmagent](./vmagent.md) and Prometheus may scrape Prometheus targets (aka `/metrics` pages)
according to the provided Prometheus-compatible [scrape configs](./sd_configs.md#scrape_configs)
and send data to multiple remote storage systems, vmagent has the following additional features:
* vmagent usually requires lower amounts of CPU, RAM and disk IO compared to Prometheus when scraping an enormous number of targets (more than 1000)
@ -59,30 +59,30 @@ and send data to multiple remote storage systems, vmagent has the following addi
with a hardcoded retention of 2 hours.
* vmagent may accept, relabel and filter data obtained via multiple data ingestion protocols in addition to data scraped from Prometheus targets.
That means it supports both `pull` and `push` protocols for data ingestion.
See [these docs](https://docs.victoriametrics.com/vmagent/#features) for details.
* vmagent may be used in different [use cases](https://docs.victoriametrics.com/vmagent/#use-cases):
* [IoT and edge monitoring](https://docs.victoriametrics.com/vmagent/#iot-and-edge-monitoring)
* [Drop-in replacement for Prometheus](https://docs.victoriametrics.com/vmagent/#drop-in-replacement-for-prometheus)
* [Statsd alternative](https://docs.victoriametrics.com/vmagent/#statsd-alternative)
* [Flexible metrics relay](https://docs.victoriametrics.com/vmagent/#flexible-metrics-relay)
* [Replication and high availability](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability)
* [Sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages)
* [Relabeling and filtering](https://docs.victoriametrics.com/vmagent/#relabeling-and-filtering)
* [Splitting data streams among multiple systems](https://docs.victoriametrics.com/vmagent/#splitting-data-streams-among-multiple-systems)
* [Prometheus remote_write proxy](https://docs.victoriametrics.com/vmagent/#prometheus-remote_write-proxy)
* [remote_write for clustered version](https://docs.victoriametrics.com/vmagent/#remote_write-for-clustered-version)
See [these docs](./vmagent.md#features) for details.
* vmagent may be used in different [use cases](./vmagent.md#use-cases):
* [IoT and edge monitoring](./vmagent.md#iot-and-edge-monitoring)
* [Drop-in replacement for Prometheus](./vmagent.md#drop-in-replacement-for-prometheus)
* [Statsd alternative](./vmagent.md#statsd-alternative)
* [Flexible metrics relay](./vmagent.md#flexible-metrics-relay)
* [Replication and high availability](./vmagent.md#replication-and-high-availability)
* [Sharding among remote storages](./vmagent.md#sharding-among-remote-storages)
* [Relabeling and filtering](./vmagent.md#relabeling-and-filtering)
* [Splitting data streams among multiple systems](./vmagent.md#splitting-data-streams-among-multiple-systems)
* [Prometheus remote_write proxy](./vmagent.md#prometheus-remote_write-proxy)
* [remote_write for clustered version](./vmagent.md#remote_write-for-clustered-version)
## What is the difference between vmagent and Prometheus agent?
Both [vmagent](https://docs.victoriametrics.com/vmagent/) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences:
Both [vmagent](./vmagent.md) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences:
* vmagent usually requires lower amounts of CPU, RAM and disk IO compared to the Prometheus agent.
* Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON – see [these docs](https://docs.victoriametrics.com/vmagent/#features).
* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets).
* vmagent supports [improved relabeling](https://docs.victoriametrics.com/vmagent/#relabeling).
* vmagent can limit the number of scraped metrics per target – see [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter).
* vmagent supports loading scrape configs from multiple files – see [these docs](https://docs.victoriametrics.com/vmagent/#loading-scrape-configs-from-multiple-files).
* vmagent supports data reading and data writing from/to Kafka – see [these docs](https://docs.victoriametrics.com/vmagent/#kafka-integration).
* Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON – see [these docs](./vmagent.md#features).
* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](./vmagent.md#scraping-big-number-of-targets).
* vmagent supports [improved relabeling](./vmagent.md#relabeling).
* vmagent can limit the number of scraped metrics per target – see [these docs](./vmagent.md#cardinality-limiter).
* vmagent supports loading scrape configs from multiple files – see [these docs](./vmagent.md#loading-scrape-configs-from-multiple-files).
* vmagent supports data reading and data writing from/to Kafka – see [these docs](./vmagent.md#kafka-integration).
* vmagent can read and update scrape configs from http and https URLs, while the Prometheus agent can read them only from the local file system.
## Is it safe to enable [remote write](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) in Prometheus?
@ -90,7 +90,7 @@ Both [vmagent](https://docs.victoriametrics.com/vmagent/) and [Prometheus agent]
Yes. Prometheus continues writing data to local storage after enabling remote write, so all the existing local storage data
and new data is available for querying via Prometheus as usual.
It is recommended using [vmagent](https://docs.victoriametrics.com/vmagent/) for scraping Prometheus targets
It is recommended using [vmagent](./vmagent.md) for scraping Prometheus targets
and writing data to VictoriaMetrics.
## How does VictoriaMetrics compare to other remote storage solutions for Prometheus such as [M3DB](https://github.com/m3db/m3), [Thanos](https://github.com/thanos-io/thanos), [Cortex](https://github.com/cortexproject/cortex), [Mimir](https://github.com/grafana/mimir), etc.?
@ -113,13 +113,13 @@ VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/t
## What is the difference between VictoriaMetrics and [QuestDB](https://questdb.io/)?
* QuestDB needs more than 20x storage space than VictoriaMetrics. This translates to higher storage costs and slower queries over historical data, which must be read from the disk.
* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics).
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
* VictoriaMetrics can be queried via the [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana.
* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/vmagent/), while QuestDB has no integration with Prometheus.
* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-import-time-series-data)).
* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](https://docs.victoriametrics.com/#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data).
* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](./#how-to-start-victoriametrics).
* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
* VictoriaMetrics can be queried via the [Prometheus querying API](./#prometheus-querying-api-usage) and via [Graphite's API](./#graphite-api-usage).
* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](./#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana.
* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](./vmagent.md), while QuestDB has no integration with Prometheus.
* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](./#how-to-import-time-series-data)).
* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](./#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data).
## What is the difference between VictoriaMetrics and [Grafana Mimir](https://github.com/grafana/mimir)?
@ -132,13 +132,13 @@ See also [Grafana Mimir vs VictoriaMetrics benchmark](https://victoriametrics.co
VictoriaMetrics is similar to Cortex in the following aspects:
* Both systems accept data from [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus
* Both systems accept data from [vmagent](./vmagent.md) or Prometheus
via the standard [remote_write API](https://prometheus.io/docs/practices/remote_write/), so there is no need for running sidecars
unlike in [Thanos](https://github.com/thanos-io/thanos)' case.
* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety).
* Both systems scale horizontally to multiple nodes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability) for details.
* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](https://docs.victoriametrics.com/vmalert/).
* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](./Cluster-VictoriaMetrics.md#multitenancy).
* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](./Cluster-VictoriaMetrics.md#replication-and-data-safety).
* Both systems scale horizontally to multiple nodes. See [these docs](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability) for details.
* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](./vmalert.md).
* Both systems can be queried via the [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/) and integrate perfectly with Grafana.
The main differences between Cortex and VictoriaMetrics:
@ -147,17 +147,17 @@ The main differences between Cortex and VictoriaMetrics:
* Cortex heavily relies on third-party services such as Consul, Memcache, DynamoDB, BigTable, Cassandra, etc.
This may increase operational complexity and reduce system reliability compared to VictoriaMetrics' case,
which doesn't use any external services. Compare [Cortex' Architecture](https://github.com/cortexproject/cortex/blob/master/docs/architecture.md)
to [VictoriaMetrics' architecture](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
* VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/),
to [VictoriaMetrics' architecture](./Cluster-VictoriaMetrics.md#architecture-overview).
* VictoriaMetrics provides [production-ready single-node solution](./Single-Server-VictoriaMetrics.md),
which is much easier to set up and operate than a Cortex cluster.
* Cortex may lose up to 12 hours of recent data on Ingestor failure – see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss).
VictoriaMetrics may lose only a few seconds of recent data, which isn't synced to persistent storage yet.
See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://docs.victoriametrics.com/casestudies/).
* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](./CaseStudies.md).
* VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details.
* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage).
## What is the difference between VictoriaMetrics and [Thanos](https://github.com/thanos-io/thanos)?
@ -171,33 +171,33 @@ The main differences between Cortex and VictoriaMetrics:
VictoriaMetrics works perfectly with HDD-based block storage – there is no need for using more expensive SSD or NVMe disks in most cases.
* Thanos may lose up to 2 hours of recent data, which wasn't uploaded yet to object storage. VictoriaMetrics may lose only a few seconds of recent data,
which hasn't been synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
* VictoriaMetrics provides a [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/),
* VictoriaMetrics provides a [production-ready single-node solution](./Single-Server-VictoriaMetrics.md),
which is much easier to set up and operate than Thanos components.
* Thanos may be harder to set up and operate compared to VictoriaMetrics, since it has more moving parts, which can be connected with fewer reliable networks.
See [this article for details](https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683).
* Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
* VictoriaMetrics accepts data via multiple popular data ingestion protocols in addition to the Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details.
* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage).
## How does VictoriaMetrics compare to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/)?
* VictoriaMetrics requires [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and it [works faster](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
* VictoriaMetrics needs lower amounts of storage space than InfluxDB for production data.
* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](https://docs.victoriametrics.com/metricsql/). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](./MetricsQL.md). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
* VictoriaMetrics accepts data in multiple popular data ingestion protocols in addition to InfluxDB – Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details.
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details.
* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage).
## How does VictoriaMetrics compare to [TimescaleDB](https://www.timescale.com/)?
* TimescaleDB insists on using SQL as a query language. While SQL is more powerful than PromQL, this power is rarely required during typical usages of a TSDB. Real-world queries usually [look clearer and simpler when written in PromQL than in SQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085).
* VictoriaMetrics requires [up to 70x less storage space compared to TimescaleDB](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) for storing the same amount of time series data. The gap in storage space usage can be lowered from 70x to 3x if [compression in TimescaleDB is properly configured](https://docs.timescale.com/latest/using-timescaledb/compression) (it isn't an easy task in general :)).
* VictoriaMetrics requires up to 10x less CPU and RAM resources than TimescaleDB for processing production data. See [this article](https://abiosgaming.com/press/high-cardinality-aggregations/) for details.
* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics)).
* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](./#how-to-start-victoriametrics)).
* VictoriaMetrics accepts data in multiple popular data ingestion protocols – InfluxDB, OpenTSDB, Graphite, CSV – while TimescaleDB supports only SQL inserts.
* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage).
## Does VictoriaMetrics use Prometheus technologies like other clustered TSDBs built on top of Prometheus such as [Thanos](https://github.com/thanos-io/thanos) or [Cortex](https://github.com/cortexproject/cortex)?
@ -208,7 +208,7 @@ The architecture is [optimized for storing and querying large amounts of time se
The following versions are open source and free:
* [Single-node version](https://docs.victoriametrics.com/single-server-victoriametrics/).
* [Single-node version](./Single-Server-VictoriaMetrics.md).
* [Cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
We provide commercial support for both versions. [Contact us](mailto:info@victoriametrics.com) for the pricing.
@ -231,12 +231,12 @@ if a query covers 1000 metrics with 10K values each, then the remote read API ha
This is slow and expensive.
Prometheus' remote read API isn't intended for querying foreign data – aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details.
So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), the [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/#grafana-setup).
So just query VictoriaMetrics directly via [vmui](./#vmui), the [Prometheus Querying API](./#prometheus-querying-api-usage)
or via [Prometheus datasource in Grafana](./#grafana-setup).
## Does VictoriaMetrics deduplicate data from Prometheus instances scraping the same targets (aka `HA pairs`)?
Yes. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication) for details.
Yes. See [these docs](./Single-Server-VictoriaMetrics.md#deduplication) for details.
## Where is the source code of VictoriaMetrics?
@ -254,21 +254,21 @@ and scales horizontally to multiple nodes.
## What is the difference between single-node and cluster versions of VictoriaMetrics?
Both [single-node](https://docs.victoriametrics.com/single-server-victoriametrics/) and
[cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) versions of VictoriaMetrics
Both [single-node](./Single-Server-VictoriaMetrics.md) and
[cluster](./Cluster-VictoriaMetrics.md) versions of VictoriaMetrics
share the core source code, so they have many common features. They have the following differences though:
* [Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) runs on a single host,
while [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can scale to many hosts.
* [Single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) runs on a single host,
while [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) can scale to many hosts.
Single-node VictoriaMetrics scales vertically though, e.g. its capacity and performance scales almost linearly when increasing
available CPU, RAM, disk IO and disk space. See [an article about vertical scalability of a single-node VictoriaMetrics](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
* Cluster version of VictoriaMetrics supports [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy),
* Cluster version of VictoriaMetrics supports [multitenancy](./Cluster-VictoriaMetrics.md#multitenancy),
while single-node VictoriaMetrics doesn't support it.
* Cluster version of VictoriaMetrics supports data replication, while single-node VictoriaMetrics relies on the durability
of the persistent storage pointed by `-storageDataPath` command-line flag.
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details.
See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details.
* Single-node VictoriaMetrics provides higher capacity and performance comparing to cluster version of VictoriaMetrics
when running on the same hardware with the same amounts of CPU and RAM, since it has no overhead on data transfer
@ -290,7 +290,7 @@ File bugs and feature requests [here](https://github.com/VictoriaMetrics/Victori
## Where can I find information about multi-tenancy?
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). Multitenancy is supported only by the [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/) of VictoriaMetrics.
See [these docs](./Cluster-VictoriaMetrics.md#multitenancy). Multitenancy is supported only by the [cluster version](./Cluster-VictoriaMetrics.md) of VictoriaMetrics.
## How to set a memory limit for VictoriaMetrics components?
@ -298,10 +298,10 @@ All the VictoriaMetrics components provide command-line flags to control the siz
Memory usage for VictoriaMetrics components can be tuned according to the following docs:
* [Resource usage limits for single-node VictoriaMetrics](https://docs.victoriametrics.com/#resource-usage-limits)
* [Resource usage limits for cluster VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits)
* [Troubleshooting for vmagent](https://docs.victoriametrics.com/vmagent/#troubleshooting)
* [Troubleshooting for single-node VictoriaMetrics](https://docs.victoriametrics.com/#troubleshooting)
* [Resource usage limits for single-node VictoriaMetrics](./#resource-usage-limits)
* [Resource usage limits for cluster VictoriaMetrics](./Cluster-VictoriaMetrics.md#resource-usage-limits)
* [Troubleshooting for vmagent](./vmagent.md#troubleshooting)
* [Troubleshooting for single-node VictoriaMetrics](./#troubleshooting)
## How can I run VictoriaMetrics on FreeBSD/OpenBSD?
@ -311,13 +311,13 @@ or use pre-built binaries from [releases page](https://github.com/VictoriaMetric
## Does VictoriaMetrics support the Graphite query language?
Yes. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage).
Yes. See [these docs](./#graphite-api-usage).
## What is an active time series?
A time series is uniquely identified by its name plus a set of its labels. For example, `temperature{city="NY",country="US"}` and `temperature{city="SF",country="US"}`
are two distinct series, since they differ by the `city` label. A time series is considered active if it receives at least a single new sample during the last hour.
The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](./#monitoring) for details.
## What is high churn rate?
@ -335,10 +335,10 @@ The main reason for high churn rate is a metric label with frequently changed va
* A `hash` or `uuid` label, which changes frequently.
The solution against high churn rate is to identify and eliminate labels with frequently changed values.
[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data
before it gets ingested into database with [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/).
[Cardinality explorer](./#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data
before it gets ingested into database with [stream aggregation](./stream-aggregation.md).
The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](./#monitoring) for details.
## What is high cardinality?
@ -350,10 +350,10 @@ a large number of unique values, which presents a big share of the ingested time
* `url`
* `ip`
The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer).
The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](./#cardinality-explorer).
The official Grafana dashboards for VictoriaMetrics contain graphs, which show the number of active time series -
see [these docs](https://docs.victoriametrics.com/#monitoring) for details.
see [these docs](./#monitoring) for details.
## What is a slow insert?
@ -361,34 +361,34 @@ VictoriaMetrics maintains in-memory cache for mapping of [active time series](#w
The cache size depends on the available memory for VictoriaMetrics in the host system. If the information about all the active time series doesn't fit the cache,
then VictoriaMetrics needs to read and unpack the information from disk on every incoming sample for time series missing in the cache.
This operation is much slower than the cache lookup, so such an insert is named a `slow insert`.
A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) indicates
A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](./#monitoring) indicates
a memory shortage for the current number of [active time series](#what-is-an-active-time-series). Such a condition usually leads
to a significant slowdown for data ingestion and to significantly increased disk IO and CPU usage.
The solution is to add more memory or to reduce the number of [active time series](#what-is-an-active-time-series).
[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can be helpful for locating the source of high number of active time series.
[Cardinality explorer](./#cardinality-explorer) can be helpful for locating the source of high number of active time series.
## How to optimize MetricsQL query?
See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986).
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing) and [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer),
VictoriaMetrics also provides [query tracer](./#query-tracing) and [cardinality explorer](./#cardinality-explorer),
which can help during query optimization.
See also [troubleshooting slow queries](https://docs.victoriametrics.com/troubleshooting/#slow-queries).
See also [troubleshooting slow queries](./Troubleshooting.md#slow-queries).
## Which VictoriaMetrics type is recommended for use in production - single-node or cluster?
Both [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and
[VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) are production-ready.
Both [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) and
[VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) are production-ready.
Single-node VictoriaMetrics is able to handle quite big workloads in production
with tens of millions of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series)
at the ingestion rate of million of samples per second. See [this case study](https://docs.victoriametrics.com/casestudies/#wixcom).
with tens of millions of [active time series](./FAQ.md#what-is-an-active-time-series)
at the ingestion rate of million of samples per second. See [this case study](./CaseStudies.md#wixcom).
Single-node VictoriaMetrics requires lower amounts of CPU and RAM for handling the same workload comparing
to cluster version of VictoriaMetrics, since it doesn't need to pass the encoded data over the network
between [cluster components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
between [cluster components](./Cluster-VictoriaMetrics.md#architecture-overview).
The performance of a single-node VictoriaMetrics scales almost perfectly with the available CPU, RAM and disk IO resources on the host where it runs -
see [this article](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
@ -399,40 +399,40 @@ Given the facts above **it is recommended to use single-node VictoriaMetrics in
Cluster version of VictoriaMetrics may be preferred over single-node VictoriaMetrics in the following relatively rare cases:
- If [multitenancy support](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) is needed,
- If [multitenancy support](./Cluster-VictoriaMetrics.md#multitenancy) is needed,
since single-node VictoriaMetrics doesn't support multitenancy. Though it is possible to run multiple single-node VictoriaMetrics
instances - one per each tenant - and route incoming requests from particular tenant to the needed VictoriaMetrics instance
via [vmauth](https://docs.victoriametrics.com/vmauth/).
via [vmauth](./vmauth.md).
- If the current workload cannot be handled by a single-node VictoriaMetrics. For example, if you are going to ingest hundreds of millions of active time series
at ingestion rates exceeding a million samples per second, then it is better to use cluster version of VictoriaMetrics,
since its capacity can [scale horizontally with the number of nodes in the cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability).
since its capacity can [scale horizontally with the number of nodes in the cluster](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability).
## How to migrate data from single-node VictoriaMetrics to cluster version?
Single-node VictoriaMetrics stores data on disk in slightly different format comparing to cluster version of VictoriaMetrics.
So it is impossible to just copy the on-disk data from `-storageDataPath` directory from single-node VictoriaMetrics to a `vmstorage` node in VictoriaMetrics cluster.
If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics).
If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](./vmctl.md#migrating-data-from-victoriametrics).
## Why isn't MetricsQL 100% compatible with PromQL?
[MetricsQL](https://docs.victoriametrics.com/metricsql/) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
[MetricsQL](./MetricsQL.md) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
## How to migrate data from Prometheus to VictoriaMetrics?
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus).
Please see [these docs](./vmctl.md#migrating-data-from-prometheus).
## How to migrate data from InfluxDB to VictoriaMetrics?
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x).
Please see [these docs](./vmctl.md#migrating-data-from-influxdb-1x).
## How to migrate data from OpenTSDB to VictoriaMetrics?
Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb).
Please see [these docs](./vmctl.md#migrating-data-from-opentsdb).
## How to migrate data from Graphite to VictoriaMetrics?
Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
## Why do the same metrics have differences in VictoriaMetrics' and Prometheus' dashboards?
@ -442,19 +442,19 @@ The query engine may behave differently for some functions. Please see [this art
## If downsampling and deduplication are enabled how will this work?
[Deduplication](https://docs.victoriametrics.com/#deduplication) is a special case of zero-offset [downsampling](https://docs.victoriametrics.com/#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling
[Deduplication](./#deduplication) is a special case of zero-offset [downsampling](./#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling
## How to upgrade or downgrade VictoriaMetrics without downtime?
Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics).
Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](./#how-to-upgrade-victoriametrics).
[Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can be restarted / upgraded / downgraded without downtime according to [these instructions](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes).
[Cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) can be restarted / upgraded / downgraded without downtime according to [these instructions](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes).
## Why VictoriaMetrics misses automatic data re-balancing between vmstorage nodes?
VictoriaMetrics doesn't rebalance data between `vmstorage` nodes when new `vmstorage` nodes are added to the cluster.
This means that newly added `vmstorage` nodes will have less data at `-storageDataPath` comparing to the old `vmstorage` nodes
until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](https://docs.victoriametrics.com/#retention).
until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](./#retention).
The automatic rebalancing is the process of moving data between `vmstorage` nodes, so every node has the same amounts of data eventually.
It is disabled by default because it may consume additional CPU, network bandwidth and disk IO at `vmstorage` nodes for long periods of time,
@ -463,22 +463,22 @@ which, in turn, can negatively impact VictoriaMetrics cluster availability.
Additionally, it is unclear how to handle the automatic re-balancing if cluster configuration changes when the re-balancing is in progress.
The amounts of data stored in `vmstorage` becomes equal among old `vmstorage` nodes and new `vmstorage` nodes
after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](https://docs.victoriametrics.com/#retention).
after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](./#retention).
The data ingestion load becomes even between old `vmstorage` nodes and new `vmstorage` nodes almost immediately
after adding new `vmstorage` nodes to the cluster, since `vminsert` nodes evenly distribute incoming time series
among the nodes specified in `-storageNode` command-line flag. The newly added `vmstorage` nodes may experience
increased load during the first couple of minutes because they need to register [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series).
increased load during the first couple of minutes because they need to register [active time series](./FAQ.md#what-is-an-active-time-series).
The query load becomes even between old `vmstorage` nodes and new `vmstorage` nodes after most of queries are executed
over time ranges with data covered by new `vmstorage` nodes. Usually the most of queries are received
from [alerting and recording rules](https://docs.victoriametrics.com/vmalert/), which query data on limited time ranges
from [alerting and recording rules](./vmalert.md), which query data on limited time ranges
such as a few hours or few days at max. This means that the query load between old `vmstorage` nodes and new `vmstorage` nodes
should become even in a few hours / days after adding new `vmstorage` nodes.
## Why VictoriaMetrics misses automatic recovery of replication factor?
VictoriaMetrics doesn't restore [replication factor](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
VictoriaMetrics doesn't restore [replication factor](./Cluster-VictoriaMetrics.md#replication-and-data-safety)
when some of `vmstorage` nodes are removed from the cluster because of the following reasons:
- Automatic replication factor recovery needs copying non-trivial amounts of data between the remaining `vmstorage` nodes.
@ -488,5 +488,5 @@ when some of `vmstorage` nodes are removed from the cluster because of the follo
- It is unclear when the automatic replication factor recovery must be started. How to distinguish the expected temporary
`vmstorage` node unavailability because of maintenance, upgrade or config changes from permanent loss of data at the `vmstorage` node?
It is recommended reading [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
It is recommended reading [replication and data safety docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety)
for more details.

444
docs/MetricsQL.md
View file

File diff suppressed because it is too large Load diff

104
docs/Quick-Start.md
View file

@ -12,10 +12,10 @@ aliases:
## How to install
VictoriaMetrics is distributed in two forms:
* [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - all-in-one
* [Single-server-VictoriaMetrics](./Single-Server-VictoriaMetrics.md) - all-in-one
binary, which is very easy to use and maintain.
Single-server-VictoriaMetrics perfectly scales vertically and easily handles millions of metrics/s;
* [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) - set of components
* [VictoriaMetrics Cluster](./Cluster-VictoriaMetrics.md) - set of components
for building horizontally scalable clusters.
Single-server-VictoriaMetrics VictoriaMetrics is available as:
@ -26,16 +26,16 @@ Single-server-VictoriaMetrics VictoriaMetrics is available as:
* [Binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
* [Ansible Roles](https://github.com/VictoriaMetrics/ansible-playbooks)
* [Source code](https://github.com/VictoriaMetrics/VictoriaMetrics).
See [How to build from sources](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-build-from-sources)
See [How to build from sources](./Single-Server-VictoriaMetrics.md#how-to-build-from-sources)
* [VictoriaMetrics on Linode](https://www.linode.com/marketplace/apps/victoriametrics/victoriametrics/)
* [VictoriaMetrics on DigitalOcean](https://marketplace.digitalocean.com/apps/victoriametrics-single)
Just download VictoriaMetrics and follow
[these instructions](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-start-victoriametrics).
Then read [Prometheus setup](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-setup)
and [Grafana setup](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup) docs.
[these instructions](./Single-Server-VictoriaMetrics.md#how-to-start-victoriametrics).
Then read [Prometheus setup](./Single-Server-VictoriaMetrics.md#prometheus-setup)
and [Grafana setup](./Single-Server-VictoriaMetrics.md#grafana-setup) docs.
VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/changelog/) and performing [regular upgrades](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics).
VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](./CHANGELOG.md) and performing [regular upgrades](how-to-upgrade-victoriametrics).
### Starting VictoriaMetrics Single Node via Docker {anchor="starting-vm-single-via-docker"}
@ -53,9 +53,9 @@ docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 842
Open <a href="http://localhost:8428">http://localhost:8428</a> in web browser
and read [these docs](https://docs.victoriametrics.com/#operation).
and read [these docs](operation).
There is also [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/)
There is also [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md)
- horizontally scalable installation, which scales to multiple nodes.
### Starting VictoriaMetrics Cluster via Docker {anchor="starting-vm-cluster-via-docker"}
@ -75,13 +75,13 @@ make docker-cluster-up
See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#readme).
* [Cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup)
* [Cluster setup](./Cluster-VictoriaMetrics.md#cluster-setup)
### Starting VictoriaMetrics Single Node from a Binary {anchor="starting-vm-single-from-a-binary"}
1. Download the correct binary for your OS and architecture from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases).
For Enterprise binaries see [this link](./enterprise.md#binary-releases).
2. Extract the archive to /usr/local/bin by running:
```sh
@ -128,9 +128,9 @@ WantedBy=multi-user.target
END
```
Extra [command-line flags](https://docs.victoriametrics.com/#list-of-command-line-flags) can be added to `ExecStart` line.
Extra [command-line flags](list-of-command-line-flags) can be added to `ExecStart` line.
If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](https://docs.victoriametrics.com/single-server-victoriametrics/#running-as-windows-service).
If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](./Single-Server-VictoriaMetrics.md#running-as-windows-service).
> Please note, `victoriametrics` service is listening on `:8428` for HTTP connections (see `-httpListenAddr` flag).
@ -146,20 +146,20 @@ sudo systemctl daemon-reload && sudo systemctl enable --now victoriametrics.serv
sudo systemctl status victoriametrics.service
```
8. After VictoriaMetrics is in `Running` state, verify [vmui](https://docs.victoriametrics.com/#vmui) is working
8. After VictoriaMetrics is in `Running` state, verify [vmui](vmui) is working
by going to `http://<ip_or_hostname>:8428/vmui`.
### Starting VictoriaMetrics Cluster from Binaries {anchor="starting-vm-cluster-from-binaries"}
VictoriaMetrics cluster consists of [3 components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
It is recommended to run these components in the same private network (for [security reasons](https://docs.victoriametrics.com/#security)),
VictoriaMetrics cluster consists of [3 components](./Cluster-VictoriaMetrics.md#architecture-overview).
It is recommended to run these components in the same private network (for [security reasons](security)),
but on the separate physical nodes for the best performance.
On all nodes you will need to do the following:
1. Download the correct binary for your OS and architecture with `-cluster` suffix from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases).
For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases).
For Enterprise binaries see [this link](./enterprise.md#binary-releases).
2. Extract the archive to /usr/local/bin by running:
@ -175,7 +175,7 @@ Replace `<victoriametrics-archive>` with the path to the archive you downloaded
sudo useradd -s /usr/sbin/nologin victoriametrics
```
See recommendations for installing each type of [cluster component](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) below.
See recommendations for installing each type of [cluster component](./Cluster-VictoriaMetrics.md#architecture-overview) below.
##### Installing vmstorage
@ -207,7 +207,7 @@ WantedBy=multi-user.target
END
```
Extra [command-line flags](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vmstorage)
Extra [command-line flags](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vmstorage)
for vmstorage can be added to `ExecStart` line.
> Please note, `vmstorage` service is listening on `:8400` for vminsert connections (see `-vminsertAddr` flag),
@ -256,7 +256,7 @@ END
Replace `<list of vmstorages>` with addresses of previously configured `vmstorage` services.
To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
in one flag. See more details in `-storageNode` flag description [here](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vminsert).
> Please note, `vminsert` service is listening on `:8480` for HTTP connections (see `-httpListenAddr` flag).
@ -310,7 +310,7 @@ END
Replace `<list of vmstorages>` with addresses of previously configured `vmstorage` services.
To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas
in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert).
in one flag. See more details in `-storageNode` flag description [here](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vminsert).
> Please note, `vmselect` service is listening on `:8481` for HTTP connections (see `-httpListenAddr` flag).
@ -327,53 +327,53 @@ sudo systemctl status vmselect.service
```
5. After `vmselect` is in `Running` state, confirm the service is healthy by visiting `http://<ip_or_hostname>:8481/select/0/vmui` link.
It should open [vmui](https://docs.victoriametrics.com/#vmui) page.
It should open [vmui](vmui) page.
## Write data
There are two main models in monitoring for data collection:
[push](https://docs.victoriametrics.com/keyconcepts/#push-model)
and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model).
[push](./keyConcepts.md#push-model)
and [pull](./keyConcepts.md#pull-model).
Both are used in modern monitoring and both are supported by VictoriaMetrics.
See more details on [writing data here](https://docs.victoriametrics.com/keyconcepts/#write-data).
See more details on [writing data here](./keyConcepts.md#write-data).
## Query data
VictoriaMetrics provides an
[HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
[HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage)
for serving read queries. The API is used in various integrations such as
[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup).
[Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup).
The same API is also used by
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - graphical User Interface
[VMUI](./Single-Server-VictoriaMetrics.md#vmui) - graphical User Interface
for querying and visualizing metrics.
[MetricsQL](https://docs.victoriametrics.com/metricsql/) - is the query language for executing read queries
[MetricsQL](./MetricsQL.md) - is the query language for executing read queries
in VictoriaMetrics. MetricsQL is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics)
-like query language with a powerful set of functions and features for working specifically with time series data.
See more details on [querying data here](https://docs.victoriametrics.com/keyconcepts/#query-data)
See more details on [querying data here](./keyConcepts.md#query-data)
## Alerting
It is not possible to physically trace all changes on graphs all the time, that is why alerting exists.
In [vmalert](https://docs.victoriametrics.com/vmalert/) it is possible to create a set of conditions
In [vmalert](./vmalert.md) it is possible to create a set of conditions
based on PromQL and MetricsQL queries that will send a notification when such conditions are met.
## Data migration
Migrating data from other TSDBs to VictoriaMetrics is as simple as importing data via any of
[supported formats](https://docs.victoriametrics.com/keyconcepts/#push-model).
[supported formats](./keyConcepts.md#push-model).
The migration might get easier when using [vmctl](https://docs.victoriametrics.com/vmctl/) - VictoriaMetrics
The migration might get easier when using [vmctl](./vmctl.md) - VictoriaMetrics
command line tool. It supports the following databases for migration to VictoriaMetrics:
* [Prometheus using snapshot API](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus);
* [Thanos](https://docs.victoriametrics.com/vmctl/#migrating-data-from-thanos);
* [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x);
* [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb);
* [Migrate data between VictoriaMetrics single and cluster versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics).
* [Prometheus using snapshot API](./vmctl.md#migrating-data-from-prometheus);
* [Thanos](./vmctl.md#migrating-data-from-thanos);
* [InfluxDB](./vmctl.md#migrating-data-from-influxdb-1x);
* [OpenTSDB](./vmctl.md#migrating-data-from-opentsdb);
* [Migrate data between VictoriaMetrics single and cluster versions](./vmctl.md#migrating-data-from-victoriametrics).
## Productionization
@ -384,7 +384,7 @@ When going to production with VictoriaMetrics we recommend following the recomme
Each VictoriaMetrics component emits its own metrics with various details regarding performance
and health state. Docs for the components also contain a `Monitoring` section with an explanation
of what and how should be monitored. For example,
[Single-server-VictoriaMetrics Monitoring](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring).
[Single-server-VictoriaMetrics Monitoring](./Cluster-VictoriaMetrics.md#monitoring).
VictoriaMetric team prepared a list of [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards)
for the main components. Each dashboard contains a lot of useful information and tips. It is recommended
@ -401,33 +401,33 @@ See more details in the article [VictoriaMetrics Monitoring](https://victoriamet
### Capacity planning
See capacity planning sections in [docs](https://docs.victoriametrics.com) for
[Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning)
and [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning).
See capacity planning sections in [docs](./README.md) for
[Single-server-VictoriaMetrics](./Single-Server-VictoriaMetrics.md#capacity-planning)
and [VictoriaMetrics Cluster](./Cluster-VictoriaMetrics.md#capacity-planning).
Capacity planning isn't possible without [monitoring](#monitoring), so consider configuring it first.
Understanding resource usage and performance of VictoriaMetrics also requires knowing the tech terms
[active series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series),
[churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate),
[cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality),
[slow inserts](https://docs.victoriametrics.com/faq/#what-is-a-slow-insert).
[active series](./FAQ.md#what-is-an-active-time-series),
[churn rate](./FAQ.md#what-is-high-churn-rate),
[cardinality](./FAQ.md#what-is-high-cardinality),
[slow inserts](./FAQ.md#what-is-a-slow-insert).
All of them are present in [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards).
### Data safety
It is recommended to read [Replication and data safety](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety),
It is recommended to read [Replication and data safety](./Cluster-VictoriaMetrics.md#replication-and-data-safety),
[Why replication doesn’t save from disaster?](https://valyala.medium.com/speeding-up-backups-for-big-time-series-databases-533c1a927883)
and [backups](https://docs.victoriametrics.com/single-server-victoriametrics/#backups).
and [backups](./Single-Server-VictoriaMetrics.md#backups).
### Configuring limits
To avoid excessive resource usage or performance degradation limits must be in place:
* [Resource usage limits](https://docs.victoriametrics.com/faq/#how-to-set-a-memory-limit-for-victoriametrics-components);
* [Cardinality limiter](https://docs.victoriametrics.com/single-server-victoriametrics/#cardinality-limiter).
* [Resource usage limits](./FAQ.md#how-to-set-a-memory-limit-for-victoriametrics-components);
* [Cardinality limiter](./Single-Server-VictoriaMetrics.md#cardinality-limiter).
### Security recommendations
* [Security recommendations for single-node VictoriaMetrics](https://docs.victoriametrics.com/#security)
* [Security recommendations for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#security)
* [Security recommendations for single-node VictoriaMetrics](security)
* [Security recommendations for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#security)

570
docs/README.md
View file

File diff suppressed because it is too large Load diff

2
docs/Release-Guide.md
View file

@ -50,7 +50,7 @@ Bumping the limits may significantly improve build speed.
1. Add `(available starting from v1.xx.y)` line to feature docs introduced in the upcoming release.
1. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md)
and make it merged. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7).
1. Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/lts-releases/).
1. Cherry-pick bug fixes relevant for [LTS releases](./LTS-releases.md).
1. Make sure you get all changes fetched `git fetch --all`.
1. Create the following release tags:
* `git tag -s v1.xx.y` in `master` branch

572
docs/Single-server-VictoriaMetrics.md
View file

File diff suppressed because it is too large Load diff

130
docs/Troubleshooting.md
View file

@ -26,21 +26,21 @@ If you hit some issue or have some question about VictoriaMetrics components,
then please follow the following steps in order to quickly find the solution:
1. Check the version of VictoriaMetrics component, which needs to be troubleshot and compare
it to [the latest available version](https://docs.victoriametrics.com/changelog/).
it to [the latest available version](./CHANGELOG.md).
If the used version is lower than the latest available version, then there are high chances
that the issue is already resolved in newer versions. Carefully read [the changelog](https://docs.victoriametrics.com/changelog/)
that the issue is already resolved in newer versions. Carefully read [the changelog](./CHANGELOG.md)
between your version and the latest version and check whether the issue is already fixed there.
If the issue is already fixed in newer versions, then upgrade to the newer version and verify whether the issue is fixed:
- [How to upgrade single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics)
- [How to upgrade VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes)
- [How to upgrade single-node VictoriaMetrics](./README.md#how-to-upgrade-victoriametrics)
- [How to upgrade VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes)
Upgrade procedure for other VictoriaMetrics components is as simple as gracefully stopping the component
by sending `SIGINT` signal to it and starting the new version of the component.
There may be breaking changes between different versions of VictoriaMetrics components in rare cases.
These cases are documented in [the changelog](https://docs.victoriametrics.com/changelog/).
These cases are documented in [the changelog](./CHANGELOG.md).
So please read the changelog before the upgrade.
1. Inspect command-line flags passed to VictoriaMetrics components and remove flags which unclear outcomes for your workload.
@ -78,7 +78,7 @@ then please follow the following steps in order to quickly find the solution:
before posting your question to VictoriaMetrics Slack chat, since GitHub issues are indexed by Google,
while Slack messages aren't indexed by Google. This simplifies searching for the solution to the issue for future VictoriaMetrics users.
1. Pro tip 1: if you see that [VictoriaMetrics docs](https://docs.victoriametrics.com/) contain incomplete or incorrect information,
1. Pro tip 1: if you see that [VictoriaMetrics docs](./Single-server-VictoriaMetrics.md) contain incomplete or incorrect information,
then please create a pull request with the relevant changes. This will help VictoriaMetrics community.
All the docs published at `https://docs.victoriametrics.com` are located in the [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs)
@ -121,12 +121,12 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
to reduce the number of returned series.
Sometimes the query may be improperly constructed, so it returns unexpected results.
It is recommended reading and understanding [MetricsQL docs](https://docs.victoriametrics.com/metricsql/),
especially [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries)
and [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) sections.
It is recommended reading and understanding [MetricsQL docs](./MetricsQL.md),
especially [subqueries](./MetricsQL.md#subqueries)
and [rollup functions](./MetricsQL.md#rollup-functions) sections.
1. If the simplest query continues returning unexpected / unreliable results, then try verifying correctness
of raw unprocessed samples for this query via [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
of raw unprocessed samples for this query via [/api/v1/export](./README.md#how-to-export-data-in-json-line-format)
on the given `[start..end]` time range and check whether they are expected:
```sh
@ -134,16 +134,16 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
cluster: curl http://<vmselect>:8481/select/<tenantID>/prometheus/api/v1/export -d 'match[]=http_requests_total' -d 'start=...' -d 'end=...'
```
Note that responses returned from [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
and from [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) contain **evaluated** data
Note that responses returned from [/api/v1/query](./keyConcepts.md#instant-query)
and from [/api/v1/query_range](./keyConcepts.md#range-query) contain **evaluated** data
instead of raw samples stored in VictoriaMetrics. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness)
for details.
If you migrate from InfluxDB, then pass `-search.setLookbackToStep` command-line flag to single-node VictoriaMetrics
or to `vmselect` in VictoriaMetrics cluster. See also [how to migrate from InfluxDB to VictoriaMetrics](https://docs.victoriametrics.com/guides/migrate-from-influx.html).
or to `vmselect` in VictoriaMetrics cluster. See also [how to migrate from InfluxDB to VictoriaMetrics](./guides/migrate-from-influx.md).
1. Sometimes response caching may lead to unexpected results when samples with older timestamps
are ingested into VictoriaMetrics (aka [backfilling](https://docs.victoriametrics.com/#backfilling)).
are ingested into VictoriaMetrics (aka [backfilling](./README.md#backfilling)).
Try disabling response cache and see whether this helps. This can be done in the following ways:
- By passing `-search.disableCache` command-line flag to a single-node VictoriaMetrics
@ -153,10 +153,10 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
If you use Grafana, then this query arg can be specified in `Custom Query Parameters` field
at Prometheus datasource settings - see [these docs](https://grafana.com/docs/grafana/latest/datasources/prometheus/) for details.
If the problem was in the cache, try resetting it via [resetRollupCache handler](https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache).
If the problem was in the cache, try resetting it via [resetRollupCache handler](./url-examples.md#internalresetrollupresultcache).
1. If you use cluster version of VictoriaMetrics, then it may return partial responses by default
when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability)
when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](./Cluster-VictoriaMetrics.md#cluster-availability)
for details. If you want to prioritize query consistency over cluster availability,
then you can pass `-search.denyPartialResponse` command-line flag to all the `vmselect` nodes.
In this case VictoriaMetrics returns an error during querying if at least a single `vmstorage` node is unavailable.
@ -171,20 +171,20 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
1. If you observe gaps when plotting time series try simplifying your query according to p2 and follow the list.
If problem still remains, then it is likely caused by irregular intervals for metrics collection (network delays
or targets unavailability on scrapes, irregular pushes, irregular timestamps).
VictoriaMetrics automatically [fills the gaps](https://docs.victoriametrics.com/keyconcepts/#range-query)
based on median interval between [data samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
VictoriaMetrics automatically [fills the gaps](./keyConcepts.md#range-query)
based on median interval between [data samples](./keyConcepts.md#raw-samples).
This might work incorrect for irregular data as median will be skewed. In this case it is recommended to switch
to the static interval for gaps filling by setting `-search.minStalenessInterval=5m` cmd-line flag (`5m` is
the static interval used by Prometheus).
1. If you observe recently written data is not immediately visible/queryable, then read more about
[query latency](https://docs.victoriametrics.com/keyconcepts/#query-latency) behavior.
[query latency](./keyConcepts.md#query-latency) behavior.
1. Try upgrading to the [latest available version of VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
and verifying whether the issue is fixed there.
1. Try executing the query with `trace=1` query arg. This enables query tracing, which may contain
useful information on why the query returns unexpected data. See [query tracing docs](https://docs.victoriametrics.com/#query-tracing) for details.
useful information on why the query returns unexpected data. See [query tracing docs](./README.md#query-tracing) for details.
1. Inspect command-line flags passed to VictoriaMetrics components. If you don't understand clearly the purpose
or the effect of some flags, then remove them from the list of flags passed to VictoriaMetrics components,
@ -193,15 +193,15 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try
1. If the steps above didn't help identifying the root cause of unexpected query results,
then [file a bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new) with details on how to reproduce the issue.
Instead of sharing screenshots in the issue, consider sharing query and [trace](https://docs.victoriametrics.com/#query-tracing)
results in [VMUI](https://docs.victoriametrics.com/#vmui) by clicking on `Export query` button in top right corner of the graph area.
Instead of sharing screenshots in the issue, consider sharing query and [trace](./README.md#query-tracing)
results in [VMUI](./README.md#vmui) by clicking on `Export query` button in top right corner of the graph area.
## Slow data ingestion
There are the following most commons reasons for slow data ingestion in VictoriaMetrics:
1. Memory shortage for the given amounts of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series).
1. Memory shortage for the given amounts of [active time series](./FAQ.md#what-is-an-active-time-series).
VictoriaMetrics (or `vmstorage` in cluster version of VictoriaMetrics) maintains an in-memory cache
for quick search for internal series ids per each incoming metric.
@ -210,10 +210,10 @@ There are the following most commons reasons for slow data ingestion in Victoria
for holding all the entries for active time series, then VictoriaMetrics locates the needed data on disk,
unpacks it, re-constructs the missing entry and puts it into the cache. This takes additional CPU time and disk read IO.
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring)
contain `Slow inserts` graph, which shows the cache miss percentage for `storage/tsid` cache
during data ingestion. If `slow inserts` graph shows values greater than 5% for more than 10 minutes,
then it is likely the current number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series)
then it is likely the current number of [active time series](./FAQ.md#what-is-an-active-time-series)
cannot fit the `storage/tsid` cache.
There are the following solutions exist for this issue:
@ -223,28 +223,28 @@ There are the following most commons reasons for slow data ingestion in Victoria
memory at `vmstorage` nodes. This can be done in two ways: either to increase the available memory
per each existing `vmstorage` node or to add more `vmstorage` nodes to the cluster.
- To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
- To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring)
contain a graph showing the number of active time series. Recent versions of VictoriaMetrics
provide [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer),
which can help determining and fixing the source of [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality).
provide [cardinality explorer](./README.md#cardinality-explorer),
which can help determining and fixing the source of [high cardinality](./FAQ.md#what-is-high-cardinality).
1. [High churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate),
1. [High churn rate](./FAQ.md#what-is-high-churn-rate),
e.g. when old time series are substituted with new time series at a high rate.
When VictoriaMetrics encounters a sample for new time series, it needs to register the time series
in the internal index (aka `indexdb`), so it can be quickly located on subsequent select queries.
The process of registering new time series in the internal index is an order of magnitude slower
than the process of adding new sample to already registered time series.
So VictoriaMetrics may work slower than expected under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
So VictoriaMetrics may work slower than expected under [high churn rate](./FAQ.md#what-is-high-churn-rate).
The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring)
provides `Churn rate` graph, which shows the average number of new time series registered
during the last 24 hours. If this number exceeds the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series),
then you need to identify and fix the source of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
during the last 24 hours. If this number exceeds the number of [active time series](./FAQ.md#what-is-an-active-time-series),
then you need to identify and fix the source of [high churn rate](./FAQ.md#what-is-high-churn-rate).
The most commons source of high churn rate is a label, which frequently changes its value. Try avoiding such labels.
The [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help identifying
The [cardinality explorer](./README.md#cardinality-explorer) can help identifying
such labels.
1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring)
1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring)
contain `resource usage` graphs, which show memory usage, CPU usage, disk IO usage and free disk size.
Make sure VictoriaMetrics has enough free resources for graceful handling of potential spikes in workload
according to the following recommendations:
@ -270,7 +270,7 @@ There are the following most commons reasons for slow data ingestion in Victoria
- If free disk space is lower than 20%, then VictoriaMetrics is unable to perform optimal
background merge of the incoming data. This leads to increased number of data files on disk,
which, in turn, slows down both data ingestion and querying. See [these docs](https://docs.victoriametrics.com/#storage) for details.
which, in turn, slows down both data ingestion and querying. See [these docs](./README.md#storage) for details.
1. If you run cluster version of VictoriaMetrics, then make sure `vminsert` and `vmstorage` components
are located in the same network with small network latency between them.
@ -279,7 +279,7 @@ There are the following most commons reasons for slow data ingestion in Victoria
If the network latency between `vminsert` and `vmstorage` is high (for example, if they run in different datacenters),
then this may become limiting factor for data ingestion speed.
The [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
The [official Grafana dashboard for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring)
contain `connection saturation` graph for `vminsert` components. If this graph reaches 100% (1s),
then it is likely you have issues with network latency between `vminsert` and `vmstorage`.
Another possible issue for 100% connection saturation between `vminsert` and `vmstorage`
@ -288,10 +288,10 @@ There are the following most commons reasons for slow data ingestion in Victoria
1. Noisy neighbor. Make sure VictoriaMetrics components run in an environments without other resource-hungry apps.
Such apps may steal RAM, CPU, disk IO and network bandwidth, which is needed for VictoriaMetrics components.
Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring)
and proper diagnosis would require checking resource usage on the instances where VictoriaMetrics runs.
1. If you see `TooHighSlowInsertsRate` [alert](https://docs.victoriametrics.com/#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough
1. If you see `TooHighSlowInsertsRate` [alert](./README.md#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough
free CPU and RAM, then increase `-cacheExpireDuration` command-line flag at single-node VictoriaMetrics or at `vmstorage` to the value,
which exceeds the interval between ingested samples for the same time series (aka `scrape_interval`).
See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183) for more details.
@ -308,7 +308,7 @@ Some queries may take more time and resources (CPU, RAM, network bandwidth) than
VictoriaMetrics logs slow queries if their execution time exceeds the duration passed
to `-search.logSlowQueryDuration` command-line flag (5s by default).
VictoriaMetrics provides [`top queries` page at VMUI](https://docs.victoriametrics.com/#top-queries), which shows
VictoriaMetrics provides [`top queries` page at VMUI](./README.md#top-queries), which shows
queries that took the most time to execute.
There are the following solutions exist for improving performance of slow queries:
@ -318,33 +318,33 @@ There are the following solutions exist for improving performance of slow querie
with more CPU and RAM should help improving speed for slow queries. Query performance
is always limited by resources of one `vmselect` which processes the query. For example, if 2vCPU cores on `vmselect`
isn't enough to process query fast enough, then migrating `vmselect` to a machine with 4vCPU cores should increase heavy query performance by up to 2x.
If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring)
If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring)
is close to the limit, then prefer adding more `vmselect` nodes to the cluster.
Sometimes adding more `vmstorage` nodes also can help improving the speed for slow queries.
- Rewriting slow queries, so they become faster. Unfortunately it is hard determining
whether the given query is slow by just looking at it.
The main source of slow queries in practice is [alerting and recording rules](https://docs.victoriametrics.com/vmalert/#rules)
The main source of slow queries in practice is [alerting and recording rules](./vmalert.md#rules)
with long lookbehind windows in square brackets. These queries are frequently used in SLI/SLO calculations such as [Sloth](https://github.com/slok/sloth).
For example, `avg_over_time(up[30d]) > 0.99` needs to read and process
all the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
for `up` [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the last 30 days
all the [raw samples](./keyConcepts.md#raw-samples)
for `up` [time series](./keyConcepts.md#time-series) over the last 30 days
each time it executes. If this query is executed frequently, then it can take significant share of CPU, disk read IO, network bandwidth and RAM.
Such queries can be optimized in the following ways:
- To reduce the lookbehind window in square brackets. For example, `avg_over_time(up[10d])` takes up to 3x less compute resources
than `avg_over_time(up[30d])` at VictoriaMetrics.
- To increase evaluation interval for alerting and recording rules, so they are executed less frequently.
For example, increasing `-evaluationInterval` command-line flag value at [vmalert](https://docs.victoriametrics.com/vmalert/)
For example, increasing `-evaluationInterval` command-line flag value at [vmalert](./vmalert.md)
from `1m` to `2m` should reduce compute resource usage at VictoriaMetrics by 2x.
Another source of slow queries is improper use of [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries).
Another source of slow queries is improper use of [subqueries](./MetricsQL.md#subqueries).
It is recommended avoiding subqueries if you don't understand clearly how they work.
It is easy to create a subquery without knowing about it.
For example, `rate(sum(some_metric))` is implicitly transformed into the following subquery
according to [implicit conversion rules for MetricsQL queries](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions):
according to [implicit conversion rules for MetricsQL queries](./MetricsQL.md#implicit-query-conversions):
```metricsql
rate(
@ -357,7 +357,7 @@ There are the following solutions exist for improving performance of slow querie
It is likely this query won't return the expected results. Instead, `sum(rate(some_metric))` must be used instead.
See [this article](https://www.robustperception.io/rate-then-sum-never-sum-then-rate/) for more details.
VictoriaMetrics provides [query tracing](https://docs.victoriametrics.com/#query-tracing) feature,
VictoriaMetrics provides [query tracing](./README.md#query-tracing) feature,
which can help determining the source of slow query.
See also [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986),
which explains how to determine and optimize slow queries.
@ -374,24 +374,24 @@ There are the following most common sources of out of memory (aka OOM) crashes i
VictoriaMetrics is optimized for running with default flag values (e.g. when they aren't set explicitly).
For example, it isn't recommended tuning cache sizes in VictoriaMetrics, since it frequently leads to OOM exceptions.
[These docs](https://docs.victoriametrics.com/#cache-tuning) refer command-line flags, which aren't
[These docs](./README.md#cache-tuning) refer command-line flags, which aren't
recommended to tune. If you see that VictoriaMetrics needs increasing some cache sizes for the current workload,
then it is better migrating to a host with more memory instead of trying to tune cache sizes manually.
1. Unexpected heavy queries. The query is considered as heavy if it needs to select and process millions of unique time series.
Such query may lead to OOM exception, since VictoriaMetrics needs to keep some of per-series data in memory.
VictoriaMetrics provides [various settings](https://docs.victoriametrics.com/#resource-usage-limits),
VictoriaMetrics provides [various settings](./README.md#resource-usage-limits),
which can help limit resource usage.
For more context, see [How to optimize PromQL and MetricsQL queries](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986).
VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing)
VictoriaMetrics also provides [query tracer](./README.md#query-tracing)
to help identify the source of heavy query.
1. Lack of free memory for processing workload spikes. If VictoriaMetrics components use almost all the available memory
under the current workload, then it is recommended migrating to a host with bigger amounts of memory.
This would protect from possible OOM crashes on workload spikes. It is recommended to have at least 50%
of free memory for graceful handling of possible workload spikes.
See [capacity planning for single-node VictoriaMetrics](https://docs.victoriametrics.com/#capacity-planning)
and [capacity planning for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning).
See [capacity planning for single-node VictoriaMetrics](./README.md#capacity-planning)
and [capacity planning for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#capacity-planning).
## Cluster instability
@ -405,7 +405,7 @@ The most common sources of cluster instability are:
the cluster has no enough free resources for processing the increased workload,
then it may become unstable.
VictoriaMetrics provides various configuration settings, which can be used for limiting unexpected workload spikes.
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) for details.
See [these docs](./Cluster-VictoriaMetrics.md#resource-usage-limits) for details.
- Various maintenance tasks such as rolling upgrades or rolling restarts during configuration changes.
For example, if a cluster contains `N=3` `vmstorage` nodes and they are restarted one-by-one (aka rolling restart),
@ -423,10 +423,10 @@ The most common sources of cluster instability are:
`N=11` `vmstorage` nodes, then the workload increase during rolling restart of `vmstorage` nodes
would be `100%/(N-1)=10%`. It is recommended to have at least 8 `vmstorage` nodes in the cluster.
The recommended number of `vmstorage` nodes should be multiplied by `-replicationFactor` if replication is enabled -
see [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety)
see [replication and data safety docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety)
for details.
- Time series sharding. Received time series [are consistently sharded](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview)
- Time series sharding. Received time series [are consistently sharded](./Cluster-VictoriaMetrics.md#architecture-overview)
by `vminsert` between configured `vmstorage` nodes. As a sharding key `vminsert` is using time series name and labels,
respecting their order. If the order of labels in time series is constantly changing, this could cause wrong sharding
calculation and result in un-even and sub-optimal time series distribution across available vmstorages. It is expected
@ -436,18 +436,18 @@ The most common sources of cluster instability are:
The obvious solution against VictoriaMetrics cluster instability is to make sure cluster components
have enough free resources for graceful processing of the increased workload.
See [capacity planning docs](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)
and [cluster resizing and scalability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability)
See [capacity planning docs](./Cluster-VictoriaMetrics.md#capacity-planning)
and [cluster resizing and scalability docs](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability)
for details.
## Too much disk space used
If too much disk space is used by a [single-node VictoriaMetrics](https://docs.victoriametrics.com/) or by `vmstorage` component
at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/), then please check the following:
If too much disk space is used by a [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md) or by `vmstorage` component
at [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md), then please check the following:
- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots)
and [snapshot troubleshooting](https://docs.victoriametrics.com/#snapshot-troubleshooting).
- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](./README.md#how-to-work-with-snapshots)
and [snapshot troubleshooting](./README.md#snapshot-troubleshooting).
- Under normal conditions the size of `<-storageDataPath>/indexdb` folder must be smaller than the size of `<-storageDataPath>/data` folder, where `-storageDataPath`
is the corresponding command-line flag value. This can be checked by the following query if [VictoriaMetrics monitoring](#monitoring) is properly set up:
@ -456,14 +456,14 @@ at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriame
/
sum(vm_data_size_bytes{type=~"(storage|indexdb)/.+"}) without(type)
```
If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issue,
If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](./FAQ.md#what-is-high-churn-rate) issue,
which results in excess disk space usage for both `indexdb` and `data` folders under `-storageDataPath` folder.
The solution is to identify and fix the source of high churn rate with [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer).
The solution is to identify and fix the source of high churn rate with [cardinality explorer](./README.md#cardinality-explorer).
## Monitoring
Having proper [monitoring](https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring)
Having proper [monitoring](./Single-Server-VictoriaMetrics.md#monitoring)
would help identify and prevent most of the issues listed above.
[Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards) contain panels reflecting the

10
docs/goals.md
View file

@ -9,15 +9,15 @@ menu:
---
## Goals
1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/enterprise/) resolving issues with VictoriaMetrics components,
1. The main goal - **to help users and [clients](./enterprise.md) resolving issues with VictoriaMetrics components,
so they could use these components in the most efficient way**.
1. Fixing bugs in the essential functionality of VictoriaMetrics components. Small usability bugs are usually the most annoying,
so they **must be fixed first**.
1. Improving [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs) for VictoriaMetrics components,
so users could find answers to their questions via Google or [Perplexity](https://www.perplexity.ai/) without the need
to ask these questions at our [support channels](https://docs.victoriametrics.com/#community-and-contributions).
to ask these questions at our [support channels](./#community-and-contributions).
1. Simplifying usage of VictoriaMetrics components without breaking backwards compatibility, so users could regularly
upgrade to [the latest available release](https://docs.victoriametrics.com/CHANGELOG.html) and remain happy.
upgrade to [the latest available release](./CHANGELOG.md) and remain happy.
1. Improving usability for the existing functionality of VictoriaMetrics components.
1. Improving the readability and maintainability of the code base by removing unnecessary abstractions and simplifying the code whenever possible.
1. Improving development velocity by optimizing and simplifying CI/CD tasks, so they take less time to execute and debug.
@ -26,7 +26,7 @@ menu:
1. Convincing people to use VictoriaMetrics components when there are better suited solutions exist for their tasks,
since users will become angry at VictoriaMetrics after they discover better solutions.
1. Breaking links to [VictoriaMetrics docs](https://docs.victoriametrics.com/), since users will be unhappy seeing 404 page
1. Breaking links to [VictoriaMetrics docs](./Single-server-VictoriaMetrics.md), since users will be unhappy seeing 404 page
or unexpected results after they click some old link somewhere on the Internet or in the internal knowledge base.
1. Breaking backwards compatibility in new releases, since users will be unhappy when their working setup breaks after the upgrade.
1. Adding non-trivial features, which require significant changes in the code and the architecture,
@ -50,7 +50,7 @@ menu:
with VictoriaMetrics components in the most efficient way. Nobody uses new shiny feature if it isn't documented properly.
- **Happy users are more important than the momentary revenue.** Happy users spread the word about VictoriaMetrics,
so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](https://docs.victoriametrics.com/enterprise/).
so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](./enterprise.md).
This increases long-term revenue.
- **Simple solution is better than smart solution.** Simple solution is easier to setup, operate, debug and troubleshoot than the smart solution.

74
docs/guides/migrate-from-influx.md
View file

@ -18,8 +18,8 @@ sometimes old known solutions just can't keep up with the new expectations.
VictoriaMetrics is a high-performance opensource time series database specifically designed to deal with huge volumes of
monitoring data while remaining cost-efficient at the same time. Many companies are choosing to migrate from InfluxDB to
VictoriaMetrics specifically for performance and scalability reasons. Along them see case studies provided by
[ARNES](https://docs.victoriametrics.com/casestudies/#arnes)
and [Brandwatch](https://docs.victoriametrics.com/casestudies/#brandwatch).
[ARNES](./CaseStudies.md#arnes)
and [Brandwatch](./CaseStudies.md#brandwatch).
This guide will cover the differences between two solutions, most commonly asked questions, and approaches for migrating
from InfluxDB to VictoriaMetrics.
@ -28,13 +28,13 @@ from InfluxDB to VictoriaMetrics.
While readers are likely familiar
with [InfluxDB key concepts](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/), the data model of
VictoriaMetrics is something [new to explore](https://docs.victoriametrics.com/keyconcepts/#data-model). Let's start
VictoriaMetrics is something [new to explore](./keyConcepts.md#data-model). Let's start
with similarities and differences:
* both solutions are **schemaless**, which means there is no need to define metrics or their tags in advance;
* multidimensional data support is implemented
via [tags](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#tags)
in InfluxDB and via [labels](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric) in
in InfluxDB and via [labels](./keyConcepts.md#structure-of-a-metric) in
VictoriaMetrics. However, labels in VictoriaMetrics are always `strings`, while InfluxDB supports multiple data types;
* timestamps are stored with nanosecond resolution in InfluxDB, while in VictoriaMetrics it is **milliseconds**;
* in VictoriaMetrics metric value is always `float64`, while InfluxDB supports multiple data types.
@ -47,8 +47,8 @@ with similarities and differences:
[buckets](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#bucket)
or [organizations](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#organization). All
data in VictoriaMetrics is stored in a global namespace or within
a [tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
See more about multi-tenancy [here](https://docs.victoriametrics.com/keyconcepts/#multi-tenancy).
a [tenant](./Cluster-VictoriaMetrics.md#multitenancy).
See more about multi-tenancy [here](./keyConcepts.md#multi-tenancy).
Let's consider the
following [sample data](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#sample-data)
@ -78,7 +78,7 @@ VictoriaMetrics, so lookups by names or labels have the same query speed.
## Write data
VictoriaMetrics
supports [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
supports [InfluxDB line protocol](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
for data ingestion. For example, to write a measurement to VictoriaMetrics we need to send an HTTP POST request with
payload in a line protocol format:
@ -116,7 +116,7 @@ The expected response is the following:
```
Please note, VictoriaMetrics performed additional
[data mapping](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
[data mapping](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
to the data ingested via InfluxDB line protocol.
Support of InfluxDB line protocol also means VictoriaMetrics is compatible with
@ -129,20 +129,20 @@ add `http://<victoriametric-addr>:8428` URL to Telegraf configs:
```
In addition to InfluxDB line protocol, VictoriaMetrics supports many other ways for
[metrics collection](https://docs.victoriametrics.com/keyconcepts/#write-data).
[metrics collection](./keyConcepts.md#write-data).
## Query data
VictoriaMetrics does not have a command-line interface (CLI). Instead, it provides
an [HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
an [HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage)
for serving read queries. This API is used in various integrations such as
[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup). The same API is also used
by [VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for
[Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup). The same API is also used
by [VMUI](./Single-Server-VictoriaMetrics.md#vmui) - a graphical User Interface for
querying and visualizing metrics:
![Migrate from Influx](migrate-from-influx_vmui.webp)
See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyconcepts/#query-data).
See more about [how to query data in VictoriaMetrics](./keyConcepts.md#query-data).
### Basic concepts
@ -186,19 +186,19 @@ Having this, let's import the same data sample in VictoriaMetrics and plot it in
InfluxQL query might be translated to MetricsQL let's break it into components first:
* `SELECT last("bar") FROM "foo"` - all requests
to [instant](https://docs.victoriametrics.com/keyconcepts/#instant-query)
or [range](https://docs.victoriametrics.com/keyconcepts/#range-query) VictoriaMetrics APIs are reads, so no need
to [instant](./keyConcepts.md#instant-query)
or [range](./keyConcepts.md#range-query) VictoriaMetrics APIs are reads, so no need
to specify the `SELECT` statement. There are no `measurements` or `fields` in VictoriaMetrics, so the whole expression
can be replaced with `foo_bar` in MetricsQL;
* `WHERE ("instance" = 'localhost')`- [filtering by labels](https://docs.victoriametrics.com/keyconcepts/#filtering)
* `WHERE ("instance" = 'localhost')`- [filtering by labels](./keyConcepts.md#filtering)
in MetricsQL requires specifying the filter in curly braces next to the metric name. So in MetricsQL filter expression
will be translated to `{instance="localhost"}`;
* `WHERE $timeFilter` - filtering by time is done via request params sent along with query, so in MetricsQL no need to
specify this filter;
* `GROUP BY time(1m)` - grouping by time is done by default
in [range](https://docs.victoriametrics.com/keyconcepts/#range-query) API according to specified `step` param.
in [range](./keyConcepts.md#range-query) API according to specified `step` param.
This param is also a part of params sent along with request. See how to perform additional
[aggregations and grouping via MetricsQL](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions)
[aggregations and grouping via MetricsQL](./keyConcepts.md#aggregation-and-grouping-functions)
.
In result, executing the `foo_bar{instance="localhost"}` MetricsQL expression with `step=1m` for the same set of data in
@ -208,13 +208,13 @@ Grafana will have the following form:
Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points
filling the gaps in the graph. This behavior is described in more
detail [here](https://docs.victoriametrics.com/keyconcepts/#range-query). In InfluxDB, we can achieve a similar
detail [here](./keyConcepts.md#range-query). In InfluxDB, we can achieve a similar
behavior by adding `fill(previous)` to the query.
VictoriaMetrics fills the gaps on the graph assuming time series are always continuous and not discrete.
To limit the interval on which VictoriaMetrics will try to fill the gaps, set `-search.setLookbackToStep`
command-line flag. This limits the gap filling to a single `step` interval passed to
[/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query).
[/api/v1/query_range](./keyConcepts.md#range-query).
This behavior is close to InfluxDB data model.
@ -227,56 +227,56 @@ about 230 PromQL queries in it! But a closer look at those queries shows the fol
* ~120 queries are just selecting a metric with label filters,
e.g. `node_textfile_scrape_error{instance="$node",job="$job"}`;
* ~80 queries are using [rate](https://docs.victoriametrics.com/metricsql/#rate) function for selected metric,
* ~80 queries are using [rate](./MetricsQL.md#rate) function for selected metric,
e.g. `rate(node_netstat_Tcp_InSegs{instance=\"$node\",job=\"$job\"})`
* and the rest
are [aggregation functions](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions)
like [sum](https://docs.victoriametrics.com/metricsql/#sum)
or [count](https://docs.victoriametrics.com/metricsql/#count).
are [aggregation functions](./keyConcepts.md#aggregation-and-grouping-functions)
like [sum](./MetricsQL.md#sum)
or [count](./MetricsQL.md#count).
To get a better understanding of how MetricsQL works, see the following resources:
* [MetricsQL concepts](https://docs.victoriametrics.com/keyconcepts/#metricsql);
* [MetricsQL functions](https://docs.victoriametrics.com/metricsql/);
* [MetricsQL concepts](./keyConcepts.md#metricsql);
* [MetricsQL functions](./MetricsQL.md);
* [PromQL tutorial for beginners](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085).
## How to migrate current data from InfluxDB to VictoriaMetrics
Migrating data from other TSDBs to VictoriaMetrics is as simple as importing data via any of
[supported formats](https://docs.victoriametrics.com/keyconcepts/#push-model).
[supported formats](./keyConcepts.md#push-model).
But migration from InfluxDB might get easier when using [vmctl](https://docs.victoriametrics.com/vmctl/) -
But migration from InfluxDB might get easier when using [vmctl](./vmctl.md) -
VictoriaMetrics command-line tool. See more about
migrating [from InfluxDB v1.x versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x).
migrating [from InfluxDB v1.x versions](./vmctl.md#migrating-data-from-influxdb-1x).
Migrating data from InfluxDB v2.x is not supported yet. But there is
useful [3rd party solution](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-2x) for this.
useful [3rd party solution](./vmctl.md#migrating-data-from-influxdb-2x) for this.
Please note, that data migration is a backfilling process. So, please
consider [backfilling tips](https://docs.victoriametrics.com/single-server-victoriametrics/#backfilling).
consider [backfilling tips](./Single-Server-VictoriaMetrics.md#backfilling).
## Frequently asked questions
* How does VictoriaMetrics compare to InfluxDB?
* _[Answer](https://docs.victoriametrics.com/faq/#how-does-victoriametrics-compare-to-influxdb)_
* _[Answer](./FAQ.md#how-does-victoriametrics-compare-to-influxdb)_
* Why don't VictoriaMetrics support Remote Read API, so I don't need to learn MetricsQL?
* _[Answer](https://docs.victoriametrics.com/faq/#why-doesnt-victoriametrics-support-the-prometheus-remote-read-api)_
* _[Answer](./FAQ.md#why-doesnt-victoriametrics-support-the-prometheus-remote-read-api)_
* The PromQL and MetricsQL are often mentioned together - why is that?
* _MetricsQL - query language inspired by PromQL. MetricsQL is backward-compatible with PromQL, so Grafana
dashboards backed by Prometheus datasource should work the same after switching from Prometheus to
VictoriaMetrics. Both languages mostly share the same concepts with slight differences._
* Query returns more data points than expected - why?
* _VictoriaMetrics may return non-existing data points if `step` param is lower than the actual data resolution. See
more about this [here](https://docs.victoriametrics.com/keyconcepts/#range-query)._
more about this [here](./keyConcepts.md#range-query)._
* How do I get the `real` last data point, not `ephemeral`?
* _[last_over_time](https://docs.victoriametrics.com/metricsql/#last_over_time) function can be used for
* _[last_over_time](./MetricsQL.md#last_over_time) function can be used for
limiting the lookbehind window for calculated data. For example, `last_over_time(metric[10s])` would return
calculated samples only if the real samples are located closer than 10 seconds to the calculated timestamps
according to
`start`, `end` and `step` query args passed
to [range query](https://docs.victoriametrics.com/keyconcepts/#range-query)._
to [range query](./keyConcepts.md#range-query)._
* How do I get raw data points with MetricsQL?
* _For getting raw data points specify the interval at which you want them in square brackets and send
as [instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query). For
as [instant query](./keyConcepts.md#instant-query). For
example, `GET api/v1/query?query=my_metric[5m]&time=<time>` will return raw samples for `my_metric` in interval
from `<time>` to `<time>-5m`._
* Can you have multiple aggregators in a MetricsQL query, e.g. `SELECT MAX(field), MIN(field) ...`?

126
docs/keyConcepts.md
View file

@ -54,10 +54,10 @@ requests_total{path="/", code="200"}
```
Labels can be automatically attached to the [time series](#time-series)
written via [vmagent](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics)
or [Prometheus](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-setup).
VictoriaMetrics supports enforcing of label filters for [query API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-enhancements)
to emulate data isolation. However, the real data isolation can be achieved via [multi-tenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy).
written via [vmagent](./vmagent.md#adding-labels-to-metrics)
or [Prometheus](./Single-Server-VictoriaMetrics.md#prometheus-setup).
VictoriaMetrics supports enforcing of label filters for [query API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-enhancements)
to emulate data isolation. However, the real data isolation can be achieved via [multi-tenancy](./Cluster-VictoriaMetrics.md#multitenancy).
#### Time series
@ -66,14 +66,14 @@ A combination of a metric name and its labels defines a `time series`. For examp
are two different time series because they have different values for `code` label.
The number of unique time series has an impact on database resource usage.
See [what is an active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) and
[what is high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) docs for details.
See [what is an active time series](./FAQ.md#what-is-an-active-time-series) and
[what is high churn rate](./FAQ.md#what-is-high-churn-rate) docs for details.
#### Cardinality
The number of unique [time series](#time-series) is named `cardinality`. Too big number of unique time series is named `high cardinality`.
High cardinality may result in increased resource usage at VictoriaMetrics.
See [these docs](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) for more details.
See [these docs](./FAQ.md#what-is-high-cardinality) for more details.
#### Raw samples
@ -99,8 +99,8 @@ requests_total{path="/", code="200"} 123 4567890
#### Time series resolution
Resolution is the minimum interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
of the [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). Consider the following example:
Resolution is the minimum interval between [raw samples](./keyConcepts.md#raw-samples)
of the [time series](./keyConcepts.md#time-series). Consider the following example:
```
----------------------------------------------------------------------
| <time series> | <value> | <timestamp> |
@ -113,9 +113,9 @@ of the [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
Here we have a time series `requests_total{path="/health", code="200"}` which has a value update each `30s`.
This means, its resolution is also a `30s`.
> In terms of [pull model](https://docs.victoriametrics.com/keyconcepts/#pull-model), resolution is equal
> In terms of [pull model](./keyConcepts.md#pull-model), resolution is equal
> to `scrape_interval` and is controlled by the monitoring system (server).
> For [push model](https://docs.victoriametrics.com/keyconcepts/#push-model), resolution is an interval between
> For [push model](./keyConcepts.md#push-model), resolution is an interval between
> samples timestamps and is controlled by a client (metrics collector).
Try to keep time series resolution consistent, since some [MetricsQL](#metricsql) functions may expect it to be so.
@ -144,9 +144,9 @@ was rapidly changing from 1:38 pm to 1:39 pm, then there were no changes until 1
Counter is used for measuring the number of events, like the number of requests, errors, logs, messages, etc.
The most common [MetricsQL](#metricsql) functions used with counters are:
* [rate](https://docs.victoriametrics.com/metricsql/#rate) - calculates the average per-second speed of metric change.
* [rate](./MetricsQL.md#rate) - calculates the average per-second speed of metric change.
For example, `rate(requests_total)` shows how many requests are served per second on average;
* [increase](https://docs.victoriametrics.com/metricsql/#increase) - calculates the growth of a metric on the given
* [increase](./MetricsQL.md#increase) - calculates the growth of a metric on the given
time period specified in square brackets.
For example, `increase(requests_total[1h])` shows the number of requests served over the last hour.
@ -176,7 +176,7 @@ Gauge is used in the following scenarios:
can store the timestamp of the last successful configuration reload.
The most common [MetricsQL](#metricsql) functions used with gauges are [aggregation functions](#aggregation-and-grouping-functions)
and [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions).
and [rollup functions](./MetricsQL.md#rollup-functions).
#### Histogram
@ -203,7 +203,7 @@ that there were 2 queries with the number of rows in the range `(408.4 - 464.2]`
since the last VictoriaMetrics start.
The counters ending with `_bucket` suffix allow estimating arbitrary percentile
for the observed measurement with the help of [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile)
for the observed measurement with the help of [histogram_quantile](./MetricsQL.md#histogram_quantile)
function. For example, the following query returns the estimated 99th percentile
on the number of rows read per each query during the last hour (see `1h` in square brackets):
@ -261,7 +261,7 @@ and calculating [quantiles](https://prometheus.io/docs/practices/histograms/#qua
![histogram](keyConcepts_histogram.webp)
Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/metricsql/#prometheus_buckets)
Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](./MetricsQL.md#prometheus_buckets)
function must be used for converting buckets with `vmrange` labels to buckets with `le` labels before building heatmaps in Grafana.
Histograms are usually used for measuring the distribution of latency, sizes of elements (batch size, for example) etc. There are two
@ -354,13 +354,13 @@ avoid excessive resource usage and database slowdown.
### Multi-tenancy
[Cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/) of VictoriaMetrics
supports [multi-tenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
[Cluster version](./Cluster-VictoriaMetrics.md) of VictoriaMetrics
supports [multi-tenancy](./Cluster-VictoriaMetrics.md#multitenancy)
for data isolation.
Multi-tenancy can be emulated for [single-server](https://docs.victoriametrics.com/single-server-victoriametrics/)
Multi-tenancy can be emulated for [single-server](./Single-Server-VictoriaMetrics.md)
version of VictoriaMetrics by adding [labels](#labels) on [write path](#write-data)
and enforcing [labels filtering](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-enhancements)
and enforcing [labels filtering](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-enhancements)
on [read path](#query-data).
@ -375,13 +375,13 @@ Client regularly sends the collected metrics to the server in the push model:
![push model](keyConcepts_push_model.webp)
The client (application) decides when and where to send its metrics. VictoriaMetrics supports many protocols
for data ingestion (aka `push protocols`) - see [the full list here](https://docs.victoriametrics.com/#how-to-import-time-series-data).
for data ingestion (aka `push protocols`) - see [the full list here](./#how-to-import-time-series-data).
All the protocols are fully compatible with VictoriaMetrics [data model](#data-model) and can be used in production.
We recommend using the [github.com/VictoriaMetrics/metrics](https://github.com/VictoriaMetrics/metrics) package
for pushing application metrics to VictoriaMetrics.
It is also possible to use already existing clients compatible with the protocols listed above
like [Telegraf](https://github.com/influxdata/telegraf)
for [InfluxDB line protocol](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
for [InfluxDB line protocol](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf).
Creating custom clients or instrumenting the application for metrics writing is as easy as sending a POST request:
@ -389,14 +389,14 @@ Creating custom clients or instrumenting the application for metrics writing is
curl -d '{"metric":{"__name__":"foo","job":"node_exporter"},"values":[0,1,2],"timestamps":[1549891472010,1549891487724,1549891503438]}' -X POST 'http://localhost:8428/api/v1/import'
```
It is allowed to push/write metrics to [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/),
to [cluster component vminsert](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview)
and to [vmagent](https://docs.victoriametrics.com/vmagent/).
It is allowed to push/write metrics to [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md),
to [cluster component vminsert](./Cluster-VictoriaMetrics.md#architecture-overview)
and to [vmagent](./vmagent.md).
The pros of push model:
* Simpler configuration at VictoriaMetrics side - there is no need to configure VictoriaMetrics with locations of the monitored applications.
There is no need in complex [service discovery schemes](https://docs.victoriametrics.com/sd_configs/).
There is no need in complex [service discovery schemes](./sd_configs.md).
* Simpler security setup - there is no need to set up access from VictoriaMetrics to each monitored application.
See [Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics](https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/)
@ -423,10 +423,10 @@ In pull model, the monitoring system needs to be aware of all the applications i
scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
VictoriaMetrics supports discovering Prometheus-compatible targets and scraping metrics from them in the same way as Prometheus does -
see [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
see [these docs](./#how-to-scrape-prometheus-exporters-such-as-node-exporter).
Metrics scraping is supported by [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
and by [vmagent](https://docs.victoriametrics.com/vmagent/).
Metrics scraping is supported by [single-node VictoriaMetrics](./#how-to-scrape-prometheus-exporters-such-as-node-exporter)
and by [vmagent](./vmagent.md).
The pros of the pull model:
@ -439,7 +439,7 @@ The pros of the pull model:
The cons of the pull model:
* Harder security setup - monitoring system needs to have access to applications it monitors.
* Pull model needs non-trivial [service discovery schemes](https://docs.victoriametrics.com/sd_configs/).
* Pull model needs non-trivial [service discovery schemes](./sd_configs.md).
### Common approaches for data collection
@ -450,7 +450,7 @@ The most common approach for data collection is using both models:
![data collection](keyConcepts_data_collection.webp)
In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent/). Vmagent is
In this approach the additional component is used - [vmagent](./vmagent.md). Vmagent is
a lightweight agent whose main purpose is to collect, filter, relabel and deliver metrics to VictoriaMetrics.
It supports all [push](#push-model) and [pull](#pull-model) protocols mentioned above.
@ -465,17 +465,17 @@ VictoriaMetrics components allow building more advanced topologies. For example,
![two dcs](keyConcepts_two_dcs.webp)
VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). Vmagent also allows
[replicating the same data to multiple destinations](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability).
VictoriaMetrics in this example may be either [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
or [VictoriaMetrics Cluster](./Cluster-VictoriaMetrics.md). Vmagent also allows
[replicating the same data to multiple destinations](./vmagent.md#replication-and-high-availability).
## Query data
VictoriaMetrics provides
an [HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
an [HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage)
for serving read queries. The API is used in various integrations such as
[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup). The same API is also used by
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for querying
[Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup). The same API is also used by
[VMUI](./Single-Server-VictoriaMetrics.md#vmui) - a graphical User Interface for querying
and visualizing metrics.
The API consists of two main handlers for serving [instant queries](#instant-query) and [range queries](#range-query).
@ -490,10 +490,10 @@ GET | POST /api/v1/query?query=...&time=...&step=...&timeout=...
Params:
* `query` - [MetricsQL](https://docs.victoriametrics.com/metricsql/) expression.
* `time` - optional, [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
* `query` - [MetricsQL](./MetricsQL.md) expression.
* `time` - optional, [timestamp](./Single-Server-VictoriaMetrics.md#timestamp-formats)
in second precision to evaluate the `query` at. If omitted, `time` is set to `now()` (current timestamp).
The `time` param can be specified in [multiple allowed formats](https://docs.victoriametrics.com/#timestamp-formats).
The `time` param can be specified in [multiple allowed formats](./#timestamp-formats).
* `step` - optional [interval](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-durations)
for searching for raw samples in the past when executing the `query` (used when a sample is missing at the specified `time`).
For example, the request `/api/v1/query?query=up&step=1m` looks for the last written raw sample for the metric `up`
@ -502,7 +502,7 @@ Params:
By default the timeout is set to the value of `-search.maxQueryDuration` command-line flag passed to single-node VictoriaMetrics
or to `vmselect` component of VictoriaMetrics cluster.
The result of Instant query is a list of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
The result of Instant query is a list of [time series](./keyConcepts.md#time-series)
matching the filter in `query` expression. Each returned series contains exactly one `(timestamp, value)` entry,
where `timestamp` equals to the `time` query arg, while the `value` contains `query` result at the requested `time`.
@ -572,7 +572,7 @@ Instant queries can return multiple time series, but always only one data sample
the following scenarios:
* Getting the last recorded value;
* For [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) such as `count_over_time`;
* For [rollup functions](./MetricsQL.md#rollup-functions) such as `count_over_time`;
* For alerts and recording rules evaluation;
* Plotting Stat or Table panels in Grafana.
@ -585,10 +585,10 @@ GET | POST /api/v1/query_range?query=...&start=...&end=...&step=...&timeout=...
```
Params:
* `query` - [MetricsQL](https://docs.victoriametrics.com/metricsql/) expression.
* `start` - the starting [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
* `query` - [MetricsQL](./MetricsQL.md) expression.
* `start` - the starting [timestamp](./Single-Server-VictoriaMetrics.md#timestamp-formats)
of the time range for `query` evaluation.
* `end` - the ending [timestamp](https://docs.victoriametrics.com/single-server-victoriametrics/#timestamp-formats)
* `end` - the ending [timestamp](./Single-Server-VictoriaMetrics.md#timestamp-formats)
of the time range for `query` evaluation.
If the `end` isn't set, then the `end` is automatically set to the current time.
* `step` - the [interval](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-durations)
@ -599,7 +599,7 @@ Params:
By default the timeout is set to the value of `-search.maxQueryDuration` command-line flag passed to single-node VictoriaMetrics
or to `vmselect` component in VictoriaMetrics cluster.
The result of Range query is a list of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
The result of Range query is a list of [time series](./keyConcepts.md#time-series)
matching the filter in `query` expression. Each returned series contains `(timestamp, value)` results for the `query` executed
at `start`, `start+step`, `start+2*step`, ..., `end` timestamps. In other words, Range query is an [Instant query](#instant-query)
executed independently at `start`, `start+step`, ..., `end` timestamps.
@ -733,7 +733,7 @@ useful in the following scenarios:
* Correlate changes between multiple metrics on the time interval;
* Observe trends and dynamics of the metric change.
If you need to export raw samples from VictoriaMetrics, then take a look at [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series).
If you need to export raw samples from VictoriaMetrics, then take a look at [export APIs](./#how-to-export-time-series).
### Query latency
@ -760,14 +760,14 @@ VictoriaMetrics buffers recently ingested samples in memory for up to a few seco
This bufferring improves data ingestion performance. The buffered samples are invisible in query results, even if `-search.latencyOffset` command-line flag is set to 0,
or if `latency_offset` query arg is set to 0.
You can send GET request to `/internal/force_flush` http handler at single-node VictoriaMetrics
or to `vmstorage` at [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)
or to `vmstorage` at [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md)
in order to forcibly flush the buffered samples to disk, so they become visible for querying. The `/internal/force_flush` handler
is provided for debugging and testing purposes only. Do not call it in production, since this may significantly slow down data ingestion
performance and increase resource usage.
### MetricsQL
VictoriaMetrics provide a special query language for executing read queries - [MetricsQL](https://docs.victoriametrics.com/metricsql/).
VictoriaMetrics provide a special query language for executing read queries - [MetricsQL](./MetricsQL.md).
It is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics)-like query language with a powerful set of
functions and features for working specifically with time series data. MetricsQL is backward-compatible with PromQL,
so it shares most of the query concepts. The basic concepts for PromQL and MetricsQL are
@ -825,7 +825,7 @@ The query above returns series for two metrics: `requests_error_total` and `requ
#### Filtering by multiple "or" filters
[MetricsQL](https://docs.victoriametrics.com/metricsql/) supports selecting time series, which match at least one of multiple "or" filters.
[MetricsQL](./MetricsQL.md) supports selecting time series, which match at least one of multiple "or" filters.
Such filters must be delimited by `or` inside curly braces. For example, the following query selects time series with
`{job="app1",env="prod"}` or `{job="app2",env="dev"}` labels:
@ -836,9 +836,9 @@ Such filters must be delimited by `or` inside curly braces. For example, the fol
The number of `or` groups can be arbitrary. The number of `,`-delimited label filters per each `or` group can be arbitrary.
Per-group filters are applied with `and` operation, e.g. they select series simultaneously matching all the filters in the group.
This functionality allows passing the selected series to [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions)
such as [rate()](https://docs.victoriametrics.com/metricsql/#rate)
without the need to use [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries):
This functionality allows passing the selected series to [rollup functions](./MetricsQL.md#rollup-functions)
such as [rate()](./MetricsQL.md#rate)
without the need to use [subqueries](./MetricsQL.md#subqueries):
```metricsql
rate({job="app1",env="prod" or job="app2",env="dev"}[5m])
@ -909,12 +909,12 @@ summary memory usage for each `job`:
sum(process_resident_memory_bytes) by (job)
```
See [docs for aggregate functions in MetricsQL](https://docs.victoriametrics.com/metricsql/#aggregate-functions).
See [docs for aggregate functions in MetricsQL](./MetricsQL.md#aggregate-functions).
#### Calculating rates
One of the most widely used functions for [counters](#counter)
is [rate](https://docs.victoriametrics.com/metricsql/#rate). It calculates the average per-second increase rate individually
is [rate](./MetricsQL.md#rate). It calculates the average per-second increase rate individually
per each matching time series. For example, the following query shows the average per-second data receive speed
per each monitored `node_exporter` instance, which exposes the `node_network_receive_bytes_total` metric:
@ -935,7 +935,7 @@ In this case VictoriaMetrics uses the specified lookbehind window - `5m` (5 minu
Bigger lookbehind windows usually lead to smoother graphs.
`rate` strips metric name while leaving all the labels for the inner time series. If you need to keep the metric name,
then add [keep_metric_names](https://docs.victoriametrics.com/metricsql/#keep_metric_names) modifier
then add [keep_metric_names](./MetricsQL.md#keep_metric_names) modifier
after the `rate(..)`. For example, the following query leaves metric names after calculating the `rate()`:
```metricsql
@ -947,13 +947,13 @@ rate(node_network_receive_bytes_total) keep_metric_names
### Visualizing time series
VictoriaMetrics has a built-in graphical User Interface for querying and visualizing metrics -
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui).
[VMUI](./Single-Server-VictoriaMetrics.md#vmui).
Open `http://victoriametrics:8428/vmui` page, type the query and see the results:
![vmui](keyConcepts_vmui.webp)
VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup)
VictoriaMetrics supports [Prometheus HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage)
which makes it possible to [query it with Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup)
in the same way as Grafana queries Prometheus.
## Modify data
@ -965,20 +965,20 @@ it is stored. Due to this limitation, VictoriaMetrics does not support direct da
### Deletion
See [How to delete time series](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series)
See [How to delete time series](./Single-Server-VictoriaMetrics.md#how-to-delete-time-series)
.
### Relabeling
Relabeling is a powerful mechanism for modifying time series before they have been written to the database. Relabeling
may be applied for both [push](#push-model) and [pull](#pull-model) models. See more
details [here](https://docs.victoriametrics.com/single-server-victoriametrics/#relabeling).
details [here](./Single-Server-VictoriaMetrics.md#relabeling).
### Deduplication
VictoriaMetrics supports data deduplication. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication).
VictoriaMetrics supports data deduplication. See [these docs](./Single-Server-VictoriaMetrics.md#deduplication).
### Downsampling
VictoriaMetrics supports data downsampling. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#downsampling).
VictoriaMetrics supports data downsampling. See [these docs](./Single-Server-VictoriaMetrics.md#downsampling).

74
docs/relabeling.md
View file

@ -9,13 +9,13 @@ menu:
aliases:
- /relabeling.html
---
VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) support
[Prometheus-compatible relabeling](https://docs.victoriametrics.com/vmagent/#relabeling)
with [additional enhancements](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements).
VictoriaMetrics and [vmagent](./vmagent.md) support
[Prometheus-compatible relabeling](./vmagent.md#relabeling)
with [additional enhancements](./vmagent.md#relabeling-enhancements).
The relabeling is mostly used for the following tasks:
* Dropping unneeded scrape targets during [service discovery](https://docs.victoriametrics.com/sd_configs/#prometheus-service-discovery).
* Dropping unneeded scrape targets during [service discovery](./sd_configs.md#prometheus-service-discovery).
See [how to drop discovered targets](#how-to-drop-discovered-targets).
* Adding or updating static labels at scrape targets. See [how to add labels to scrape targets](#how-to-add-labels-to-scrape-targets).
* Copying target labels from another labels. See [how to copy labels in scrape targets](#how-to-copy-labels-in-scrape-targets).
@ -32,7 +32,7 @@ The relabeling is mostly used for the following tasks:
* Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
See also [relabeling docs at vmagent](https://docs.victoriametrics.com/vmagent/#relabeling).
See also [relabeling docs at vmagent](./vmagent.md#relabeling).
## How to remove labels from metrics subset
@ -62,7 +62,7 @@ See also:
## How to rename scraped metrics
Metric name is a regular label with special name - `__name__` (see [these docs](https://docs.victoriametrics.com/keyconcepts/#labels)).
Metric name is a regular label with special name - `__name__` (see [these docs](./keyConcepts.md#labels)).
So renaming of metric name is performed in the same way as changing label value.
Let's look at a few examples.
@ -114,7 +114,7 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
## How to add labels to scraped metrics
The following config sets `foo="bar"` [label](https://docs.victoriametrics.com/keyconcepts/#labels) across all the scraped metrics:
The following config sets `foo="bar"` [label](./keyConcepts.md#labels) across all the scraped metrics:
```yaml
scrape_configs:
@ -126,7 +126,7 @@ scrape_configs:
replacement: bar
```
The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](./keyConcepts.md#filtering):
```yaml
scrape_configs:
@ -158,7 +158,7 @@ scrape_configs:
```
The following config adds `foo_` prefix to `job` label values only for metrics
matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
matching `{job=~"my-app-.*",env!="dev"}` [series selector](./keyConcepts.md#filtering):
```yaml
scrape_configs:
@ -178,8 +178,8 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
## How to remove labels from scraped metrics
Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels
lead to [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)
or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues,
lead to [high cardinality](./FAQ.md#what-is-high-cardinality)
or [high churn rate](./FAQ.md#what-is-high-churn-rate) issues,
then it may be a good idea to drop these labels during scrapes.
This can be done with `action: labeldrop` or `action: labelkeep` relabeling rules at `metric_relabel_configs` section:
@ -215,8 +215,8 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
## How to drop metrics during scrape
Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result
in [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)
or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues,
in [high cardinality](./FAQ.md#what-is-high-cardinality)
or [high churn rate](./FAQ.md#what-is-high-churn-rate) issues,
then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep`
relabeling rules at `metric_relabel_configs` section:
@ -245,12 +245,12 @@ See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
## How to remove labels from a subset of targets
Sometimes it is needed to remove some labels from a subset of [discovered targets](https://docs.victoriametrics.com/sd_configs/),
Sometimes it is needed to remove some labels from a subset of [discovered targets](./sd_configs.md),
while leaving these labels in the rest of discovered targets.
In this case the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
can be added to `action: labeldrop` or `action: labelkeep` relabeling rule.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
For example, the following config discovers pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names for targets matching `{__address__=~"pod123.+"}` selector:
@ -273,13 +273,13 @@ See also [how to remove labels from targets](#how-to-remove-labels-from-targets)
## How to remove labels from targets
Sometimes it is needed to remove some labels from [discovered targets](https://docs.victoriametrics.com/sd_configs/).
Sometimes it is needed to remove some labels from [discovered targets](./sd_configs.md).
In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used:
* `action: labeldrop` drops all the labels with names matching the `regex` option
* `action: labelkeep` drops all the labels with names not matching the `regex` option
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
For example, the following config discovers pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs),
[extracts pod-level labels](#how-to-remove-prefixes-from-target-label-names) into labels with `foo_` prefix and then drops all the labels
with `foo_bar_` prefix in their names:
@ -310,8 +310,8 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
## How to remove prefixes from target label names
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](https://docs.victoriametrics.com/sd_configs/).
For example, [Kubernetes service discovery](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
Sometimes it is needed to remove `__meta_*` prefixes from meta-labels of the [discovered targets](./sd_configs.md).
For example, [Kubernetes service discovery](./sd_configs.md#kubernetes_sd_configs) adds `__meta_kubernetes_pod_label_<labelname>`
labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names,
while removing the `__meta_kubernetes_pod_label_` prefix. This can be done with `action: labelmap` relabeling option:
@ -341,7 +341,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
Relabeling allows extracting parts from label values and storing them into arbitrary labels.
This is performed with `regex` and `replacement` options in relabeling rules.
For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs),
For example, the following config discovers pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs),
extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix:
```yaml
@ -372,16 +372,16 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
## How to modify instance and job
Single-node VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) automatically add `instance` and `job` labels per each discovered target:
Single-node VictoriaMetrics and [vmagent](./vmagent.md) automatically add `instance` and `job` labels per each discovered target:
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs).
* The `job` label is set to `job_name` value specified in the corresponding [scrape_config](./sd_configs.md#scrape_configs).
* The `instance` label is set to the `host:port` part of `__address__` label value after target-level relabeling.
The `__address__` label value is automatically set to the most suitable value depending
on the used [service discovery type](https://docs.victoriametrics.com/sd_configs/#supported-service-discovery-configs).
on the used [service discovery type](./sd_configs.md#supported-service-discovery-configs).
The `__address__` label can be overridden during relabeling - see [these docs](#how-to-modify-scrape-urls-in-targets).
Both `instance` and `job` labels can be overridden during relabeling. For example, the following config discovers pod targets
in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
in [Kubernetes](./sd_configs.md#kubernetes_sd_configs) and overrides `job` label from `k8s` to `foo`:
```yaml
scrape_configs:
@ -402,10 +402,10 @@ URLs for scrape targets are composed of the following parts:
* Scheme (e.g. `http` or `https`). The scheme is available during target relabeling in a special label - `__scheme__`.
By default, the scheme is set to `http`. It can be overridden either by specifying the `scheme` option
at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs) level
at [scrape_config](./sd_configs.md#scrape_configs) level
or by updating the `__scheme__` label during relabeling.
* Host and port (e.g. `host12:3456`). This information is available during target relabeling in a special label - `__address__`.
Every [supported service discovery type](https://docs.victoriametrics.com/sd_configs/#supported-service-discovery-configs)
Every [supported service discovery type](./sd_configs.md#supported-service-discovery-configs)
sets the `__address__` label to the most suitable value. Sometimes this value needs to be modified. In this case
just update the `__address__` label during relabeling to the needed value.
The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending
@ -416,12 +416,12 @@ URLs for scrape targets are composed of the following parts:
In this case the `__scheme__` and `__metrics_path__` labels are ignored.
* URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `__metrics_path__`.
By default, the `__metrics_path__` is set to `/metrics`. It can be overridden either by specifying the `metrics_path`
option at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
option at [scrape_config](./sd_configs.md#scrape_configs)
or by updating the `__metrics_path__` label during relabeling.
* Query args (e.g. `?foo=bar&baz=xyz`). This information is available during target relabeling in special labels
with `__param_` prefix. For example, `__param_foo` would have the `bar` value, while `__param_baz` would have the `xyz` value
for `?foo=bar&baz=xyz` query string. The query args can be specified either via `params` section
at [scrape_config](https://docs.victoriametrics.com/sd_configs/#scrape_configs)
at [scrape_config](./sd_configs.md#scrape_configs)
or by updating/setting the corresponding `__param_*` labels during relabeling.
The resulting scrape url looks like the following:
@ -435,7 +435,7 @@ in [Prometheus text exposition format](https://github.com/prometheus/docs/blob/m
at the resulting scrape url.
Given the scrape url construction rules above, the following config discovers pod targets
in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
in [Kubernetes](./sd_configs.md#kubernetes_sd_configs)
and constructs per-target scrape url as `https://<pod_name>/foo/bar?baz=<container_name>`:
```yaml
@ -461,7 +461,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
Labels can be copied by specifying the source labels via `source_labels` relabeling option
and specifying the target label via `target_label` relabeling option.
For example, the following config copies `__meta_kubernetes_pod_name` label to `pod` label
for all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
for all the discovered pods in [Kubernetes](./sd_configs.md#kubernetes_sd_configs):
```yaml
scrape_configs:
@ -477,7 +477,7 @@ Note that the `source_labels` option accepts a list of labels in square brackets
in the `source_labels` list, then the specified label values are joined into a single string with `;` delimiter by default.
The delimiter can be modified by specifying it via `separator` option.
For example, the following config sets the `pod_name:container_port` value to the `host_port` label
for all the discovered pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
for all the discovered pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs):
```yaml
scrape_configs:
@ -499,7 +499,7 @@ Additional labels can be added to scrape targets by specifying the label name in
and by specifying the label value in `replacement` relabeling option.
The same approach can be used for updating already existing label values at target level.
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs):
For example, the following config adds `{foo="bar"}` label to all the discovered pods in [Kubernetes](./sd_configs.md#kubernetes_sd_configs):
```yaml
scrape_configs:
@ -540,7 +540,7 @@ If a particular discovered target shouldn't be scraped, then `action: keep` or `
must be used inside `relabel_configs` section.
The `action: keep` keeps only scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
while dropping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs)
and scrapes only pods with names starting with `foo` prefix:
```yaml
@ -554,7 +554,7 @@ scrape_configs:
```
The `action: drop` drops all the scrape targets with labels matching the `if` [selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors),
while keeping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs)
while keeping the rest of targets. For example, the following config discovers pod targets in [Kubernetes](./sd_configs.md#kubernetes_sd_configs)
and scrapes only pods with names starting with prefixes other than `foo`:
```yaml
@ -573,10 +573,10 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
## Useful tips for target relabeling
* Target relabeling can be debugged by clicking the `debug` link for the needed target on the `http://vmagent:8429/target`
or on the `http://vmagent:8429/service-discovery` pages. See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug).
or on the `http://vmagent:8429/service-discovery` pages. See [these docs](./vmagent.md#relabel-debug).
* Every discovered target contains a set of meta-labels, which start with `__meta_` prefix.
The specific sets of labels per each supported service discovery option are listed
[here](https://docs.victoriametrics.com/sd_configs/#prometheus-service-discovery).
[here](./sd_configs.md#prometheus-service-discovery).
* Every discovered target contains additional labels with `__` prefix other than `__meta_` labels.
See [these docs](#how-to-modify-scrape-urls-in-targets) for more details.
* All the labels, which start with `__` prefix, are automatically removed from targets after the relabeling.
@ -591,7 +591,7 @@ See also [useful tips for target relabeling](#useful-tips-for-target-relabeling)
## Useful tips for metric relabeling
* Metric relabeling can be debugged at `http://vmagent:8429/metric-relabel-debug` page.
See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug).
See [these docs](./vmagent.md#relabel-debug).
* All the labels, which start with `__` prefix, are automatically removed from metrics after the relabeling.
So it is common practice to store temporary labels with names starting with `__` during metrics relabeling.
* All the target-level labels are automatically added to all the metrics scraped from targets,

176
docs/sd_configs.md
View file

@ -11,7 +11,7 @@ aliases:
---
# Supported service discovery configs
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
[vmagent](./vmagent.md) and [single-node VictoriaMetrics](./README.md#how-to-scrape-prometheus-exporters-such-as-node-exporter)
supports the following Prometheus-compatible service discovery options for Prometheus-compatible scrape targets in the file pointed by `-promscrape.config` command-line flag:
* `azure_sd_configs` is for scraping the targets registered in [Azure Cloud](https://azure.microsoft.com/en-us/). See [these docs](#azure_sd_configs).
@ -89,14 +89,14 @@ scrape_configs:
# port: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<private_ip>:<port>`, where `<private_ip>` is the machine's private IP and the `<port>` is the `port`
option specified in the `azure_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_azure_machine_id`: the machine ID
* `__meta_azure_machine_location`: the location the machine runs in
@ -197,14 +197,14 @@ scrape_configs:
# allow_stale: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<service_or_node_addr>:<service_port>`, where `<service_or_node_addr>` is the service address. If the service address is empty,
then the node address is used instead. The `<service_port>` is the service port.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_consul_address`: the address of the target
* `__meta_consul_dc`: the datacenter name for the target
@ -289,14 +289,14 @@ scrape_configs:
# filter: "..."
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<service_or_node_addr>:<service_port>`, where `<service_or_node_addr>` is the service address. If the service address is empty,
then the node address is used instead. The `<service_port>` is the service port.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_consulagent_address`: the address of the target
* `__meta_consulagent_dc`: the datacenter name for the target
@ -337,13 +337,13 @@ scrape_configs:
# port: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<public_ip>:<port>`, where `<public_ip>` is a public ipv4 address of the droplet, while `<port>` is the port specified in the `digitalocean_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_digitalocean_droplet_id`: the id of the droplet
* `__meta_digitalocean_droplet_name`: the name of the droplet
@ -389,11 +389,11 @@ scrape_configs:
# port: ...
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to the `<addr>:<port>`, where `<addr>` is the discovered DNS address, while `<port>` is either the discovered port for SRV records or the port
specified in the `dns_sd_config`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_dns_name`: the record name that produced the discovered target.
* `__meta_dns_srv_record_target`: the target field of the SRV record
@ -435,15 +435,15 @@ scrape_configs:
# values: ["...", "..."]
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip_address>:<port>`, where `<ip_address>` is the exposed ip address of the docker container, while the `<port>` is either the exposed port
of the docker container or the port specified in the `docker_sd_configs` if the docker container has no exposed ports.
If a container exposes multiple ip addresses, then multiple targets will be discovered - one per each exposed ip address.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_docker_container_id`: the id of the container
* `__meta_docker_container_name`: the name of the container
@ -498,7 +498,7 @@ scrape_configs:
# values: ["...", "..."]
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
One of the following roles can be configured to discover targets:
@ -507,12 +507,12 @@ One of the following roles can be configured to discover targets:
The `services` role discovers all Swarm services.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip>:<port>`, where `<ip>` is the endpoint's virtual IP, while the `<port>` is the published port of the service.
If the service has multiple published ports, then multiple targets are generated - one per each port.
If the service has no published ports, then the `<port>` is set to the `port` value obtained from `dockerswarm_sd_configs`.
Available meta labels for `role: services` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: services` during [relabeling](./vmagent.md#relabeling):
* `__meta_dockerswarm_service_id`: the id of the service
* `__meta_dockerswarm_service_name`: the name of the service
@ -534,12 +534,12 @@ One of the following roles can be configured to discover targets:
The `tasks` role discovers all Swarm tasks.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip>:<port>`, where the `<ip>` is the node IP, while the `<port>` is the published port of the task.
If the task has multiple published ports, then multiple targets are generated - one per each port.
If the task has no published ports, then the `<port>` is set to the `port` value obtained from `dockerswarm_sd_configs`.
Available meta labels for `role: tasks` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: tasks` during [relabeling](./vmagent.md#relabeling):
* `__meta_dockerswarm_container_label_<labelname>`: each label of the container
* `__meta_dockerswarm_task_id`: the id of the task
@ -575,10 +575,10 @@ One of the following roles can be configured to discover targets:
The `nodes` role is used to discover Swarm nodes.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip>:<port>`, where `<ip>` is the node IP, while the `<port>` is the `port` value obtained from the `dockerswarm_sd_configs`.
Available meta labels for `role: nodes` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: nodes` during [relabeling](./vmagent.md#relabeling):
* `__meta_dockerswarm_node_address`: the address of the node
* `__meta_dockerswarm_node_availability`: the availability of the node
@ -660,11 +660,11 @@ scrape_configs:
# values: ["...", "..."]
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<instance_ip>:<port>`, where `<instance_ip>` is the private IP of the instance, while the `<port>` is set to the `port` value
obtain from `ec2_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_ec2_ami`: the EC2 Amazon Machine Image
* `__meta_ec2_architecture`: the architecture of the instance
@ -706,14 +706,14 @@ scrape_configs:
- server: "..."
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<instance_host>:<instance_port>`, where `<instance_host>` is the discovered instance hostname, while the `<instance_port>`
is the discovered instance port. If the instance has no port, then port 80 is used.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_eureka_app_name`: the name of the app
* `__meta_eureka_app_instance_id`: the ID of the app instance
@ -758,7 +758,7 @@ scrape_configs:
- "http://central-config-server/targets?type=foobar"
```
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#file-based-target-discovery) on how to configure file-based target discovery.
See [these examples](./scrape_config_examples.md#file-based-target-discovery) on how to configure file-based target discovery.
The referred files and urls must contain a list of static configs in one of the following formats:
@ -787,10 +787,10 @@ The referred files and urls must contain a list of static configs in one of the
...
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to one of the `target` value specified in the target files.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_filepath`: the filepath from which the target was extracted
@ -843,11 +843,11 @@ Credentials are discovered by looking in the following places, preferring the fi
1. a JSON file in the well-known path `$HOME/.config/gcloud/application_default_credentials.json`
1. fetched from the GCE metadata server
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<iface_ip>:<port>`, where `<iface_ip>` is private IP of the discovered instance, while `<port>` is the `port` value
specified in the `gce_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_gce_instance_id`: the numeric id of the instance
* `__meta_gce_instance_name`: the name of the instance
@ -902,13 +902,13 @@ scrape_configs:
# port: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<FQDN>:<port>`, where FQDN is discovered instance address and `<port>` is the port from the `hetzner_sd_configs` (default port is `80`).
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
Common labels for both `hcloud` and `robot` roles:
@ -960,10 +960,10 @@ scrape_configs:
- url: "http://..."
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#http-based-target-discovery) on how to configure http-based target discovery.
See [these examples](./scrape_config_examples.md#http-based-target-discovery) on how to configure http-based target discovery.
The service at `url` must return JSON response in the following format:
@ -983,10 +983,10 @@ The service at `url` must return JSON response in the following format:
The `url` is queried periodically with the interval specified in `-promscrape.httpSDCheckInterval` command-line flag.
Discovery errors are tracked in `promscrape_discovery_http_errors_total` metric.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to one of the targets returned by the http service.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_url`: the URL from which the target was extracted
@ -1051,10 +1051,10 @@ scrape_configs:
# node: <boolean>
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#kubernetes-target-discovery) on how to discover and scrape Kubernetes targets.
See [these examples](./scrape_config_examples.md#kubernetes-target-discovery) on how to discover and scrape Kubernetes targets.
One of the following `role` types can be configured to discover targets:
@ -1062,12 +1062,12 @@ One of the following `role` types can be configured to discover targets:
The `role: node` discovers one target per cluster node.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip>:<port>`, where `<ip>` is to the first existing address of the Kubernetes node object in the address type order
of `NodeInternalIP`, `NodeExternalIP`, `NodeLegacyHostIP` and `NodeHostName`,
while `<port>` is the kubelet port on the given node.
Available meta labels for `role: node` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: node` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_node_name`: The name of the node object.
* `__meta_kubernetes_node_provider_id`: The cloud provider's name for the node object.
@ -1083,7 +1083,7 @@ One of the following `role` types can be configured to discover targets:
The `role: service` discovers Kubernetes services.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<service_name>.<namespace>:<port>`, where `<service_name>` is the service name, `<namespace>` is the service namespace
and `<port>` is the service port.
If the service has multiple ports, then multiple targets are discovered for the service - one per each port.
@ -1091,7 +1091,7 @@ One of the following `role` types can be configured to discover targets:
This is generally useful for blackbox monitoring of a service. The target address will be set to the Kubernetes DNS name
of the service and respective service port.
Available meta labels for `role: service` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: service` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_namespace`: The namespace of the service object.
* `__meta_kubernetes_service_annotation_<annotationname>`: Each annotation from the service object.
@ -1110,12 +1110,12 @@ One of the following `role` types can be configured to discover targets:
The `role: pod` discovers all pods and exposes their containers as targets.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<ip>:<port>`, where `<ip>` is pod IP, while `<port>` is the exposed container port.
If the pod has multiple container ports, then multiple targets are generated for the pod - one per each exposed container port.
If the pod has no exposed container ports, then the `__address__` for pod target is set to the pod IP.
Available meta labels for `role: pod` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: pod` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_namespace`: The namespace of the pod object.
* `__meta_kubernetes_pod_name`: The name of the pod object.
@ -1144,12 +1144,12 @@ One of the following `role` types can be configured to discover targets:
The `role: endpoints` discovers targets from listed endpoints of a service.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<addr>:<port>`, where `<addr>` is the endpoint address, while `<port>` is the endpoint port.
If the endpoint has multiple ports, then a single target per each port is generated.
If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
Available meta labels for `role: endpoints` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: endpoints` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_namespace`: The namespace of the endpoints object.
* `__meta_kubernetes_endpoints_name`: The names of the endpoints object.
@ -1173,12 +1173,12 @@ One of the following `role` types can be configured to discover targets:
The `role: endpointslice` discovers targets from existing endpointslices.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<addr>:<port>`, where `<addr>` is the endpoint address, while `<port>` is the endpoint port.
If the endpoint has multiple ports, then a single target per each port is generated.
If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.
Available meta labels for `role: endpointslice` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: endpointslice` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_namespace`: The namespace of the endpointslice object.
* `__meta_kubernetes_endpointslice_name`: The name of endpointslice object.
@ -1202,13 +1202,13 @@ One of the following `role` types can be configured to discover targets:
The `role: ingress` discovers a target for each path of each ingress.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to the host obtained from ingress spec.
If the ingress has multiple specs with multiple hosts, then a target per each host is created.
This is generally useful for blackbox monitoring of an ingress.
Available meta labels for `role: ingress` during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
Available meta labels for `role: ingress` during [relabeling](./vmagent.md#relabeling):
* `__meta_kubernetes_namespace`: The namespace of the ingress object.
* `__meta_kubernetes_ingress_name`: The name of the ingress object.
@ -1248,16 +1248,16 @@ scrape_configs:
# client_id: "..."
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
The `server` is queried periodically with the interval specified in `-promscrape.kumaSDCheckInterval` command-line flag.
Discovery errors are tracked in `promscrape_discovery_kuma_errors_total` metric.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to one of the targets returned by the http service.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_kuma_mesh`: the name of the mesh
* `__meta_kuma_dataplane`: the name of the proxy
@ -1307,13 +1307,13 @@ scrape_configs:
# allow_stale: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<addr>:<port>`, where `<addr>` is the service address, while `<port>` is the service port.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_nomad_address`: the address of the target
* `__meta_nomad_dc`: the datacenter name for the target
@ -1426,10 +1426,10 @@ One of the following `role` types can be configured to discover targets:
The `role: hypervisor` discovers one target per Nova hypervisor node.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<host>:<port>`, where `<host>` is the discovered node IP, while `<port>` is the port specified in the `openstack_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_openstack_hypervisor_host_ip`: the hypervisor node's IP address.
* `__meta_openstack_hypervisor_hostname`: the hypervisor node's name.
@ -1442,10 +1442,10 @@ One of the following `role` types can be configured to discover targets:
The `role: instance` discovers one target per network interface of Nova instance.
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<host>:<port>`, where `<host>` is the private IP address of the discovered instance, while `<port>` is the port specified in the `openstack_sd_configs`.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_openstack_address_pool`: the pool of the private IP.
* `__meta_openstack_instance_flavor`: the flavor of the OpenStack instance.
@ -1475,10 +1475,10 @@ scrape_configs:
# The `http://host:port/metrics` endpoint is scraped per each configured target then.
# The `http` scheme can be changed to `https` by setting it via `scheme` field at `scrape_config` level.
# The `/metrics` path can be changed to arbitrary path via `metrics_path` field at `scrape_config` level.
# See https://docs.victoriametrics.com/sd_configs/#scrape_configs .
# See {{% ref "./sd_configs.md#scrape_configs" %}}.
#
# Alternatively the scheme and path can be changed via `relabel_configs` section at `scrape_config` level.
# See https://docs.victoriametrics.com/vmagent/#relabeling .
# See {{% ref "./vmagent.md#relabeling" %}}.
#
# It is also possible specifying full target urls here, e.g. "http://host:port/metrics/path?query_args"
#
@ -1494,7 +1494,7 @@ scrape_configs:
# <labelnameN>: "<labelvalueN>"
```
See [these examples](https://docs.victoriametrics.com/scrape_config_examples/#static-configs) on how to configure scraping for static targets.
See [these examples](./scrape_config_examples.md#static-configs) on how to configure scraping for static targets.
## vultr_sd_configs
Vultr SD configuration discovers scrape targets from [Vultr](https://www.vultr.com/) Instances.
@ -1539,15 +1539,15 @@ scrape_configs:
# port: ...
# Additional HTTP API client options can be specified here.
# See https://docs.victoriametrics.com/sd_configs.html#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to `<FQDN>:<port>`, where FQDN is discovered instance address and `<port>` is the port from the `vultr_sd_configs` (default port is `80`).
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_vultr_instance_allowed_bandwidth_gb`: monthly bandwidth quota in GB.
* `__meta_vultr_instance_disk_gb`: the size of the disk in GB.
@ -1625,10 +1625,10 @@ scrape_configs:
replacement: "$1:9100"
```
Each discovered target has an [`__address__`](https://docs.victoriametrics.com/relabeling/#how-to-modify-scrape-urls-in-targets) label set
Each discovered target has an [`__address__`](./relabeling.md#how-to-modify-scrape-urls-in-targets) label set
to the FQDN of the discovered instance.
The following meta labels are available on discovered targets during [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling):
The following meta labels are available on discovered targets during [relabeling](./vmagent.md#relabeling):
* `__meta_yandexcloud_instance_name`: the name of instance
* `__meta_yandexcloud_instance_id`: the id of instance
@ -1670,7 +1670,7 @@ scrape_configs:
# - "2m" - 2 minutes
# The scrape_interval can be set on a per-target basis by specifying `__scrape_interval__`
# label during target relabeling phase.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# scrape_interval: <duration>
@ -1685,7 +1685,7 @@ scrape_configs:
# The `scrape_timeout` cannot exceed the `scrape_interval`.
# The scrape_timeout can be set on a per-target basis by specifying `__scrape_timeout__`
# label during target relabeling phase.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# scrape_timeout: <duration>
@ -1751,14 +1751,14 @@ scrape_configs:
# relabel_configs is an optional relabeling configurations
# for the specified and discovered scrape targets.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# relabel_configs:
# - <relabel_config> ...
# metric_relabel_configs is an optional relabeling configs
# for the collected metrics from active scrape targets.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# metric_relabel_configs:
# - <relabel_config> ...
@ -1774,23 +1774,23 @@ scrape_configs:
# disable_compression allows disabling HTTP compression for responses received from scrape targets.
# By default, scrape targets are queried with `Accept-Encoding: gzip` http request header,
# so targets could send compressed responses in order to save network bandwidth.
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
# See {{% ref "./vmagent.md#scrape_config-enhancements" %}}
#
# disable_compression: <boolean>
# disable_keepalive allows disabling HTTP keep-alive when scraping targets.
# By default, HTTP keep-alive is enabled, so TCP connections to scrape targets
# could be re-used.
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
# See {{% ref "./vmagent.md#scrape_config-enhancements" %}}
#
# disable_keepalive: <boolean>
# stream_parse allows enabling stream parsing mode when scraping targets.
# By default, stream parsing mode is disabled for targets which return up to a few thousands samples.
# See https://docs.victoriametrics.com/vmagent/#stream-parsing-mode .
# See {{% ref "./vmagent.md#stream-parsing-mode" %}}.
# The stream_parse can be set on a per-target basis by specifying `__stream_parse__`
# label during target relabeling phase.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# stream_parse: <boolean>
@ -1798,7 +1798,7 @@ scrape_configs:
# Example values:
# - "5m" - align scrapes to every 5 minutes.
# - "1h" - align scrapes to every hour.
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
# See {{% ref "./vmagent.md#scrape_config-enhancements" %}}
#
# scrape_align_interval: <duration>
@ -1806,28 +1806,28 @@ scrape_configs:
# Example values:
# - "5m" - align scrapes to every 5 minutes.
# - "1h" - align scrapes to every hour.
# See https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
# See {{% ref "./vmagent.md#scrape_config-enhancements" %}}
#
# scrape_offset: <duration>
# series_limit is an optional limit on the number of unique time series
# a single target can expose during all the scrapes on the time window of 24h.
# By default, there is no limit on the number of exposed series.
# See https://docs.victoriametrics.com/vmagent/#cardinality-limiter .
# See {{% ref "./vmagent.md#cardinality-limiter" %}}.
# The series_limit can be set on a per-target basis by specifying `__series_limit__`
# label during target relabeling phase.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
#
# series_limit: ...
# no_stale_markers allows disabling staleness tracking.
# By default, staleness tracking is enabled for all the discovered scrape targets.
# See https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers
# See {{% ref "./vmagent.md#prometheus-staleness-markers" %}}
#
# no_stale_markers: <boolean>
# Additional HTTP client options for target scraping can be specified here.
# See https://docs.victoriametrics.com/sd_configs/#http-api-client-options
# See {{% ref "./sd_configs.md#http-api-client-options" %}}
```
## HTTP API client options

256
docs/stream-aggregation.md
View file

@ -9,23 +9,23 @@ menu:
aliases:
- /stream-aggregation.html
---
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage
[vmagent](./vmagent.md) and [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
can aggregate incoming [samples](./keyConcepts.md#raw-samples) in streaming mode by time and by labels before data is written to remote storage
(or local storage for single-node VictoriaMetrics).
The aggregation is applied to all the metrics received via any [supported data ingestion protocol](https://docs.victoriametrics.com/#how-to-import-time-series-data)
and/or scraped from [Prometheus-compatible targets](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)
after applying all the configured [relabeling stages](https://docs.victoriametrics.com/vmagent/#relabeling).
The aggregation is applied to all the metrics received via any [supported data ingestion protocol](./#how-to-import-time-series-data)
and/or scraped from [Prometheus-compatible targets](./#how-to-scrape-prometheus-exporters-such-as-node-exporter)
after applying all the configured [relabeling stages](./vmagent.md#relabeling).
**By default, stream aggregation ignores timestamps associated with the input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
**By default, stream aggregation ignores timestamps associated with the input [samples](./keyConcepts.md#raw-samples).
It expects that the ingested samples have timestamps close to the current time. See [how to ignore old samples](#ignoring-old-samples).**
## Configuration
Stream aggregation can be configured via the following command-line flags:
- `-streamAggr.config` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and at [vmagent](https://docs.victoriametrics.com/vmagent/).
- `-remoteWrite.streamAggr.config` at [vmagent](https://docs.victoriametrics.com/vmagent/) only. This flag can be specified individually
- `-streamAggr.config` at [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
and at [vmagent](./vmagent.md).
- `-remoteWrite.streamAggr.config` at [vmagent](./vmagent.md) only. This flag can be specified individually
per each `-remoteWrite.url`, so the aggregation happens independently per each remote storage destination.
This allows writing different aggregates to different remote storage systems.
@ -39,29 +39,29 @@ By default, the following data is written to the storage when stream aggregation
This behaviour can be changed via the following command-line flags:
- `-streamAggr.keepInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
- `-streamAggr.keepInput` at [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
and [vmagent](./vmagent.md). At [vmagent](./vmagent.md)
`-remoteWrite.streamAggr.keepInput` flag can be specified individually per each `-remoteWrite.url`.
If one of these flags is set, then all the input samples are written to the storage alongside the aggregated samples.
- `-streamAggr.dropInput` at [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
- `-streamAggr.dropInput` at [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
and [vmagent](./vmagent.md). At [vmagent](./vmagent.md)
`-remoteWrite.streamAggr.dropInput` flag can be specified individually per each `-remoteWrite.url`.
If one of these flags are set, then all the input samples are dropped, while only the aggregated samples are written to the storage.
## Routing
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports relabeling,
[Single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) supports relabeling,
deduplication and stream aggregation for all the received data, scraped or pushed.
The processed data is then stored in local storage and **can't be forwarded further**.
[vmagent](https://docs.victoriametrics.com/vmagent/) supports relabeling, deduplication and stream aggregation for all
[vmagent](./vmagent.md) supports relabeling, deduplication and stream aggregation for all
the received data, scraped or pushed. Then, the collected data will be forwarded to specified `-remoteWrite.url` destinations.
The data processing order is the following:
1. all the received data is relabeled according to the specified [`-remoteWrite.relabelConfig`](https://docs.victoriametrics.com/vmagent/#relabeling) (if it is set)
1. all the received data is deduplicated according to specified [`-streamAggr.dedupInterval`](https://docs.victoriametrics.com/stream-aggregation/#deduplication)
1. all the received data is relabeled according to the specified [`-remoteWrite.relabelConfig`](./vmagent.md#relabeling) (if it is set)
1. all the received data is deduplicated according to specified [`-streamAggr.dedupInterval`](./stream-aggregation.md#deduplication)
(if it is set to duration bigger than 0)
1. all the received data is aggregated according to specified [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) (if it is set)
1. all the received data is aggregated according to specified [`-streamAggr.config`](./stream-aggregation.md#configuration) (if it is set)
1. the resulting data is then replicated to each `-remoteWrite.url`
1. data sent to each `-remoteWrite.url` can be additionally relabeled according to the corresponding `-remoteWrite.urlRelabelConfig` (set individually per URL)
1. data sent to each `-remoteWrite.url` can be additionally deduplicated according to the corresponding `-remoteWrite.streamAggr.dedupInterval` (set individually per URL)
@ -70,22 +70,22 @@ The data processing order is the following:
Typical scenarios for data routing with `vmagent`:
1. **Aggregate incoming data and replicate to N destinations**. Specify [`-streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration) command-line flag
1. **Aggregate incoming data and replicate to N destinations**. Specify [`-streamAggr.config`](./stream-aggregation.md#configuration) command-line flag
to aggregate the incoming data before replicating it to all the configured `-remoteWrite.url` destinations.
2. **Individually aggregate incoming data for each destination**. Specify [`-remoteWrite.streamAggr.config`](https://docs.victoriametrics.com/stream-aggregation/#configuration)
command-line flag for each `-remoteWrite.url` destination. [Relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) via `-remoteWrite.urlRelabelConfig`
2. **Individually aggregate incoming data for each destination**. Specify [`-remoteWrite.streamAggr.config`](./stream-aggregation.md#configuration)
command-line flag for each `-remoteWrite.url` destination. [Relabeling](./vmagent.md#relabeling) via `-remoteWrite.urlRelabelConfig`
can be used for routing only the selected metrics to each `-remoteWrite.url` destination.
## Deduplication
[vmagent](https://docs.victoriametrics.com/vmagent/) supports online [de-duplication](https://docs.victoriametrics.com/#deduplication) of samples
[vmagent](./vmagent.md) supports online [de-duplication](./#deduplication) of samples
before sending them to the configured `-remoteWrite.url`. The de-duplication can be enabled via the following options:
- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag for all received data
or via `-remoteWrite.streamAggr.dedupInterval` command-line flag for the particular `-remoteWrite.url` destination.
For example, `./vmagent -remoteWrite.url=http://remote-storage/api/v1/write -remoteWrite.streamAggr.dedupInterval=30s` instructs `vmagent` to leave
only the last sample per each seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds.
The de-deduplication is performed after applying [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) and
only the last sample per each seen [time series](./keyConcepts.md#time-series) per every 30 seconds.
The de-deduplication is performed after applying [relabeling](./vmagent.md#relabeling) and
before performing the aggregation.
If the `-remoteWrite.streamAggr.config` and / or `-streamAggr.config` is set, then the de-duplication is performed individually per each
[stream aggregation config](#stream-aggregation-config) for the matching samples after applying [input_relabel_configs](#relabeling).
@ -93,13 +93,13 @@ before sending them to the configured `-remoteWrite.url`. The de-duplication can
- By specifying `dedup_interval` option individually per each [stream aggregation config](#stream-aggregation-config)
in `-remoteWrite.streamAggr.config` or `-streamAggr.config` configs.
[Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) supports two types of de-duplication:
- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](https://docs.victoriametrics.com/#deduplication) command-line option.
[Single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) supports two types of de-duplication:
- After storing the duplicate samples to local storage. See [`-dedup.minScrapeInterval`](./#deduplication) command-line option.
- Before storing the duplicate samples to local storage. This type of de-duplication can be enabled via the following options:
- By specifying the desired de-duplication interval via `-streamAggr.dedupInterval` command-line flag.
For example, `./victoria-metrics -streamAggr.dedupInterval=30s` instructs VictoriaMetrics to leave only the last sample per each
seen [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per every 30 seconds.
The de-duplication is performed after applying `-relabelConfig` [relabeling](https://docs.victoriametrics.com/#relabeling).
seen [time series](./keyConcepts.md#time-series) per every 30 seconds.
The de-duplication is performed after applying `-relabelConfig` [relabeling](./#relabeling).
If the `-streamAggr.config` is set, then the de-duplication is performed individually per each [stream aggregation config](#stream-aggregation-config)
for the matching samples after applying [input_relabel_configs](#relabeling).
@ -108,15 +108,15 @@ before sending them to the configured `-remoteWrite.url`. The de-duplication can
It is possible to drop the given labels before applying the de-duplication. See [these docs](#dropping-unneeded-labels).
The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](https://docs.victoriametrics.com/#deduplication) at VictoriaMetrics.
The online de-duplication uses the same logic as [`-dedup.minScrapeInterval` command-line flag](./#deduplication) at VictoriaMetrics.
## Ignoring old samples
By default, all the input samples are taken into account during stream aggregation. If samples with old timestamps
outside the current [aggregation interval](#stream-aggregation-config) must be ignored, then the following options can be used:
- To pass `-streamAggr.ignoreOldSamples` command-line flag to [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
or to [vmagent](https://docs.victoriametrics.com/vmagent/). At [vmagent](https://docs.victoriametrics.com/vmagent/)
- To pass `-streamAggr.ignoreOldSamples` command-line flag to [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md)
or to [vmagent](./vmagent.md). At [vmagent](./vmagent.md)
`-remoteWrite.streamAggr.ignoreOldSamples` flag can be specified individually per each `-remoteWrite.url`.
This enables ignoring old samples for all the [aggregation configs](#stream-aggregation-config).
@ -125,16 +125,16 @@ outside the current [aggregation interval](#stream-aggregation-config) must be i
## Ignore aggregation intervals on start
Streaming aggregation results may be incorrect for some time after the restart of [vmagent](https://docs.victoriametrics.com/vmagent/)
or [single-node VictoriaMetrics](https://docs.victoriametrics.com/) until all the buffered [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent).
Streaming aggregation results may be incorrect for some time after the restart of [vmagent](./vmagent.md)
or [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md) until all the buffered [samples](./keyConcepts.md#raw-samples)
are sent from remote sources to the `vmagent` or single-node VictoriaMetrics via [supported data ingestion protocols](./vmagent.md#how-to-push-data-to-vmagent).
In this case it may be a good idea to drop the aggregated data during the first `N` [aggregation intervals](#stream-aggregation-config)
just after the restart of `vmagent` or single-node VictoriaMetrics. This can be done via the following options:
- The `-streamAggr.ignoreFirstIntervals=N` command-line flag at `vmagent` and single-node VictoriaMetrics. This flag instructs skipping the first `N`
[aggregation intervals](#stream-aggregation-config) just after the restart across all the [configured stream aggregation configs](#configuration).
The `-remoteWrite.streamAggr.ignoreFirstIntervals=N` command-line flag can be specified individually per each `-remoteWrite.url` at [vmagent](https://docs.victoriametrics.com/vmagent/).
The `-remoteWrite.streamAggr.ignoreFirstIntervals=N` command-line flag can be specified individually per each `-remoteWrite.url` at [vmagent](./vmagent.md).
- The `ignore_first_intervals: N` option at the particular [aggregation config](#stream-aggregation-config).
@ -183,17 +183,17 @@ Stream aggregation can be used as [statsd](https://github.com/statsd/statsd) alt
* [Histograms over input metrics](#histograms-over-input-metrics)
* [Aggregating histograms](#aggregating-histograms)
Currently, streaming aggregation is available only for [supported data ingestion protocols](https://docs.victoriametrics.com/#how-to-import-time-series-data)
Currently, streaming aggregation is available only for [supported data ingestion protocols](./#how-to-import-time-series-data)
and not available for [Statsd metrics format](https://github.com/statsd/statsd/blob/master/docs/metric_types.md).
### Recording rules alternative
Sometimes [alerting queries](https://docs.victoriametrics.com/vmalert/#alerting-rules) may require non-trivial amounts of CPU, RAM,
Sometimes [alerting queries](./vmalert.md#alerting-rules) may require non-trivial amounts of CPU, RAM,
disk IO and network bandwidth at metrics storage side. For example, if `http_request_duration_seconds` histogram is generated by thousands
of application instances, then the alerting query `histogram_quantile(0.99, sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)) > 0.5`
can become slow, since it needs to scan too big number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
can become slow, since it needs to scan too big number of unique [time series](./keyConcepts.md#time-series)
with `http_request_duration_seconds_bucket` name. This alerting query can be accelerated by pre-calculating
the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](https://docs.victoriametrics.com/vmalert/#recording-rules).
the `sum(increase(http_request_duration_seconds_bucket[5m])) without (instance)` via [recording rule](./vmalert.md#recording-rules).
But this recording rule may take too much time to execute too. In this case the slow recording rule can be substituted
with the following [stream aggregation config](#stream-aggregation-config):
@ -205,7 +205,7 @@ with the following [stream aggregation config](#stream-aggregation-config):
```
This stream aggregation generates `http_request_duration_seconds_bucket:5m_without_instance_total` output series according to [output metric naming](#output-metric-names).
Then these series can be used in [alerting rules](https://docs.victoriametrics.com/vmalert/#alerting-rules):
Then these series can be used in [alerting rules](./vmalert.md#alerting-rules):
```metricsql
histogram_quantile(0.99, last_over_time(http_request_duration_seconds_bucket:5m_without_instance_total[5m])) > 0.5
@ -221,7 +221,7 @@ Field `interval` is recommended to be set to a value at least several times high
### Reducing the number of stored samples
If per-[series](https://docs.victoriametrics.com/keyconcepts/#time-series) samples are ingested at high frequency,
If per-[series](./keyConcepts.md#time-series) samples are ingested at high frequency,
then this may result in high disk space usage, since too much data must be stored to disk. This also may result
in slow queries, since too much data must be processed during queries.
@ -232,13 +232,13 @@ to one sample per 5 minutes per each input time series (this operation is also k
```yaml
# Aggregate metrics ending with _total with `total` output.
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
# See {{% ref "./stream-aggregation.md#aggregation-outputs" %}}
- match: '{__name__=~".+_total"}'
interval: 5m
outputs: [total]
# Downsample other metrics with `count_samples`, `sum_samples`, `min` and `max` outputs
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
# See {{% ref "./stream-aggregation.md#aggregation-outputs" %}}
- match: '{__name__!~".+_total"}'
interval: 5m
outputs: [count_samples, sum_samples, min, max]
@ -262,7 +262,7 @@ See also [aggregating histograms](#aggregating-histograms) and [aggregating by l
### Reducing the number of stored series
Sometimes applications may generate too many [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
Sometimes applications may generate too many [time series](./keyConcepts.md#time-series).
For example, the `http_requests_total` metric may have `path` or `user` label with too big number of unique values.
In this case the following stream aggregation can be used for reducing the number metrics stored in VictoriaMetrics:
@ -372,7 +372,7 @@ See also [histograms over input metrics](#histograms-over-input-metrics) and [ag
### Histograms over input metrics
If the monitored application generates measurement metrics per each request, then it may be useful to calculate
a [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) over these metrics.
a [histogram](./keyConcepts.md#histogram) over these metrics.
For example, if the monitored application generates `request_duration_seconds N` and `response_size_bytes M` metrics
per each incoming request, then the following [stream aggregation config](#stream-aggregation-config)
@ -401,7 +401,7 @@ response_size_bytes:60s_histogram_bucket{vmrange="start2...end2"} count2
response_size_bytes:60s_histogram_bucket{vmrange="startN...endN"} countN
```
The resulting histogram buckets can be queried with [MetricsQL](https://docs.victoriametrics.com/metricsql/) in the following ways:
The resulting histogram buckets can be queried with [MetricsQL](./MetricsQL.md) in the following ways:
1. An estimated 50th and 99th [percentiles](https://en.wikipedia.org/wiki/Percentile) of the request duration over the last hour:
@ -409,7 +409,7 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
histogram_quantiles("quantile", 0.50, 0.99, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
```
This query uses [histogram_quantiles](https://docs.victoriametrics.com/metricsql/#histogram_quantiles) function.
This query uses [histogram_quantiles](./MetricsQL.md#histogram_quantiles) function.
1. An estimated [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of the request duration over the last hour:
@ -417,7 +417,7 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
histogram_stddev(sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
```
This query uses [histogram_stddev](https://docs.victoriametrics.com/metricsql/#histogram_stddev) function.
This query uses [histogram_stddev](./MetricsQL.md#histogram_stddev) function.
1. An estimated share of requests with the duration smaller than `0.5s` over the last hour:
@ -425,16 +425,16 @@ The resulting histogram buckets can be queried with [MetricsQL](https://docs.vic
histogram_share(0.5, sum(increase(request_duration_seconds:60s_histogram_bucket[1h])) by (vmrange))
```
This query uses [histogram_share](https://docs.victoriametrics.com/metricsql/#histogram_share) function.
This query uses [histogram_share](./MetricsQL.md#histogram_share) function.
See [the list of aggregate output](#aggregation-outputs), which can be specified at `output` field.
See also [quantiles over input metrics](#quantiles-over-input-metrics) and [aggregating by labels](#aggregating-by-labels).
### Aggregating histograms
[Histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) is a set of [counter](https://docs.victoriametrics.com/keyconcepts/#counter)
[Histogram](./keyConcepts.md#histogram) is a set of [counter](./keyConcepts.md#counter)
metrics with different `vmrange` or `le` labels. As they're counters, the applicable aggregation output is
[total](https://docs.victoriametrics.com/stream-aggregation/#total):
[total](./stream-aggregation.md#total):
```yaml
- match: 'http_request_duration_seconds_bucket'
@ -454,7 +454,7 @@ http_request_duration_seconds_bucket:1m_without_instance_total{le="3"} value5
http_request_duration_seconds_bucket:1m_without_instance_total{le="+Inf" value6
```
The resulting metrics can be passed to [histogram_quantile](https://docs.victoriametrics.com/metricsql/#histogram_quantile)
The resulting metrics can be passed to [histogram_quantile](./MetricsQL.md#histogram_quantile)
function:
```metricsql
@ -492,7 +492,7 @@ The `keep_metric_names` option can be used if only a single output is set in [`o
## Relabeling
It is possible to apply [arbitrary relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) to input and output metrics
It is possible to apply [arbitrary relabeling](./vmagent.md#relabeling) to input and output metrics
during stream aggregation via `input_relabel_configs` and `output_relabel_configs` options in [stream aggregation config](#stream-aggregation-config).
Relabeling rules inside `input_relabel_configs` are applied to samples matching the `match` filters before optional [deduplication](#deduplication).
@ -575,10 +575,10 @@ Below are aggregation functions that can be put in the `outputs` list at [stream
### avg
`avg` returns the average over input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples).
`avg` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`avg` returns the average over input [sample values](./keyConcepts.md#raw-samples).
`avg` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `avg` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `avg` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(sum_over_time(some_metric[interval])) / sum(count_over_time(some_metric[interval]))
@ -598,9 +598,9 @@ See also:
### count_samples
`count_samples` counts the number of input [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`count_samples` counts the number of input [samples](./keyConcepts.md#raw-samples) over the given `interval`.
The results of `count_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `count_samples` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(count_over_time(some_metric[interval]))
@ -613,9 +613,9 @@ See also:
### count_series
`count_series` counts the number of unique [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
`count_series` counts the number of unique [time series](./keyConcepts.md#time-series) over the given `interval`.
The results of `count_series` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `count_series` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
count(last_over_time(some_metric[interval]))
@ -629,11 +629,11 @@ See also:
### histogram_bucket
`histogram_bucket` returns [VictoriaMetrics histogram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350)
for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`histogram_bucket` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
for the input [sample values](./keyConcepts.md#raw-samples) over the given `interval`.
`histogram_bucket` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
See how to aggregate regular histograms [here](#aggregating-histograms).
The results of `histogram_bucket` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `histogram_bucket` is equal to the following [MetricsQL](./MetricsQL.md) query:
Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/)
or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option.
@ -651,16 +651,16 @@ See also:
### increase
`increase` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given 'interval'.
`increase` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`increase` returns the increase of input [time series](./keyConcepts.md#time-series) over the given 'interval'.
`increase` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `increase` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `increase` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(increase_pure(some_counter[interval]))
```
`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
`increase` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](./keyConcepts.md#time-series)
is `10`, then `increase` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series,
then take a look at [increase_prometheus](#increase_prometheus).
@ -680,16 +680,16 @@ See also:
### increase_prometheus
`increase_prometheus` returns the increase of input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
`increase_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`increase_prometheus` returns the increase of input [time series](./keyConcepts.md#time-series) over the given `interval`.
`increase_prometheus` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `increase_prometheus` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `increase_prometheus` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(increase_prometheus(some_counter[interval]))
```
`increase_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
`increase_prometheus` skips the first seen sample value per each [time series](./keyConcepts.md#time-series).
If you need taking into account the first sample per time series, then take a look at [increase](#increase).
Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/)
@ -705,9 +705,9 @@ See also:
### last
`last` returns the last input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`last` returns the last input [sample value](./keyConcepts.md#raw-samples) over the given `interval`.
The results of `last` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `last` is roughly equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
last_over_time(some_metric[interval])
@ -722,9 +722,9 @@ See also:
### max
`max` returns the maximum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`max` returns the maximum input [sample value](./keyConcepts.md#raw-samples) over the given `interval`.
The results of `max` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `max` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
max(max_over_time(some_metric[interval]))
@ -743,9 +743,9 @@ See also:
### min
`min` returns the minimum input [sample value](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`min` returns the minimum input [sample value](./keyConcepts.md#raw-samples) over the given `interval`.
The results of `min` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `min` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
min(min_over_time(some_metric[interval]))
@ -764,10 +764,10 @@ See also:
### rate_avg
`rate_avg` returns the average of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
`rate_avg` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`rate_avg` returns the average of average per-second increase rates across input [time series](./keyConcepts.md#time-series) over the given `interval`.
`rate_avg` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `rate_avg` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `rate_avg` are equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
avg(rate(some_counter[interval]))
@ -781,10 +781,10 @@ See also:
### rate_sum
`rate_sum` returns the sum of average per-second increase rates across input [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the given `interval`.
`rate_sum` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`rate_sum` returns the sum of average per-second increase rates across input [time series](./keyConcepts.md#time-series) over the given `interval`.
`rate_sum` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `rate_sum` are equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `rate_sum` are equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(rate(some_counter[interval]))
@ -798,11 +798,11 @@ See also:
### stddev
`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`stddev` returns [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) for the input [sample values](./keyConcepts.md#raw-samples)
over the given `interval`.
`stddev` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`stddev` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `stddev` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `stddev` is roughly equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
histogram_stddev(sum(histogram_over_time(some_metric[interval])) by (vmrange))
@ -816,11 +816,11 @@ See also:
### stdvar
`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples)
`stdvar` returns [standard variance](https://en.wikipedia.org/wiki/Variance) for the input [sample values](./keyConcepts.md#raw-samples)
over the given `interval`.
`stdvar` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`stdvar` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `stdvar` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `stdvar` is roughly equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
histogram_stdvar(sum(histogram_over_time(some_metric[interval])) by (vmrange))
@ -838,10 +838,10 @@ See also:
### sum_samples
`sum_samples` sums input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given `interval`.
`sum_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`sum_samples` sums input [sample values](./keyConcepts.md#raw-samples) over the given `interval`.
`sum_samples` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `sum_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `sum_samples` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(sum_over_time(some_metric[interval]))
@ -858,16 +858,16 @@ See also:
### total
`total` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`.
`total` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`total` generates output [counter](./keyConcepts.md#counter) by summing the input counters over the given `interval`.
`total` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `total` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `total` is roughly equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(running_sum(increase_pure(some_counter)))
```
`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
`total` assumes that all the counters start from 0. For example, if the first seen sample for new [time series](./keyConcepts.md#time-series)
is `10`, then `total` assumes that the time series has been increased by `10`. If you need ignoring the first sample for new time series,
then take a look at [total_prometheus](#total_prometheus).
@ -875,7 +875,7 @@ For example, see below time series produced by config with aggregation interval
![total aggregation](stream-aggregation-check-total.webp)
`total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) -
`total` is not affected by [counter resets](./keyConcepts.md#counter) -
it continues to increase monotonically with respect to the previous value.
The counters are most often reset when the application is restarted.
@ -901,19 +901,19 @@ See also:
### total_prometheus
`total_prometheus` generates output [counter](https://docs.victoriametrics.com/keyconcepts/#counter) by summing the input counters over the given `interval`.
`total_prometheus` makes sense only for aggregating [counters](https://docs.victoriametrics.com/keyconcepts/#counter).
`total_prometheus` generates output [counter](./keyConcepts.md#counter) by summing the input counters over the given `interval`.
`total_prometheus` makes sense only for aggregating [counters](./keyConcepts.md#counter).
The results of `total_prometheus` is roughly equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `total_prometheus` is roughly equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
sum(running_sum(increase_prometheus(some_counter)))
```
`total_prometheus` skips the first seen sample value per each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
`total_prometheus` skips the first seen sample value per each [time series](./keyConcepts.md#time-series).
If you need taking into account the first sample per time series, then take a look at [total](#total).
`total_prometheus` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) -
`total_prometheus` is not affected by [counter resets](./keyConcepts.md#counter) -
it continues to increase monotonically with respect to the previous value.
The counters are most often reset when the application is restarted.
@ -931,9 +931,9 @@ See also:
### unique_samples
`unique_samples` counts the number of unique sample values over the given `interval`.
`unique_samples` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`unique_samples` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `unique_samples` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `unique_samples` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
count(count_values_over_time(some_metric[interval]))
@ -947,11 +947,11 @@ See also:
### quantiles
`quantiles(phi1, ..., phiN)` returns [percentiles](https://en.wikipedia.org/wiki/Percentile) for the given `phi*`
over the input [sample values](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given `interval`.
over the input [sample values](./keyConcepts.md#raw-samples) on the given `interval`.
`phi` must be in the range `[0..1]`, where `0` means `0th` percentile, while `1` means `100th` percentile.
`quantiles(...)` makes sense only for aggregating [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge).
`quantiles(...)` makes sense only for aggregating [gauges](./keyConcepts.md#gauge).
The results of `quantiles(phi1, ..., phiN)` is equal to the following [MetricsQL](https://docs.victoriametrics.com/metricsql/) query:
The results of `quantiles(phi1, ..., phiN)` is equal to the following [MetricsQL](./MetricsQL.md) query:
```metricsql
histogram_quantiles("quantile", phi1, ..., phiN, sum(histogram_over_time(some_metric[interval])) by (vmrange))
@ -1008,8 +1008,8 @@ See also [aggregation outputs](#aggregation-outputs).
## Stream aggregation config
Below is the format for stream aggregation config file, which may be referred via `-streamAggr.config` command-line flag at
[single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and [vmagent](https://docs.victoriametrics.com/vmagent/).
At [vmagent](https://docs.victoriametrics.com/vmagent/) `-remoteWrite.streamAggr.config` command-line flag can be
[single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) and [vmagent](./vmagent.md).
At [vmagent](./vmagent.md) `-remoteWrite.streamAggr.config` command-line flag can be
specified individually per each `-remoteWrite.url`:
```yaml
@ -1018,12 +1018,12 @@ specified individually per each `-remoteWrite.url`:
#
# If it is set, then it is used as `name` label in the exposed metrics
# for the given aggregation config at /metrics page.
# See https://docs.victoriametrics.com/vmagent/#monitoring and https://docs.victoriametrics.com/#monitoring
# See {{% ref "./vmagent.md#monitoring" %}} and {{% ref "./#monitoring" %}}
- name: 'foobar'
# match is an optional filter for incoming samples to aggregate.
# It can contain arbitrary Prometheus series selector
# according to https://docs.victoriametrics.com/keyconcepts/#filtering .
# according to {{% ref "./keyConcepts.md#filtering" %}}.
# If match isn't set, then all the incoming samples are aggregated.
#
# match also can contain a list of series selectors. Then the incoming samples are aggregated
@ -1037,8 +1037,8 @@ specified individually per each `-remoteWrite.url`:
interval: 1m
# dedup_interval is an optional interval for de-duplication of input samples before the aggregation.
# Samples are de-duplicated on a per-series basis. See https://docs.victoriametrics.com/keyconcepts/#time-series
# and https://docs.victoriametrics.com/#deduplication
# Samples are de-duplicated on a per-series basis. See {{% ref "./keyConcepts.md#time-series" %}}
# and {{% ref "./#deduplication" %}}
# The deduplication is performed after input_relabel_configs relabeling is applied.
# By default, the deduplication is disabled unless -remoteWrite.streamAggr.dedupInterval or -streamAggr.dedupInterval
# command-line flags are set.
@ -1054,7 +1054,7 @@ specified individually per each `-remoteWrite.url`:
# - rate_sum
# - total
# - total_prometheus
# See https://docs.victoriametrics.com/stream-aggregation/#staleness for more details.
# See {{% ref "./stream-aggregation.md#staleness" %}} for more details.
#
# staleness_interval: 2m
@ -1073,42 +1073,42 @@ specified individually per each `-remoteWrite.url`:
# flush_on_shutdown: false
# without is an optional list of labels, which must be removed from the output aggregation.
# See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels
# See {{% ref "./stream-aggregation.md#aggregating-by-labels" %}}
#
# without: [instance]
# by is an optional list of labels, which must be preserved in the output aggregation.
# See https://docs.victoriametrics.com/stream-aggregation/#aggregating-by-labels
# See {{% ref "./stream-aggregation.md#aggregating-by-labels" %}}
#
# by: [job, vmrange]
# outputs is the list of unique aggregations to perform on the input data.
# See https://docs.victoriametrics.com/stream-aggregation/#aggregation-outputs
# See {{% ref "./stream-aggregation.md#aggregation-outputs" %}}
#
outputs: [total]
# keep_metric_names instructs keeping the original metric names for the aggregated samples.
# This option can be set only if outputs list contains only a single output.
# By default, a special suffix is added to original metric names in the aggregated samples.
# See https://docs.victoriametrics.com/stream-aggregation/#output-metric-names
# See {{% ref "./stream-aggregation.md#output-metric-names" %}}
#
# keep_metric_names: false
# ignore_old_samples instructs ignoring input samples with old timestamps outside the current aggregation interval.
# See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples
# See {{% ref "./stream-aggregation.md#ignoring-old-samples" %}}
# See also -remoteWrite.streamAggr.ignoreOldSamples and -streamAggr.ignoreOldSamples command-line flag.
#
# ignore_old_samples: false
# ignore_first_intervals instructs ignoring the first N aggregation intervals after process start.
# See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start
# See {{% ref "./stream-aggregation.md#ignore-aggregation-intervals-on-start" %}}
# See also -remoteWrite.streamAggr.ignoreFirstIntervals and -streamAggr.ignoreFirstIntervals command-line flags.
#
# ignore_first_intervals: N
# drop_input_labels instructs dropping the given labels from input samples.
# The labels' dropping is performed before input_relabel_configs are applied.
# This also means that the labels are dropped before de-duplication ( https://docs.victoriametrics.com/stream-aggregation/#deduplication )
# This also means that the labels are dropped before de-duplication ({{% ref "./stream-aggregation.md#deduplication" %}})
# and stream aggregation.
#
# drop_input_labels: [replica, availability_zone]
@ -1116,7 +1116,7 @@ specified individually per each `-remoteWrite.url`:
# input_relabel_configs is an optional relabeling rules,
# which are applied to the incoming samples after they pass the match filter
# and before being aggregated.
# See https://docs.victoriametrics.com/stream-aggregation/#relabeling
# See {{% ref "./stream-aggregation.md#relabeling" %}}
#
input_relabel_configs:
- target_label: vmaggr
@ -1135,7 +1135,7 @@ per each specified config entry.
### Configuration update
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
[vmagent](./vmagent.md) and [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
support the following approaches for hot reloading stream aggregation configs from `-remoteWrite.streamAggr.config` and `-streamAggr.config`:
* By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process:
@ -1187,23 +1187,23 @@ These issues can be fixed in the following ways:
The following solutions can help reducing memory usage and CPU usage durting streaming aggregation:
- To use more specific `match` filters at [streaming aggregation config](#stream-aggregation-config), so only the really needed
[raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) are aggregated.
[raw samples](./keyConcepts.md#raw-samples) are aggregated.
- To increase aggregation interval by specifying bigger duration for the `interval` option at [streaming aggregation config](#stream-aggregation-config).
- To generate lower number of output time series by using less specific [`by` list](#aggregating-by-labels) or more specific [`without` list](#aggregating-by-labels).
- To drop unneeded long labels in input samples via [input_relabel_configs](#relabeling).
### Cluster mode
If you use [vmagent in cluster mode](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets) for streaming aggregation
If you use [vmagent in cluster mode](./vmagent.md#scraping-big-number-of-targets) for streaming aggregation
then be careful when using [`by` or `without` options](#aggregating-by-labels) or when modifying sample labels
via [relabeling](#relabeling), since incorrect usage may result in duplicates and data collision.
For example, if more than one `vmagent` instance calculates [increase](#increase) for `http_requests_total` metric
with `by: [path]` option, then all the `vmagent` instances will aggregate samples to the same set of time series with different `path` labels.
The proper fix would be [adding an unique label](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics) for all the output samples
produced by each `vmagent`, so they are aggregated into distinct sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series).
The proper fix would be [adding an unique label](./vmagent.md#adding-labels-to-metrics) for all the output samples
produced by each `vmagent`, so they are aggregated into distinct sets of [time series](./keyConcepts.md#time-series).
These time series then can be aggregated later as needed during querying.
If `vmagent` instances run in Docker or Kubernetes, then you can refer `POD_NAME` or `HOSTNAME` environment variables
as an unique label value per each `vmagent` via `-remoteWrite.label=vmagent=%{HOSTNAME}` command-line flag.
See [these docs](https://docs.victoriametrics.com/#environment-variables) on how to refer environment variables in VictoriaMetrics components.
See [these docs](./#environment-variables) on how to refer environment variables in VictoriaMetrics components.

112
docs/url-examples.md
View file

@ -72,8 +72,8 @@ The expected output should return [HTTP Status 204](https://datatracker.ietf.org
Additional information:
* [How to delete time series](https://docs.victoriametrics.com/#how-to-delete-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to delete time series](./#how-to-delete-time-series)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/export
@ -95,10 +95,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export -d 'match[]=vm_htt
Additional information:
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export data in JSON line format](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to export time series](./#how-to-export-time-series)
* [How to import time series](./#how-to-import-time-series-data)
* [How to export data in JSON line format](./#how-to-export-data-in-json-line-format)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/export/csv
@ -120,9 +120,9 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/csv -d 'format=__n
Additional information:
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to export time series](./#how-to-export-time-series)
* [How to import time series](./#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/export/native
@ -144,9 +144,9 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/export/native -d 'match[]
More information:
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to export time series](./#how-to-export-time-series)
* [How to import time series](./#how-to-import-time-series-data)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/import
@ -168,9 +168,9 @@ curl -H 'Content-Type: application/json' --data-binary "@filename.json" -X POST
More information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to import time series](./#how-to-import-time-series-data)
* [How to export time series](./#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/import/csv
@ -192,9 +192,9 @@ curl -d "GOOG,1.23,4.56,NYSE" 'http://<vminsert>:8480/insert/0/prometheus/api/v1
Additional information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to import time series](./#how-to-import-time-series-data)
* [How to export time series](./#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/import/native
@ -214,9 +214,9 @@ curl -X POST http://<vminsert>:8480/insert/0/prometheus/api/v1/import/native -T
Additional information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to import time series](./#how-to-import-time-series-data)
* [How to export time series](./#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/import/prometheus
@ -237,9 +237,9 @@ curl -d 'metric_name{foo="bar"} 123' -X POST http://<vminsert>:8480/insert/0/pro
Additional information:
* [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
* [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to import time series](./#how-to-import-time-series-data)
* [How to export time series](./#how-to-export-time-series)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/labels
@ -259,13 +259,13 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/labels
```
By default, VictoriaMetrics returns labels seen during the last day starting at 00:00 UTC because of performance reasons.
An arbitrary time range can be set via [`start` and `end` query args](https://docs.victoriametrics.com/#timestamp-formats).
An arbitrary time range can be set via [`start` and `end` query args](./#timestamp-formats).
The specified `start..end` time range is rounded to UTC day granularity because of performance reasons.
Additional information:
* [Getting label names](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/label/.../values
@ -289,8 +289,8 @@ The specified `start..end` time range is rounded to UTC day granularity because
Additional information:
* [Querying label values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/query
@ -311,10 +311,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/query -d 'query=vm_http_r
Additional information:
* [Instant queries](https://docs.victoriametrics.com/keyconcepts/#instant-query)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Query language](https://docs.victoriametrics.com/keyconcepts/#metricsql)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Instant queries](./keyConcepts.md#instant-query)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [Query language](./keyConcepts.md#metricsql)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/query_range
@ -335,10 +335,10 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/query_range -d 'query=sum
Additional information:
* [Range queries](https://docs.victoriametrics.com/keyconcepts/#range-query)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [Query language](https://docs.victoriametrics.com/keyconcepts/#metricsql)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Range queries](./keyConcepts.md#range-query)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [Query language](./keyConcepts.md#metricsql)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /api/v1/series
@ -363,8 +363,8 @@ The specified `start..end` time range is rounded to UTC day granularity because
Additional information:
* [Finding series by label matchers](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
VictoriaMetrics accepts `limit` query arg for `/api/v1/series` handlers for limiting the number of returned entries. For example, the query to `/api/v1/series?limit=5` returns a sample of up to 5 series, while ignoring the rest. If the provided `limit` value exceeds the corresponding `-search.maxSeries` command-line flag values, then limits specified in the command-line flags are used.
### /api/v1/status/tsdb
@ -387,8 +387,8 @@ curl http://<vmselect>:8481/select/0/prometheus/api/v1/status/tsdb
Additional information:
* [TSDB Stats](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats)
* [Prometheus querying API usage](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Prometheus querying API usage](./#prometheus-querying-api-usage)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /datadog
@ -464,8 +464,8 @@ echo '
Additional information:
* [How to send data from DataDog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to send data from DataDog agent](./#how-to-send-data-from-datadog-agent)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /datadog/api/v2/series
@ -532,8 +532,8 @@ echo '
Additional information:
* [How to send data from DataDog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to send data from DataDog agent](./#how-to-send-data-from-datadog-agent)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /federate
@ -555,9 +555,9 @@ curl http://<vmselect>:8481/select/0/prometheus/federate -d 'match[]=vm_http_req
Additional information:
* [Federation](https://docs.victoriametrics.com/#federation)
* [Federation](./#federation)
* [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/#configuring-federation)
* [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [URL format for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#url-format)
### /graphite/metrics/find
@ -580,9 +580,9 @@ curl http://<vmselect>:8481/select/0/graphite/metrics/find -d 'query=vm_http_req
Additional information:
* [Metrics find API in Graphite](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
* [Graphite API in VictoriaMetrics](https://docs.victoriametrics.com/#graphite-api-usage)
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [URL Format](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [Graphite API in VictoriaMetrics](./#graphite-api-usage)
* [How to send Graphite data to VictoriaMetrics](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [URL Format](./Cluster-VictoriaMetrics.md#url-format)
### /influx/write
@ -604,12 +604,12 @@ curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST htt
Additional information:
* [How to send Influx data to VictoriaMetrics](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
* [URL Format](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format)
* [How to send Influx data to VictoriaMetrics](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
* [URL Format](./Cluster-VictoriaMetrics.md#url-format)
### /internal/resetRollupResultCache
**Resets the response cache for previously served queries. It is recommended to invoke after [backfilling](https://docs.victoriametrics.com/#backfilling) procedure.**
**Resets the response cache for previously served queries. It is recommended to invoke after [backfilling](./#backfilling) procedure.**
Single-node VictoriaMetrics:
@ -670,7 +670,7 @@ curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"m
Additional information:
* [OpenTSDB http put API](http://opentsdb.net/docs/build/html/api_http/put.html)
* [How to send data OpenTSDB data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-opentsdb-compatible-agents)
* [How to send data OpenTSDB data to VictoriaMetrics](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-opentsdb-compatible-agents)
#### How to send Graphite data to VictoriaMetrics
@ -692,5 +692,5 @@ echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" | nc -N http://<vminse
Additional information:
* [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [Multitenancy in cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
* [How to send Graphite data to VictoriaMetrics](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
* [Multitenancy in cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#multitenancy)

464
docs/vmagent.md
View file

File diff suppressed because it is too large Load diff

142
docs/vmalert.md
View file

@ -18,8 +18,8 @@ protocol and require `-remoteWrite.url` to be configured.
Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
implementation and aims to be compatible with its syntax.
A [single-node](https://docs.victoriametrics.com/single-server-victoriametrics/#vmalert)
or [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/#vmalert)
A [single-node](./Single-Server-VictoriaMetrics.md#vmalert)
or [cluster version](./Cluster-VictoriaMetrics.md#vmalert)
of VictoriaMetrics are capable of proxying requests to vmalert via `-vmalert.proxyURL` command-line flag.
Use this feature for the following cases:
* for proxying requests from [Grafana Alerting UI](https://grafana.com/docs/grafana/latest/alerting/);
@ -28,7 +28,7 @@ Use this feature for the following cases:
## Features
* Integration with [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) TSDB;
* VictoriaMetrics [MetricsQL](https://docs.victoriametrics.com/metricsql/)
* VictoriaMetrics [MetricsQL](./MetricsQL.md)
support and expressions validation;
* Prometheus [alerting rules definition format](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#defining-alerting-rules)
support;
@ -92,7 +92,7 @@ alerting/recording rules into a different storage than the initial data that's
queried. This allows using `vmalert` to aggregate data from a short-term,
high-frequency, high-cardinality storage into a long-term storage with
decreased cardinality and a bigger interval between samples.
See also [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/).
See also [stream aggregation](./stream-aggregation.md).
See the full list of configuration flags in [configuration](#configuration) section.
@ -136,7 +136,7 @@ name: <string>
# Adjust the `time` parameter of group evaluation requests to compensate intentional query delay from the datasource.
# By default, the value is inherited from the `-rule.evalDelay` cmd-line flag - see its description for details.
# If group has `latency_offset` set in `params`, then it is recommended to set `eval_delay` equal to `latency_offset`.
# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and https://docs.victoriametrics.com/keyconcepts/#query-latency.
# See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155 and {{% ref "./keyConcepts.md#query-latency" %}}.
[ eval_delay: <duration> ]
# Limit limits the number of alerts or recording results the rule within this group can produce.
@ -171,7 +171,7 @@ name: <string>
# nocache: ["1"] # disable caching for vmselect
# denyPartialResponse: ["true"] # fail if one or more vmstorage nodes returned an error
# extra_label: ["env=dev"] # apply additional label filter "env=dev" for all requests
# see more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements
# see more details at {{% ref "./#prometheus-querying-api-enhancements" %}}
params:
[ <string>: [<string>, ...]]
@ -208,7 +208,7 @@ rules:
### Rules
Every rule contains `expr` field for [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/)
or [MetricsQL](https://docs.victoriametrics.com/metricsql/) expression. `vmalert` will execute the configured
or [MetricsQL](./MetricsQL.md) expression. `vmalert` will execute the configured
expression and then act according to the Rule type.
There are two types of Rules:
@ -252,13 +252,13 @@ expr: <string>
# Please note, that if rule's query params contain sensitive
# information - it will be printed to logs.
# Is applicable to alerting rules only.
# Available starting from https://docs.victoriametrics.com/changelog/#v1820
# Available starting from {{% ref "./CHANGELOG.md#v1820" %}}
[ debug: <bool> | default = false ]
# Defines the number of rule's updates entries stored in memory
# and available for view on rule's Details page.
# Overrides `rule.updateEntriesLimit` value for this specific rule.
# Available starting from https://docs.victoriametrics.com/changelog/#v1860
# Available starting from {{% ref "./CHANGELOG.md#v1860" %}}
[ update_entries_limit: <integer> | default 0 ]
# Labels to add or overwrite for each alert.
@ -312,7 +312,7 @@ Additionally, `vmalert` provides some extra templating functions listed [here](#
- `parseDurationTime` - parses the input string into [time.Duration](https://pkg.go.dev/time#Duration).
- `pathEscape` - escapes the input string, so it can be safely put inside path part of URL.
- `pathPrefix` - returns the path part of the `-external.url` command-line flag.
- `query` - executes the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query against `-datasource.url` and returns the query result.
- `query` - executes the [MetricsQL](./MetricsQL.md) query against `-datasource.url` and returns the query result.
For example, `{{ query "sort_desc(process_resident_memory_bytes)" | first | value }}` executes the `sort_desc(process_resident_memory_bytes)`
query at `-datasource.url` and returns the first result.
- `queryEscape` - escapes the input string, so it can be safely put inside [query arg](https://en.wikipedia.org/wiki/Percent-encoding) part of URL.
@ -402,7 +402,7 @@ in the memory. To prevent `vmalert` from losing the state on restarts configure
to the remote database via the following flags:
* `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
to the configured address in the form of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series)
to the configured address in the form of [time series](./keyConcepts.md#time-series)
`ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
These time series can be queried from VictoriaMetrics just as any other time series.
The state will be persisted to the configured address on each evaluation.
@ -420,9 +420,9 @@ with `restored` label in [web UI](#web).
Alerting notifications sent by vmalert always contain a `source` link. By default, the link format
is the following `http://<vmalert-addr>/vmalert/alert?group_id=<group_id>&alert_id=<alert_id>`. On click, it opens
vmalert [web UI](https://docs.victoriametrics.com/vmalert/#web) to show the alert status and its fields.
vmalert [web UI](./vmalert.md#web) to show the alert status and its fields.
It is possible to override the link format. For example, to make the link to [vmui](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui)
It is possible to override the link format. For example, to make the link to [vmui](./Single-Server-VictoriaMetrics.md#vmui)
specify the following cmd-line flags:
```
./bin/vmalert \
@ -433,7 +433,7 @@ specify the following cmd-line flags:
Now, all `source` links will lead to `http://<vmui-addr>/vmui/#/?g0.expr=$expr`, where $expr is an alerting rule
expression.
The `-external.alert.source` cmd-line flag supports [templating](https://docs.victoriametrics.com/vmalert/#templating)
The `-external.alert.source` cmd-line flag supports [templating](./vmalert.md#templating)
and allows using labels and extra data related to the alert. For example, see the following link to Grafana:
```
./bin/vmalert \
@ -444,18 +444,18 @@ and allows using labels and extra data related to the alert. For example, see th
In this example, `-external.alert.source` will lead to Grafana's Explore page with `expr` field equal to alert expression,
and time range will be selected starting from `"from":"{{ .ActiveAt.UnixMilli }}"` when alert became active.
In addition to `source` link, some extra links could be added to alert's [annotations](https://docs.victoriametrics.com/vmalert/#alerting-rules)
In addition to `source` link, some extra links could be added to alert's [annotations](./vmalert.md#alerting-rules)
field. See [how we use them](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/839596c00df123c639d1244b28ee8137dfc9609c/deployment/docker/alerts-cluster.yml#L43)
to link alerting rule and the corresponding panel on Grafana dashboard.
### Multitenancy
There are the following approaches exist for alerting and recording rules across
[multiple tenants](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy):
[multiple tenants](./Cluster-VictoriaMetrics.md#multitenancy):
* To run a separate `vmalert` instance per each tenant.
The corresponding tenant must be specified in `-datasource.url` command-line flag
according to [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format).
according to [these docs](./Cluster-VictoriaMetrics.md#url-format).
For example, `/path/to/vmalert -datasource.url=http://vmselect:8481/select/123/prometheus`
would run alerts against `AccountID=123`. For recording rules the `-remoteWrite.url` command-line
flag must contain the url for the specific tenant as well.
@ -463,7 +463,7 @@ There are the following approaches exist for alerting and recording rules across
rules to `AccountID=123`.
* To specify `tenant` parameter per each alerting and recording group if
[enterprise version of vmalert](https://docs.victoriametrics.com/enterprise/) is used
[enterprise version of vmalert](./enterprise.md) is used
with `-clusterMode` command-line flag. For example:
```yaml
@ -480,7 +480,7 @@ groups:
```
The results of alerting and recording rules contain `vm_account_id` and `vm_project_id` labels
if `-clusterMode` is enabled. These labels can be used during [templating](https://docs.victoriametrics.com/vmalert/#templating),
if `-clusterMode` is enabled. These labels can be used during [templating](./vmalert.md#templating),
and help to identify to which account or project the triggered alert or produced recording belongs.
If `-clusterMode` is enabled, then `-datasource.url`, `-remoteRead.url` and `-remoteWrite.url` must
@ -496,7 +496,7 @@ tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags).
### Reading rules from object storage
[Enterprise version](https://docs.victoriametrics.com/enterprise/) of `vmalert` may read alerting and recording rules
[Enterprise version](./enterprise.md) of `vmalert` may read alerting and recording rules
from object storage:
- `./bin/vmalert -rule=s3://bucket/dir/alert.rules` would read rules from the given path at S3 bucket
@ -544,7 +544,7 @@ rules execution, storing recording rules results and alerts state.
#### Cluster VictoriaMetrics
In [cluster mode](https://docs.victoriametrics.com/cluster-victoriametrics/)
In [cluster mode](./Cluster-VictoriaMetrics.md)
VictoriaMetrics has separate components for writing and reading path:
`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions
and `vminsert` is used to persist recording rules results and alerts state.
@ -564,7 +564,7 @@ Cluster mode could have multiple `vminsert` and `vmselect` components.
In case when you want to spread the load on these components - add balancers before them and configure
`vmalert` with balancer addresses. Please, see more about VM's cluster architecture
[here](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
[here](./Cluster-VictoriaMetrics.md#architecture-overview).
#### HA vmalert
@ -588,7 +588,7 @@ Alertmanagers.
{width="800px"}
To avoid recording rules results and alerts state duplication in VictoriaMetrics server
don't forget to configure [deduplication](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication).
don't forget to configure [deduplication](./Single-Server-VictoriaMetrics.md#deduplication).
Multiple equally configured vmalerts should evaluate rules at the same timestamps, so it is recommended
to set `-dedup.minScrapeInterval` as equal to vmalert's `-evaluationInterval`.
@ -613,7 +613,7 @@ Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics)
#### Downsampling and aggregation via vmalert
_Please note, [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) might be more efficient
_Please note, [stream aggregation](./stream-aggregation.md) might be more efficient
for cases when downsampling or aggregation need to be applied **before data gets into the TSDB.**_
`vmalert` can't modify existing data. But it can run arbitrary PromQL/MetricsQL queries
@ -667,21 +667,21 @@ Please note, [replay](#rules-backfilling) feature may be used for transforming h
Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used.
See also [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) and [downsampling](https://docs.victoriametrics.com/#downsampling).
See also [stream aggregation](./stream-aggregation.md) and [downsampling](downsampling).
#### Multiple remote writes
For persisting recording or alerting rule results `vmalert` requires `-remoteWrite.url` to be set.
But this flag supports only one destination. To persist rule results to multiple destinations
we recommend using [vmagent](https://docs.victoriametrics.com/vmagent/) as fan-out proxy:
we recommend using [vmagent](./vmagent.md) as fan-out proxy:
![vmalert multiple remote write destinations](vmalert_multiple_rw.webp)
In this topology, `vmalert` is configured to persist rule results to `vmagent`. And `vmagent`
is configured to fan-out received data to two or more destinations.
Using `vmagent` as a proxy provides additional benefits such as
[data persisting when storage is unreachable](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability),
or time series modification via [relabeling](https://docs.victoriametrics.com/vmagent/#relabeling).
[data persisting when storage is unreachable](./vmagent.md#replication-and-high-availability),
or time series modification via [relabeling](./vmagent.md#relabeling).
### Web
@ -699,12 +699,12 @@ or time series modification via [relabeling](https://docs.victoriametrics.com/vm
* `http://<vmalert-addr>/metrics` - application metrics.
* `http://<vmalert-addr>/-/reload` - hot configuration reload.
`vmalert` web UI can be accessed from [single-node version of VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and from [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/).
`vmalert` web UI can be accessed from [single-node version of VictoriaMetrics](./Single-Server-VictoriaMetrics.md)
and from [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md).
This may be used for better integration with Grafana unified alerting system. See the following docs for details:
* [How to query vmalert from single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#vmalert)
* [How to query vmalert from VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#vmalert)
* [How to query vmalert from single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md#vmalert)
* [How to query vmalert from VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#vmalert)
## Graphite
@ -724,7 +724,7 @@ data source for backfilling.
Please note, that response caching may lead to unexpected results during and after backfilling process.
In order to avoid this you need to reset cache contents or disable caching when using backfilling
as described in [backfilling docs](https://docs.victoriametrics.com/single-server-victoriametrics/#backfilling).
as described in [backfilling docs](./Single-Server-VictoriaMetrics.md#backfilling).
See a blogpost about [Rules backfilling via vmalert](https://victoriametrics.com/blog/rules-replay/).
@ -771,7 +771,7 @@ max range per request: 8h20m0s
In `replay` mode all groups are executed sequentially one-by-one. Rules within the group are
executed sequentially as well (`concurrency` setting is ignored). vmalert sends rule's expression
to [/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) endpoint
to [/query_range](./keyConcepts.md#range-query) endpoint
of the configured `-datasource.url`. Returned data is then processed according to the rule type and
backfilled to `-remoteWrite.url` via [remote Write protocol](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations).
vmalert respects `evaluationInterval` value set by flag or per-group during the replay.
@ -822,13 +822,13 @@ See full description for these flags in `./vmalert -help`.
## Unit Testing for Rules
You can use `vmalert-tool` to test your alerting and recording rules like [promtool does](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/).
See more details [here](https://docs.victoriametrics.com/vmalert-tool.html#Unit-testing-for-rules).
See more details [here](./vmalert-tool.md#Unit-testing-for-rules).
## Monitoring
`vmalert` exports various metrics in Prometheus exposition format at `http://vmalert-host:8880/metrics` page.
The default list of alerting rules for these metric can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker).
We recommend setting up regular scraping of this page either through [vmagent](https://docs.victoriametrics.com/vmagent/) or by Prometheus-compatible scraper,
We recommend setting up regular scraping of this page either through [vmagent](./vmagent.md) or by Prometheus-compatible scraper,
so that the exported metrics may be analyzed later.
If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass `-metrics.exposeMetadata`
@ -857,19 +857,19 @@ may get empty response from the datasource and produce empty recording rules or
Try the following recommendations to reduce the chance of hitting the data delay issue:
* Always configure group's `-evaluationInterval` to be bigger or at least equal to
[time series resolution](https://docs.victoriametrics.com/keyconcepts/#time-series-resolution);
[time series resolution](./keyConcepts.md#time-series-resolution);
* Ensure that `[duration]` value is at least twice bigger than
[time series resolution](https://docs.victoriametrics.com/keyconcepts/#time-series-resolution). For example,
[time series resolution](./keyConcepts.md#time-series-resolution). For example,
if expression is `rate(my_metric[2m]) > 0` then ensure that `my_metric` resolution is at least `1m` or better `30s`.
* Extend `[duration]` in expr to help tolerate the delay. For example, `max_over_time(errors_total[10m]) > 0` will be active even if there is no data in datasource for last `9m`.
* If [time series resolution](https://docs.victoriametrics.com/keyconcepts/#time-series-resolution)
* If [time series resolution](./keyConcepts.md#time-series-resolution)
in datasource is inconsistent or `>=5min` - try changing vmalerts `-datasource.queryStep` command-line flag to specify
how far search query can look back for the recent datapoint. The recommendation is to have the step
at least two times bigger than the resolution.
> Please note, data delay is inevitable in distributed systems. And it is better to account for it instead of ignoring.
By default, recently written samples to VictoriaMetrics [aren't visible for queries](https://docs.victoriametrics.com/keyconcepts/#query-latency)
By default, recently written samples to VictoriaMetrics [aren't visible for queries](./keyConcepts.md#query-latency)
for up to 30s (see `-search.latencyOffset` command-line flag at vmselect or VictoriaMetrics single-node). Such delay is needed to eliminate risk of
incomplete data on the moment of querying, due to chance that metrics collectors won't be able to deliver that data in time.
To compensate the latency in timestamps for produced evaluation results, `-rule.evalDelay` is also set to `30s` by default.
@ -885,11 +885,11 @@ becomes false, then alert's state resets to the initial state.
If `-remoteWrite.url` command-line flag is configured, vmalert will persist alert's state in form of time series
`ALERTS` and `ALERTS_FOR_STATE` to the specified destination. Such time series can be then queried via
[vmui](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) or Grafana to track how alerts state
[vmui](./Single-Server-VictoriaMetrics.md#vmui) or Grafana to track how alerts state
changed in time.
vmalert stores last `-rule.updateEntriesLimit` (or `update_entries_limit` [per-rule config](https://docs.victoriametrics.com/vmalert/#alerting-rules))
state updates for each rule starting from [v1.86](https://docs.victoriametrics.com/changelog/#v1860).
vmalert stores last `-rule.updateEntriesLimit` (or `update_entries_limit` [per-rule config](./vmalert.md#alerting-rules))
state updates for each rule starting from [v1.86](./CHANGELOG.md#v1860).
To check updates, click on `Details` link next to rule's name on `/vmalert/groups` page
and check the `Last updates` section:
@ -903,7 +903,7 @@ for more details.
### Debug mode
vmalert allows configuring more detailed logging for specific alerting rule starting from [v1.82](https://docs.victoriametrics.com/changelog/#v1820).
vmalert allows configuring more detailed logging for specific alerting rule starting from [v1.82](./CHANGELOG.md#v1820).
Just set `debug: true` in rule's configuration and vmalert will start printing additional log messages:
```shell-session
2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms)
@ -920,7 +920,7 @@ Sensitive info is stripped from the `curl` examples - see [security](#security)
### Never-firing alerts
vmalert can detect if alert's expression doesn't match any time series in runtime
starting from [v1.91](https://docs.victoriametrics.com/changelog/#v1910). This problem usually happens
starting from [v1.91](./CHANGELOG.md#v1910). This problem usually happens
when alerting expression selects time series which aren't present in the datasource (i.e. wrong `job` label)
or there is a typo in the series selector (i.e. `env=prod`). Such alerting rules will be marked with special icon in
vmalerts UI and exposed via `vmalert_alerting_rules_last_evaluation_series_fetched` metric. The metric value will
@ -941,7 +941,7 @@ vmalert can produce the following error message:
result contains metrics with the same labelset during evaluation
```
The error means there is a collision between [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series)
The error means there is a collision between [time series](./keyConcepts.md#time-series)
during evaluation.
For example, a rule with `expr: {__name__=~"vmalert_alerts_.*"} > 0` returns two distinct time series in response:
@ -951,13 +951,13 @@ For example, a rule with `expr: {__name__=~"vmalert_alerts_.*"} > 0` returns two
```
As label `__name__` will be dropped during evaluation, leads to duplicated time series.
To fix this, one could use function like [label_replace](https://docs.victoriametrics.com/metricsql/#label_replace) to preserve the distinct labelset.
To fix this, one could use function like [label_replace](./MetricsQL.md#label_replace) to preserve the distinct labelset.
## mTLS protection
By default `vmalert` accepts http requests at `8880` port (this port can be changed via `-httpListenAddr` command-line flags),
since it is expected it runs in an isolated trusted network.
[Enterprise version of vmagent](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
[Enterprise version of vmagent](./enterprise.md) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `vmalert`, which accepts only mTLS requests at port `8880`:
```
@ -969,7 +969,7 @@ It is possible to specify custom TLS Root CA via `-mtlsCAFile` command-line flag
## Security
See general recommendations regarding security [here](https://docs.victoriametrics.com/single-server-victoriametrics/#security).
See general recommendations regarding security [here](./Single-Server-VictoriaMetrics.md#security).
vmalert [web UI](#web) exposes configuration details such as list of [Groups](#groups), active alerts,
[alerts state](#alerts-state), [notifiers](#notifier-configuration-file). Notifier addresses (sanitized) are attached
@ -1018,7 +1018,7 @@ The shortlist of configuration flags is the following:
```sh
-clusterMode
If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See https://docs.victoriametrics.com/vmalert/#multitenancy . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
If clusterMode is enabled, then vmalert automatically adds the tenant specified in config groups to -datasource.url, -remoteWrite.url and -remoteRead.url. See {{% ref "./vmalert.md#multitenancy" %}}. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-configCheckInterval duration
Interval for checking for changes in '-rule' or '-notifier.config' files. By default, the checking is disabled. Send SIGHUP signal in order to force config check for changes.
-datasource.appendTypePrefix
@ -1078,9 +1078,9 @@ The shortlist of configuration flags is the following:
-datasource.url string
Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. Supports address in the form of IP address with a port (e.g., 127.0.0.1:8428) or DNS SRV record. See also -remoteRead.disablePathAppend and -datasource.showURL
-defaultTenant.graphite string
Default tenant for Graphite alerting groups. See https://docs.victoriametrics.com/vmalert/#multitenancy .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Default tenant for Graphite alerting groups. See {{% ref "./vmalert.md#multitenancy" %}}.This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-defaultTenant.prometheus string
Default tenant for Prometheus alerting groups. See https://docs.victoriametrics.com/vmalert/#multitenancy . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Default tenant for Prometheus alerting groups. See {{% ref "./vmalert.md#multitenancy" %}}. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-disableAlertgroupLabel
Whether to disable adding group's Name as label to generated alerts and time series.
-dryRun
@ -1088,15 +1088,15 @@ The shortlist of configuration flags is the following:
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See environment-variables for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-evaluationInterval duration
How often to evaluate the rules (default 1m0s)
-external.alert.source string
External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see https://docs.victoriametrics.com/vmalert/#templating . For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
External Alert Source allows to override the Source link for alerts sent to AlertManager for cases where you want to build a custom link to Grafana, Prometheus or any other service. Supports templating - see {{% ref "./vmalert.md#templating" %}}. For example, link to Grafana: -external.alert.source='explore?orgId=1&left={"datasource":"VictoriaMetrics","queries":[{"expr":{{.Expr|jsonEscape|queryEscape}},"refId":"A"}],"range":{"from":"now-1h","to":"now"}}'. Link to VMUI: -external.alert.source='vmui/#/?g0.expr={{.Expr|queryEscape}}'. If empty 'vmalert/alert?group_id={{.GroupID}}&alert_id={{.AlertID}}' is used.
-external.label array
Optional label in the form 'Name=value' to add to all generated recording rules and alerts. Pass multiple -label flags in order to add multiple label sets.
Supports an array of values separated by comma or specified via multiple flags.
@ -1182,11 +1182,11 @@ The shortlist of configuration flags is the following:
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-mtlsCAFile array
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-notifier.basicAuth.password array
@ -1271,13 +1271,13 @@ The shortlist of configuration flags is the following:
-promscrape.consul.waitTime duration
Wait time used by Consul service discovery. Default value is used if not set
-promscrape.consulSDCheckInterval duration
Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#consul_sd_configs for details (default 30s)
Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#consul_sd_configs" %}} for details (default 30s)
-promscrape.discovery.concurrency int
The maximum number of concurrent requests to Prometheus autodiscovery API (Consul, Kubernetes, etc.) (default 100)
-promscrape.discovery.concurrentWaitTime duration
The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s)
-promscrape.dnsSDCheckInterval duration
Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#dns_sd_configs for details (default 30s)
Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#dns_sd_configs" %}} for details (default 30s)
-pushmetrics.disableCompression
Whether to disable request body compression when pushing metrics to every -pushmetrics.url
-pushmetrics.extraLabel array
@ -1291,7 +1291,7 @@ The shortlist of configuration flags is the following:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-reloadAuthKey value
@ -1433,7 +1433,7 @@ The shortlist of configuration flags is the following:
-rule.resendDelay duration
MiniMum amount of time to wait before resending an alert to notifier
-rule.stripFilePath
Whether to strip file path in responses from the api/v1/rules API for files configured via -rule cmd-line flag. For example, the file path '/path/to/tenant_id/rules.yml' will be stripped to just 'rules.yml'. This flag might be useful to hide sensitive information in file path such as tenant ID. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to strip file path in responses from the api/v1/rules API for files configured via -rule cmd-line flag. For example, the file path '/path/to/tenant_id/rules.yml' will be stripped to just 'rules.yml'. This flag might be useful to hide sensitive information in file path such as tenant ID. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-rule.templates array
Path or glob pattern to location with go template definitions for rules annotations templating. Flag can be specified multiple times.
Examples:
@ -1451,26 +1451,26 @@ The shortlist of configuration flags is the following:
Whether to validate annotation and label templates (default true)
-s3.configFilePath string
Path to file with S3 configs. Configs are loaded from default location if not set.
See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
See https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-s3.configProfile string
Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Profile name for S3 configs. If no set, the value of the environment variable will be loaded (AWS_PROFILE or AWS_DEFAULT_PROFILE), or if both not set, DefaultSharedConfigProfile is used. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-s3.credsFilePath string
Path to file with GCS or S3 credentials. Credentials are loaded from default locations if not set.
See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-s3.customEndpoint string
Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-s3.forcePathStyle
Prefixing endpoint with bucket name when set false, true by default. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ (default true)
Prefixing endpoint with bucket name when set false, true by default. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} (default true)
-tls array
Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsAutocertCacheDir string
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertEmail string
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertHosts array
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCertFile array
@ -1646,13 +1646,13 @@ dns_sd_configs:
# List of relabel configurations for entities discovered via service discovery.
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
relabel_configs:
[ - <relabel_config> ... ]
# List of relabel configurations for alert labels sent via Notifier.
# Supports the same relabeling features as the rest of VictoriaMetrics components.
# See https://docs.victoriametrics.com/vmagent/#relabeling
# See {{% ref "./vmagent.md#relabeling" %}}
alert_relabel_configs:
[ - <relabel_config> ... ]
```

114
docs/vmauth.md
View file

@ -9,8 +9,8 @@ title: vmauth
aliases:
- /vmauth.html
---
`vmauth` is an HTTP proxy, which can [authorize](https://docs.victoriametrics.com/vmauth/#authorization), [route](https://docs.victoriametrics.com/vmauth/#routing)
and [load balance](https://docs.victoriametrics.com/vmauth/#load-balancing) requests across [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) components
`vmauth` is an HTTP proxy, which can [authorize](./vmauth.md#authorization), [route](./vmauth.md#routing)
and [load balance](./vmauth.md#load-balancing) requests across [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) components
or any other HTTP backends.
## Quick start
@ -35,7 +35,7 @@ See how `vmauth` used in [docker-compose env](https://github.com/VictoriaMetrics
Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions.
Feel free [contacting us](mailto:info@victoriametrics.com) if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML,
accounting and rate limiting such as [vmgateway](https://docs.victoriametrics.com/vmgateway/).
accounting and rate limiting such as [vmgateway](./vmgateway.md).
## Use cases
@ -134,7 +134,7 @@ See also [authorization](#authorization) and [routing](#routing) docs.
### Load balancer for vmagent
If [vmagent](https://docs.victoriametrics.com/vmagent/) is used for processing [data push requests](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent),
If [vmagent](./vmagent.md) is used for processing [data push requests](./vmagent.md#how-to-push-data-to-vmagent),
then it is possible to scale the performance of data processing at `vmagent` by spreading load among multiple identically configured `vmagent` instances.
This can be done with the following [config](#auth-config) for `vmauth`:
@ -158,8 +158,8 @@ See also [authorization](#authorization) and [routing](#routing) docs.
### Load balancer for VictoriaMetrics cluster
[VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) accepts incoming data via `vminsert` nodes
and processes incoming requests via `vmselect` nodes according to [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
[VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) accepts incoming data via `vminsert` nodes
and processes incoming requests via `vmselect` nodes according to [these docs](./Cluster-VictoriaMetrics.md#architecture-overview).
`vmauth` can be used for balancing both `insert` and `select` requests among `vminsert` and `vmselect` nodes, when the following [`-auth.config`](#auth-config) is used:
```yaml
@ -223,7 +223,7 @@ See also [authorization](#authorization), [routing](#routing) and [load balancin
### Basic Auth proxy
`vmauth` can authorize access to backends depending on the provided [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication) request headers.
For example, the following [config](#auth-config) proxies requests to [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
For example, the following [config](#auth-config) proxies requests to [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md)
if they contain Basic Auth header with the given `username` and `password`:
```yaml
@ -238,7 +238,7 @@ See also [authorization](#authorization), [routing](#routing) and [load balancin
### Bearer Token auth proxy
`vmauth` can authorize access to backends depending on the provided `Bearer Token` request headers.
For example, the following [config](#auth-config) proxies requests to [single-node VictoriaMetrics](https://docs.victoriametrics.com/)
For example, the following [config](#auth-config) proxies requests to [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md)
if they contain the given `bearer_token`:
```yaml
@ -252,7 +252,7 @@ See also [authorization](#authorization), [routing](#routing) and [load balancin
### Per-tenant authorization
The following [`-auth.config`](#auth-config) instructs proxying `insert` and `select` requests from the [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication)
user `tenant1` to the [tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) `1`,
user `tenant1` to the [tenant](./Cluster-VictoriaMetrics.md#multitenancy) `1`,
while requests from the user `tenant2` are sent to tenant `2`:
```yaml
@ -289,7 +289,7 @@ See also [authorization](#authorization), [routing](#routing) and [load balancin
### mTLS-based request routing
[Enterprise version of `vmauth`](https://docs.victoriametrics.com/enterprise/) can be configured for routing requests
[Enterprise version of `vmauth`](./enterprise.md) can be configured for routing requests
to different backends depending on the following [subject fields](https://en.wikipedia.org/wiki/Public_key_certificate#Common_fields) in the TLS certificate provided by client:
* `organizational_unit` aka `OU`
@ -317,8 +317,8 @@ See also [authorization](#authorization), [routing](#routing) and [load balancin
### Enforcing query args
`vmauth` can be configured for adding some mandatory query args before proxying requests to backends.
For example, the following [config](#auth-config) adds [`extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements)
to all the requests, which are proxied to [single-node VictoriaMetrics](https://docs.victoriametrics.com/):
For example, the following [config](#auth-config) adds [`extra_label`](./#prometheus-querying-api-enhancements)
to all the requests, which are proxied to [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md):
```yaml
unauthorized_user:
@ -333,8 +333,8 @@ By default `vmauth` doesn't drop the path prefix from the original request when
Sometimes it is needed to drop path prefix before proxying the request to the backend. This can be done by specifying the number of `/`-delimited
prefix parts to drop from the request path via `drop_src_path_prefix_parts` option at `url_map` level or at `user` level or [`-auth.config`](#auth-config).
For example, if you need serving requests to [vmalert](https://docs.victoriametrics.com/vmalert/) at `/vmalert/` path prefix,
while serving requests to [vmagent](https://docs.victoriametrics.com/vmagent/) at `/vmagent/` path prefix,
For example, if you need serving requests to [vmalert](./vmalert.md) at `/vmalert/` path prefix,
while serving requests to [vmagent](./vmagent.md) at `/vmagent/` path prefix,
then the following [-auth.config](#auth-config) can be used:
```yaml
@ -362,11 +362,11 @@ unauthorized_user:
`vmauth` supports the following authorization mechanisms:
- [No authorization](https://docs.victoriametrics.com/vmauth/#simple-http-proxy)
- [Basic Auth](https://docs.victoriametrics.com/vmauth/#basic-auth-proxy)
- [Bearer token](https://docs.victoriametrics.com/vmauth/#bearer-token-auth-proxy)
- [Client TLS certificate verification aka mTLS](https://docs.victoriametrics.com/vmauth/#mtls-based-request-routing)
- [Auth tokens via Arbitrary HTTP request headers](https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers)
- [No authorization](./vmauth.md#simple-http-proxy)
- [Basic Auth](./vmauth.md#basic-auth-proxy)
- [Bearer token](./vmauth.md#bearer-token-auth-proxy)
- [Client TLS certificate verification aka mTLS](./vmauth.md#mtls-based-request-routing)
- [Auth tokens via Arbitrary HTTP request headers](./vmauth.md#reading-auth-tokens-from-other-http-headers)
See also [security docs](#security), [routing docs](#routing) and [load balancing docs](#load-balancing).
@ -553,7 +553,7 @@ Each `url_prefix` in the [-auth.config](#auth-config) can be specified in the fo
Load balancing feature can be used in the following cases:
- Balancing the load among multiple `vmselect` and/or `vminsert` nodes in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/).
- Balancing the load among multiple `vmselect` and/or `vminsert` nodes in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md).
The following [`-auth.config`](#auth-config) can be used for spreading incoming requests among 3 vmselect nodes and re-trying failed requests
or requests with 500 and 502 response status codes:
@ -569,7 +569,7 @@ Load balancing feature can be used in the following cases:
- Sending select queries to the closest availability zone (AZ), while falling back to other AZs with identical data if the closest AZ is unavailable.
For example, the following [`-auth.config`](#auth-config) sends select queries to `https://vmselect-az1/` and uses the `https://vmselect-az2/` as a fallback
when `https://vmselect-az1/` is temporarily unavailable or cannot return full responses.
See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability) for details about `deny_partial_response` query arg,
See [these docs](./Cluster-VictoriaMetrics.md#cluster-availability) for details about `deny_partial_response` query arg,
which is added to requests before they are proxied to backends.
```yaml
@ -589,7 +589,7 @@ See also [discovering backend IPs](#discovering-backend-ips), [authorization](#a
By default `vmauth` spreads load among the listed backends at `url_prefix` as described in [load balancing docs](#load-balancing).
Sometimes multiple backend instances can be hidden behind a single hostname. For example, `vmselect-service` hostname
may point to a cluster of `vmselect` instances in [VictoriaMetrics cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview).
may point to a cluster of `vmselect` instances in [VictoriaMetrics cluster setup](./Cluster-VictoriaMetrics.md#architecture-overview).
So the following config may fail spreading load among available `vmselect` instances, since `vmauth` will send all the requests to the same url, which may end up
to a single backend instance:
@ -800,7 +800,7 @@ The file is checked for modifications every second and is automatically re-read
## IP filters
[Enterprise version](https://docs.victoriametrics.com/enterprise/) of `vmauth` can be configured to allow / deny incoming requests via global and per-user IP filters.
[Enterprise version](./enterprise.md) of `vmauth` can be configured to allow / deny incoming requests via global and per-user IP filters.
For example, the following config allows requests to `vmauth` from `10.0.0.0/24` network and from `1.2.3.4` IP address, while denying requests from `10.0.0.42` IP address:
@ -999,7 +999,7 @@ This may be useful for passing secrets to the config.
## mTLS protection
By default `vmauth` accepts http requests at `8427` port (this port can be changed via `-httpListenAddr` command-line flags).
[Enterprise version of vmauth](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
[Enterprise version of vmauth](./enterprise.md) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)
requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `vmauth`, which accepts only mTLS requests at port `8427`:
```
@ -1042,7 +1042,7 @@ It is recommended protecting the following endpoints with authKeys:
## Automatic issuing of TLS certificates
`vmauth` [Enterprise](https://docs.victoriametrics.com/enterprise/) supports automatic issuing of TLS certificates via [Let's Encrypt service](https://letsencrypt.org/).
`vmauth` [Enterprise](./enterprise.md) supports automatic issuing of TLS certificates via [Let's Encrypt service](https://letsencrypt.org/).
The following command-line flags must be set in order to enable automatic issuing of TLS certificates:
- `-httpListenAddr` must be set for listening TCP port `443`. For example, `-httpListenAddr=:443`. This port must be accessible by the [Let's Encrypt service](https://letsencrypt.org/).
@ -1052,14 +1052,14 @@ The following command-line flags must be set in order to enable automatic issuin
- `-tlsAutocertCacheDir` may be set to the directory path for persisting the issued TLS certificates between `vmauth` restarts. If this flag isn't set,
then TLS certificates are re-issued on every restart.
This functionality can be evaluated for free according to [these docs](https://docs.victoriametrics.com/enterprise/).
This functionality can be evaluated for free according to [these docs](./enterprise.md).
See also [security recommendations](#security).
## Monitoring
`vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page
either via [vmagent](https://docs.victoriametrics.com/vmagent/) or via Prometheus-compatible scraper, so the exported metrics could be analyzed later.
either via [vmagent](./vmagent.md) or via Prometheus-compatible scraper, so the exported metrics could be analyzed later.
Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/21394) and [alerting rules](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-vmauth.yml)
for `vmauth` monitoring.
@ -1069,14 +1069,14 @@ See [these docs](https://cloud.google.com/stackdriver/docs/managed-prometheus/tr
`vmauth` exports the following metrics per each defined user in [`-auth.config`](#auth-config):
* `vmauth_user_requests_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of requests served for the given `username`
* `vmauth_user_request_backend_errors_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of request errors for the given `username`
* `vmauth_user_request_duration_seconds` [summary](https://docs.victoriametrics.com/keyconcepts/#summary) - the duration of requests for the given `username`
* `vmauth_user_concurrent_requests_limit_reached_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of failed requests
* `vmauth_user_requests_total` [counter](./keyConcepts.md#counter) - the number of requests served for the given `username`
* `vmauth_user_request_backend_errors_total` [counter](./keyConcepts.md#counter) - the number of request errors for the given `username`
* `vmauth_user_request_duration_seconds` [summary](./keyConcepts.md#summary) - the duration of requests for the given `username`
* `vmauth_user_concurrent_requests_limit_reached_total` [counter](./keyConcepts.md#counter) - the number of failed requests
for the given `username` because of exceeded [concurrency limits](#concurrency-limiting)
* `vmauth_user_concurrent_requests_capacity` [gauge](https://docs.victoriametrics.com/keyconcepts/#gauge) - the maximum number of [concurrent requests](#concurrency-limiting)
* `vmauth_user_concurrent_requests_capacity` [gauge](./keyConcepts.md#gauge) - the maximum number of [concurrent requests](#concurrency-limiting)
for the given `username`
* `vmauth_user_concurrent_requests_current` [gauge](https://docs.victoriametrics.com/keyconcepts/#gauge) - the current number of [concurrent requests](#concurrency-limiting)
* `vmauth_user_concurrent_requests_current` [gauge](./keyConcepts.md#gauge) - the current number of [concurrent requests](#concurrency-limiting)
for the given `username`
By default, per-user metrics contain only `username` label. This label is set to `username` field value at the corresponding user section in the [`-auth.config`](#auth-config) file.
@ -1104,14 +1104,14 @@ users:
`vmauth` exports the following metrics if `unauthorized_user` section is defined in [`-auth.config`](#auth-config):
* `vmauth_unauthorized_user_requests_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of unauthorized requests served
* `vmauth_unauthorized_user_request_backend_errors_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of unauthorized request errors
* `vmauth_unauthorized_user_request_duration_seconds` [summary](https://docs.victoriametrics.com/keyconcepts/#summary) - the duration of unauthorized requests
* `vmauth_unauthorized_user_concurrent_requests_limit_reached_total` [counter](https://docs.victoriametrics.com/keyconcepts/#counter) - the number of failed unauthorized requests
* `vmauth_unauthorized_user_requests_total` [counter](./keyConcepts.md#counter) - the number of unauthorized requests served
* `vmauth_unauthorized_user_request_backend_errors_total` [counter](./keyConcepts.md#counter) - the number of unauthorized request errors
* `vmauth_unauthorized_user_request_duration_seconds` [summary](./keyConcepts.md#summary) - the duration of unauthorized requests
* `vmauth_unauthorized_user_concurrent_requests_limit_reached_total` [counter](./keyConcepts.md#counter) - the number of failed unauthorized requests
because of exceeded [concurrency limits](#concurrency-limiting)
* `vmauth_unauthorized_user_concurrent_requests_capacity` [gauge](https://docs.victoriametrics.com/keyconcepts/#gauge) - the maximum number
* `vmauth_unauthorized_user_concurrent_requests_capacity` [gauge](./keyConcepts.md#gauge) - the maximum number
of [concurrent unauthorized requests](#concurrency-limiting)
* `vmauth_unauthorized_user_concurrent_requests_current` [gauge](https://docs.victoriametrics.com/keyconcepts/#gauge) - the current number of [concurrent unauthorized requests](#concurrency-limiting)
* `vmauth_unauthorized_user_concurrent_requests_current` [gauge](./keyConcepts.md#gauge) - the current number of [concurrent unauthorized requests](#concurrency-limiting)
## How to build from sources
@ -1176,34 +1176,34 @@ Pass `-help` command-line arg to `vmauth` in order to see all the configuration
vmauth authenticates and authorizes incoming requests and proxies them to VictoriaMetrics.
See the docs at https://docs.victoriametrics.com/vmauth/ .
See the docs at {{% ref "./vmauth.md" %}}.
-auth.config string
Path to auth config. It can point either to local file or to http url. See https://docs.victoriametrics.com/vmauth/ for details on the format of this auth config
Path to auth config. It can point either to local file or to http url. See {{% ref "./vmauth.md" %}} for details on the format of this auth config
-backend.TLSCAFile string
Optional path to TLS root CA file, which is used for TLS verification when connecting to backends over HTTPS. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup
Optional path to TLS root CA file, which is used for TLS verification when connecting to backends over HTTPS. See {{% ref "./vmauth.md#backend-tls-setup" %}}
-backend.TLSCertFile string
Optional path to TLS client certificate file, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup
Optional path to TLS client certificate file, which must be sent to HTTPS backend. See {{% ref "./vmauth.md#backend-tls-setup" %}}
-backend.TLSKeyFile string
Optional path to TLS client key file, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup
Optional path to TLS client key file, which must be sent to HTTPS backend. See {{% ref "./vmauth.md#backend-tls-setup" %}}
-backend.TLSServerName string
Optional TLS ServerName, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup
Optional TLS ServerName, which must be sent to HTTPS backend. See {{% ref "./vmauth.md#backend-tls-setup" %}}
-backend.tlsInsecureSkipVerify
Whether to skip TLS verification when connecting to backends over HTTPS. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup
Whether to skip TLS verification when connecting to backends over HTTPS. See {{% ref "./vmauth.md#backend-tls-setup" %}}
-configCheckInterval duration
interval for config file re-read. Zero value disables config re-reading. By default, refreshing is disabled, send SIGHUP for config refresh.
-discoverBackendIPs
Whether to discover backend IPs via periodic DNS queries to hostnames specified in url_prefix. This may be useful when url_prefix points to a hostname with dynamically scaled instances behind it. See https://docs.victoriametrics.com/vmauth/#discovering-backend-ips
Whether to discover backend IPs via periodic DNS queries to hostnames specified in url_prefix. This may be useful when url_prefix points to a hostname with dynamically scaled instances behind it. See {{% ref "./vmauth.md#discovering-backend-ips" %}}
-discoverBackendIPsInterval duration
The interval for re-discovering backend IPs if -discoverBackendIPs command-line flag is set. Too low value may lead to DNS errors (default 10s)
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables for more details" %}}
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-failTimeout duration
Sets a delay period for load balancing to skip a malfunctioning backend (default 3s)
-filestream.disableFadvise
@ -1263,7 +1263,7 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
-licenseFile string
Path to file with license key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/ . Trial Enterprise license can be obtained from https://victoriametrics.com/products/enterprise/trial/ . This flag is available only in Enterprise binaries. The license key can be also passed inline via -license command-line flag
-loadBalancingPolicy string
The default load balancing policy to use for backend urls specified inside url_prefix section. Supported policies: least_loaded, first_available. See https://docs.victoriametrics.com/vmauth/#load-balancing (default "least_loaded")
The default load balancing policy to use for backend urls specified inside url_prefix section. Supported policies: least_loaded, first_available. See {{% ref "./vmauth.md#load-balancing" %}} (default "least_loaded")
-logInvalidAuthTokens
Whether to log requests with invalid auth tokens. Such requests are always counted at vmauth_http_request_errors_total{reason="invalid_auth_token"} metric, which is exposed at /metrics page
-loggerDisableTimestamps
@ -1304,11 +1304,11 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-mtlsCAFile array
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-pprofAuthKey value
@ -1327,7 +1327,7 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-reloadAuthKey value
@ -1336,7 +1336,7 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
-responseTimeout duration
The timeout for receiving a response from backend (default 5m0s)
-retryStatusCodes array
Comma-separated list of default HTTP response status codes when vmauth re-tries the request on other backends. See https://docs.victoriametrics.com/vmauth/#load-balancing for details (default 0)
Comma-separated list of default HTTP response status codes when vmauth re-tries the request on other backends. See {{% ref "./vmauth.md#load-balancing" %}} for details (default 0)
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to default value.
-tls array
@ -1344,11 +1344,11 @@ See the docs at https://docs.victoriametrics.com/vmauth/ .
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsAutocertCacheDir string
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertEmail string
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertHosts array
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCertFile array

42
docs/vmbackup.md
View file

@ -9,7 +9,7 @@ title: vmbackup
aliases:
- /vmbackup.html
---
`vmbackup` creates VictoriaMetrics data backups from [instant snapshots](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots).
`vmbackup` creates VictoriaMetrics data backups from [instant snapshots](./Single-Server-VictoriaMetrics.md#how-to-work-with-snapshots).
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
Full backups can be accelerated with `-origin` pointing to an already existing backup on the same remote storage. In this case `vmbackup` makes server-side copy for the shared
@ -17,11 +17,11 @@ data between the existing backup and new backup. It saves time and costs on data
Backup process can be interrupted at any time. It is automatically resumed from the interruption point when restarting `vmbackup` with the same args.
Backed up data can be restored with [vmrestore](https://docs.victoriametrics.com/vmrestore/).
Backed up data can be restored with [vmrestore](./vmrestore.md).
See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883) for more details.
See also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) tool built on top of `vmbackup`. This tool simplifies
See also [vmbackupmanager](./vmbackupmanager.md) tool built on top of `vmbackup`. This tool simplifies
creation of hourly, daily, weekly and monthly backups.
## Supported storage types
@ -45,8 +45,8 @@ Regular backup can be performed with the following command:
```
* `</path/to/victoria-metrics-data>` - path to VictoriaMetrics data pointed by `-storageDataPath` command-line flag in single-node VictoriaMetrics or in cluster `vmstorage`.
There is no need to stop VictoriaMetrics for creating backups since they are performed from immutable [instant snapshots](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots).
* `http://victoriametrics:8428/snapshot/create` is the url for creating snapshots according to [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots). `vmbackup` creates a snapshot by querying the provided `-snapshot.createURL`, then performs the backup and then automatically removes the created snapshot.
There is no need to stop VictoriaMetrics for creating backups since they are performed from immutable [instant snapshots](./Single-Server-VictoriaMetrics.md#how-to-work-with-snapshots).
* `http://victoriametrics:8428/snapshot/create` is the url for creating snapshots according to [these docs](./Single-Server-VictoriaMetrics.md#how-to-work-with-snapshots). `vmbackup` creates a snapshot by querying the provided `-snapshot.createURL`, then performs the backup and then automatically removes the created snapshot.
* `<bucket>` is an already existing name for [GCS bucket](https://cloud.google.com/storage/docs/creating-buckets).
* `<path/to/new/backup>` is the destination path where new backup will be placed.
@ -84,7 +84,7 @@ Smart backups mean storing full daily backups into `YYYYMMDD` folders and creati
./vmbackup -storageDataPath=</path/to/victoria-metrics-data> -snapshot.createURL=http://localhost:8428/snapshot/create -dst=gs://<bucket>/latest
```
This command creates an [instant snapshot](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots)
This command creates an [instant snapshot](./Single-Server-VictoriaMetrics.md#how-to-work-with-snapshots)
and uploads it to `gs://<bucket>/latest`. It uploads only the changed data (aka incremental backup). This saves network bandwidth costs and time
when backing up large amounts of data.
@ -107,7 +107,7 @@ Note that hourly backup shouldn't run when creating daily backup.
Do not forget to remove old backups when they are no longer needed in order to save storage costs.
See also [vmbackupmanager tool](https://docs.victoriametrics.com/vmbackupmanager/) for automating smart backups.
See also [vmbackupmanager tool](./vmbackupmanager.md) for automating smart backups.
### Server-side copy of the existing backup
@ -161,8 +161,8 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
* If `vmbackup` eats all the network bandwidth or CPU, then either decrease the `-concurrency` command-line flag value or set `-maxBytesPerSecond` command-line flag value to lower value.
* If `vmbackup` consumes all the CPU on systems with big number of CPU cores, then try running it with `-filestream.disableFadvise` command-line flag.
* If `vmbackup` has been interrupted due to temporary error, then just restart it with the same args. It will resume the backup process.
* Backups created from [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) cannot be restored
at [cluster VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) and vice versa.
* Backups created from [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) cannot be restored
at [cluster VictoriaMetrics](./Cluster-VictoriaMetrics.md) and vice versa.
## Advanced usage
@ -213,7 +213,7 @@ Obtaining credentials from env variables.
The `AZURE_STORAGE_DOMAIN` can be used for optionally overriding the default domain for the Azure storage service.
Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](https://docs.victoriametrics.com/vmbackup/#using-cloud-providers-metadata-service).
Please, note that `vmbackup` will use credentials provided by cloud providers metadata service [when applicable](./vmbackup.md#using-cloud-providers-metadata-service).
### Using cloud providers metadata service
@ -296,7 +296,7 @@ You have to add a custom url endpoint via flag:
### Permanent deletion of objects in S3-compatible storages
`vmbackup` and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) use standard delete operation
`vmbackup` and [vmbackupmanager](./vmbackupmanager.md) use standard delete operation
for S3-compatible object storage when performing [incremental backups](#incremental-backups).
This operation removes only the current version of the object. This works OK in most cases.
@ -323,18 +323,18 @@ Run `vmbackup -help` in order to see all the available options:
-customS3Endpoint string
Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set
-deleteAllObjectVersions
Whether to prune previous object versions when deleting an object. By default, when object storage has versioning enabled deleting the file removes only current version. This option forces removal of all previous versions. See: https://docs.victoriametrics.com/vmbackup/#permanent-deletion-of-objects-in-s3-compatible-storages
Whether to prune previous object versions when deleting an object. By default, when object storage has versioning enabled deleting the file removes only current version. This option forces removal of all previous versions. See: {{ ref "./vmbackup.md#permanent-deletion-of-objects-in-s3-compatible-storages" %}}
-dst string
Where to put the backup on the remote storage. Example: gs://bucket/path/to/backup, s3://bucket/path/to/backup, azblob://container/path/to/backup or fs:///path/to/local/backup/dir
-dst can point to the previous backup. In this case incremental backup is performed, i.e. only changed data is uploaded
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-filestream.disableFadvise
Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
-flagsAuthKey value
@ -411,11 +411,11 @@ Run `vmbackup -help` in order to see all the available options:
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-mtlsCAFile array
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-origin string
@ -436,7 +436,7 @@ Run `vmbackup -help` in order to see all the available options:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-s3ForcePathStyle
@ -461,7 +461,7 @@ Run `vmbackup -help` in order to see all the available options:
-snapshot.tlsServerName string
Optional TLS server name to use for connections to -snapshotCreateURL. By default, the server name from -snapshotCreateURL is used
-snapshotName string
Name for the snapshot to backup. See https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots. There is no need in setting -snapshotName if -snapshot.createURL is set
Name for the snapshot to backup. See {{% ref "./Single-Server-VictoriaMetrics.md#how-to-work-with-snapshots" %}}. There is no need in setting -snapshotName if -snapshot.createURL is set
-storageDataPath string
Path to VictoriaMetrics data. Must match -storageDataPath from VictoriaMetrics or vmstorage (default "victoria-metrics-data")
-tls array
@ -469,11 +469,11 @@ Run `vmbackup -help` in order to see all the available options:
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsAutocertCacheDir string
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertEmail string
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertHosts array
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}/
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCertFile array

26
docs/vmctl.md
View file

@ -37,10 +37,10 @@ COMMANDS:
prometheus Migrate timeseries from Prometheus
vm-native Migrate time series between VictoriaMetrics installations via native binary format
remote-read Migrate timeseries by Prometheus remote read protocol
verify-block Verifies correctness of data blocks exported via VictoriaMetrics Native format. See https://docs.victoriametrics.com/#how-to-export-data-in-native-format
verify-block Verifies correctness of data blocks exported via VictoriaMetrics Native format. See {{% ref "./#how-to-export-data-in-native-format" %}}
```
Each command has its own unique set of flags specific (e.g. prefixed with `influx-` for [influx](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x))
Each command has its own unique set of flags specific (e.g. prefixed with `influx-` for [influx](./vmctl.md#migrating-data-from-influxdb-1x))
to the data source and common list of flags for destination (prefixed with `vm-` for VictoriaMetrics):
```sh
@ -213,7 +213,7 @@ One important note for OpenTSDB migration: Queries/HBase scans can "get stuck" w
## Migrating data from InfluxDB (1.x)
`vmctl` supports the `influx` mode for [migrating data from InfluxDB to VictoriaMetrics](https://docs.victoriametrics.com/guides/migrate-from-influx.html)
`vmctl` supports the `influx` mode for [migrating data from InfluxDB to VictoriaMetrics](./guides/migrate-from-influx.md)
time-series database.
See `./vmctl influx --help` for details and full list of flags.
@ -326,7 +326,7 @@ You may find useful a 3rd party solution for this - <https://github.com/jonppe/i
[Promscale](https://github.com/timescale/promscale) supports [Prometheus Remote Read API](https://prometheus.io/docs/prometheus/latest/querying/remote_read_api/).
To migrate historical data from Promscale to VictoriaMetrics we recommend using `vmctl`
in [remote-read](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol) mode.
in [remote-read](./vmctl.md#migrating-data-by-remote-read-protocol) mode.
See the example of migration command below:
```sh
@ -789,9 +789,9 @@ requires an Authentication header like `X-Scope-OrgID`. You can define it via th
## Migrating data from VictoriaMetrics
The simplest way to migrate data between VM instances is [to copy data between instances](https://docs.victoriametrics.com/single-server-victoriametrics/#data-migration).
The simplest way to migrate data between VM instances is [to copy data between instances](./Single-Server-VictoriaMetrics.md#data-migration).
vmctl uses [native binary protocol](https://docs.victoriametrics.com/#how-to-export-data-in-native-format)
vmctl uses [native binary protocol](./#how-to-export-data-in-native-format)
(available since [1.42.0 release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.42.0))
to migrate data between VM instances: single to single, cluster to cluster, single to cluster and vice versa.
@ -837,11 +837,11 @@ Importing tips:
1. Migrating big volumes of data may result in reaching the safety limits on `src` side.
Please verify that `-search.maxExportDuration` and `-search.maxExportSeries` were set with
proper values for `src`. If hitting the limits, follow the recommendations
[here](https://docs.victoriametrics.com/#how-to-export-data-in-native-format).
[here](./#how-to-export-data-in-native-format).
If hitting `the number of matching timeseries exceeds...` error, adjust filters to match less time series or
update `-search.maxSeries` command-line flag on vmselect/vmsingle;
1. Using smaller intervals via `--vm-native-step-interval` cmd-line flag can reduce the number of matched series per-request
for sources with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate).
for sources with [high churn rate](./FAQ.md#what-is-high-churn-rate).
See more about [step interval here](#using-time-based-chunking-of-migration).
1. Migrating all the metrics from one VM to another may collide with existing application metrics
(prefixed with `vm_`) at destination and lead to confusion when using
@ -852,7 +852,7 @@ Importing tips:
This will instruct `vmselect`/`vmstorage` to ignore duplicates with identical timestamps. Ignore this recommendation
if you already have `-dedup.minScrapeInterval` set to 1ms or higher values at destination.
1. When migrating data from one VM cluster to another, consider using [cluster-to-cluster mode](#cluster-to-cluster-migration-mode).
Or manually specify addresses according to [URL format](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format):
Or manually specify addresses according to [URL format](./Cluster-VictoriaMetrics.md#url-format):
```sh
# Migrating from cluster specific tenantID to single
--vm-native-src-addr=http://<src-vmselect>:8481/select/0/prometheus
@ -874,7 +874,7 @@ Importing tips:
of data copies for the destination database, and will result only in creating duplicates. To remove duplicates,
destination database need to be configured with `-dedup.minScrapeInterval=1ms`. To restore the replication factor
the destination `vminsert` component need to be configured with the according `-replicationFactor` value.
See more about replication [here](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety).
See more about replication [here](./Cluster-VictoriaMetrics.md#replication-and-data-safety).
1. Migration speed can be adjusted via `--vm-concurrency` cmd-line flag, which controls the number of concurrent
workers busy with processing. Please note, that each worker can load up to a single vCPU core on VictoriaMetrics.
So try to set it according to allocated CPU resources of your VictoriaMetrics destination installation.
@ -930,7 +930,7 @@ Requests to make: 45 / 45 [█████████████████
Using cluster-to-cluster migration mode helps to migrate all tenants data in a single `vmctl` run.
Cluster-to-cluster uses `/admin/tenants` endpoint (available starting from [v1.84.0](https://docs.victoriametrics.com/changelog/#v1840)) to discover list of tenants from source cluster.
Cluster-to-cluster uses `/admin/tenants` endpoint (available starting from [v1.84.0](./CHANGELOG.md#v1840)) to discover list of tenants from source cluster.
To use this mode you need to set `--vm-intercluster` flag to `true`, `--vm-native-src-addr` flag to 'http://vmselect:8481/' and `--vm-native-dst-addr` value to http://vminsert:8480/:
@ -984,7 +984,7 @@ Run the following command to get all configuration options:
## Verifying exported blocks from VictoriaMetrics
In this mode, `vmctl` allows verifying correctness and integrity of data exported via
[native format](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-export-data-in-native-format)
[native format](./Single-Server-VictoriaMetrics.md#how-to-export-data-in-native-format)
from VictoriaMetrics.
You can verify exported data at disk before uploading it by `vmctl verify-block` command:
@ -1080,7 +1080,7 @@ results such as `average`, `rate`, etc.
Limiting the rate of data transfer could help to reduce pressure on disk or on destination database.
The rate limit may be set in bytes-per-second via `--vm-rate-limit` flag.
Please note, you can also use [vmagent](https://docs.victoriametrics.com/vmagent/)
Please note, you can also use [vmagent](./vmagent.md)
as a proxy between `vmctl` and destination with `-remoteWrite.rateLimit` flag enabled.
## How to build

20
docs/vmgateway.md
View file

@ -9,7 +9,7 @@ title: vmgateway
aliases:
- /vmgateway.html
---
***vmgateway is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/).
***vmgateway is a part of [enterprise package](./enterprise.md).
It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/).***
@ -24,7 +24,7 @@ See how to request a free trial license [here](https://victoriametrics.com/produ
* Provides access by tenantID in the Cluster version
* Allows for separate write/read/admin access to data
`vmgateway` is included in our [enterprise packages](https://docs.victoriametrics.com/enterprise/).
`vmgateway` is included in our [enterprise packages](./enterprise.md).
## Access Control
@ -353,11 +353,11 @@ Below is the list of configuration flags (it can be viewed by running `./vmgatew
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-filestream.disableFadvise
Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
-flagsAuthKey value
@ -439,11 +439,11 @@ Below is the list of configuration flags (it can be viewed by running `./vmgatew
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-mtlsCAFile array
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-pprofAuthKey value
@ -462,7 +462,7 @@ Below is the list of configuration flags (it can be viewed by running `./vmgatew
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-ratelimit.config string
@ -484,11 +484,11 @@ Below is the list of configuration flags (it can be viewed by running `./vmgatew
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsAutocertCacheDir string
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertEmail string
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertHosts array
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCertFile array

26
docs/vmrestore.md
View file

@ -9,7 +9,7 @@ title: vmrestore
aliases:
- /vmrestore.html
---
`vmrestore` restores data from backups created by [vmbackup](https://docs.victoriametrics.com/vmbackup/).
`vmrestore` restores data from backups created by [vmbackup](./vmbackup.md).
Restore process can be interrupted at any time. It is automatically resumed from the interruption point
when restarting `vmrestore` with the same args.
@ -24,7 +24,7 @@ Run the following command to restore backup from the given `-src` into the given
./vmrestore -src=<storageType>://<path/to/backup> -storageDataPath=<local/path/to/restore>
```
* `<storageType>://<path/to/backup>` is the path to backup made with [vmbackup](https://docs.victoriametrics.com/vmbackup/).
* `<storageType>://<path/to/backup>` is the path to backup made with [vmbackup](./vmbackup.md).
`vmrestore` can restore backups from the following storage types:
* [GCS](https://cloud.google.com/storage/). Example: `-src=gs://<bucket>/<path/to/backup>`
* [S3](https://aws.amazon.com/s3/). Example: `-src=s3://<bucket>/<path/to/backup>`
@ -42,13 +42,13 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
## Troubleshooting
* See [how to setup credentials via environment variables](https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables).
* See [how to setup credentials via environment variables](./vmbackup.md#providing-credentials-via-env-variables).
* If `vmrestore` eats all the network bandwidth, then set `-maxBytesPerSecond` to the desired value.
* If `vmrestore` has been interrupted due to temporary error, then just restart it with the same args. It will resume the restore process.
## Advanced usage
Please, see [vmbackup docs](https://docs.victoriametrics.com/vmbackup/#advanced-usage) for examples of authentication
Please, see [vmbackup docs](./vmbackup.md#advanced-usage) for examples of authentication
with different storage types.
Run `vmrestore -help` in order to see all the available options:
@ -67,15 +67,15 @@ Run `vmrestore -help` in order to see all the available options:
-customS3Endpoint string
Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set
-deleteAllObjectVersions
Whether to prune previous object versions when deleting an object. By default, when object storage has versioning enabled deleting the file removes only current version. This option forces removal of all previous versions. See: https://docs.victoriametrics.com/vmbackup/#permanent-deletion-of-objects-in-s3-compatible-storages
Whether to prune previous object versions when deleting an object. By default, when object storage has versioning enabled deleting the file removes only current version. This option forces removal of all previous versions. See: {{% ref "./vmbackup.md#permanent-deletion-of-objects-in-s3-compatible-storages" %}}
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details
-envflag.prefix string
Prefix for environment variables if -envflag.enable is set
-eula
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-filestream.disableFadvise
Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU
-flagsAuthKey value
@ -152,11 +152,11 @@ Run `vmrestore -help` in order to see all the available options:
Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.*
Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path
-mtls array
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-mtlsCAFile array
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-pprofAuthKey value
@ -175,7 +175,7 @@ Run `vmrestore -help` in order to see all the available options:
-pushmetrics.interval duration
Interval for pushing metrics to every -pushmetrics.url (default 10s)
-pushmetrics.url array
Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-s3ForcePathStyle
@ -196,11 +196,11 @@ Run `vmrestore -help` in order to see all the available options:
Supports array of values separated by comma or specified via multiple flags.
Empty values are set to false.
-tlsAutocertCacheDir string
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertEmail string
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
-tlsAutocertHosts array
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/
Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}}
Supports an array of values separated by comma or specified via multiple flags.
Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.
-tlsCertFile array