diff --git a/docs/Articles.md b/docs/Articles.md index 386a2bbbf..552c21e81 100644 --- a/docs/Articles.md +++ b/docs/Articles.md @@ -9,7 +9,7 @@ menu: aliases: - /Articles.html --- -See also [case studies](./CaseStudies.md). +See also [case studies](https://docs.victoriametrics.com/casestudies/). ## Third-party articles and slides about VictoriaMetrics diff --git a/docs/BestPractices.md b/docs/BestPractices.md index 4f0934a7f..9fa7f1834 100644 --- a/docs/BestPractices.md +++ b/docs/BestPractices.md @@ -64,7 +64,7 @@ The following steps must be performed during the upgrade / downgrade procedure: ## Backup Recommendations -VictoriaMetrics supports backups via [vmbackup](./vmbackup.md) and [vmrestore](./vmrestore.md) tools. There is also [vmbackupmanager](./vmbackupmanager.md), which simplifies backup automation. +VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/vmbackup/) and [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. There is also [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/), which simplifies backup automation. ## Technical Support and Services @@ -74,4 +74,4 @@ There are the following channels for providing technical support for VictoriaMet * [Slack Inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/) * [Telegram channel](https://t.me/VictoriaMetrics_en) -We also provide [Enterprise support](./enterprise.md). +We also provide [Enterprise support](https://docs.victoriametrics.com/enterprise/). diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md index 76441b98d..193bf75f4 100644 --- a/docs/Cluster-VictoriaMetrics.md +++ b/docs/Cluster-VictoriaMetrics.md @@ -112,9 +112,9 @@ The `vm_account_id` and `vm_project_id` labels are extracted after applying the set via `-relabelConfig` command-line flag, so these labels can be set at this stage. The `vm_account_id` and `vm_project_id` labels are also taken into account when ingesting data via non-http-based protocols -such as [Graphite](./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd), -[InfluxDB line protocol via TCP and UDP](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and -[OpenTSDB telnet put protocol](./#sending-data-via-telnet-put-protocol). +such as [Graphite](./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd), +[InfluxDB line protocol via TCP and UDP](./README.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) and +[OpenTSDB telnet put protocol](./README.md#sending-data-via-telnet-put-protocol). **Security considerations:** it is recommended restricting access to `multitenant` endpoints only to trusted sources, since untrusted source may break per-tenant data by writing unwanted samples to arbitrary tenants. @@ -278,7 +278,7 @@ General security recommendations: - Prefer using lists of allowed [API endpoints](#url-format), while disallowing access to other endpoints when configuring auth proxy in front of `vminsert` and `vmselect`. This minimizes attack surface. -See also [security recommendation for single-node VictoriaMetrics](./#security) +See also [security recommendation for single-node VictoriaMetrics](./README.md#security) and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/). @@ -344,7 +344,7 @@ See more details in the article [VictoriaMetrics Monitoring](https://victoriamet Note that these limits are set and applied individually per each `vmstorage` node in the cluster. So, if the cluster has `N` `vmstorage` nodes, then the cluster-level limits will be `N` times bigger than the per-`vmstorage` limits. -See more details about cardinality limiter in [these docs](./#cardinality-limiter). +See more details about cardinality limiter in [these docs](./README.md#cardinality-limiter). ## Troubleshooting @@ -387,7 +387,7 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md). - `datadog/api/v2/series` - for ingesting data with [DataDog submit metrics API](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-datadog-agent) for details. - `datadog/api/beta/sketches` - for ingesting data with [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/). - `influx/write` and `influx/api/v2/write` - for ingesting data with [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/). TCP and UDP receiver is disabled by default. It is exposed on a distinct TCP address set via `-influxListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) for details. - - `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./#how-to-send-data-from-newrelic-agent) for details. + - `newrelic/infra/v2/metrics/events/bulk` - for accepting data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](./README.md#how-to-send-data-from-newrelic-agent) for details. - `opentsdb/api/put` - for accepting [OpenTSDB HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html). This handler is disabled by default. It is exposed on a distinct TCP address set via `-opentsdbHTTPListenAddr` command-line flag. See [these docs](./Single-Server-VictoriaMetrics.md#sending-opentsdb-data-via-http-apiput-requests) for details. - URLs for [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): `http://:8481/select//prometheus/`, where: @@ -403,7 +403,7 @@ Check practical examples of VictoriaMetrics API [here](./url-examples.md). - `api/v1/export/native` - exports raw data in native binary format. It may be imported into another VictoriaMetrics via `api/v1/import/native` (see above). - `api/v1/export/csv` - exports data in CSV. It may be imported into another VictoriaMetrics via `api/v1/import/csv` (see above). - `api/v1/series/count` - returns the total number of series. - - `api/v1/status/tsdb` - for time series stats. See [these docs](./#tsdb-stats) for details. + - `api/v1/status/tsdb` - for time series stats. See [these docs](./README.md#tsdb-stats) for details. - `api/v1/status/active_queries` - for currently executed active queries. Note that every `vmselect` maintains an independent list of active queries, which is returned in the response. - `api/v1/status/top_queries` - for listing the most frequently executed queries and queries taking the most duration. @@ -440,7 +440,7 @@ The `start` and `end` query args are optional. If they are missing, then all the This URL works only when `-vmalert.proxyURL` flag is set. See more about vmalert [here](#vmalert). - `vmstorage` nodes provide the following HTTP endpoints on `8482` port: - - `/internal/force_merge` - initiate [forced compactions](./#forced-merge) on the given `vmstorage` node. + - `/internal/force_merge` - initiate [forced compactions](./README.md#forced-merge) on the given `vmstorage` node. - `/snapshot/create` - create [instant snapshot](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282), which can be used for backups in background. Snapshots are created in `/snapshots` folder, where `` is the corresponding command-line flag value. @@ -620,7 +620,7 @@ The cluster works in the following way when some of `vmstorage` nodes are unavai `vmselect` doesn't serve partial responses for API handlers returning [raw datapoints](./keyConcepts.md#raw-samples), since users usually expect this data is always complete. The following handlers return raw samples: -- [`/api/v1/export*` endpoints](./#how-to-export-time-series) +- [`/api/v1/export*` endpoints](./README.md#how-to-export-time-series) - [`/api/v1/query`](./url-examples.md#apiv1query) when the `query` contains [series selector](./keyConcepts.md#filtering) ending with some duration in square brackets. For example, `/api/v1/query?query=up[1h]&time=2024-01-02T03:00:00Z`. This query returns [raw samples](./keyConcepts.md#raw-samples) for [time series](./keyConcepts.md#time-series) @@ -633,7 +633,7 @@ Data replication can be used for increasing storage durability. See [these docs] VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](./CaseStudies/README.md). -Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), [series churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](./#troubleshooting). +Each node type - `vminsert`, `vmselect` and `vmstorage` - can run on the most suitable hardware. Cluster capacity scales linearly with the available resources. The needed amounts of CPU and RAM per each node type highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), [series churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics cluster for your production workload and iteratively scaling per-node resources and the number of nodes per node type until the cluster becomes stable. It is recommended setting up [monitoring for the cluster](#monitoring). It helps to determine bottlenecks in cluster setup. It is also recommended following [the troubleshooting docs](./README.md#troubleshooting). The needed storage space for the given retention (the retention is set via `-retentionPeriod` command-line flag at `vmstorage`) can be extrapolated from disk space usage in a test run. For example, if the storage space usage is 10GB after a day-long test run on a production workload, then it will need at least `10GB*100=1TB` of disk space for `-retentionPeriod=100d` (100-days retention period). Storage space usage can be monitored with [the official Grafana dashboard for VictoriaMetrics cluster](#monitoring). @@ -685,7 +685,7 @@ Some workloads may need fine-grained resource usage limits. In these cases the f See also `-search.maxMemoryPerQuery` command-line flag at `vmselect`. - `-search.maxQueueDuration` at `vmselect` and `vmstorage` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests` concurrent queries are executed. -- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./#prometheus-querying-api-enhancements) +- `-search.ignoreExtraFiltersAtLabelsAPI` at `vmselect` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./README.md#prometheus-querying-api-enhancements) query args at [/api/v1/labels](./url-examples.md#apiv1labels) and [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). This may be useful for reducing the load on `vmstorage` if the provided extra filters match too many time series. @@ -880,7 +880,7 @@ HDD-based persistent disks should be enough for the majority of use cases. It is ## Deduplication -Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](./#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects: +Cluster version of VictoriaMetrics supports data deduplication in the same way as single-node version do. See [these docs](./README.md#deduplication) for details. The only difference is that the same `-dedup.minScrapeInterval` command-line flag value must be passed to both `vmselect` and `vmstorage` nodes because of the following aspects: By default, `vminsert` tries to route all the samples for a single time series to a single `vmstorage` node. But samples for a single time series can be spread among multiple `vmstorage` nodes under certain conditions: * when adding/removing `vmstorage` nodes. Then new samples for a part of time series will be routed to another `vmstorage` nodes; @@ -911,7 +911,7 @@ Restoring from backup: ## Retention filters [VictoriaMetrics enterprise](./enterprise.md) supports configuring multiple retentions for distinct sets of time series -by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](./#retention-filters) for details on this feature. +by passing `-retentionFilter` command-line flag to `vmstorage` nodes. See [these docs](./README.md#retention-filters) for details on this feature. Additionally, enterprise version of VictoriaMetrics cluster supports multiple retentions for distinct sets of [tenants](#multitenancy) by specifying filters on `vm_account_id` and/or `vm_project_id` pseudo-labels in `-retentionFilter` command-line flag. @@ -932,7 +932,7 @@ For example, the following config sets retention to 5 days for time series with -retentionFilter='{vm_account_id="5",env="dev"}:5d' ``` -See also [these docs](./#retention-filters) for additional details on retention filters. +See also [these docs](./README.md#retention-filters) for additional details on retention filters. See also [downsampling](#downsampling). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). @@ -941,7 +941,7 @@ See how to request a free trial license [here](https://victoriametrics.com/produ ## Downsampling Downsampling is available in [enterprise version of VictoriaMetrics](./enterprise.md). -It is configured with `-downsampling.period` command-line flag according to [these docs](./#downsampling). +It is configured with `-downsampling.period` command-line flag according to [these docs](./README.md#downsampling). It is possible to downsample series, which belong to a particular [tenant](#multitenancy) by using [filters](./keyConcepts.md#filtering) on `vm_account_id` or `vm_project_id` pseudo-labels in `-downsampling.period` command-line flag. For example, the following config leaves the last sample per each minute for samples @@ -1064,7 +1064,7 @@ Below is the output for `/path/to/vminsert -help`: -datadog.sanitizeMetricName Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true) -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}} + Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} -disableRerouting Whether to disable re-routing when some of vmstorage nodes accept incoming data at slower speed compared to other storage nodes. Disabled re-routing limits the ingestion rate by the slowest vmstorage node. On the other side, disabled re-routing minimizes the number of active time series in the cluster during rolling restarts and during spikes in series churn rate. See also -disableReroutingOnUnavailable and -dropSamplesOnOverload (default true) -disableReroutingOnUnavailable @@ -1074,7 +1074,7 @@ Below is the output for `/path/to/vminsert -help`: -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula @@ -1087,7 +1087,7 @@ Below is the output for `/path/to/vminsert -help`: -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -graphite.sanitizeMetricName - Sanitize metric names for the ingested Graphite data. See {{% ref "./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}} + Sanitize metric names for the ingested Graphite data. See {{% ref "./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}} -graphiteListenAddr string TCP and UDP address to listen for Graphite plaintext data. Usually :2003 must be set. Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol -graphiteListenAddr.useProxyProtocol @@ -1212,7 +1212,7 @@ Below is the output for `/path/to/vminsert -help`: The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864) -opentelemetry.usePrometheusNaming - Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./#sending-data-via-opentelemetry" %}} + Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./README.md#sending-data-via-opentelemetry" %}} -opentsdbHTTPListenAddr string TCP address to listen for OpenTSDB HTTP put requests. Usually :4242 must be set. Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol -opentsdbHTTPListenAddr.useProxyProtocol @@ -1246,11 +1246,11 @@ Below is the output for `/path/to/vminsert -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}} . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -relabelConfig string - Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./#relabeling" %}} for details. The config is reloaded on SIGHUP signal + Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./README.md#relabeling" %}} for details. The config is reloaded on SIGHUP signal -relabelConfigCheckInterval duration Interval for checking for changes in '-relabelConfig' file. By default the checking is disabled. Send SIGHUP signal in order to force config check for changes -replicationFactor int @@ -1355,20 +1355,20 @@ Below is the output for `/path/to/vmselect -help`: -clusternativeListenAddr string TCP address to listen for requests from other vmselect nodes in multi-level cluster setup. See {{% ref "./Cluster-VictoriaMetrics.md#multi-level-cluster-setup" %}}. Usually :8401 should be set to match default vmstorage port for vmselect. Disabled work if empty -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./#deduplication" %}} for details + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details -deleteAuthKey value authKey for metrics' deletion via /prometheus/api/v1/admin/tsdb/delete_series and /graphite/tags/delSeries Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}} + Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula @@ -1479,7 +1479,7 @@ Below is the output for `/path/to/vmselect -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -replicationFactor array @@ -1490,7 +1490,7 @@ Below is the output for `/path/to/vmselect -help`: -search.denyPartialResponse Whether to deny partial responses if a part of -storageNode instances fail to perform queries; this trades availability over consistency; see also -search.maxQueryDuration -search.disableCache - Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup + Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./README.md#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup -search.disableImplicitConversion Whether to return an error for queries that rely on implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. See also -search.logImplicitConversion -search.graphiteMaxPointsPerSeries int @@ -1517,11 +1517,11 @@ Below is the output for `/path/to/vmselect -help`: -search.maxFederateSeries int The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000) -search.maxGraphiteSeries int - The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./#graphite-render-api-usage" %}} (default 300000) + The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./README.md#graphite-render-api-usage" %}} (default 300000) -search.maxGraphiteTagKeys int - The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) -search.maxGraphiteTagValues int - The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) -search.maxLabelsAPIDuration duration The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s) -search.maxLabelsAPISeries int @@ -1581,7 +1581,7 @@ Below is the output for `/path/to/vmselect -help`: Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path -search.resetRollupResultCacheOnStartup - Whether to reset rollup result cache on startup. See {{% ref "./#rollup-result-cache" %}}. See also -search.disableCache + Whether to reset rollup result cache on startup. See {{% ref "./README.md#rollup-result-cache" %}}. See also -search.disableCache -search.setLookbackToStep Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored -search.skipSlowReplicas @@ -1668,19 +1668,19 @@ Below is the output for `/path/to/vmstorage -help`: -cluster.tlsKeyFile string Path to server-side TLS key file to use when accepting connections from vminsert and vmselect if -cluster.tls flag is set. See {{% ref "./Cluster-VictoriaMetrics.md#mtls-protection" %}}. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./#deduplication" %}} for details + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See {{% ref "./README.md#deduplication" %}} for details -denyQueriesOutsideRetention Whether to deny queries outside of the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}} + Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula @@ -1807,11 +1807,11 @@ Below is the output for `/path/to/vmstorage -help`: -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionFilter array - Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./README.md#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionPeriod value @@ -1856,9 +1856,9 @@ Below is the output for `/path/to/vmstorage -help`: Overrides max size for storage/tsid cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.maxDailySeries int - The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxHourlySeries + The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxHourlySeries -storage.maxHourlySeries int - The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxDailySeries + The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxDailySeries -storage.minFreeDiskSpaceBytes size The minimum free disk space at -storageDataPath after which the storage stops accepting new data Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 10000000) diff --git a/docs/ExtendedPromQL.md b/docs/ExtendedPromQL.md index 56d919a4b..2c81204df 100644 --- a/docs/ExtendedPromQL.md +++ b/docs/ExtendedPromQL.md @@ -5,4 +5,4 @@ weight: 100 # MetricsQL old page -The page has been moved to [MetricsQL](./MetricsQL.md). +The page has been moved to [MetricsQL](https://docs.victoriametrics.com/metricsql/). diff --git a/docs/FAQ.md b/docs/FAQ.md index e9b1e8da3..e8f91c2c8 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -16,40 +16,40 @@ To provide the best monitoring solution. ## Who uses VictoriaMetrics? -See [case studies](./CaseStudies.md). +See [case studies](https://docs.victoriametrics.com/casestudies/). ## Which features does VictoriaMetrics have? -See [these docs](./Single-Server-VictoriaMetrics.md#prominent-features). +See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#prominent-features). ## Are there performance comparisons with other solutions? -Yes. See [these benchmarks](./Articles.md#benchmarks). +Yes. See [these benchmarks](https://docs.victoriametrics.com/articles/#benchmarks). ## How to start using VictoriaMetrics? -See [these docs](./Quick-Start.md). +See [these docs](https://docs.victoriametrics.com/quick-start/). ## Hot to contribute to VictoriaMetrics? -See [these docs](./CONTRIBUTING.md). +See [these docs](https://docs.victoriametrics.com/contributing/). ## Does VictoriaMetrics support replication? -Yes. See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details. +Yes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details. ## Can I use VictoriaMetrics instead of Prometheus? Yes in most cases. VictoriaMetrics can substitute Prometheus in the following aspects: -* Prometheus-compatible service discovery and target scraping can be done with [vmagent](./vmagent.md) and with single-node VictoriaMetrics. See [these docs](./#how-to-scrape-prometheus-exporters-such-as-node-exporter). -* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](./vmalert.md). -* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](./#grafana-setup). +* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/vmagent/) and with single-node VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter). +* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](https://docs.victoriametrics.com/vmalert/). +* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#grafana-setup). ## What is the difference between vmagent and Prometheus? -While both [vmagent](./vmagent.md) and Prometheus may scrape Prometheus targets (aka `/metrics` pages) -according to the provided Prometheus-compatible [scrape configs](./sd_configs.md#scrape_configs) +While both [vmagent](https://docs.victoriametrics.com/vmagent/) and Prometheus may scrape Prometheus targets (aka `/metrics` pages) +according to the provided Prometheus-compatible [scrape configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) and send data to multiple remote storage systems, vmagent has the following additional features: * vmagent usually requires lower amounts of CPU, RAM and disk IO compared to Prometheus when scraping an enormous number of targets (more than 1000) @@ -59,30 +59,30 @@ and send data to multiple remote storage systems, vmagent has the following addi with a hardcoded retention of 2 hours. * vmagent may accept, relabel and filter data obtained via multiple data ingestion protocols in addition to data scraped from Prometheus targets. That means it supports both `pull` and `push` protocols for data ingestion. - See [these docs](./vmagent.md#features) for details. -* vmagent may be used in different [use cases](./vmagent.md#use-cases): - * [IoT and edge monitoring](./vmagent.md#iot-and-edge-monitoring) - * [Drop-in replacement for Prometheus](./vmagent.md#drop-in-replacement-for-prometheus) - * [Statsd alternative](./vmagent.md#statsd-alternative) - * [Flexible metrics relay](./vmagent.md#flexible-metrics-relay) - * [Replication and high availability](./vmagent.md#replication-and-high-availability) - * [Sharding among remote storages](./vmagent.md#sharding-among-remote-storages) - * [Relabeling and filtering](./vmagent.md#relabeling-and-filtering) - * [Splitting data streams among multiple systems](./vmagent.md#splitting-data-streams-among-multiple-systems) - * [Prometheus remote_write proxy](./vmagent.md#prometheus-remote_write-proxy) - * [remote_write for clustered version](./vmagent.md#remote_write-for-clustered-version) + See [these docs](https://docs.victoriametrics.com/vmagent/#features) for details. +* vmagent may be used in different [use cases](https://docs.victoriametrics.com/vmagent/#use-cases): + * [IoT and edge monitoring](https://docs.victoriametrics.com/vmagent/#iot-and-edge-monitoring) + * [Drop-in replacement for Prometheus](https://docs.victoriametrics.com/vmagent/#drop-in-replacement-for-prometheus) + * [Statsd alternative](https://docs.victoriametrics.com/vmagent/#statsd-alternative) + * [Flexible metrics relay](https://docs.victoriametrics.com/vmagent/#flexible-metrics-relay) + * [Replication and high availability](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability) + * [Sharding among remote storages](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages) + * [Relabeling and filtering](https://docs.victoriametrics.com/vmagent/#relabeling-and-filtering) + * [Splitting data streams among multiple systems](https://docs.victoriametrics.com/vmagent/#splitting-data-streams-among-multiple-systems) + * [Prometheus remote_write proxy](https://docs.victoriametrics.com/vmagent/#prometheus-remote_write-proxy) + * [remote_write for clustered version](https://docs.victoriametrics.com/vmagent/#remote_write-for-clustered-version) ## What is the difference between vmagent and Prometheus agent? -Both [vmagent](./vmagent.md) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences: +Both [vmagent](https://docs.victoriametrics.com/vmagent/) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficiently scrape Prometheus-compatible targets at the edge. They have the following differences: * vmagent usually requires lower amounts of CPU, RAM and disk IO compared to the Prometheus agent. -* Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON – see [these docs](./vmagent.md#features). -* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](./vmagent.md#scraping-big-number-of-targets). -* vmagent supports [improved relabeling](./vmagent.md#relabeling). -* vmagent can limit the number of scraped metrics per target – see [these docs](./vmagent.md#cardinality-limiter). -* vmagent supports loading scrape configs from multiple files – see [these docs](./vmagent.md#loading-scrape-configs-from-multiple-files). -* vmagent supports data reading and data writing from/to Kafka – see [these docs](./vmagent.md#kafka-integration). +* Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON – see [these docs](https://docs.victoriametrics.com/vmagent/#features). +* vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). +* vmagent supports [improved relabeling](https://docs.victoriametrics.com/vmagent/#relabeling). +* vmagent can limit the number of scraped metrics per target – see [these docs](https://docs.victoriametrics.com/vmagent/#cardinality-limiter). +* vmagent supports loading scrape configs from multiple files – see [these docs](https://docs.victoriametrics.com/vmagent/#loading-scrape-configs-from-multiple-files). +* vmagent supports data reading and data writing from/to Kafka – see [these docs](https://docs.victoriametrics.com/vmagent/#kafka-integration). * vmagent can read and update scrape configs from http and https URLs, while the Prometheus agent can read them only from the local file system. ## Is it safe to enable [remote write](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) in Prometheus? @@ -90,7 +90,7 @@ Both [vmagent](./vmagent.md) and [Prometheus agent](https://prometheus.io/blog/2 Yes. Prometheus continues writing data to local storage after enabling remote write, so all the existing local storage data and new data is available for querying via Prometheus as usual. -It is recommended using [vmagent](./vmagent.md) for scraping Prometheus targets +It is recommended using [vmagent](https://docs.victoriametrics.com/vmagent/) for scraping Prometheus targets and writing data to VictoriaMetrics. ## How does VictoriaMetrics compare to other remote storage solutions for Prometheus such as [M3DB](https://github.com/m3db/m3), [Thanos](https://github.com/thanos-io/thanos), [Cortex](https://github.com/cortexproject/cortex), [Mimir](https://github.com/grafana/mimir), etc.? @@ -113,13 +113,13 @@ VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/t ## What is the difference between VictoriaMetrics and [QuestDB](https://questdb.io/)? * QuestDB needs more than 20x storage space than VictoriaMetrics. This translates to higher storage costs and slower queries over historical data, which must be read from the disk. -* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](./#how-to-start-victoriametrics). -* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details. -* VictoriaMetrics can be queried via the [Prometheus querying API](./#prometheus-querying-api-usage) and via [Graphite's API](./#graphite-api-usage). -* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](./#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana. -* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](./vmagent.md), while QuestDB has no integration with Prometheus. -* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](./#how-to-import-time-series-data)). -* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](./#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data). +* QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics). +* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details. +* VictoriaMetrics can be queried via the [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage). +* Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana. +* Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/vmagent/), while QuestDB has no integration with Prometheus. +* QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-import-time-series-data)). +* [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](https://docs.victoriametrics.com/#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data). ## What is the difference between VictoriaMetrics and [Grafana Mimir](https://github.com/grafana/mimir)? @@ -132,13 +132,13 @@ See also [Grafana Mimir vs VictoriaMetrics benchmark](https://victoriametrics.co VictoriaMetrics is similar to Cortex in the following aspects: -* Both systems accept data from [vmagent](./vmagent.md) or Prometheus +* Both systems accept data from [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus via the standard [remote_write API](https://prometheus.io/docs/practices/remote_write/), so there is no need for running sidecars unlike in [Thanos](https://github.com/thanos-io/thanos)' case. -* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](./Cluster-VictoriaMetrics.md#multitenancy). -* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](./Cluster-VictoriaMetrics.md#replication-and-data-safety). -* Both systems scale horizontally to multiple nodes. See [these docs](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability) for details. -* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](./vmalert.md). +* Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). +* Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety). +* Both systems scale horizontally to multiple nodes. See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability) for details. +* Both systems support alerting and recording rules via the corresponding tools such as [vmalert](https://docs.victoriametrics.com/vmalert/). * Both systems can be queried via the [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/) and integrate perfectly with Grafana. The main differences between Cortex and VictoriaMetrics: @@ -147,17 +147,17 @@ The main differences between Cortex and VictoriaMetrics: * Cortex heavily relies on third-party services such as Consul, Memcache, DynamoDB, BigTable, Cassandra, etc. This may increase operational complexity and reduce system reliability compared to VictoriaMetrics' case, which doesn't use any external services. Compare [Cortex' Architecture](https://github.com/cortexproject/cortex/blob/master/docs/architecture.md) - to [VictoriaMetrics' architecture](./Cluster-VictoriaMetrics.md#architecture-overview). -* VictoriaMetrics provides [production-ready single-node solution](./Single-Server-VictoriaMetrics.md), + to [VictoriaMetrics' architecture](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview). +* VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/), which is much easier to set up and operate than a Cortex cluster. * Cortex may lose up to 12 hours of recent data on Ingestor failure – see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss). VictoriaMetrics may lose only a few seconds of recent data, which isn't synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704). -* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](./CaseStudies.md). +* Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://docs.victoriametrics.com/casestudies/). * VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary. - See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details. -* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. -* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage). + See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details. +* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. +* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage). ## What is the difference between VictoriaMetrics and [Thanos](https://github.com/thanos-io/thanos)? @@ -171,33 +171,33 @@ The main differences between Cortex and VictoriaMetrics: VictoriaMetrics works perfectly with HDD-based block storage – there is no need for using more expensive SSD or NVMe disks in most cases. * Thanos may lose up to 2 hours of recent data, which wasn't uploaded yet to object storage. VictoriaMetrics may lose only a few seconds of recent data, which hasn't been synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704). -* VictoriaMetrics provides a [production-ready single-node solution](./Single-Server-VictoriaMetrics.md), +* VictoriaMetrics provides a [production-ready single-node solution](https://docs.victoriametrics.com/single-server-victoriametrics/), which is much easier to set up and operate than Thanos components. * Thanos may be harder to set up and operate compared to VictoriaMetrics, since it has more moving parts, which can be connected with fewer reliable networks. See [this article for details](https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683). * Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/). * VictoriaMetrics accepts data via multiple popular data ingestion protocols in addition to the Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary. - See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details. -* VictoriaMetrics provides the [MetricsQL](./MetricsQL.md) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. -* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage). + See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details. +* VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/metricsql/) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language. +* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage). ## How does VictoriaMetrics compare to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/)? * VictoriaMetrics requires [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and it [works faster](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae). * VictoriaMetrics needs lower amounts of storage space than InfluxDB for production data. -* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](./MetricsQL.md). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details. +* VictoriaMetrics doesn't support InfluxQL or Flux but provides a better query language – [MetricsQL](https://docs.victoriametrics.com/metricsql/). See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details. * VictoriaMetrics accepts data in multiple popular data ingestion protocols in addition to InfluxDB – Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary. - See [these docs](./Single-Server-VictoriaMetrics.md#how-to-import-time-series-data) for details. -* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage). + See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-import-time-series-data) for details. +* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage). ## How does VictoriaMetrics compare to [TimescaleDB](https://www.timescale.com/)? * TimescaleDB insists on using SQL as a query language. While SQL is more powerful than PromQL, this power is rarely required during typical usages of a TSDB. Real-world queries usually [look clearer and simpler when written in PromQL than in SQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085). * VictoriaMetrics requires [up to 70x less storage space compared to TimescaleDB](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) for storing the same amount of time series data. The gap in storage space usage can be lowered from 70x to 3x if [compression in TimescaleDB is properly configured](https://docs.timescale.com/latest/using-timescaledb/compression) (it isn't an easy task in general :)). * VictoriaMetrics requires up to 10x less CPU and RAM resources than TimescaleDB for processing production data. See [this article](https://abiosgaming.com/press/high-cardinality-aggregations/) for details. -* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](./#how-to-start-victoriametrics)). +* TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics)). * VictoriaMetrics accepts data in multiple popular data ingestion protocols – InfluxDB, OpenTSDB, Graphite, CSV – while TimescaleDB supports only SQL inserts. -* VictoriaMetrics can be queried via [Graphite's API](./#graphite-api-usage). +* VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage). ## Does VictoriaMetrics use Prometheus technologies like other clustered TSDBs built on top of Prometheus such as [Thanos](https://github.com/thanos-io/thanos) or [Cortex](https://github.com/cortexproject/cortex)? @@ -208,7 +208,7 @@ The architecture is [optimized for storing and querying large amounts of time se The following versions are open source and free: -* [Single-node version](./Single-Server-VictoriaMetrics.md). +* [Single-node version](https://docs.victoriametrics.com/single-server-victoriametrics/). * [Cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster). We provide commercial support for both versions. [Contact us](mailto:info@victoriametrics.com) for the pricing. @@ -231,12 +231,12 @@ if a query covers 1000 metrics with 10K values each, then the remote read API ha This is slow and expensive. Prometheus' remote read API isn't intended for querying foreign data – aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details. -So just query VictoriaMetrics directly via [vmui](./#vmui), the [Prometheus Querying API](./#prometheus-querying-api-usage) -or via [Prometheus datasource in Grafana](./#grafana-setup). +So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), the [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) +or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/#grafana-setup). ## Does VictoriaMetrics deduplicate data from Prometheus instances scraping the same targets (aka `HA pairs`)? -Yes. See [these docs](./Single-Server-VictoriaMetrics.md#deduplication) for details. +Yes. See [these docs](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication) for details. ## Where is the source code of VictoriaMetrics? @@ -254,21 +254,21 @@ and scales horizontally to multiple nodes. ## What is the difference between single-node and cluster versions of VictoriaMetrics? -Both [single-node](./Single-Server-VictoriaMetrics.md) and -[cluster](./Cluster-VictoriaMetrics.md) versions of VictoriaMetrics +Both [single-node](https://docs.victoriametrics.com/single-server-victoriametrics/) and +[cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) versions of VictoriaMetrics share the core source code, so they have many common features. They have the following differences though: -* [Single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) runs on a single host, - while [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) can scale to many hosts. +* [Single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) runs on a single host, + while [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can scale to many hosts. Single-node VictoriaMetrics scales vertically though, e.g. its capacity and performance scales almost linearly when increasing available CPU, RAM, disk IO and disk space. See [an article about vertical scalability of a single-node VictoriaMetrics](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae). -* Cluster version of VictoriaMetrics supports [multitenancy](./Cluster-VictoriaMetrics.md#multitenancy), +* Cluster version of VictoriaMetrics supports [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy), while single-node VictoriaMetrics doesn't support it. * Cluster version of VictoriaMetrics supports data replication, while single-node VictoriaMetrics relies on the durability of the persistent storage pointed by `-storageDataPath` command-line flag. - See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details. + See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details. * Single-node VictoriaMetrics provides higher capacity and performance comparing to cluster version of VictoriaMetrics when running on the same hardware with the same amounts of CPU and RAM, since it has no overhead on data transfer @@ -290,7 +290,7 @@ File bugs and feature requests [here](https://github.com/VictoriaMetrics/Victori ## Where can I find information about multi-tenancy? -See [these docs](./Cluster-VictoriaMetrics.md#multitenancy). Multitenancy is supported only by the [cluster version](./Cluster-VictoriaMetrics.md) of VictoriaMetrics. +See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). Multitenancy is supported only by the [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/) of VictoriaMetrics. ## How to set a memory limit for VictoriaMetrics components? @@ -298,10 +298,10 @@ All the VictoriaMetrics components provide command-line flags to control the siz Memory usage for VictoriaMetrics components can be tuned according to the following docs: -* [Resource usage limits for single-node VictoriaMetrics](./#resource-usage-limits) -* [Resource usage limits for cluster VictoriaMetrics](./Cluster-VictoriaMetrics.md#resource-usage-limits) -* [Troubleshooting for vmagent](./vmagent.md#troubleshooting) -* [Troubleshooting for single-node VictoriaMetrics](./#troubleshooting) +* [Resource usage limits for single-node VictoriaMetrics](https://docs.victoriametrics.com/#resource-usage-limits) +* [Resource usage limits for cluster VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) +* [Troubleshooting for vmagent](https://docs.victoriametrics.com/vmagent/#troubleshooting) +* [Troubleshooting for single-node VictoriaMetrics](https://docs.victoriametrics.com/#troubleshooting) ## How can I run VictoriaMetrics on FreeBSD/OpenBSD? @@ -311,13 +311,13 @@ or use pre-built binaries from [releases page](https://github.com/VictoriaMetric ## Does VictoriaMetrics support the Graphite query language? -Yes. See [these docs](./#graphite-api-usage). +Yes. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage). ## What is an active time series? A time series is uniquely identified by its name plus a set of its labels. For example, `temperature{city="NY",country="US"}` and `temperature{city="SF",country="US"}` are two distinct series, since they differ by the `city` label. A time series is considered active if it receives at least a single new sample during the last hour. -The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](./#monitoring) for details. +The number of active time series is displayed on the official Grafana dashboard for VictoriaMetrics - see [these docs](https://docs.victoriametrics.com/#monitoring) for details. ## What is high churn rate? @@ -335,10 +335,10 @@ The main reason for high churn rate is a metric label with frequently changed va * A `hash` or `uuid` label, which changes frequently. The solution against high churn rate is to identify and eliminate labels with frequently changed values. -[Cardinality explorer](./#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data -before it gets ingested into database with [stream aggregation](./stream-aggregation.md). +[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help determining these labels. If labels can't be removed, try pre-aggregating data +before it gets ingested into database with [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/). -The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](./#monitoring) for details. +The official Grafana dashboards for VictoriaMetrics contain graphs for churn rate - see [these docs](https://docs.victoriametrics.com/#monitoring) for details. ## What is high cardinality? @@ -350,10 +350,10 @@ a large number of unique values, which presents a big share of the ingested time * `url` * `ip` -The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](./#cardinality-explorer). +The solution is to identify and remove the source of high cardinality with the help of [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer). The official Grafana dashboards for VictoriaMetrics contain graphs, which show the number of active time series - -see [these docs](./#monitoring) for details. +see [these docs](https://docs.victoriametrics.com/#monitoring) for details. ## What is a slow insert? @@ -361,34 +361,34 @@ VictoriaMetrics maintains in-memory cache for mapping of [active time series](#w The cache size depends on the available memory for VictoriaMetrics in the host system. If the information about all the active time series doesn't fit the cache, then VictoriaMetrics needs to read and unpack the information from disk on every incoming sample for time series missing in the cache. This operation is much slower than the cache lookup, so such an insert is named a `slow insert`. -A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](./#monitoring) indicates +A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) indicates a memory shortage for the current number of [active time series](#what-is-an-active-time-series). Such a condition usually leads to a significant slowdown for data ingestion and to significantly increased disk IO and CPU usage. The solution is to add more memory or to reduce the number of [active time series](#what-is-an-active-time-series). -[Cardinality explorer](./#cardinality-explorer) can be helpful for locating the source of high number of active time series. +[Cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can be helpful for locating the source of high number of active time series. ## How to optimize MetricsQL query? See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986). -VictoriaMetrics also provides [query tracer](./#query-tracing) and [cardinality explorer](./#cardinality-explorer), +VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing) and [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer), which can help during query optimization. -See also [troubleshooting slow queries](./Troubleshooting.md#slow-queries). +See also [troubleshooting slow queries](https://docs.victoriametrics.com/troubleshooting/#slow-queries). ## Which VictoriaMetrics type is recommended for use in production - single-node or cluster? -Both [single-node VictoriaMetrics](./Single-Server-VictoriaMetrics.md) and -[VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) are production-ready. +Both [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) and +[VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) are production-ready. Single-node VictoriaMetrics is able to handle quite big workloads in production -with tens of millions of [active time series](./FAQ.md#what-is-an-active-time-series) -at the ingestion rate of million of samples per second. See [this case study](./CaseStudies.md#wixcom). +with tens of millions of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) +at the ingestion rate of million of samples per second. See [this case study](https://docs.victoriametrics.com/casestudies/#wixcom). Single-node VictoriaMetrics requires lower amounts of CPU and RAM for handling the same workload comparing to cluster version of VictoriaMetrics, since it doesn't need to pass the encoded data over the network -between [cluster components](./Cluster-VictoriaMetrics.md#architecture-overview). +between [cluster components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview). The performance of a single-node VictoriaMetrics scales almost perfectly with the available CPU, RAM and disk IO resources on the host where it runs - see [this article](https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae). @@ -399,40 +399,40 @@ Given the facts above **it is recommended to use single-node VictoriaMetrics in Cluster version of VictoriaMetrics may be preferred over single-node VictoriaMetrics in the following relatively rare cases: -- If [multitenancy support](./Cluster-VictoriaMetrics.md#multitenancy) is needed, +- If [multitenancy support](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) is needed, since single-node VictoriaMetrics doesn't support multitenancy. Though it is possible to run multiple single-node VictoriaMetrics instances - one per each tenant - and route incoming requests from particular tenant to the needed VictoriaMetrics instance - via [vmauth](./vmauth.md). + via [vmauth](https://docs.victoriametrics.com/vmauth/). - If the current workload cannot be handled by a single-node VictoriaMetrics. For example, if you are going to ingest hundreds of millions of active time series at ingestion rates exceeding a million samples per second, then it is better to use cluster version of VictoriaMetrics, - since its capacity can [scale horizontally with the number of nodes in the cluster](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability). + since its capacity can [scale horizontally with the number of nodes in the cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability). ## How to migrate data from single-node VictoriaMetrics to cluster version? Single-node VictoriaMetrics stores data on disk in slightly different format comparing to cluster version of VictoriaMetrics. So it is impossible to just copy the on-disk data from `-storageDataPath` directory from single-node VictoriaMetrics to a `vmstorage` node in VictoriaMetrics cluster. -If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](./vmctl.md#migrating-data-from-victoriametrics). +If you need migrating data from single-node VictoriaMetrics to cluster version, then [follow these instructions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). ## Why isn't MetricsQL 100% compatible with PromQL? -[MetricsQL](./MetricsQL.md) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. +[MetricsQL](https://docs.victoriametrics.com/metricsql/) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. ## How to migrate data from Prometheus to VictoriaMetrics? -Please see [these docs](./vmctl.md#migrating-data-from-prometheus). +Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus). ## How to migrate data from InfluxDB to VictoriaMetrics? -Please see [these docs](./vmctl.md#migrating-data-from-influxdb-1x). +Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x). ## How to migrate data from OpenTSDB to VictoriaMetrics? -Please see [these docs](./vmctl.md#migrating-data-from-opentsdb). +Please see [these docs](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb). ## How to migrate data from Graphite to VictoriaMetrics? -Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). +Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). ## Why do the same metrics have differences in VictoriaMetrics' and Prometheus' dashboards? @@ -442,19 +442,19 @@ The query engine may behave differently for some functions. Please see [this art ## If downsampling and deduplication are enabled how will this work? -[Deduplication](./#deduplication) is a special case of zero-offset [downsampling](./#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling +[Deduplication](https://docs.victoriametrics.com/#deduplication) is a special case of zero-offset [downsampling](https://docs.victoriametrics.com/#downsampling). So, if both downsampling and deduplication are enabled, then deduplication is replaced by zero-offset downsampling ## How to upgrade or downgrade VictoriaMetrics without downtime? -Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](./#how-to-upgrade-victoriametrics). +Single-node VictoriaMetrics cannot be restarted / upgraded or downgraded without downtime, since it needs to be gracefully shut down and then started again. See [how to upgrade VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics). -[Cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) can be restarted / upgraded / downgraded without downtime according to [these instructions](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes). +[Cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) can be restarted / upgraded / downgraded without downtime according to [these instructions](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes). ## Why VictoriaMetrics misses automatic data re-balancing between vmstorage nodes? VictoriaMetrics doesn't rebalance data between `vmstorage` nodes when new `vmstorage` nodes are added to the cluster. This means that newly added `vmstorage` nodes will have less data at `-storageDataPath` comparing to the old `vmstorage` nodes -until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](./#retention). +until the historical data is removed from the old `vmstorage` nodes when it goes outside the configured [retention](https://docs.victoriametrics.com/#retention). The automatic rebalancing is the process of moving data between `vmstorage` nodes, so every node has the same amounts of data eventually. It is disabled by default because it may consume additional CPU, network bandwidth and disk IO at `vmstorage` nodes for long periods of time, @@ -463,22 +463,22 @@ which, in turn, can negatively impact VictoriaMetrics cluster availability. Additionally, it is unclear how to handle the automatic re-balancing if cluster configuration changes when the re-balancing is in progress. The amounts of data stored in `vmstorage` becomes equal among old `vmstorage` nodes and new `vmstorage` nodes -after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](./#retention). +after historical data is removed from the old `vmstorage` nodes because it goes outside of configured [retention](https://docs.victoriametrics.com/#retention). The data ingestion load becomes even between old `vmstorage` nodes and new `vmstorage` nodes almost immediately after adding new `vmstorage` nodes to the cluster, since `vminsert` nodes evenly distribute incoming time series among the nodes specified in `-storageNode` command-line flag. The newly added `vmstorage` nodes may experience -increased load during the first couple of minutes because they need to register [active time series](./FAQ.md#what-is-an-active-time-series). +increased load during the first couple of minutes because they need to register [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). The query load becomes even between old `vmstorage` nodes and new `vmstorage` nodes after most of queries are executed over time ranges with data covered by new `vmstorage` nodes. Usually the most of queries are received -from [alerting and recording rules](./vmalert.md), which query data on limited time ranges +from [alerting and recording rules](https://docs.victoriametrics.com/vmalert/), which query data on limited time ranges such as a few hours or few days at max. This means that the query load between old `vmstorage` nodes and new `vmstorage` nodes should become even in a few hours / days after adding new `vmstorage` nodes. ## Why VictoriaMetrics misses automatic recovery of replication factor? -VictoriaMetrics doesn't restore [replication factor](./Cluster-VictoriaMetrics.md#replication-and-data-safety) +VictoriaMetrics doesn't restore [replication factor](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) when some of `vmstorage` nodes are removed from the cluster because of the following reasons: - Automatic replication factor recovery needs copying non-trivial amounts of data between the remaining `vmstorage` nodes. @@ -488,5 +488,5 @@ when some of `vmstorage` nodes are removed from the cluster because of the follo - It is unclear when the automatic replication factor recovery must be started. How to distinguish the expected temporary `vmstorage` node unavailability because of maintenance, upgrade or config changes from permanent loss of data at the `vmstorage` node? -It is recommended reading [replication and data safety docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) +It is recommended reading [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for more details. diff --git a/docs/MetricsQL.md b/docs/MetricsQL.md index 4158d3d8c..64b343f53 100644 --- a/docs/MetricsQL.md +++ b/docs/MetricsQL.md @@ -19,11 +19,11 @@ However, there are some [intentional differences](https://medium.com/@romanhavro [Standalone MetricsQL package](https://godoc.org/github.com/VictoriaMetrics/metricsql) can be used for parsing MetricsQL in external apps. If you are unfamiliar with PromQL, then it is suggested reading [this tutorial for beginners](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) -and introduction into [basic querying via MetricsQL](./keyConcepts.md#metricsql). +and introduction into [basic querying via MetricsQL](https://docs.victoriametrics.com/keyconcepts/#metricsql). The following functionality is implemented differently in MetricsQL compared to PromQL. This improves user experience: -* MetricsQL takes into account the last [raw sample](./keyConcepts.md#raw-samples) before the lookbehind window +* MetricsQL takes into account the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) before the lookbehind window in square brackets for [increase](#increase) and [rate](#rate) functions. This allows returning the exact results users expect for `increase(metric[$__interval])` queries instead of incomplete results Prometheus returns for such queries. Prometheus misses the increase between the last sample before the lookbehind window and the first sample inside the lookbehind window. @@ -32,9 +32,9 @@ The following functionality is implemented differently in MetricsQL compared to which may significantly differ from the expected results. This addresses [this issue from Prometheus](https://github.com/prometheus/prometheus/issues/3746). See technical details about VictoriaMetrics and Prometheus calculations for [rate](#rate) and [increase](#increase) [in this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1215#issuecomment-850305711). -* MetricsQL returns the expected non-empty responses for [rate](#rate) function when Grafana or [vmui](./README.md#vmui) - passes `step` values smaller than the interval between [raw samples](./keyConcepts.md#raw-samples) - to [/api/v1/query_range](./keyConcepts.md#range-query). +* MetricsQL returns the expected non-empty responses for [rate](#rate) function when Grafana or [vmui](https://docs.victoriametrics.com/#vmui) + passes `step` values smaller than the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) + to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). This addresses [this issue from Grafana](https://github.com/grafana/grafana/issues/11451). See also [this blog post](https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/). * MetricsQL treats `scalar` type the same as `instant vector` without labels, since subtle differences between these types usually confuse users. @@ -58,24 +58,24 @@ and provides additional functionality mentioned below, which is aimed towards so Feel free [filing a feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you think MetricsQL misses certain useful functionality. This functionality can be evaluated at [VictoriaMetrics playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/) -or at your own [VictoriaMetrics instance](./README.md#how-to-start-victoriametrics). +or at your own [VictoriaMetrics instance](https://docs.victoriametrics.com/#how-to-start-victoriametrics). The list of MetricsQL features on top of PromQL: * Graphite-compatible filters can be passed via `{__graphite__="foo.*.bar"}` syntax. - See [these docs](./README.md#selecting-graphite-metrics). - VictoriaMetrics can be used as Graphite datasource in Grafana. See [these docs](./README.md#graphite-api-usage) for details. + See [these docs](https://docs.victoriametrics.com/#selecting-graphite-metrics). + VictoriaMetrics can be used as Graphite datasource in Grafana. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage) for details. See also [label_graphite_group](#label_graphite_group) function, which can be used for extracting the given groups from Graphite metric name. * Lookbehind window in square brackets for [rollup functions](#rollup-functions) may be omitted. VictoriaMetrics automatically selects the lookbehind window - depending on the `step` query arg passed to [/api/v1/query_range](./keyConcepts.md#range-query) - and the real interval between [raw samples](./keyConcepts.md#raw-samples) (aka `scrape_interval`). + depending on the `step` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) + and the real interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) (aka `scrape_interval`). For instance, the following query is valid in VictoriaMetrics: `rate(node_network_receive_bytes_total)`. It is roughly equivalent to `rate(node_network_receive_bytes_total[$__interval])` when used in Grafana. The difference is documented in [rate() docs](#rate). * Numeric values can contain `_` delimiters for better readability. For example, `1_234_567_890` can be used in queries instead of `1234567890`. -* [Series selectors](./keyConcepts.md#filtering) accept multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}` +* [Series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering) accept multiple `or` filters. For example, `{env="prod",job="a" or env="dev",job="b"}` selects series with `{env="prod",job="a"}` or `{env="dev",job="b"}` labels. - See [these docs](./keyConcepts.md#filtering-by-multiple-or-filters) for details. + See [these docs](https://docs.victoriametrics.com/keyconcepts/#filtering-by-multiple-or-filters) for details. * Support for matching against multiple numeric constants via `q == (C1, ..., CN)` and `q != (C1, ..., CN)` syntax. For example, `status_code == (300, 301, 304)` returns `status_code` metrics with one of `300`, `301` or `304` values. * Support for `group_left(*)` and `group_right(*)` for copying all the labels from time series on the `one` side @@ -151,27 +151,27 @@ MetricsQL provides the following functions: ### Rollup functions -**Rollup functions** (aka range functions or window functions) calculate rollups over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window for the [selected time series](./keyConcepts.md#filtering). -For example, `avg_over_time(temperature[24h])` calculates the average temperature over [raw samples](./keyConcepts.md#raw-samples) for the last 24 hours. +**Rollup functions** (aka range functions or window functions) calculate rollups over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window for the [selected time series](https://docs.victoriametrics.com/keyconcepts/#filtering). +For example, `avg_over_time(temperature[24h])` calculates the average temperature over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for the last 24 hours. Additional details: * If rollup functions are used for building graphs in Grafana, then the rollup is calculated independently per each point on the graph. For example, every point for `avg_over_time(temperature[24h])` graph shows the average temperature for the last 24 hours ending at this point. - The interval between points is set as `step` query arg passed by Grafana to [/api/v1/query_range](./keyConcepts.md#range-query). -* If the given [series selector](./keyConcepts.md#filtering) returns multiple time series, + The interval between points is set as `step` query arg passed by Grafana to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). +* If the given [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) returns multiple time series, then rollups are calculated individually per each returned series. * If lookbehind window in square brackets is missing, then it is automatically set to the following value: - - To `step` value passed to [/api/v1/query_range](./keyConcepts.md#range-query) or [/api/v1/query](./keyConcepts.md#instant-query) + - To `step` value passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) for all the [rollup functions](#rollup-functions) except of [default_rollup](#default_rollup) and [rate](#rate). This value is known as `$__interval` in Grafana or `1i` in MetricsQL. For example, `avg_over_time(temperature)` is automatically transformed to `avg_over_time(temperature[1i])`. - - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](./keyConcepts.md#raw-samples) + - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for [default_rollup](#default_rollup) and [rate](#rate) functions. This allows avoiding unexpected gaps on the graph when `step` is smaller than `scrape_interval`. -* Every [series selector](./keyConcepts.md#filtering) in MetricsQL must be wrapped into a rollup function. +* Every [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) in MetricsQL must be wrapped into a rollup function. Otherwise, it is automatically wrapped into [default_rollup](#default_rollup). For example, `foo{bar="baz"}` is automatically converted to `default_rollup(foo{bar="baz"})` before performing the calculations. -* If something other than [series selector](./keyConcepts.md#filtering) is passed to rollup function, +* If something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) is passed to rollup function, then the inner arg is automatically converted to a [subquery](#subqueries). * All the rollup functions accept optional `keep_metric_names` modifier. If it is set, then the function keeps metric names in results. See [these docs](#keep_metric_names). @@ -183,7 +183,7 @@ The list of supported rollup functions: #### absent_over_time `absent_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns 1 -if the given lookbehind window `d` doesn't contain [raw samples](./keyConcepts.md#raw-samples). Otherwise, it returns an empty result. +if the given lookbehind window `d` doesn't contain [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples). Otherwise, it returns an empty result. This function is supported by PromQL. @@ -192,9 +192,9 @@ See also [present_over_time](#present_over_time). #### aggr_over_time `aggr_over_time(("rollup_func1", "rollup_func2", ...), series_selector[d])` is a [rollup function](#rollup-functions), -which calculates all the listed `rollup_func*` for [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d`. +which calculates all the listed `rollup_func*` for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. The calculations are performed individually per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). `rollup_func*` can contain any rollup function. For instance, `aggr_over_time(("min_over_time", "max_over_time", "rate"), m[d])` would calculate [min_over_time](#min_over_time), [max_over_time](#max_over_time) and [rate](#rate) for `m[d]`. @@ -202,8 +202,8 @@ would calculate [min_over_time](#min_over_time), [max_over_time](#max_over_time) #### ascent_over_time `ascent_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates -ascent of [raw sample](./keyConcepts.md#raw-samples) values on the given lookbehind window `d`. The calculations are performed individually -per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +ascent of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) values on the given lookbehind window `d`. The calculations are performed individually +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is useful for tracking height gains in GPS tracking. Metric names are stripped from the resulting rollups. @@ -214,10 +214,10 @@ See also [descent_over_time](#descent_over_time). #### avg_over_time `avg_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average value -over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -226,8 +226,8 @@ See also [median_over_time](#median_over_time), [min_over_time](#min_over_time) #### changes `changes(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of times -the [raw samples](./keyConcepts.md#raw-samples) changed on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) changed on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Unlike `changes()` in Prometheus it takes into account the change from the last sample before the given lookbehind window `d`. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. @@ -241,8 +241,8 @@ See also [changes_prometheus](#changes_prometheus). #### changes_prometheus `changes_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of times -the [raw samples](./keyConcepts.md#raw-samples) changed on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) changed on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It doesn't take into account the change from the last sample before the given lookbehind window `d` in the same way as Prometheus does. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. @@ -255,56 +255,56 @@ See also [changes](#changes). #### count_eq_over_time -`count_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](./keyConcepts.md#raw-samples) +`count_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which are equal to `eq`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_over_time](#count_over_time), [share_eq_over_time](#share_eq_over_time) and [count_values_over_time](#count_values_over_time). #### count_gt_over_time -`count_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](./keyConcepts.md#raw-samples) +`count_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which are bigger than `gt`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_over_time](#count_over_time) and [share_gt_over_time](#share_gt_over_time). #### count_le_over_time -`count_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](./keyConcepts.md#raw-samples) +`count_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which don't exceed `le`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_over_time](#count_over_time) and [share_le_over_time](#share_le_over_time). #### count_ne_over_time -`count_ne_over_time(series_selector[d], ne)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](./keyConcepts.md#raw-samples) +`count_ne_over_time(series_selector[d], ne)` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which aren't equal to `ne`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_over_time](#count_over_time) and [count_eq_over_time](#count_eq_over_time). #### count_over_time -`count_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`count_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -314,20 +314,20 @@ See also [count_le_over_time](#count_le_over_time), [count_gt_over_time](#count_ #### count_values_over_time -`count_values_over_time("label", series_selector[d])` is a [rollup function](#rollup-functions), which counts the number of [raw samples](./keyConcepts.md#raw-samples) +`count_values_over_time("label", series_selector[d])` is a [rollup function](#rollup-functions), which counts the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) with the same value over the given lookbehind window and stores the counts in a time series with an additional `label`, which contains each initial value. -The results are calculated independently per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The results are calculated independently per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_eq_over_time](#count_eq_over_time), [count_values](#count_values) and [distinct_over_time](#distinct_over_time) and [label_match](#label_match). #### decreases_over_time -`decreases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](./keyConcepts.md#raw-samples) -value decreases over the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`decreases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +value decreases over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -335,20 +335,20 @@ See also [increases_over_time](#increases_over_time). #### default_rollup -`default_rollup(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last [raw sample](./keyConcepts.md#raw-samples) -value on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). -Compared to [last_over_time](#last_over_time) it accounts for [staleness markers](./vmagent.md#prometheus-staleness-markers) to detect stale series. +`default_rollup(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +value on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). +Compared to [last_over_time](#last_over_time) it accounts for [staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) to detect stale series. If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value -passed to [/api/v1/query_range](./keyConcepts.md#range-query) or [/api/v1/query](./keyConcepts.md#instant-query), -while `scrape_interval` is the interval between [raw samples](./keyConcepts.md#raw-samples) for the selected time series. +passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query), +while `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for the selected time series. This allows avoiding unexpected gaps on the graph when `step` is smaller than the `scrape_interval`. #### delta `delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the difference between the last sample before the given lookbehind window `d` and the last sample at the given lookbehind window `d` -per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The behaviour of `delta()` function in MetricsQL is slightly different to the behaviour of `delta()` function in Prometheus. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. @@ -363,7 +363,7 @@ See also [increase](#increase) and [delta_prometheus](#delta_prometheus). `delta_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the difference between the first and the last samples at the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The behaviour of `delta_prometheus()` is close to the behaviour of `delta()` function in Prometheus. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. @@ -375,7 +375,7 @@ See also [delta](#delta). #### deriv `deriv(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second derivative over the given lookbehind window `d` -per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The derivative is calculated using linear regression. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -387,8 +387,8 @@ See also [deriv_fast](#deriv_fast) and [ideriv](#ideriv). #### deriv_fast `deriv_fast(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second derivative -using the first and the last [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +using the first and the last [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -396,9 +396,9 @@ See also [deriv](#deriv) and [ideriv](#ideriv). #### descent_over_time -`descent_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates descent of [raw sample](./keyConcepts.md#raw-samples) +`descent_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates descent of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) values on the given lookbehind window `d`. The calculations are performed individually per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is useful for tracking height loss in GPS tracking. @@ -408,8 +408,8 @@ See also [ascent_over_time](#ascent_over_time). #### distinct_over_time -`distinct_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number of unique [raw sample](./keyConcepts.md#raw-samples) -values on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`distinct_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number of unique [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +values on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -418,7 +418,7 @@ See also [count_values_over_time](#count_values_over_time). #### duration_over_time `duration_over_time(series_selector[d], max_interval)` is a [rollup function](#rollup-functions), which returns the duration in seconds -when time series returned from the given [series_selector](./keyConcepts.md#filtering) were present +when time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering) were present over the given lookbehind window `d`. It is expected that intervals between adjacent samples per each series don't exceed the `max_interval`. Otherwise, such intervals are considered as gaps and aren't counted. @@ -428,40 +428,40 @@ See also [lifetime](#lifetime) and [lag](#lag). #### first_over_time -`first_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the first [raw sample](./keyConcepts.md#raw-samples) -value on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`first_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the first [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +value on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). See also [last_over_time](#last_over_time) and [tfirst_over_time](#tfirst_over_time). #### geomean_over_time `geomean_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates [geometric mean](https://en.wikipedia.org/wiki/Geometric_mean) -over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### histogram_over_time `histogram_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates -[VictoriaMetrics histogram](https://godoc.org/github.com/VictoriaMetrics/metrics#Histogram) over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d`. It is calculated individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +[VictoriaMetrics histogram](https://godoc.org/github.com/VictoriaMetrics/metrics#Histogram) over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d`. It is calculated individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The resulting histograms are useful to pass to [histogram_quantile](#histogram_quantile) for calculating quantiles -over multiple [gauges](./keyConcepts.md#gauge). +over multiple [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). For example, the following query calculates median temperature by country over the last 24 hours: `histogram_quantile(0.5, sum(histogram_over_time(temperature[24h])) by (vmrange,country))`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### hoeffding_bound_lower `hoeffding_bound_lower(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates lower [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [hoeffding_bound_upper](#hoeffding_bound_upper). @@ -470,18 +470,18 @@ See also [hoeffding_bound_upper](#hoeffding_bound_upper). `hoeffding_bound_upper(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates upper [Hoeffding bound](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) for the given `phi` in the range `[0...1]`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [hoeffding_bound_lower](#hoeffding_bound_lower). #### holt_winters `holt_winters(series_selector[d], sf, tf)` is a [rollup function](#rollup-functions), which calculates Holt-Winters value -(aka [double exponential smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing)) for [raw samples](./keyConcepts.md#raw-samples) +(aka [double exponential smoothing](https://en.wikipedia.org/wiki/Exponential_smoothing#Double_exponential_smoothing)) for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given lookbehind window `d` using the given smoothing factor `sf` and the given trend factor `tf`. Both `sf` and `tf` must be in the range `[0...1]`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -489,8 +489,8 @@ See also [range_linear_regression](#range_linear_regression). #### idelta -`idelta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the difference between the last two [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`idelta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the difference between the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -501,9 +501,9 @@ See also [delta](#delta). #### ideriv `ideriv(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the per-second derivative based -on the last two [raw samples](./keyConcepts.md#raw-samples) +on the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) over the given lookbehind window `d`. The derivative is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -512,14 +512,14 @@ See also [deriv](#deriv). #### increase `increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase over the given lookbehind window `d` -per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Unlike Prometheus, it takes into account the last sample before the given lookbehind window `d` when calculating the result. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). This function is supported by PromQL. @@ -528,30 +528,30 @@ See also [increase_pure](#increase_pure), [increase_prometheus](#increase_promet #### increase_prometheus `increase_prometheus(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the increase -over the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It doesn't take into account the last sample before the given lookbehind window `d` when calculating the result in the same way as Prometheus does. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). See also [increase_pure](#increase_pure) and [increase](#increase). #### increase_pure `increase_pure(series_selector[d])` is a [rollup function](#rollup-functions), which works the same as [increase](#increase) except -of the following corner case - it assumes that [counters](./keyConcepts.md#counter) always start from 0, +of the following corner case - it assumes that [counters](https://docs.victoriametrics.com/keyconcepts/#counter) always start from 0, while [increase](#increase) ignores the first value in a series if it is too big. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). See also [increase](#increas) and [increase_prometheus](#increase_prometheus). #### increases_over_time -`increases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](./keyConcepts.md#raw-samples) -value increases over the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`increases_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +value increases over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -559,22 +559,22 @@ See also [decreases_over_time](#decreases_over_time). #### integrate -`integrate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the integral over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`integrate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the integral over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### irate `irate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the "instant" per-second increase rate over -the last two [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +the last two [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). This function is supported by PromQL. @@ -584,7 +584,7 @@ See also [rate](#rate) and [rollup_rate](#rollup_rate). `lag(series_selector[d])` is a [rollup function](#rollup-functions), which returns the duration in seconds between the last sample on the given lookbehind window `d` and the timestamp of the current point. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -592,8 +592,8 @@ See also [lifetime](#lifetime) and [duration_over_time](#duration_over_time). #### last_over_time -`last_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last [raw sample](./keyConcepts.md#raw-samples) -value on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`last_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +value on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is supported by PromQL. @@ -602,7 +602,7 @@ See also [first_over_time](#first_over_time) and [tlast_over_time](#tlast_over_t #### lifetime `lifetime(series_selector[d])` is a [rollup function](#rollup-functions), which returns the duration in seconds between the last and the first sample -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -611,19 +611,19 @@ See also [duration_over_time](#duration_over_time) and [lag](#lag). #### mad_over_time `mad_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) -over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [mad](#mad), [range_mad](#range_mad) and [outlier_iqr_over_time](#outlier_iqr_over_time). #### max_over_time -`max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`max_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the maximum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -631,20 +631,20 @@ See also [tmax_over_time](#tmax_over_time) and [min_over_time](#min_over_time). #### median_over_time -`median_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates median value over [raw samples](./keyConcepts.md#raw-samples) +`median_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates median value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [avg_over_time](#avg_over_time). #### min_over_time -`min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`min_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the minimum value over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -653,31 +653,31 @@ See also [tmin_over_time](#tmin_over_time) and [max_over_time](#max_over_time). #### mode_over_time `mode_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates [mode](https://en.wikipedia.org/wiki/Mode_(statistics)) -for [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d`. It is calculated individually per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). It is expected that [raw sample](./keyConcepts.md#raw-samples) +for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. It is calculated individually per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). It is expected that [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) values are discrete. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### outlier_iqr_over_time `outlier_iqr_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the last sample on the given lookbehind window `d` if its value is either smaller than the `q25-1.5*iqr` or bigger than `q75+1.5*iqr` where: -- `iqr` is an [Interquartile range](https://en.wikipedia.org/wiki/Interquartile_range) over [raw samples](./keyConcepts.md#raw-samples) on the lookbehind window `d` -- `q25` and `q75` are 25th and 75th [percentiles](https://en.wikipedia.org/wiki/Percentile) over [raw samples](./keyConcepts.md#raw-samples) on the lookbehind window `d`. +- `iqr` is an [Interquartile range](https://en.wikipedia.org/wiki/Interquartile_range) over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the lookbehind window `d` +- `q25` and `q75` are 25th and 75th [percentiles](https://en.wikipedia.org/wiki/Percentile) over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the lookbehind window `d`. The `outlier_iqr_over_time()` is useful for detecting anomalies in gauge values based on the previous history of values. For example, `outlier_iqr_over_time(memory_usage_bytes[1h])` triggers when `memory_usage_bytes` suddenly goes outside the usual value range for the last hour. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [outliers_iqr](#outliers_iqr). #### predict_linear `predict_linear(series_selector[d], t)` is a [rollup function](#rollup-functions), which calculates the value `t` seconds in the future using -linear interpolation over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d`. -The predicted value is calculated individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +linear interpolation over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. +The predicted value is calculated individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is supported by PromQL. @@ -685,7 +685,7 @@ See also [range_linear_regression](#range_linear_regression). #### present_over_time -`present_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns 1 if there is at least a single [raw sample](./keyConcepts.md#raw-samples) +`present_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns 1 if there is at least a single [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. Otherwise, an empty result is returned. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -694,11 +694,11 @@ This function is supported by PromQL. #### quantile_over_time -`quantile_over_time(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates `phi`-quantile over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`quantile_over_time(phi, series_selector[d])` is a [rollup function](#rollup-functions), which calculates `phi`-quantile over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The `phi` value must be in the range `[0...1]`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -707,32 +707,32 @@ See also [quantiles_over_time](#quantiles_over_time). #### quantiles_over_time `quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])` is a [rollup function](#rollup-functions), which calculates `phi*`-quantiles -over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). The function returns individual series per each `phi*` with `{phiLabel="phi*"}` label. `phi*` values must be in the range `[0...1]`. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [quantile_over_time](#quantile_over_time). #### range_over_time -`range_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates value range over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`range_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates value range over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). E.g. it calculates `max_over_time(series_selector[d]) - min_over_time(series_selector[d])`. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### rate `rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average per-second increase rate -over the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +over the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). If the lookbehind window is skipped in square brackets, then it is automatically calculated as `max(step, scrape_interval)`, where `step` is the query arg value -passed to [/api/v1/query_range](./keyConcepts.md#range-query) or [/api/v1/query](./keyConcepts.md#instant-query), -while `scrape_interval` is the interval between [raw samples](./keyConcepts.md#raw-samples) for the selected time series. +passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query), +while `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for the selected time series. This allows avoiding unexpected gaps on the graph when `step` is smaller than the `scrape_interval`. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -743,58 +743,58 @@ See also [irate](#irate) and [rollup_rate](#rollup_rate). #### rate_over_sum -`rate_over_sum(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second rate over the sum of [raw samples](./keyConcepts.md#raw-samples) +`rate_over_sum(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second rate over the sum of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. The calculations are performed individually per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. #### resets `resets(series_selector[d])` is a [rollup function](#rollup-functions), which returns the number -of [counter](./keyConcepts.md#counter) resets over the given lookbehind window `d` -per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +of [counter](https://docs.victoriametrics.com/keyconcepts/#counter) resets over the given lookbehind window `d` +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). This function is supported by PromQL. #### rollup -`rollup(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `min`, `max` and `avg` values for [raw samples](./keyConcepts.md#raw-samples) +`rollup(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `min`, `max` and `avg` values for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -These values are calculated individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +These values are calculated individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [rollup_rate](#rollup_rate). #### rollup_candlestick `rollup_candlestick(series_selector[d])` is a [rollup function](#rollup-functions), which calculates `open`, `high`, `low` and `close` values (aka OHLC) -over [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` and returns them in time series +over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns them in time series with `rollup="open"`, `rollup="high"`, `rollup="low"` and `rollup="close"` additional labels. The calculations are performed individually per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). This function is useful for financial applications. +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is useful for financial applications. Optional 2nd argument `"open"`, `"high"` or `"low"` or `"close"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### rollup_delta -`rollup_delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates differences between adjacent [raw samples](./keyConcepts.md#raw-samples) +`rollup_delta(series_selector[d])` is a [rollup function](#rollup-functions), which calculates differences between adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated differences and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -The calculations are performed individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). @@ -806,9 +806,9 @@ See also [rollup_increase](#rollup_increase). #### rollup_deriv `rollup_deriv(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second derivatives -for adjacent [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values +for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated per-second derivatives and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -The calculations are performed individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). @@ -819,27 +819,27 @@ See also [rollup](#rollup) and [rollup_rate](#rollup_rate). #### rollup_increase -`rollup_increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates increases for adjacent [raw samples](./keyConcepts.md#raw-samples) +`rollup_increase(series_selector[d])` is a [rollup function](#rollup-functions), which calculates increases for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated increases and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -The calculations are performed individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. See also [rollup_delta](#rollup_delta). -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). See also [rollup](#rollup) and [rollup_rate](#rollup_rate). #### rollup_rate `rollup_rate(series_selector[d])` is a [rollup function](#rollup-functions), which calculates per-second change rates -for adjacent [raw samples](./keyConcepts.md#raw-samples) +for adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated per-second change rates and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -The calculations are performed individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). See [this article](https://valyala.medium.com/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832) in order to understand better when to use `rollup_rate()`. @@ -849,16 +849,16 @@ See also [label_match](#label_match). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [counters](./keyConcepts.md#counter). +This function is usually applied to [counters](https://docs.victoriametrics.com/keyconcepts/#counter). See also [rollup](#rollup) and [rollup_increase](#rollup_increase). #### rollup_scrape_interval `rollup_scrape_interval(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the interval in seconds between -adjacent [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated interval +adjacent [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d` and returns `min`, `max` and `avg` values for the calculated interval and returns them in time series with `rollup="min"`, `rollup="max"` and `rollup="avg"` additional labels. -The calculations are performed individually per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +The calculations are performed individually per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Optional 2nd argument `"min"`, `"max"` or `"avg"` can be passed to keep only one calculation result and without adding a label. See also [label_match](#label_match). @@ -868,8 +868,8 @@ Metric names are stripped from the resulting rollups. Add [keep_metric_names](#k #### scrape_interval `scrape_interval(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the average interval in seconds -between [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -878,63 +878,63 @@ See also [rollup_scrape_interval](#rollup_scrape_interval). #### share_gt_over_time `share_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-functions), which returns share (in the range `[0...1]`) -of [raw samples](./keyConcepts.md#raw-samples) +of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which are bigger than `gt`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is useful for calculating SLI and SLO. Example: `share_gt_over_time(up[24h], 0)` - returns service availability for the last 24 hours. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [share_le_over_time](#share_le_over_time) and [count_gt_over_time](#count_gt_over_time). #### share_le_over_time `share_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-functions), which returns share (in the range `[0...1]`) -of [raw samples](./keyConcepts.md#raw-samples) +of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which are smaller or equal to `le`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). This function is useful for calculating SLI and SLO. Example: `share_le_over_time(memory_usage_bytes[24h], 100*1024*1024)` returns the share of time series values for the last 24 hours when memory usage was below or equal to 100MB. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [share_gt_over_time](#share_gt_over_time) and [count_le_over_time](#count_le_over_time). #### share_eq_over_time `share_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-functions), which returns share (in the range `[0...1]`) -of [raw samples](./keyConcepts.md#raw-samples) +of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`, which are equal to `eq`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [count_eq_over_time](#count_eq_over_time). #### stale_samples_over_time `stale_samples_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the number -of [staleness markers](./vmagent.md#prometheus-staleness-markers) on the given lookbehind window `d` -per each time series matching the given [series_selector](./keyConcepts.md#filtering). +of [staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) on the given lookbehind window `d` +per each time series matching the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. #### stddev_over_time -`stddev_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates standard deviation over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`stddev_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates standard deviation over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -942,12 +942,12 @@ See also [stdvar_over_time](#stdvar_over_time). #### stdvar_over_time -`stdvar_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates standard variance over [raw samples](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`stdvar_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates standard variance over [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. @@ -955,62 +955,62 @@ See also [stddev_over_time](#stddev_over_time). #### sum_eq_over_time -`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](./keyConcepts.md#raw-samples) -values equal to `eq` on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`sum_eq_over_time(series_selector[d], eq)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +values equal to `eq` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [sum_over_time](#sum_over_time) and [count_eq_over_time](#count_eq_over_time). #### sum_gt_over_time -`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](./keyConcepts.md#raw-samples) -values bigger than `gt` on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`sum_gt_over_time(series_selector[d], gt)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +values bigger than `gt` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [sum_over_time](#sum_over_time) and [count_gt_over_time](#count_gt_over_time). #### sum_le_over_time -`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](./keyConcepts.md#raw-samples) -values smaller or equal to `le` on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`sum_le_over_time(series_selector[d], le)` is a [rollup function](#rollup-function), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +values smaller or equal to `le` on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [sum_over_time](#sum_over_time) and [count_le_over_time](#count_le_over_time). #### sum_over_time -`sum_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](./keyConcepts.md#raw-samples) values -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`sum_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the sum of [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) values +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). This function is supported by PromQL. #### sum2_over_time -`sum2_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the sum of squares for [raw sample](./keyConcepts.md#raw-samples) -values on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +`sum2_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which calculates the sum of squares for [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +values on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). #### timestamp `timestamp(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision -for the last [raw sample](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +for the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -1021,8 +1021,8 @@ See also [time](#time) and [now](#now). #### timestamp_with_name `timestamp_with_name(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision -for the last [raw sample](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +for the last [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are preserved in the resulting rollups. @@ -1031,8 +1031,8 @@ See also [timestamp](#timestamp) and [keep_metric_names](#keep_metric_names) mod #### tfirst_over_time `tfirst_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision -for the first [raw sample](./keyConcepts.md#raw-samples) -on the given lookbehind window `d` per each time series returned from the given [series_selector](./keyConcepts.md#filtering). +for the first [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +on the given lookbehind window `d` per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -1041,7 +1041,7 @@ See also [first_over_time](#first_over_time). #### tlast_change_over_time `tlast_change_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision for the last change -per each time series returned from the given [series_selector](./keyConcepts.md#filtering) on the given lookbehind window `d`. +per each time series returned from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering) on the given lookbehind window `d`. Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -1056,9 +1056,9 @@ See also [tlast_change_over_time](#tlast_change_over_time). #### tmax_over_time `tmax_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision -for the [raw sample](./keyConcepts.md#raw-samples) +for the [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) with the maximum value on the given lookbehind window `d`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -1067,9 +1067,9 @@ See also [max_over_time](#max_over_time). #### tmin_over_time `tmin_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns the timestamp in seconds with millisecond precision -for the [raw sample](./keyConcepts.md#raw-samples) +for the [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) with the minimum value on the given lookbehind window `d`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. @@ -1078,12 +1078,12 @@ See also [min_over_time](#min_over_time). #### zscore_over_time `zscore_over_time(series_selector[d])` is a [rollup function](#rollup-functions), which returns [z-score](https://en.wikipedia.org/wiki/Standard_score) -for [raw samples](./keyConcepts.md#raw-samples) on the given lookbehind window `d`. It is calculated independently per each time series returned -from the given [series_selector](./keyConcepts.md#filtering). +for [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) on the given lookbehind window `d`. It is calculated independently per each time series returned +from the given [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering). Metric names are stripped from the resulting rollups. Add [keep_metric_names](#keep_metric_names) modifier in order to keep metric names. -This function is usually applied to [gauges](./keyConcepts.md#gauge). +This function is usually applied to [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge). See also [zscore](#zscore), [range_trim_zscore](#range_trim_zscore) and [outlier_iqr_over_time](#outlier_iqr_over_time). @@ -1096,7 +1096,7 @@ returned from the rollup `delta(temperature[24h])`. Additional details: -* If transform function is applied directly to a [series selector](./keyConcepts.md#filtering), +* If transform function is applied directly to a [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering), then the [default_rollup()](#default_rollup) function is automatically applied before calculating the transformations. For example, `abs(temperature)` is implicitly transformed to `abs(default_rollup(temperature))`. * All the transform functions accept optional `keep_metric_names` modifier. If it is set, @@ -1332,7 +1332,7 @@ by replacing all the values bigger or equal to 30 with 40. #### end `end()` is a [transform function](#transform-functions), which returns the unix timestamp in seconds for the last point. -It is known as `end` query arg passed to [/api/v1/query_range](./keyConcepts.md#range-query). +It is known as `end` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See also [start](#start), [time](#time) and [now](#now). @@ -1755,14 +1755,14 @@ This function is supported by PromQL. `start()` is a [transform function](#transform-functions), which returns unix timestamp in seconds for the first point. -It is known as `start` query arg passed to [/api/v1/query_range](./keyConcepts.md#range-query). +It is known as `start` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See also [end](#end), [time](#time) and [now](#now). #### step `step()` is a [transform function](#transform-functions), which returns the step in seconds (aka interval) between the returned points. -It is known as `step` query arg passed to [/api/v1/query_range](./keyConcepts.md#range-query). +It is known as `step` query arg passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See also [start](#start) and [end](#end). @@ -1819,7 +1819,7 @@ This function is supported by PromQL. Additional details: -* If label manipulation function is applied directly to a [series_selector](./keyConcepts.md#filtering), +* If label manipulation function is applied directly to a [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering), then the [default_rollup()](#default_rollup) function is automatically applied before performing the label transformation. For example, `alias(temperature, "foo")` is implicitly transformed to `alias(default_rollup(temperature), "foo")`. @@ -1996,7 +1996,7 @@ Additional details: and calculate the [count](#count) aggregate function independently per each group, while `count(up) without (instance)` would group [rollup results](#rollup-functions) by all the labels except `instance` before calculating [count](#count) aggregate function independently per each group. Multiple labels can be put in `by` and `without` modifiers. -* If the aggregate function is applied directly to a [series_selector](./keyConcepts.md#filtering), +* If the aggregate function is applied directly to a [series_selector](https://docs.victoriametrics.com/keyconcepts/#filtering), then the [default_rollup()](#default_rollup) function is automatically applied before calculating the aggregate. For example, `count(up)` is implicitly transformed to `count(default_rollup(up))`. * Aggregate functions accept arbitrary number of args. For example, `avg(q1, q2, q3)` would return the average values for every point @@ -2206,7 +2206,7 @@ See also [quantile](#quantile). `share(q) by (group_labels)` is [aggregate function](#aggregate-functions), which returns shares in the range `[0..1]` for every non-negative points returned by `q` per each timestamp, so the sum of shares per each `group_labels` equals 1. -This function is useful for normalizing [histogram bucket](./keyConcepts.md#histogram) shares +This function is useful for normalizing [histogram bucket](https://docs.victoriametrics.com/keyconcepts/#histogram) shares into `[0..1]` range: ```metricsql @@ -2310,12 +2310,12 @@ See also [zscore_over_time](#zscore_over_time), [range_trim_zscore](#range_trim_ ## Subqueries MetricsQL supports and extends PromQL subqueries. See [this article](https://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3) for details. -Any [rollup function](#rollup-functions) for something other than [series selector](./keyConcepts.md#filtering) form a subquery. +Any [rollup function](#rollup-functions) for something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) form a subquery. Nested rollup functions can be implicit thanks to the [implicit query conversions](#implicit-query-conversions). For example, `delta(sum(m))` is implicitly converted to `delta(sum(default_rollup(m))[1i:1i])`, so it becomes a subquery, since it contains [default_rollup](#default_rollup) nested into [delta](#delta). This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags -starting from [`v1.101.0` release](./CHANGELOG.md). +starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/). VictoriaMetrics performs subqueries in the following way: @@ -2323,19 +2323,19 @@ VictoriaMetrics performs subqueries in the following way: For example, for expression `max_over_time(rate(http_requests_total[5m])[1h:30s])` the inner function `rate(http_requests_total[5m])` is calculated with `step=30s`. The resulting data points are aligned by the `step`. * It calculates the outer rollup function over the results of the inner rollup function using the `step` value - passed by Grafana to [/api/v1/query_range](./keyConcepts.md#range-query). + passed by Grafana to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). ## Implicit query conversions VictoriaMetrics performs the following implicit conversions for incoming queries before starting the calculations: * If lookbehind window in square brackets is missing inside [rollup function](#rollup-functions), then it is automatically set to the following value: - - To `step` value passed to [/api/v1/query_range](./keyConcepts.md#range-query) or [/api/v1/query](./keyConcepts.md#instant-query) + - To `step` value passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) or [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) for all the [rollup functions](#rollup-functions) except of [default_rollup](#default_rollup) and [rate](#rate). This value is known as `$__interval` in Grafana or `1i` in MetricsQL. For example, `avg_over_time(temperature)` is automatically transformed to `avg_over_time(temperature[1i])`. - - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](./keyConcepts.md#raw-samples) + - To the `max(step, scrape_interval)`, where `scrape_interval` is the interval between [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) for [default_rollup](#default_rollup) and [rate](#rate) functions. This allows avoiding unexpected gaps on the graph when `step` is smaller than `scrape_interval`. -* All the [series selectors](./keyConcepts.md#filtering), +* All the [series selectors](https://docs.victoriametrics.com/keyconcepts/#filtering), which aren't wrapped into [rollup functions](#rollup-functions), are automatically wrapped into [default_rollup](#default_rollup) function. Examples: * `foo` is transformed to `default_rollup(foo)` @@ -2346,8 +2346,8 @@ VictoriaMetrics performs the following implicit conversions for incoming queries it is [transform function](#transform-functions) * If `step` in square brackets is missing inside [subquery](#subqueries), then `1i` step is automatically added there. For example, `avg_over_time(rate(http_requests_total[5m])[1h])` is automatically converted to `avg_over_time(rate(http_requests_total[5m])[1h:1i])`. -* If something other than [series selector](./keyConcepts.md#filtering) +* If something other than [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering) is passed to [rollup function](#rollup-functions), then a [subquery](#subqueries) with `1i` lookbehind window and `1i` step is automatically formed. For example, `rate(sum(up))` is automatically converted to `rate((sum(default_rollup(up)))[1i:1i])`. This behavior can be disabled or logged via `-search.disableImplicitConversion` and `-search.logImplicitConversion` command-line flags - starting from [`v1.101.0` release](./CHANGELOG.md). + starting from [`v1.101.0` release](https://docs.victoriametrics.com/changelog/). diff --git a/docs/Quick-Start.md b/docs/Quick-Start.md index 404be75c7..f5bbc3072 100644 --- a/docs/Quick-Start.md +++ b/docs/Quick-Start.md @@ -12,10 +12,10 @@ aliases: ## How to install VictoriaMetrics is distributed in two forms: -* [Single-server-VictoriaMetrics](./Single-Server-VictoriaMetrics.md) - all-in-one +* [Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) - all-in-one binary, which is very easy to use and maintain. Single-server-VictoriaMetrics perfectly scales vertically and easily handles millions of metrics/s; -* [VictoriaMetrics Cluster](./Cluster-VictoriaMetrics.md) - set of components +* [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) - set of components for building horizontally scalable clusters. Single-server-VictoriaMetrics VictoriaMetrics is available as: @@ -26,16 +26,16 @@ Single-server-VictoriaMetrics VictoriaMetrics is available as: * [Binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) * [Ansible Roles](https://github.com/VictoriaMetrics/ansible-playbooks) * [Source code](https://github.com/VictoriaMetrics/VictoriaMetrics). - See [How to build from sources](./Single-Server-VictoriaMetrics.md#how-to-build-from-sources) + See [How to build from sources](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-build-from-sources) * [VictoriaMetrics on Linode](https://www.linode.com/marketplace/apps/victoriametrics/victoriametrics/) * [VictoriaMetrics on DigitalOcean](https://marketplace.digitalocean.com/apps/victoriametrics-single) Just download VictoriaMetrics and follow -[these instructions](./Single-Server-VictoriaMetrics.md#how-to-start-victoriametrics). -Then read [Prometheus setup](./Single-Server-VictoriaMetrics.md#prometheus-setup) -and [Grafana setup](./Single-Server-VictoriaMetrics.md#grafana-setup) docs. +[these instructions](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-start-victoriametrics). +Then read [Prometheus setup](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-setup) +and [Grafana setup](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup) docs. -VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](./CHANGELOG.md) and performing [regular upgrades](how-to-upgrade-victoriametrics). +VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the [CHANGELOG](https://docs.victoriametrics.com/changelog/) and performing [regular upgrades](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics). ### Starting VictoriaMetrics Single Node via Docker {anchor="starting-vm-single-via-docker"} @@ -53,9 +53,9 @@ docker run -it --rm -v `pwd`/victoria-metrics-data:/victoria-metrics-data -p 842 Open http://localhost:8428 in web browser -and read [these docs](operation). +and read [these docs](https://docs.victoriametrics.com/#operation). -There is also [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md) +There is also [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) - horizontally scalable installation, which scales to multiple nodes. ### Starting VictoriaMetrics Cluster via Docker {anchor="starting-vm-cluster-via-docker"} @@ -75,13 +75,13 @@ make docker-cluster-up See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#readme). -* [Cluster setup](./Cluster-VictoriaMetrics.md#cluster-setup) +* [Cluster setup](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-setup) ### Starting VictoriaMetrics Single Node from a Binary {anchor="starting-vm-single-from-a-binary"} 1. Download the correct binary for your OS and architecture from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). -For Enterprise binaries see [this link](./enterprise.md#binary-releases). +For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases). 2. Extract the archive to /usr/local/bin by running: ```sh @@ -128,9 +128,9 @@ WantedBy=multi-user.target END ``` -Extra [command-line flags](list-of-command-line-flags) can be added to `ExecStart` line. +Extra [command-line flags](https://docs.victoriametrics.com/#list-of-command-line-flags) can be added to `ExecStart` line. -If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](./Single-Server-VictoriaMetrics.md#running-as-windows-service). +If you want to deploy VictoriaMetrics Single Node as a Windows Service review the [running as a Windows service docs](https://docs.victoriametrics.com/single-server-victoriametrics/#running-as-windows-service). > Please note, `victoriametrics` service is listening on `:8428` for HTTP connections (see `-httpListenAddr` flag). @@ -146,20 +146,20 @@ sudo systemctl daemon-reload && sudo systemctl enable --now victoriametrics.serv sudo systemctl status victoriametrics.service ``` -8. After VictoriaMetrics is in `Running` state, verify [vmui](vmui) is working +8. After VictoriaMetrics is in `Running` state, verify [vmui](https://docs.victoriametrics.com/#vmui) is working by going to `http://:8428/vmui`. ### Starting VictoriaMetrics Cluster from Binaries {anchor="starting-vm-cluster-from-binaries"} -VictoriaMetrics cluster consists of [3 components](./Cluster-VictoriaMetrics.md#architecture-overview). -It is recommended to run these components in the same private network (for [security reasons](security)), +VictoriaMetrics cluster consists of [3 components](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview). +It is recommended to run these components in the same private network (for [security reasons](https://docs.victoriametrics.com/#security)), but on the separate physical nodes for the best performance. On all nodes you will need to do the following: 1. Download the correct binary for your OS and architecture with `-cluster` suffix from [GitHub](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). -For Enterprise binaries see [this link](./enterprise.md#binary-releases). +For Enterprise binaries see [this link](https://docs.victoriametrics.com/enterprise/#binary-releases). 2. Extract the archive to /usr/local/bin by running: @@ -175,7 +175,7 @@ Replace `` with the path to the archive you downloaded sudo useradd -s /usr/sbin/nologin victoriametrics ``` -See recommendations for installing each type of [cluster component](./Cluster-VictoriaMetrics.md#architecture-overview) below. +See recommendations for installing each type of [cluster component](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) below. ##### Installing vmstorage @@ -207,7 +207,7 @@ WantedBy=multi-user.target END ``` -Extra [command-line flags](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vmstorage) +Extra [command-line flags](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vmstorage) for vmstorage can be added to `ExecStart` line. > Please note, `vmstorage` service is listening on `:8400` for vminsert connections (see `-vminsertAddr` flag), @@ -256,7 +256,7 @@ END Replace `` with addresses of previously configured `vmstorage` services. To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas -in one flag. See more details in `-storageNode` flag description [here](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vminsert). +in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert). > Please note, `vminsert` service is listening on `:8480` for HTTP connections (see `-httpListenAddr` flag). @@ -310,7 +310,7 @@ END Replace `` with addresses of previously configured `vmstorage` services. To specify multiple addresses you can repeat the flag multiple times, or separate addresses with commas -in one flag. See more details in `-storageNode` flag description [here](./Cluster-VictoriaMetrics.md#list-of-command-line-flags-for-vminsert). +in one flag. See more details in `-storageNode` flag description [here](https://docs.victoriametrics.com/cluster-victoriametrics/#list-of-command-line-flags-for-vminsert). > Please note, `vmselect` service is listening on `:8481` for HTTP connections (see `-httpListenAddr` flag). @@ -327,53 +327,53 @@ sudo systemctl status vmselect.service ``` 5. After `vmselect` is in `Running` state, confirm the service is healthy by visiting `http://:8481/select/0/vmui` link. -It should open [vmui](vmui) page. +It should open [vmui](https://docs.victoriametrics.com/#vmui) page. ## Write data There are two main models in monitoring for data collection: -[push](./keyConcepts.md#push-model) -and [pull](./keyConcepts.md#pull-model). +[push](https://docs.victoriametrics.com/keyconcepts/#push-model) +and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model). Both are used in modern monitoring and both are supported by VictoriaMetrics. -See more details on [writing data here](./keyConcepts.md#write-data). +See more details on [writing data here](https://docs.victoriametrics.com/keyconcepts/#write-data). ## Query data VictoriaMetrics provides an -[HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage) +[HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage) for serving read queries. The API is used in various integrations such as -[Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup). +[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup). The same API is also used by -[VMUI](./Single-Server-VictoriaMetrics.md#vmui) - graphical User Interface +[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - graphical User Interface for querying and visualizing metrics. -[MetricsQL](./MetricsQL.md) - is the query language for executing read queries +[MetricsQL](https://docs.victoriametrics.com/metricsql/) - is the query language for executing read queries in VictoriaMetrics. MetricsQL is a [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics) -like query language with a powerful set of functions and features for working specifically with time series data. -See more details on [querying data here](./keyConcepts.md#query-data) +See more details on [querying data here](https://docs.victoriametrics.com/keyconcepts/#query-data) ## Alerting It is not possible to physically trace all changes on graphs all the time, that is why alerting exists. -In [vmalert](./vmalert.md) it is possible to create a set of conditions +In [vmalert](https://docs.victoriametrics.com/vmalert/) it is possible to create a set of conditions based on PromQL and MetricsQL queries that will send a notification when such conditions are met. ## Data migration Migrating data from other TSDBs to VictoriaMetrics is as simple as importing data via any of -[supported formats](./keyConcepts.md#push-model). +[supported formats](https://docs.victoriametrics.com/keyconcepts/#push-model). -The migration might get easier when using [vmctl](./vmctl.md) - VictoriaMetrics +The migration might get easier when using [vmctl](https://docs.victoriametrics.com/vmctl/) - VictoriaMetrics command line tool. It supports the following databases for migration to VictoriaMetrics: -* [Prometheus using snapshot API](./vmctl.md#migrating-data-from-prometheus); -* [Thanos](./vmctl.md#migrating-data-from-thanos); -* [InfluxDB](./vmctl.md#migrating-data-from-influxdb-1x); -* [OpenTSDB](./vmctl.md#migrating-data-from-opentsdb); -* [Migrate data between VictoriaMetrics single and cluster versions](./vmctl.md#migrating-data-from-victoriametrics). +* [Prometheus using snapshot API](https://docs.victoriametrics.com/vmctl/#migrating-data-from-prometheus); +* [Thanos](https://docs.victoriametrics.com/vmctl/#migrating-data-from-thanos); +* [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x); +* [OpenTSDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-opentsdb); +* [Migrate data between VictoriaMetrics single and cluster versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). ## Productionization @@ -384,7 +384,7 @@ When going to production with VictoriaMetrics we recommend following the recomme Each VictoriaMetrics component emits its own metrics with various details regarding performance and health state. Docs for the components also contain a `Monitoring` section with an explanation of what and how should be monitored. For example, -[Single-server-VictoriaMetrics Monitoring](./Cluster-VictoriaMetrics.md#monitoring). +[Single-server-VictoriaMetrics Monitoring](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring). VictoriaMetric team prepared a list of [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards) for the main components. Each dashboard contains a lot of useful information and tips. It is recommended @@ -401,33 +401,33 @@ See more details in the article [VictoriaMetrics Monitoring](https://victoriamet ### Capacity planning -See capacity planning sections in [docs](./README.md) for -[Single-server-VictoriaMetrics](./Single-Server-VictoriaMetrics.md#capacity-planning) -and [VictoriaMetrics Cluster](./Cluster-VictoriaMetrics.md#capacity-planning). +See capacity planning sections in [docs](https://docs.victoriametrics.com) for +[Single-server-VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning) +and [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning). Capacity planning isn't possible without [monitoring](#monitoring), so consider configuring it first. Understanding resource usage and performance of VictoriaMetrics also requires knowing the tech terms -[active series](./FAQ.md#what-is-an-active-time-series), -[churn rate](./FAQ.md#what-is-high-churn-rate), -[cardinality](./FAQ.md#what-is-high-cardinality), -[slow inserts](./FAQ.md#what-is-a-slow-insert). +[active series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series), +[churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), +[cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality), +[slow inserts](https://docs.victoriametrics.com/faq/#what-is-a-slow-insert). All of them are present in [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards). ### Data safety -It is recommended to read [Replication and data safety](./Cluster-VictoriaMetrics.md#replication-and-data-safety), +It is recommended to read [Replication and data safety](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety), [Why replication doesn’t save from disaster?](https://valyala.medium.com/speeding-up-backups-for-big-time-series-databases-533c1a927883) -and [backups](./Single-Server-VictoriaMetrics.md#backups). +and [backups](https://docs.victoriametrics.com/single-server-victoriametrics/#backups). ### Configuring limits To avoid excessive resource usage or performance degradation limits must be in place: -* [Resource usage limits](./FAQ.md#how-to-set-a-memory-limit-for-victoriametrics-components); -* [Cardinality limiter](./Single-Server-VictoriaMetrics.md#cardinality-limiter). +* [Resource usage limits](https://docs.victoriametrics.com/faq/#how-to-set-a-memory-limit-for-victoriametrics-components); +* [Cardinality limiter](https://docs.victoriametrics.com/single-server-victoriametrics/#cardinality-limiter). ### Security recommendations -* [Security recommendations for single-node VictoriaMetrics](security) -* [Security recommendations for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#security) +* [Security recommendations for single-node VictoriaMetrics](https://docs.victoriametrics.com/#security) +* [Security recommendations for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#security) diff --git a/docs/README.md b/docs/README.md index ea24b4b0d..65212f14f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,30 +13,30 @@ VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. -See [case studies for VictoriaMetrics](./CaseStudies.md). +See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/). VictoriaMetrics is available in [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), [Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). -Documentation for the cluster version of VictoriaMetrics is available [here](./Cluster-VictoriaMetrics.md). +Documentation for the cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/cluster-victoriametrics/). -Learn more about [key concepts](./keyConcepts.md) of VictoriaMetrics and follow the -[quick start guide](./Quick-Start.md) for a better experience. +Learn more about [key concepts](https://docs.victoriametrics.com/keyconcepts/) of VictoriaMetrics and follow the +[quick start guide](https://docs.victoriametrics.com/quick-start/) for a better experience. If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://victoriametrics.slack.com/), you can join it via [Slack Inviter](https://slack.victoriametrics.com/). [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. -See [features available in enterprise package](./enterprise.md). +See [features available in enterprise package](https://docs.victoriametrics.com/enterprise/). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/). -VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](./CHANGELOG.md) periodically, +VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/changelog/) periodically, and to perform [regular upgrades](#how-to-upgrade-victoriametrics). -[VictoriaMetrics enterprise](./enterprise.md) provides long-term support lines of releases (LTS releases) - -see [these docs](./LTS-releases.md). +[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) - +see [these docs](https://docs.victoriametrics.com/lts-releases/). VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details. @@ -48,16 +48,16 @@ VictoriaMetrics has the following prominent features: * It can be used as long-term storage for Prometheus. See [these docs](#prometheus-setup) for details. * It can be used as a drop-in replacement for Prometheus in Grafana, because it supports the [Prometheus querying API](#prometheus-querying-api-usage). * It can be used as a drop-in replacement for Graphite in Grafana, because it supports the [Graphite API](#graphite-api-usage). - VictoriaMetrics allows reducing infrastructure costs by more than 10x comparing to Graphite - see [this case study](./CaseStudies.md#grammarly). + VictoriaMetrics allows reducing infrastructure costs by more than 10x comparing to Graphite - see [this case study](https://docs.victoriametrics.com/casestudies/#grammarly). * It is easy to setup and operate: * VictoriaMetrics consists of a single [small executable](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d) without external dependencies. * All the configuration is done via explicit command-line flags with reasonable defaults. * All the data is stored in a single directory specified by the `-storageDataPath` command-line flag. * Easy and fast backups from [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) - can be done with [vmbackup](./vmbackup.md) / [vmrestore](./vmrestore.md) tools. + can be done with [vmbackup](https://docs.victoriametrics.com/vmbackup/) / [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883) for more details. -* It implements a PromQL-like query language - [MetricsQL](./MetricsQL.md), which provides improved functionality on top of PromQL. +* It implements a PromQL-like query language - [MetricsQL](https://docs.victoriametrics.com/metricsql/), which provides improved functionality on top of PromQL. * It provides a global query view. Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics. Later this data may be queried via a single query. * It provides high performance and good vertical and horizontal scalability for both [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) @@ -65,8 +65,8 @@ VictoriaMetrics has the following prominent features: It [outperforms InfluxDB and TimescaleDB by up to 20x](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae). * It [uses 10x less RAM than InfluxDB](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and [up to 7x less RAM than Prometheus, Thanos or Cortex](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) - when dealing with millions of unique time series (aka [high cardinality](./FAQ.md#what-is-high-cardinality)). -* It is optimized for time series with [high churn rate](./FAQ.md#what-is-high-churn-rate). + when dealing with millions of unique time series (aka [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)). +* It is optimized for time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). * It provides high data compression: up to 70x more data points may be stored into limited storage compared with TimescaleDB according to [these benchmarks](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), and up to 7x less storage space is required compared to Prometheus, Thanos or Cortex. @@ -94,34 +94,34 @@ VictoriaMetrics has the following prominent features: * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent). * [NewRelic infrastructure agent](#how-to-send-data-from-newrelic-agent). * [OpenTelemetry metrics format](#sending-data-via-opentelemetry). -* It supports powerful [stream aggregation](./stream-aggregation.md), which can be used as a [statsd](https://github.com/statsd/statsd) alternative. +* It supports powerful [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/), which can be used as a [statsd](https://github.com/statsd/statsd) alternative. * It supports metrics [relabeling](#relabeling). -* It can deal with [high cardinality issues](./FAQ.md#what-is-high-cardinality) and - [high churn rate](./FAQ.md#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter). +* It can deal with [high cardinality issues](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) and + [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter). * It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data - and various [Enterprise workloads](./enterprise.md). + and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise/). * It has an open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster). * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore). -See [case studies for VictoriaMetrics](./CaseStudies.md) and [various Articles about VictoriaMetrics](./Articles.md). +See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/) and [various Articles about VictoriaMetrics](https://docs.victoriametrics.com/articles/). ## Components -VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](./README.md): +VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](https://docs.victoriametrics.com/): -- [vmagent](./vmagent.md) - lightweight agent for receiving metrics via [pull-based](./vmagent.md#how-to-collect-metrics-in-prometheus-format) - and [push-based](./vmagent.md#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible +- [vmagent](https://docs.victoriametrics.com/vmagent/) - lightweight agent for receiving metrics via [pull-based](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format) + and [push-based](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible remote storage systems such as VictoriaMetrics. -- [vmalert](./vmalert.md) - a service for processing Prometheus-compatible alerting and recording rules. -- [vmalert-tool](./vmalert-tool.md) - a tool for validating alerting and recording rules. -- [vmauth](./vmauth.md) - authorization proxy and load balancer optimized for VictoriaMetrics products. -- [vmgateway](./vmgateway.md) - authorization proxy with per-[tenant](./Cluster-VictoriaMetrics.md#multitenancy) rate limiting capabilities. -- [vmctl](./vmctl.md) - a tool for migrating and copying data between different storage systems for metrics. -- [vmbackup](./vmbackup.md), [vmrestore](./vmrestore.md) and [vmbackupmanager](./vmbackupmanager.md) - +- [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules. +- [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) - a tool for validating alerting and recording rules. +- [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products. +- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - authorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting capabilities. +- [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics. +- [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) - tools for creating backups and restoring from backups for VictoriaMetrics data. -- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md). -- [VictoriaLogs](./VictoriaLogs/README.md) - user-friendly cost-efficient database for logs. +- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). +- [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) - user-friendly cost-efficient database for logs. ## Operation @@ -129,7 +129,7 @@ VictoriaMetrics ecosystem contains the following components additionally to [sin To quickly try VictoriaMetrics, just download the [VictoriaMetrics executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) or [Docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and start it with the desired command-line flags. -See also [QuickStart guide](./Quick-Start.md) for additional information. +See also [QuickStart guide](https://docs.victoriametrics.com/quick-start/) for additional information. VictoriaMetrics can also be installed via these installation methods: @@ -149,9 +149,9 @@ The following command-line flags are used the most: Other flags have good enough default values, so set them only if you really need to. Pass `-help` to see [all the available flags with description and default values](#list-of-command-line-flags). The following docs may be useful during initial VictoriaMetrics setup: -* [How to set up scraping of Prometheus-compatible targets](./README.md#how-to-scrape-prometheus-exporters-such-as-node-exporter) +* [How to set up scraping of Prometheus-compatible targets](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) * [How to ingest data to VictoriaMetrics](#how-to-import-time-series-data) -* [How to set up Prometheus to write data to VictoriaMetrics](./README.md#prometheus-setup) +* [How to set up Prometheus to write data to VictoriaMetrics](https://docs.victoriametrics.com/#prometheus-setup) * [How to query VictoriaMetrics via Grafana](#grafana-setup) * [How to query VictoriaMetrics via Graphite API](#graphite-api-usage) * [How to handle alerts](#alerting) @@ -291,8 +291,8 @@ Read more about tuning remote write for Prometheus [here](https://prometheus.io/ It is recommended upgrading Prometheus to [v2.12.0](https://github.com/prometheus/prometheus/releases/latest) or newer, since previous versions may have issues with `remote_write`. -Take a look also at [vmagent](./vmagent.md) -and [vmalert](./vmalert.md), +Take a look also at [vmagent](https://docs.victoriametrics.com/vmagent/) +and [vmalert](https://docs.victoriametrics.com/vmalert/), which can be used as faster and less resource-hungry alternative to Prometheus. ## Grafana setup @@ -313,10 +313,10 @@ In the "Type and version" section it is recommended to set the type to "Promethe This allows Grafana to use a more efficient API to get label values. Then build graphs and dashboards for the created datasource using [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) -or [MetricsQL](./MetricsQL.md). +or [MetricsQL](https://docs.victoriametrics.com/metricsql/). Alternatively, use VictoriaMetrics [datasource plugin](https://github.com/VictoriaMetrics/victoriametrics-datasource) with support of extra features. -See more in [description](./victoriametrics-datasource.md). +See more in [description](https://docs.victoriametrics.com/victoriametrics-datasource/). Creating a datasource may require [specific permissions](https://grafana.com/docs/grafana/latest/administration/data-source-management/). If you don't see an option to create a data source - try contacting system administrator. @@ -325,7 +325,7 @@ Grafana playground is available for viewing at our [sandbox](https://play-grafan ## How to upgrade VictoriaMetrics -VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking [the CHANGELOG page](./CHANGELOG.md) and performing regular upgrades. +VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking [the CHANGELOG page](https://docs.victoriametrics.com/changelog/) and performing regular upgrades. It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) say otherwise. @@ -339,12 +339,12 @@ The following steps must be performed during the upgrade / downgrade procedure: * Wait until the process stops. This can take a few seconds. * Start the upgraded VictoriaMetrics. -Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](./vmagent.md). +Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent/). ## vmui VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui` -(or at `http://:8481/select//vmui/` in [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md)). +(or at `http://:8481/select//vmui/` in [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)). The UI allows exploring query results via graphs and tables. It also provides the following features: - Explore: @@ -358,10 +358,10 @@ The UI allows exploring query results via graphs and tables. It also provides th - [WITH expressions playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/expand-with-exprs) - test how WITH expressions work; - [Metric relabel debugger](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/relabeling) - playground for [relabeling](#relabeling) configs. -VMUI provides auto-completion for [MetricsQL](./MetricsQL.md) functions, metric names, label names and label values. The auto-completion can be enabled +VMUI provides auto-completion for [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions, metric names, label names and label values. The auto-completion can be enabled by checking the `Autocomplete` toggle. When the auto-completion is disabled, it can still be triggered for the current cursor position by pressing `ctrl+space`. -VMUI automatically switches from graph view to heatmap view when the query returns [histogram](./keyConcepts.md#histogram) buckets +VMUI automatically switches from graph view to heatmap view when the query returns [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) buckets (both [Prometheus histograms](https://prometheus.io/docs/concepts/metric_types/#histogram) and [VictoriaMetrics histograms](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) are supported). Try, for example, [this query](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.expr=sum%28rate%28vm_promscrape_scrape_duration_seconds_bucket%29%29+by+%28vmrange%29&g0.range_input=24h&g0.end_input=2023-04-10T17%3A46%3A12&g0.relative_time=last_24_hours&g0.step_input=31m). @@ -376,8 +376,8 @@ Query history can be navigated by holding `Ctrl` (or `Cmd` on MacOS) and pressin Multi-line queries can be entered by pressing `Shift-Enter` in query input field. -When querying the [backfilled data](./README.md#backfilling) -or during [query troubleshooting](./Troubleshooting.md#unexpected-query-results), +When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) +or during [query troubleshooting](https://docs.victoriametrics.com/troubleshooting/#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox. VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. @@ -390,7 +390,7 @@ Graphs for a particular query can be temporarily hidden by clicking the `eye` ic When the `eye` icon is clicked while holding the `ctrl` key, then query results for the rest of queries become hidden except of the current query results. -VMUI allows sharing query and [trace](./README.md#query-tracing) results by clicking on +VMUI allows sharing query and [trace](https://docs.victoriametrics.com/#query-tracing) results by clicking on `Export query` button in top right corner of the graph area. The query and trace will be exported as a file that later can be loaded in VMUI via `Query Analyzer` tool. @@ -411,7 +411,7 @@ This information is obtained from the `/api/v1/status/top_queries` HTTP endpoint [VMUI](#vmui) provides `active queries` tab, which shows currently execute queries. It provides the following information per each query: -- The query itself, together with the time range and step args passed to [/api/v1/query_range](./keyConcepts.md#range-query). +- The query itself, together with the time range and step args passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). - The duration of the query execution. - The client address, who initiated the query execution. @@ -438,7 +438,7 @@ VictoriaMetrics provides an ability to explore time series cardinality at `Explo - To identify values with the highest number of series for the selected label (aka `focusLabel`). - To identify label=name pairs with the highest number of series. - To identify labels with the highest number of unique values. - Note that [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) + Note that [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) may show lower than expected number of unique label values for labels with small number of unique values. This is because of [implementation limits](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5a6e617b5e41c9170e7c562aecd15ee0c901d489/app/vmselect/netstorage/netstorage.go#L1039-L1045). @@ -453,10 +453,10 @@ See the example of using the cardinality explorer [here](https://victoriametrics ## Cardinality explorer statistic inaccuracy -In [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) each vmstorage tracks the stored time series individually. +In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) each vmstorage tracks the stored time series individually. vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats. This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes -due to [replication](#replication) or [rerouting](./Cluster-VictoriaMetrics.md?highlight=re-routes#cluster-availability). +due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=re-routes#cluster-availability). ## How to apply new config to VictoriaMetrics @@ -466,26 +466,26 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh * Wait until the process stops. This can take a few seconds. * Start VictoriaMetrics with the new command-line flags. -Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](./vmagent.md). +Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent/). ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. -If the provided configuration file contains [unsupported options](./vmagent.md#unsupported-prometheus-config-sections), +If the provided configuration file contains [unsupported options](https://docs.victoriametrics.com/vmagent/#unsupported-prometheus-config-sections), then either delete them from the file or just pass `-promscrape.config.strictParse=false` command-line flag to VictoriaMetrics, so it will ignore unsupported options. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. See also: -- [scrape config examples](./scrape_config_examples.md) -- [the list of supported service discovery types for Prometheus scrape targets](./sd_configs.md) +- [scrape config examples](https://docs.victoriametrics.com/scrape_config_examples/) +- [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs/) VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format). -See also [vmagent](./vmagent.md), which can be used as drop-in replacement for Prometheus. +See also [vmagent](https://docs.victoriametrics.com/vmagent/), which can be used as drop-in replacement for Prometheus. ## How to send data from DataDog agent @@ -508,7 +508,7 @@ DD_DD_URL=http://victoriametrics:8428/datadog ``` -_Choose correct URL for VictoriaMetrics [here](./url-examples.md#datadog)._ +_Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples/#datadog)._ To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33) add the following line: @@ -517,8 +517,8 @@ add the following line: dd_url: http://victoriametrics:8428/datadog ``` -[vmagent](./vmagent.md) also can accept DataDog metrics format. Depending on where vmagent will forward data, -pick [single-node or cluster URL](./url-examples.md#datadog) formats. +[vmagent](https://docs.victoriametrics.com/vmagent/) also can accept DataDog metrics format. Depending on where vmagent will forward data, +pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples/#datadog) formats. ### Sending metrics to DataDog and VictoriaMetrics @@ -533,7 +533,7 @@ Run DataDog using the following ENV variable with VictoriaMetrics as additional DD_ADDITIONAL_ENDPOINTS='{\"http://victoriametrics:8428/datadog\": [\"apikey\"]}' ``` -_Choose correct URL for VictoriaMetrics [here](./url-examples.md#datadog)._ +_Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples/#datadog)._ To configure DataDog Dual Shipping via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files) @@ -561,9 +561,9 @@ provider: ### Send via cURL -See how to send data to VictoriaMetrics via DataDog "submit metrics" API [here](./url-examples.md#datadogapiv2series). +See how to send data to VictoriaMetrics via DataDog "submit metrics" API [here](https://docs.victoriametrics.com/url-examples/#datadogapiv2series). -The imported data can be read via [export API](./url-examples.md#apiv1export). +The imported data can be read via [export API](https://docs.victoriametrics.com/url-examples/#apiv1export). ### Additional details @@ -577,12 +577,12 @@ For example, `/datadog/api/v2/series?extra_label=foo=bar` would add `{foo="bar"} DataDog agent sends the [configured tags](https://docs.datadoghq.com/getting_started/tagging/) to undocumented endpoint - `/datadog/intake`. This endpoint isn't supported by VictoriaMetrics yet. This prevents from adding the configured tags to DataDog agent data sent into VictoriaMetrics. -The workaround is to run a sidecar [vmagent](./vmagent.md) alongside every DataDog agent, +The workaround is to run a sidecar [vmagent](https://docs.victoriametrics.com/vmagent/) alongside every DataDog agent, which must run with `DD_DD_URL=http://localhost:8429/datadog` environment variable. The sidecar `vmagent` must be configured with the needed tags via `-remoteWrite.label` command-line flag and must forward incoming data with the added tags to a centralized VictoriaMetrics specified via `-remoteWrite.url` command-line flag. -See [these docs](./vmagent.md#adding-labels-to-metrics) for details on how to add labels to metrics at `vmagent`. +See [these docs](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics) for details on how to add labels to metrics at `vmagent`. ## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) @@ -600,9 +600,9 @@ and stream plain InfluxDB line protocol data to the configured TCP and/or UDP ad VictoriaMetrics performs the following transformations to the ingested InfluxDB data: * [db query arg](https://docs.influxdata.com/influxdb/v1.7/tools/api/#write-http-endpoint) is mapped into `db` - [label](./keyConcepts.md#labels) value unless `db` tag exists in the InfluxDB line. + [label](https://docs.victoriametrics.com/keyconcepts/#labels) value unless `db` tag exists in the InfluxDB line. The `db` label name can be overridden via `-influxDBLabel` command-line flag. If more strict data isolation is required, - read more about multi-tenancy [here](./keyConcepts.md#multi-tenancy). + read more about multi-tenancy [here](https://docs.victoriametrics.com/keyconcepts/#multi-tenancy). * Field names are mapped to time series names prefixed with `{measurement}{separator}` value, where `{separator}` equals to `_` by default. It can be changed with `-influxMeasurementFieldSeparator` command-line flag. See also `-influxSkipSingleField` command-line flag. If `{measurement}` is empty or if `-influxSkipMeasurement` command-line flag is set, then time series names correspond to field names. * Field values are mapped to time series values. * Tags are mapped to Prometheus labels as-is. @@ -718,7 +718,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` -[Graphite relabeling](./vmagent.md#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](./MetricsQL.md). +[Graphite relabeling](https://docs.victoriametrics.com/vmagent/#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](https://docs.victoriametrics.com/metricsql/). ## Querying Graphite data @@ -730,7 +730,7 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t ## Selecting Graphite metrics -VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series with Graphite-compatible filters in [MetricsQL](./MetricsQL.md). For example, `{__graphite__="foo.*.bar"}` is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. See [docs for Graphite paths and wildcards](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). VictoriaMetrics also supports [label_graphite_group](./MetricsQL.md#label_graphite_group) function for extracting the given groups from Graphite metric name. +VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series with Graphite-compatible filters in [MetricsQL](https://docs.victoriametrics.com/metricsql/). For example, `{__graphite__="foo.*.bar"}` is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. See [docs for Graphite paths and wildcards](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). VictoriaMetrics also supports [label_graphite_group](https://docs.victoriametrics.com/metricsql/#label_graphite_group) function for extracting the given groups from Graphite metric name. The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `(value1|...|valueN)`. They are transparently converted to `{value1,...,valueN}` syntax [used in Graphite](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). This allows using [multi-value template variables in Grafana](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, Grafana expands `{__graphite__=~"foo.($bar).baz"}` into `{__graphite__=~"foo.(x|y).baz"}` if `$bar` template variable contains `x` and `y` values. In this case the query is automatically converted into `{__graphite__=~"foo.{x,y}.baz"}` before execution. @@ -826,7 +826,7 @@ For example, `/api/put?extra_label=foo=bar` would add `{foo="bar"}` label to all VictoriaMetrics accepts data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent) at `/newrelic/infra/v2/metrics/events/bulk` HTTP path. VictoriaMetrics receives [Events](https://docs.newrelic.com/docs/infrastructure/manage-your-data/data-instrumentation/default-infrastructure-monitoring-data/#infrastructure-events) -from NewRelic agent at the given path, transforms them to [raw samples](./keyConcepts.md#raw-samples) +from NewRelic agent at the given path, transforms them to [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) according to [these docs](#newrelic-agent-data-mapping) before storing the raw samples to the database. You need passing `COLLECTOR_URL` and `NRIA_LICENSE_KEY` environment variables to NewRelic infrastructure agent in order to send the collected metrics to VictoriaMetrics. @@ -841,11 +841,11 @@ COLLECTOR_URL="http://localhost:8428/newrelic" NRIA_LICENSE_KEY="NEWRELIC_LICENS ### NewRelic agent data mapping VictoriaMetrics maps [NewRelic Events](https://docs.newrelic.com/docs/infrastructure/manage-your-data/data-instrumentation/default-infrastructure-monitoring-data/#infrastructure-events) -to [raw samples](./keyConcepts.md#raw-samples) in the following way: +to [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in the following way: 1. Every numeric field is converted into a raw sample with the corresponding name. -1. The `eventType` and all the other fields with `string` value type are attached to every raw sample as [metric labels](./keyConcepts.md#labels). -1. The `timestamp` field is used as timestamp for the ingested [raw sample](./keyConcepts.md#raw-samples). +1. The `eventType` and all the other fields with `string` value type are attached to every raw sample as [metric labels](https://docs.victoriametrics.com/keyconcepts/#labels). +1. The `timestamp` field is used as timestamp for the ingested [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples). The `timestamp` field may be specified either in seconds or in milliseconds since the [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time). If the `timestamp` field is missing, then the raw sample is stored with the current timestamp. @@ -898,11 +898,11 @@ curl http://localhost:8428/api/v1/export -d 'match={eventType="SystemSample"}' VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): -* [/api/v1/query](./keyConcepts.md#instant-query) -* [/api/v1/query_range](./keyConcepts.md#range-query) -* [/api/v1/series](./url-examples.md#apiv1series) -* [/api/v1/labels](./url-examples.md#apiv1labels) -* [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) +* [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) +* [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) +* [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) +* [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) +* [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) * [/api/v1/status/tsdb](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). See [these docs](#tsdb-stats) for details. * [/api/v1/targets](https://prometheus.io/docs/prometheus/latest/querying/api/#targets) - see [these docs](#how-to-scrape-prometheus-exporters-such-as-node-exporter) for more details. * [/federate](https://prometheus.io/docs/prometheus/latest/federation/) - see [these docs](#federation) for more details. @@ -917,38 +917,38 @@ for enforcing additional label filters for queries. For example, `/api/v1/query_ would automatically add `{user_id="123",group_id="456"}` label filters to the given ``. This functionality can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query args are automatically set by auth proxy sitting in front of VictoriaMetrics. -See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. VictoriaMetrics accepts optional `extra_filters[]=series_selector` query arg, which can be used for enforcing arbitrary label filters for queries. For example, `/api/v1/query_range?extra_filters[]={env=~"prod|staging",user="xyz"}&query=` would automatically add `{env=~"prod|staging",user="xyz"}` label filters to the given ``. This functionality can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_filters[]` query args are automatically set by auth proxy sitting in front of VictoriaMetrics. -See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. VictoriaMetrics accepts multiple formats for `time`, `start` and `end` query args - see [these docs](#timestamp-formats). -VictoriaMetrics accepts `round_digits` query arg for [/api/v1/query](./keyConcepts.md#instant-query) -and [/api/v1/query_range](./keyConcepts.md#range-query) handlers. It can be used for rounding response values +VictoriaMetrics accepts `round_digits` query arg for [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) +and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) handlers. It can be used for rounding response values to the given number of digits after the decimal point. For example, `/api/v1/query?query=avg_over_time(temperature[1h])&round_digits=2` would round response values to up to two digits after the decimal point. -VictoriaMetrics accepts `limit` query arg for [/api/v1/labels](./url-examples.md#apiv1labels) -and [`/api/v1/label//values`](./url-examples.md#apiv1labelvalues) handlers for limiting the number of returned entries. +VictoriaMetrics accepts `limit` query arg for [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) +and [`/api/v1/label//values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) handlers for limiting the number of returned entries. For example, the query to `/api/v1/labels?limit=5` returns a sample of up to 5 unique labels, while ignoring the rest of labels. If the provided `limit` value exceeds the corresponding `-search.maxTagKeys` / `-search.maxTagValues` command-line flag values, then limits specified in the command-line flags are used. By default, VictoriaMetrics returns time series for the last day starting at 00:00 UTC -from [/api/v1/series](./url-examples.md#apiv1series), -[/api/v1/labels](./url-examples.md#apiv1labels) and -[`/api/v1/label//values`](./url-examples.md#apiv1labelvalues), +from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series), +[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and +[`/api/v1/label//values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues), while the Prometheus API defaults to all time. Explicitly set `start` and `end` to select the desired time range. VictoriaMetrics rounds the specified `start..end` time range to day granularity because of performance optimization concerns. If you need the exact set of label names and label values on the given time range, then send queries -to [/api/v1/query](./keyConcepts.md#instant-query) or to [/api/v1/query_range](./keyConcepts.md#range-query). +to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) or to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). -VictoriaMetrics accepts `limit` query arg at [/api/v1/series](./url-examples.md#apiv1series) +VictoriaMetrics accepts `limit` query arg at [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) for limiting the number of returned entries. For example, the query to `/api/v1/series?limit=5` returns a sample of up to 5 series, while ignoring the rest of series. If the provided `limit` value exceeds the corresponding `-search.maxSeries` command-line flag values, then limits specified in the command-line flags are used. @@ -973,8 +973,8 @@ Additionally, VictoriaMetrics provides the following handlers: ### Timestamp formats VictoriaMetrics accepts the following formats for `time`, `start` and `end` query args -in [query APIs](./README.md#prometheus-querying-api-usage) and -in [export APIs](./README.md#how-to-export-time-series). +in [query APIs](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and +in [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). - Unix timestamps in seconds with optional milliseconds after the point. For example, `1562529662.678`. - Unix timestamps in milliseconds. For example, `1562529662678`. @@ -996,11 +996,11 @@ VictoriaMetrics supports the following Graphite querying APIs, which are needed All the Graphite handlers can be pre-pended with `/graphite` prefix. For example, both `/graphite/metrics/find` and `/metrics/find` should work. -VictoriaMetrics accepts optional query args: `extra_label==` and `extra_filters[]=series_selector` query args for all the Graphite APIs. These args can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +VictoriaMetrics accepts optional query args: `extra_label==` and `extra_filters[]=series_selector` query args for all the Graphite APIs. These args can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. [Contact us](mailto:sales@victoriametrics.com) if you need assistance with such a proxy. -VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series with Graphite-compatible filters in [MetricsQL](./MetricsQL.md). See [these docs](#selecting-graphite-metrics). +VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series with Graphite-compatible filters in [MetricsQL](https://docs.victoriametrics.com/metricsql/). See [these docs](#selecting-graphite-metrics). ### Graphite Render API usage @@ -1108,7 +1108,7 @@ Note that `production` builds are not supported via Podman because Podman does n ## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) -helps to spin up VictoriaMetrics, [vmagent](./vmagent.md) and Grafana with one command. +helps to spin up VictoriaMetrics, [vmagent](https://docs.victoriametrics.com/vmagent/) and Grafana with one command. More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker). ## Setting up service @@ -1128,7 +1128,7 @@ The page returns the following JSON response on successful creation of snapshot: Snapshots are created under `<-storageDataPath>/snapshots` directory, where `<-storageDataPath>` is the corresponding command-line flag value. Snapshots can be archived to backup storage at any time -with [vmbackup](./vmbackup.md). +with [vmbackup](https://docs.victoriametrics.com/vmbackup/). Snapshots consist of a mix of hard-links and soft-links to various files and directories inside `-storageDataPath`. See [this article](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) @@ -1137,7 +1137,7 @@ for more details. This adds some restrictions on what can be done with the conte - Do not delete subdirectories inside `<-storageDataPath>/snapshots` with `rm` or similar commands, since this will leave some snapshot data undeleted. Prefer using the `/snapshot/delete` API for deleting snapshot. See below for more details about this API. - Do not copy subdirectories inside `<-storageDataPath>/snapshot` with `cp`, `rsync` or similar commands, since there are high chances - that these commands won't copy some data stored in the snapshot. Prefer using [vmbackup](./vmbackup.md) for making copies of snapshot data. + that these commands won't copy some data stored in the snapshot. Prefer using [vmbackup](https://docs.victoriametrics.com/vmbackup/) for making copies of snapshot data. See also [snapshot troubleshooting](#snapshot-troubleshooting). @@ -1151,7 +1151,7 @@ Navigate to `http://:8428/snapshot/delete_all` in order to ### How to restore from a snapshot 1. Stop VictoriaMetrics with `kill -INT`. -1. Restore snapshot contents from backup with [vmrestore](./vmrestore.md) +1. Restore snapshot contents from backup with [vmrestore](https://docs.victoriametrics.com/vmrestore/) to the directory pointed by `-storageDataPath`. 1. Start VictoriaMetrics. @@ -1529,7 +1529,7 @@ VictoriaMetrics supports data ingestion via [OpenTelemetry protocol for metrics] VictoriaMetrics expects `protobuf`-encoded requests at `/opentelemetry/v1/metrics`. Set HTTP request header `Content-Encoding: gzip` when sending gzip-compressed data to `/opentelemetry/v1/metrics`. -VictoriaMetrics stores the ingested OpenTelemetry [raw samples](./keyConcepts.md#raw-samples) as is without any transformations. +VictoriaMetrics stores the ingested OpenTelemetry [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) as is without any transformations. Pass `-opentelemetry.usePrometheusNaming` command-line flag to VictoriaMetrics for automatic conversion of metric names and labels into Prometheus-compatible format. Using the following exporter configuration in the opentelemetry collector will allow you to send metrics into VictoriaMetrics: @@ -1551,7 +1551,7 @@ service: receivers: - otlp ``` -See [How to use OpenTelemetry metrics with VictoriaMetrics](./guides/getting-started-with-opentelemetry.md). +See [How to use OpenTelemetry metrics with VictoriaMetrics](https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/). ## JSON line format @@ -1592,8 +1592,8 @@ Too long JSON lines may increase RAM usage at VictoriaMetrics side. [/api/v1/export](#how-to-export-data-in-json-line-format) handler accepts `max_rows_per_line` query arg, which allows limiting the number of samples per each exported line. -It is OK to split [raw samples](./keyConcepts.md#raw-samples) -for the same [time series](./keyConcepts.md#time-series) across multiple lines. +It is OK to split [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +for the same [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) across multiple lines. The number of lines in the request to [/api/v1/import](#how-to-import-data-in-json-line-format) can be arbitrary - they are imported in streaming manner. @@ -1605,7 +1605,7 @@ The `-relabelConfig` also can point to http or https url. For example, `-relabel The following docs can be useful in understanding the relabeling: -* [Cookbook for common relabeling tasks](./relabeling.md). +* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling/). * [Relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2). The `-relabelConfig` files can contain special placeholders in the form `%{ENV_VAR}`, which are replaced by the corresponding environment variable values. @@ -1624,11 +1624,11 @@ Example contents for `-relabelConfig` file: ``` VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling. -See [these docs](./vmagent.md#relabeling) for more details. +See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details. The relabeling can be debugged at `http://victoriametrics:8428/metric-relabel-debug` page or at our [public playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/relabeling). -See [these docs](./vmagent.md#relabel-debug) for more details. +See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug) for more details. ## Federation @@ -1650,9 +1650,9 @@ with scrape intervals exceeding `5m`. ## Capacity planning -VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](./CaseStudies.md). +VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](https://docs.victoriametrics.com/casestudies/). -VictoriaMetrics capacity scales linearly with the available resources. The needed amounts of CPU and RAM highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), series [churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics for your production workload and iteratively scaling CPU and RAM resources until it becomes stable according to [troubleshooting docs](#troubleshooting). A single-node VictoriaMetrics works perfectly with the following production workload according to [our case studies](./CaseStudies.md): +VictoriaMetrics capacity scales linearly with the available resources. The needed amounts of CPU and RAM highly depends on the workload - the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series), series [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics for your production workload and iteratively scaling CPU and RAM resources until it becomes stable according to [troubleshooting docs](#troubleshooting). A single-node VictoriaMetrics works perfectly with the following production workload according to [our case studies](https://docs.victoriametrics.com/casestudies/): * Ingestion rate: 1.5+ million samples per second * Active time series: 50+ million @@ -1692,57 +1692,57 @@ By default, VictoriaMetrics is tuned for an optimal resource usage under typical of additional memory. So it is better to limit the number of concurrent queries, while pausing additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for paused queries. See also `-search.maxMemoryPerQuery` command-line flag. - `-search.maxQueueDuration` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests` concurrent queries are executed. -- `-search.ignoreExtraFiltersAtLabelsAPI` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./README.md#prometheus-querying-api-enhancements) - query args at [/api/v1/labels](./url-examples.md#apiv1labels) and - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). +- `-search.ignoreExtraFiltersAtLabelsAPI` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) + query args at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This may be useful for reducing the load on VictoriaMetrics if the provided extra filters match too many time series. The downside is that the endpoints can return labels and series, which do not match the provided extra filters. - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory - and then applies the given [rollup function](./MetricsQL.md#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag + and then applies the given [rollup function](https://docs.victoriametrics.com/metricsql/#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series. - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries. -- `-search.maxResponseSeries` limits the number of time series a single query can return from [`/api/v1/query`](./keyConcepts.md#instant-query) - and [`/api/v1/query_range`](./keyConcepts.md#range-query). +- `-search.maxResponseSeries` limits the number of time series a single query can return from [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) + and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series - from [range query](./keyConcepts.md#range-query). + from [range query](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsSubqueryPerTimeseries` limits the number of calculated points, which can be generated per each matching time series - during [subquery](./MetricsQL.md#subqueries) evaluation. -- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](./MetricsQL.md#aggregate-functions) + during [subquery](https://docs.victoriametrics.com/metricsql/#subqueries) evaluation. +- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions) in a single query. -- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](./url-examples.md#apiv1series). +- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amounts - of CPU time and memory when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxSeries` to quite low value in order limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](./url-examples.md#apiv1labels). +- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). +- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxLabelsAPISeries` limits the number of time series, which can be scanned when performing [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series) requests. +- `-search.maxLabelsAPISeries` limits the number of time series, which can be scanned when performing [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) requests. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPISeries` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.ignoreExtraFiltersAtLabelsAPI`. -- `-search.maxLabelsAPIDuration` limits the duration for requests to [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series). +- `-search.maxLabelsAPIDuration` limits the duration for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). The limit can be altered for each query by passing `timeout` GET parameter, but can't exceed the limit specified via cmd-line flag. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPIDuration` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPISeries` and `-search.ignoreExtraFiltersAtLabelsAPI`. - `-search.maxTagValueSuffixesPerSearch` limits the number of entries, which may be returned from `/metrics/find` endpoint. See [Graphite Metrics API usage docs](#graphite-metrics-api-usage). -See also [resource usage limits at VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#resource-usage-limits), +See also [resource usage limits at VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits), [cardinality limiter](#cardinality-limiter) and [capacity planning docs](#capacity-planning). @@ -1751,17 +1751,17 @@ See also [resource usage limits at VictoriaMetrics cluster](./Cluster-VictoriaMe The general approach for achieving high availability is the following: - To run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones); -- To store the collected data simultaneously into these instances via [vmagent](./vmagent.md) or Prometheus. +- To store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus. - To query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable. - This can be done via [vmauth](./vmauth.md) according to [these docs](./vmauth.md#high-availability). + This can be done via [vmauth](https://docs.victoriametrics.com/vmauth/) according to [these docs](https://docs.victoriametrics.com/vmauth/#high-availability). Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable. The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying. -Both [vmagent](./vmagent.md) and Prometheus buffer the collected data locally if they cannot send it +Both [vmagent](https://docs.victoriametrics.com/vmagent/) and Prometheus buffer the collected data locally if they cannot send it to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance after it becomes available. -If you use [vmagent](./vmagent.md) for storing the data into VictoriaMetrics, +If you use [vmagent](https://docs.victoriametrics.com/vmagent/) for storing the data into VictoriaMetrics, then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances. For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics: @@ -1782,16 +1782,16 @@ remote_write: - url: http://:8428/api/v1/write ``` -It is recommended to use [vmagent](./vmagent.md) instead of Prometheus for highly loaded setups, +It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent/) instead of Prometheus for highly loaded setups, since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus. -If you use identically configured [vmagent](./vmagent.md) instances for collecting the same data +If you use identically configured [vmagent](https://docs.victoriametrics.com/vmagent/) instances for collecting the same data and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side. ## Deduplication -VictoriaMetrics leaves a single [raw sample](./keyConcepts.md#raw-samples) -with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](./keyConcepts.md#time-series) +VictoriaMetrics leaves a single [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete `60s` interval. @@ -1800,12 +1800,12 @@ This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/ If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, then the sample with **the biggest value** is kept. -[Prometheus staleness markers](./vmagent.md#prometheus-staleness-markers) are processed as any other value during de-duplication. +[Prometheus staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) are processed as any other value during de-duplication. If raw sample with the biggest timestamp on `-dedup.minScrapeInterval` contains a stale marker, then it is kept after the deduplication. This allows properly preserving staleness markers during the de-duplication. -Please note, [labels](./keyConcepts.md#labels) of raw samples should be identical -in order to be deduplicated. For example, this is why [HA pair of vmagents](./vmagent.md#high-availability) +Please note, [labels](https://docs.victoriametrics.com/keyconcepts/#labels) of raw samples should be identical +in order to be deduplicated. For example, this is why [HA pair of vmagents](https://docs.victoriametrics.com/vmagent/#high-availability) needs to be identically configured. The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. @@ -1815,21 +1815,21 @@ The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_inter It is recommended to have a single `scrape_interval` across all the scrape targets. See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details. -The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](./vmagent.md) +The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus instances in HA pair write data to the same VictoriaMetrics instance. These vmagent or Prometheus instances must have **identical** `external_labels` section in their configs, so they write data to the same time series. -See also [how to set up multiple vmagent instances for scraping the same targets](./vmagent.md#scraping-big-number-of-targets). +See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). It is recommended passing different `-promscrape.cluster.name` values to each distinct HA pair of `vmagent` instances, so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples from other `vmagent` instances. -See [these docs](./vmagent.md#high-availability) for details. +See [these docs](https://docs.victoriametrics.com/vmagent/#high-availability) for details. VictoriaMetrics stores all the ingested samples to disk even if `-dedup.minScrapeInterval` command-line flag is set. The ingested samples are de-duplicated during [background merges](#storage) and during query execution. VictoriaMetrics also supports de-duplication during data ingestion before the data is stored to disk, via `-streamAggr.dedupInterval` command-line flag - -see [these docs](./stream-aggregation.md#deduplication). +see [these docs](https://docs.victoriametrics.com/stream-aggregation/#deduplication). ## Storage @@ -1841,19 +1841,19 @@ can be configured with the `-inmemoryDataFlushInterval` command-line flag (note In-memory parts are persisted to disk into `part` directories under the `<-storageDataPath>/data/small/YYYY_MM/` folder, where `YYYY_MM` is the month partition for the stored data. For example, `2022_11` is the partition for `parts` -with [raw samples](./keyConcepts.md#raw-samples) from `November 2022`. +with [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) from `November 2022`. Each partition directory contains `parts.json` file with the actual list of parts in the partition. Every `part` directory contains `metadata.json` file with the following fields: -- `RowsCount` - the number of [raw samples](./keyConcepts.md#raw-samples) stored in the part +- `RowsCount` - the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) stored in the part - `BlocksCount` - the number of blocks stored in the part (see details about blocks below) - `MinTimestamp` and `MaxTimestamp` - minimum and maximum timestamps across raw samples stored in the part - `MinDedupInterval` - the [deduplication interval](#deduplication) applied to the given part. Each `part` consists of `blocks` sorted by internal time series id (aka `TSID`). -Each `block` contains up to 8K [raw samples](./keyConcepts.md#raw-samples), -which belong to a single [time series](./keyConcepts.md#time-series). +Each `block` contains up to 8K [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples), +which belong to a single [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). Raw samples in each block are sorted by `timestamp`. Blocks for the same time series are sorted by the `timestamp` of the first sample. Timestamps and values for all the blocks are stored in [compressed form](https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932) @@ -1882,7 +1882,7 @@ leaving the source `parts` untouched. However, due to hardware issues data on di VictoriaMetrics process. VictoriaMetrics can detect corruption during decompressing, decoding or sanity checking of the data blocks. But **it cannot fix the corrupted data**. Data parts that fail to load on startup need to be deleted or restored from backups. This is why it is recommended performing -[regular backups](./Cluster-VictoriaMetrics.md#backups). +[regular backups](https://docs.victoriametrics.com/cluster-victoriametrics/#backups). VictoriaMetrics doesn't use checksums for stored data blocks. See why [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011). @@ -1927,7 +1927,7 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar ## Multiple retentions Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters) -in [VictoriaMetrics enterprise](./enterprise.md). +in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag. If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags: @@ -1936,18 +1936,18 @@ If you need multiple retentions in community version of VictoriaMetrics, then yo * `-storageDataPath`, so the data for each retention period is saved in a separate directory * `-httpListenAddr`, so clients may reach VictoriaMetrics instance with proper retention -Then set up [vmauth](./vmauth.md) in front of VictoriaMetrics instances, +Then set up [vmauth](https://docs.victoriametrics.com/vmauth/) in front of VictoriaMetrics instances, so it could route requests from particular user to VictoriaMetrics with the desired retention. -Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md). -See [these docs](./guides/guide-vmcluster-multiple-retention-setup.md) for multi-retention setup details. +Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). +See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details. ## Retention filters -[Enterprise version of VictoriaMetrics](./enterprise.md) supports e.g. `retention filters`, -which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](./keyConcepts.md#filtering) +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports e.g. `retention filters`, +which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyconcepts/#filtering) via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be -a valid [series filter](./keyConcepts.md#filtering), while the `duration` +a valid [series filter](https://docs.victoriametrics.com/keyconcepts/#filtering), while the `duration` must contain valid [retention](#retention) for time series matching the given `filter`. The `duration` of the `-retentionFilter` must be lower or equal to [-retentionPeriod](#retention) flag value. If series doesn't match any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) @@ -1962,15 +1962,15 @@ For example, the following config sets 3 days retention for time series with `te Important notes: -- The data outside the configured retention isn't deleted instantly - it is deleted eventually during [background merges](./README.md#storage). +- The data outside the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage). - The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention). - So the `indexdb` size can grow big under [high churn rate](./FAQ.md#what-is-high-churn-rate) + So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) even for small retentions configured via `-retentionFilter`. It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually to historical data. -See [how to configure multiple retentions in VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#retention-filters). +See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#retention-filters). See also [downsampling](#downsampling). @@ -1979,60 +1979,60 @@ See how to request a free trial license [here](https://victoriametrics.com/produ ## Downsampling -[VictoriaMetrics Enterprise](./enterprise.md) supports multi-level downsampling via `-downsampling.period=offset:interval` command-line flag. -This command-line flag instructs leaving the last sample per each `interval` for [time series](./keyConcepts.md#time-series) -[samples](./keyConcepts.md#raw-samples) older than the `offset`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample +[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise/) supports multi-level downsampling via `-downsampling.period=offset:interval` command-line flag. +This command-line flag instructs leaving the last sample per each `interval` for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) +[samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) older than the `offset`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample per each 5-minute interval for samples older than 30 days, while the rest of samples are dropped. The `-downsampling.period` command-line flag can be specified multiple times in order to apply different downsampling levels for different time ranges (aka multi-level downsampling). For example, `-downsampling.period=30d:5m,180d:1h` instructs leaving the last sample per each 5-minute interval for samples older than 30 days, while leaving the last sample per each 1-hour interval for samples older than 180 days. -VictoriaMetrics supports configuring independent downsampling per different sets of [time series](./keyConcepts.md#time-series) +VictoriaMetrics supports configuring independent downsampling per different sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) via `-downsampling.period=filter:offset:interval` syntax. In this case the given `offset:interval` downsampling is applied only to time series matching the given `filter`. -The `filter` can contain arbitrary [series filter](./keyConcepts.md#filtering). +The `filter` can contain arbitrary [series filter](https://docs.victoriametrics.com/keyconcepts/#filtering). For example, `-downsampling.period='{__name__=~"(node|process)_.*"}:1d:1m` instructs VictoriaMetrics to deduplicate samples older than one day with one minute interval -only for [time series](./keyConcepts.md#time-series) with names starting with `node_` or `process_` prefixes. +only for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) with names starting with `node_` or `process_` prefixes. The de-duplication for other time series can be configured independently via additional `-downsampling.period` command-line flags. If the time series doesn't match any `filter`, then it isn't downsampled. If the time series matches multiple filters, then the downsampling for the first matching `filter` is applied. For example, `-downsampling.period='{env="prod"}:1d:30s,{__name__=~"node_.*"}:1d:5m'` de-duplicates -samples older than one day with 30 seconds interval across all the time series with `env="prod"` [label](./keyConcepts.md#labels), +samples older than one day with 30 seconds interval across all the time series with `env="prod"` [label](https://docs.victoriametrics.com/keyconcepts/#labels), even if their names start with `node_` prefix. All the other time series with names starting with `node_` prefix are de-duplicated with 5 minutes interval. If downsampling shouldn't be applied to some time series matching the given `filter`, then pass `-downsampling.period=filter:0s:0s` command-line flag to VictoriaMetrics. For example, if series with `env="prod"` label shouldn't be downsampled, then pass `-downsampling.period='{env="prod"}:0s:0s'` command-line flag in front of other `-downsampling.period` flags. -Downsampling is applied independently per each time series and leaves a single [raw sample](./keyConcepts.md#raw-samples) +Downsampling is applied independently per each time series and leaves a single [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) on the configured interval, in the same way as [deduplication](#deduplication) does. -It works the best for [counters](./keyConcepts.md#counter) and [histograms](./keyConcepts.md#histogram), -as their values are always increasing. Downsampling [gauges](./keyConcepts.md#gauge) -and [summaries](./keyConcepts.md#summary) lose some changes within the downsampling interval, +It works the best for [counters](https://docs.victoriametrics.com/keyconcepts/#counter) and [histograms](https://docs.victoriametrics.com/keyconcepts/#histogram), +as their values are always increasing. Downsampling [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge) +and [summaries](https://docs.victoriametrics.com/keyconcepts/#summary) lose some changes within the downsampling interval, since only the last sample on the given interval is left and the rest of samples are dropped. -You can use [recording rules](./vmalert.md#rules) or [steaming aggregation](./stream-aggregation.md) +You can use [recording rules](https://docs.victoriametrics.com/vmalert/#rules) or [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation/) to apply custom aggregation functions, like min/max/avg etc., in order to make gauges more resilient to downsampling. Downsampling can reduce disk space usage and improve query performance if it is applied to time series with big number of samples per each series. The downsampling doesn't improve query performance and doesn't reduce disk space if the database contains big number of time series with small number of samples per each series, since downsampling doesn't reduce the number of time series. -So there is little sense in applying downsampling to time series with [high churn rate](./FAQ.md#what-is-high-churn-rate). +So there is little sense in applying downsampling to time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case the majority of query time is spent on searching for the matching time series instead of processing the found samples. -It is possible to use [stream aggregation](./stream-aggregation.md) in [vmagent](./vmagent.md) -or [recording rules in vmalert](./vmalert.md#rules) in order to -[reduce the number of time series](./vmalert.md#downsampling-and-aggregation-via-vmalert). +It is possible to use [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) in [vmagent](https://docs.victoriametrics.com/vmagent/) +or [recording rules in vmalert](https://docs.victoriametrics.com/vmalert/#rules) in order to +[reduce the number of time series](https://docs.victoriametrics.com/vmalert/#downsampling-and-aggregation-via-vmalert). -Downsampling is performed during [background merges](./README.md#storage). -It cannot be performed if there is not enough of free disk space or if vmstorage is in [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). +Downsampling is performed during [background merges](https://docs.victoriametrics.com/#storage). +It cannot be performed if there is not enough of free disk space or if vmstorage is in [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). Please, note that intervals of `-downsampling.period` must be multiples of each other. -In case [deduplication](./README.md#deduplication) is enabled, value of `-dedup.minScrapeInterval` command-line flag must also +In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled, value of `-dedup.minScrapeInterval` command-line flag must also be multiple of `-downsampling.period` intervals. This is required to ensure consistency of deduplication and downsampling results. It is safe updating `-downsampling.period` during VictoriaMetrics restarts - the updated downsampling configuration will be -applied eventually to historical data during [background merges](./README.md#storage). +applied eventually to historical data during [background merges](https://docs.victoriametrics.com/#storage). -See [how to configure downsampling in VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#downsampling). +See [how to configure downsampling in VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#downsampling). See also [retention filters](#retention-filters). @@ -2041,7 +2041,7 @@ See [how to request a free trial license](https://victoriametrics.com/products/e ## Multi-tenancy -Single-node VictoriaMetrics doesn't support multi-tenancy. Use the [cluster version](./Cluster-VictoriaMetrics.md#multitenancy) instead. +Single-node VictoriaMetrics doesn't support multi-tenancy. Use the [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) instead. ## Scalability and cluster version @@ -2055,7 +2055,7 @@ horizontally scalable long-term remote storage for really large Prometheus deplo ## Alerting -It is recommended using [vmalert](./vmalert.md) for alerting. +It is recommended using [vmalert](https://docs.victoriametrics.com/vmalert/) for alerting. Additionally, alerting can be set up with the following tools: @@ -2066,7 +2066,7 @@ Additionally, alerting can be set up with the following tools: ## mTLS protection By default `VictoriaMetrics` accepts http requests at `8428` port (this port can be changed via `-httpListenAddr` command-line flags). -[Enterprise version of VictoriaMetrics](./enterprise.md) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `VictoriaMetrics`, which accepts only mTLS requests at port `8428`: ``` @@ -2083,11 +2083,11 @@ See also [security docs](#security). General security recommendations: - All the VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as Internet. - The exception is [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md), + The exception is [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/), which are intended for serving public requests and performing authorization with [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy). -- All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](./vmauth.md) - or [vmgateway](./vmgateway.md). The proxy must be set up with proper authentication and authorization. -- Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](./vmauth.md) +- All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](https://docs.victoriametrics.com/vmauth/) + or [vmgateway](https://docs.victoriametrics.com/vmgateway/). The proxy must be set up with proper authentication and authorization. +- Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](https://docs.victoriametrics.com/vmauth/) in front of VictoriaMetrics components. - Set reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value to all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag. - Set reasonable [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) header value to mitigate [XSS attacks](https://en.wikipedia.org/wiki/Cross-site_scripting). See `-http.header.csp` flag. @@ -2096,7 +2096,7 @@ General security recommendations: VictoriaMetrics provides the following security-related command-line flags: * `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr` (TCP port 8428 is listened by default). - [Enterprise version of VictoriaMetrics](./enterprise.md) supports automatic issuing of TLS certificates. + [Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports automatic issuing of TLS certificates. See [these docs](#automatic-issuing-of-tls-certificates). * `-mtls` and `-mtlsCAFile` for enabling [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) for requests to `-httpListenAddr`. See [these docs](#mtls-protection). * `-httpAuth.username` and `-httpAuth.password` for protecting all the HTTP endpoints @@ -2117,12 +2117,12 @@ VictoriaMetrics provides the following security-related command-line flags: Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats. For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=:2003`. This protects from unexpected requests from untrusted network interfaces. -See also [security recommendation for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#security) +See also [security recommendation for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#security) and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/). ## Automatic issuing of TLS certificates -All the VictoriaMetrics [Enterprise](./enterprise.md) components support automatic issuing of TLS certificates for public HTTPS server running at `-httpListenAddr` +All the VictoriaMetrics [Enterprise](https://docs.victoriametrics.com/enterprise/) components support automatic issuing of TLS certificates for public HTTPS server running at `-httpListenAddr` via [Let's Encrypt service](https://letsencrypt.org/). The following command-line flags must be set in order to enable automatic issuing of TLS certificates: - `-httpListenAddr` must be set for listening TCP port `443`. For example, `-httpListenAddr=:443`. This port must be accessible by the [Let's Encrypt service](https://letsencrypt.org/). @@ -2132,7 +2132,7 @@ via [Let's Encrypt service](https://letsencrypt.org/). The following command-lin - `-tlsAutocertCacheDir` may be set to the directory path for persisting the issued TLS certificates between VictoriaMetrics restarts. If this flag isn't set, then TLS certificates are re-issued on every restart. -This functionality can be evaluated for free according to [these docs](./enterprise.md). +This functionality can be evaluated for free according to [these docs](https://docs.victoriametrics.com/enterprise/). See also [security recommendations](#security). @@ -2159,7 +2159,7 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ## Monitoring VictoriaMetrics exports internal metrics in Prometheus exposition format at `/metrics` page. -These metrics can be scraped via [vmagent](./vmagent.md) or any other Prometheus-compatible scraper. +These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent/) or any other Prometheus-compatible scraper. If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass `-metrics.exposeMetadata` command-line to them, so they add `TYPE` and `HELP` comments per each exposed metric at `/metrics` page. @@ -2179,14 +2179,14 @@ created by community. Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it. We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts) -via [vmalert](./vmalert.md) or via Prometheus. +via [vmalert](https://docs.victoriametrics.com/vmalert/) or via Prometheus. VictoriaMetrics exposes currently running queries and their execution times at [`active queries` page](#active-queries). VictoriaMetrics exposes queries, which take the most time to execute, at [`top queries` page](#top-queries). See also [VictoriaMetrics Monitoring](https://victoriametrics.com/blog/victoriametrics-monitoring/) -and [troubleshooting docs](./Troubleshooting.md). +and [troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/). ## TSDB stats @@ -2198,10 +2198,10 @@ VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way simi * `match[]=SELECTOR` where `SELECTOR` is an arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) for series to take into account during stats calculation. By default all the series are taken into account. * `extra_label=LABEL=VALUE`. See [these docs](#prometheus-querying-api-enhancements) for more details. -In [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) each vmstorage tracks the stored time series individually. +In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) each vmstorage tracks the stored time series individually. vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats. This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes -due to [replication](#replication) or [rerouting](./Cluster-VictoriaMetrics.md?highlight=re-routes#cluster-availability). +due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=re-routes#cluster-availability). VictoriaMetrics provides an UI on top of `/api/v1/status/tsdb` - see [cardinality explorer docs](#cardinality-explorer). @@ -2277,8 +2277,8 @@ Query tracing is allowed by default. It can be denied by passing `-denyQueryTrac By default VictoriaMetrics doesn't limit the number of stored time series. The limit can be enforced by setting the following command-line flags: -* `-storage.maxHourlySeries` - limits the number of time series that can be added during the last hour. Useful for limiting the number of [active time series](./FAQ.md#what-is-an-active-time-series). -* `-storage.maxDailySeries` - limits the number of time series that can be added during the last day. Useful for limiting daily [churn rate](./FAQ.md#what-is-high-churn-rate). +* `-storage.maxHourlySeries` - limits the number of time series that can be added during the last hour. Useful for limiting the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). +* `-storage.maxDailySeries` - limits the number of time series that can be added during the last day. Useful for limiting daily [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). Both limits can be set simultaneously. If any of these limits is reached, then incoming samples for new time series are dropped. A sample of dropped series is put in the log with `WARNING` level. @@ -2308,7 +2308,7 @@ The exceeded limits can be [monitored](#monitoring) with the following metrics: These limits are approximate, so VictoriaMetrics can underflow/overflow the limit by a small percentage (usually less than 1%). -See also more advanced [cardinality limiter in vmagent](./vmagent.md#cardinality-limiter) +See also more advanced [cardinality limiter in vmagent](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) and [cardinality explorer docs](#cardinality-explorer). ## Troubleshooting @@ -2331,8 +2331,8 @@ and [cardinality explorer docs](#cardinality-explorer). * If you run VictoriaMetrics on a host with 16 or more CPU cores, then it may be needed to tune the `-search.maxWorkersPerQuery` command-line flag in order to improve query performance. If VictoriaMetrics serves big number of concurrent `select` queries, then try reducing the value for this flag. - If VictoriaMetrics serves heavy queries, which select `>10K` of [time series](./keyConcepts.md#time-series) and/or process `>100M` - of [raw samples](./keyConcepts.md#raw-samples) per query, then try setting the value for this flag to the number of available CPU cores. + If VictoriaMetrics serves heavy queries, which select `>10K` of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) and/or process `>100M` + of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) per query, then try setting the value for this flag to the number of available CPU cores. * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage. This may lead to the following "issues": @@ -2343,7 +2343,7 @@ and [cardinality explorer docs](#cardinality-explorer). See [storage docs](#storage) and [this article](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704) for more details. * If VictoriaMetrics works slowly and eats more than a CPU core per 100K ingested data points per second, - then it is likely you have too many [active time series](./FAQ.md#what-is-an-active-time-series) for the current amount of RAM. + then it is likely you have too many [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) for the current amount of RAM. VictoriaMetrics [exposes](#monitoring) `vm_slow_*` metrics such as `vm_slow_row_inserts_total` and `vm_slow_metric_name_loads_total`, which could be used as an indicator of low amounts of RAM. It is recommended increasing the amount of RAM on the node with VictoriaMetrics in order to improve ingestion and query performance in this case. @@ -2371,8 +2371,8 @@ and [cardinality explorer docs](#cardinality-explorer). This suppresses default gap filling algorithm used by VictoriaMetrics - by default it assumes each time series is continuous instead of discrete, so it fills gaps between real samples with regular intervals. -* Metrics and labels leading to [high cardinality](./FAQ.md#what-is-high-cardinality) - or [high churn rate](./FAQ.md#what-is-high-churn-rate) can be determined +* Metrics and labels leading to [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) + or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) can be determined via [cardinality explorer](#cardinality-explorer) and via [/api/v1/status/tsdb](#tsdb-stats) endpoint. * New time series can be logged if `-logNewSeries` command-line flag is passed to VictoriaMetrics. @@ -2390,13 +2390,13 @@ and [cardinality explorer docs](#cardinality-explorer). See also: - [Snapshot troubleshooting](#snapshot-troubleshooting). -- [General troubleshooting docs](./Troubleshooting.md). +- [General troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/). ## Push metrics All the VictoriaMetrics components support pushing their metrics exposed at `/metrics` page to remote storage in Prometheus text exposition format. -This functionality may be used instead of [classic Prometheus-like metrics scraping](./README.md#how-to-scrape-prometheus-exporters-such-as-node-exporter) -if VictoriaMetrics components are located in isolated networks, so they cannot be scraped by local [vmagent](./vmagent.md). +This functionality may be used instead of [classic Prometheus-like metrics scraping](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) +if VictoriaMetrics components are located in isolated networks, so they cannot be scraped by local [vmagent](https://docs.victoriametrics.com/vmagent/). The following command-line flags are related to pushing metrics from VictoriaMetrics components: @@ -2441,7 +2441,7 @@ It is also possible removing [rollup result cache](#rollup-result-cache) on star ## Rollup result cache VictoriaMetrics caches query responses by default. This allows increasing performance for repeated queries -to [`/api/v1/query`](./keyConcepts.md#instant-query) and [`/api/v1/query_range`](./keyConcepts.md#range-query) +to [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query) with the increasing `time`, `start` and `end` query args. This cache may work incorrectly when ingesting historical data into VictoriaMetrics. See [these docs](#backfilling) for details. @@ -2496,30 +2496,30 @@ Things to consider when copying data: 1. Copying data folder means complete replacement of the previous data on destination VictoriaMetrics. For more complex scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction -of data - see [vmctl. Migrating data from VictoriaMetrics](./vmctl.md#migrating-data-from-victoriametrics). +of data - see [vmctl. Migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). ### From other systems -Use [vmctl](./vmctl.md) for data migration. It supports the following data migration types: +Use [vmctl](https://docs.victoriametrics.com/vmctl/) for data migration. It supports the following data migration types: * From Prometheus to VictoriaMetrics * From InfluxDB to VictoriaMetrics * From VictoriaMetrics to VictoriaMetrics * From OpenTSDB to VictoriaMetrics -See [vmctl docs](./vmctl.md) for more details. +See [vmctl docs](https://docs.victoriametrics.com/vmctl/) for more details. ## Backfilling VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data). -See [how to backfill data with recording rules in vmalert](./vmalert.md#rules-backfilling). +See [how to backfill data with recording rules in vmalert](https://docs.victoriametrics.com/vmalert/#rules-backfilling). Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data. It is recommended disabling [query cache](#rollup-result-cache) with `-search.disableCache` command-line flag when writing historical data with timestamps from the past, since the cache assumes that the data is written with the current timestamps. Query cache can be enabled after the backfilling is complete. -An alternative solution is to query [/internal/resetRollupResultCache](./url-examples.md#internalresetrollupresultcache) +An alternative solution is to query [/internal/resetRollupResultCache](https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache) after the backfilling is complete. This will reset the [query cache](#rollup-result-cache), which could contain incomplete data cached during the backfilling. Yet another solution is to increase `-search.cacheTimestampOffset` flag value in order to disable caching @@ -2536,7 +2536,7 @@ should be used only for one-off updates. It shouldn't be used for frequent updat ## Replication Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead. -See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details. +See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details. Storage-level replication may be offloaded to durable persistent storage such as [Google Cloud disks](https://cloud.google.com/compute/docs/disks#pdspecs). @@ -2544,15 +2544,15 @@ See also [high availability docs](#high-availability) and [backup docs](#backups ## Backups -VictoriaMetrics supports backups via [vmbackup](./vmbackup.md) -and [vmrestore](./vmrestore.md) tools. -We also provide [vmbackupmanager](./vmbackupmanager.md) tool for enterprise subscribers. +VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/vmbackup/) +and [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. +We also provide [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) tool for enterprise subscribers. Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/). ## vmalert -A single-node VictoriaMetrics is capable of proxying requests to [vmalert](./vmalert.md) +A single-node VictoriaMetrics is capable of proxying requests to [vmalert](https://docs.victoriametrics.com/vmalert/) when `-vmalert.proxyURL` flag is set. Use this feature for the following cases: * for proxying requests from [Grafana Alerting UI](https://grafana.com/docs/grafana/latest/alerting/); * for accessing vmalerts UI through single-node VictoriaMetrics Web interface. @@ -2567,7 +2567,7 @@ the best parts of their product, while highlighting the worst parts of competing So we encourage users and all independent third parties to conduct their benchmarks for various products they are evaluating in production and publish the results. -As a reference, please see [benchmarks](./Articles.md#benchmarks) conducted by +As a reference, please see [benchmarks](https://docs.victoriametrics.com/articles/#benchmarks) conducted by VictoriaMetrics team. Please also see the [helm chart](https://github.com/VictoriaMetrics/benchmark) for running ingestion benchmarks based on node_exporter metrics. @@ -2602,7 +2602,7 @@ It is safe sharing the collected profiles from security point of view, since the See [this example](https://github.com/go-graphite/carbonapi/blob/main/cmd/carbonapi/carbonapi.example.victoriametrics.yaml). * [netdata](https://github.com/netdata/netdata) can push data into VictoriaMetrics via `Prometheus remote_write API`. See [these docs](https://github.com/netdata/netdata#integrations). -* [vmalert-cli](https://github.com/aorfanos/vmalert-cli) - a CLI application for managing [vmalert](./vmalert.md). +* [vmalert-cli](https://github.com/aorfanos/vmalert-cli) - a CLI application for managing [vmalert](https://docs.victoriametrics.com/vmalert/). ## Third-party contributions @@ -2628,7 +2628,7 @@ Feel free asking any questions regarding VictoriaMetrics: * [Google groups](https://groups.google.com/forum/#!forum/victorametrics-users) * [Mastodon](https://mastodon.social/@victoriametrics/) -If you like VictoriaMetrics and want to contribute, then please [read these docs](./CONTRIBUTING.md). +If you like VictoriaMetrics and want to contribute, then please [read these docs](https://docs.victoriametrics.com/contributing/). ## Reporting bugs @@ -2644,7 +2644,7 @@ To update the documentation follow the steps below: VictoriaMetrics repo and apply changes to the docs: - To update [the main page](https://docs.victoriametrics.com/) modify [this file](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md). - To update other pages, apply changes to the corresponding file in [docs folder](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs). -- If your changes contain an image then see [images in documentation](./README.md#images-in-documentation). +- If your changes contain an image then see [images in documentation](https://docs.victoriametrics.com/#images-in-documentation). - Once changes are made, execute the command below to finalize and sync the changes: ```sh @@ -2721,7 +2721,7 @@ Files included in each folder: Pass `-help` to VictoriaMetrics in order to see the list of supported command-line flags with their description: -``` +```sh -bigMergeConcurrency int Deprecated: this flag does nothing -blockcache.missesBeforeCaching int @@ -2739,16 +2739,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -datadog.sanitizeMetricName Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true) -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and {{% ref "./README.md#deduplication" %}} + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication -deleteAuthKey value authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path -denyQueriesOutsideRetention Whether to deny queries outside the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./README.md#query-tracing" %}} + Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./README.md#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -dryRun @@ -2756,11 +2756,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./README.md#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula - Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -finalMergeDelay duration @@ -2777,7 +2777,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -graphite.sanitizeMetricName - Sanitize metric names for the ingested Graphite data. See {{% ref "./README.md#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}} + Sanitize metric names for the ingested Graphite data. See https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd -graphiteListenAddr string TCP and UDP address to listen for Graphite plaintext data. Usually :2003 must be set. Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol -graphiteListenAddr.useProxyProtocol @@ -2895,18 +2895,18 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array - Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array - Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -newrelic.maxInsertRequestSize size The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864) -opentelemetry.usePrometheusNaming - Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./README.md#sending-data-via-opentelemetry" %}} + Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see https://docs.victoriametrics.com/#sending-data-via-opentelemetry -opentsdbHTTPListenAddr string TCP address to listen for OpenTSDB HTTP put requests. Usually :4242 must be set. Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol -opentsdbHTTPListenAddr.useProxyProtocol @@ -2930,35 +2930,35 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -prevCacheRemovalPercent float Items in the previous caches are removed when the percent of requests it serves becomes lower than this value. Higher values reduce memory usage at the cost of higher CPU usage. See also -cacheExpireDuration (default 0.1) -promscrape.azureSDCheckInterval duration - Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#azure_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#azure_sd_configs for details (default 1m0s) -promscrape.cluster.memberLabel string - If non-empty, then the label with this name and the -promscrape.cluster.memberNum value is added to all the scraped metrics. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info + If non-empty, then the label with this name and the -promscrape.cluster.memberNum value is added to all the scraped metrics. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info -promscrape.cluster.memberNum string - The number of vmagent instance in the cluster of scrapers. It must be a unique value in the range 0 ... promscrape.cluster.membersCount-1 across scrapers in the cluster. Can be specified as pod name of Kubernetes StatefulSet - pod-name-Num, where Num is a numeric part of pod name. See also -promscrape.cluster.memberLabel . See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default "0") + The number of vmagent instance in the cluster of scrapers. It must be a unique value in the range 0 ... promscrape.cluster.membersCount-1 across scrapers in the cluster. Can be specified as pod name of Kubernetes StatefulSet - pod-name-Num, where Num is a numeric part of pod name. See also -promscrape.cluster.memberLabel . See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default "0") -promscrape.cluster.memberURLTemplate string - An optional template for URL to access vmagent instance with the given -promscrape.cluster.memberNum value. Every %d occurrence in the template is substituted with -promscrape.cluster.memberNum at urls to vmagent instances responsible for scraping the given target at /service-discovery page. For example -promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more details + An optional template for URL to access vmagent instance with the given -promscrape.cluster.memberNum value. Every %d occurrence in the template is substituted with -promscrape.cluster.memberNum at urls to vmagent instances responsible for scraping the given target at /service-discovery page. For example -promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more details -promscrape.cluster.membersCount int - The number of members in a cluster of scrapers. Each member must have a unique -promscrape.cluster.memberNum in the range 0 ... promscrape.cluster.membersCount-1 . Each member then scrapes roughly 1/N of all the targets. By default, cluster scraping is disabled, i.e. a single scraper scrapes all the targets. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default 1) + The number of members in a cluster of scrapers. Each member must have a unique -promscrape.cluster.memberNum in the range 0 ... promscrape.cluster.membersCount-1 . Each member then scrapes roughly 1/N of all the targets. By default, cluster scraping is disabled, i.e. a single scraper scrapes all the targets. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1) -promscrape.cluster.name string - Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info + Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info -promscrape.cluster.replicationFactor int - The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default 1) + The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1) -promscrape.config string - Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See {{% ref "./README.md#how-to-scrape-prometheus-exporters-such-as-node-exporter" %}} for details + Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details -promscrape.config.dryRun Checks -promscrape.config file for errors and unsupported fields and then exits. Returns non-zero exit code on parsing errors and emits these errors to stderr. See also -promscrape.config.strictParse command-line flag. Pass -loggerLevel=ERROR if you don't need to see info messages in the output. -promscrape.config.strictParse Whether to deny unsupported fields in -promscrape.config . Set to false in order to silently skip unsupported fields (default true) -promscrape.configCheckInterval duration - Interval for checking for changes in -promscrape.config file. By default, the checking is disabled. See how to reload -promscrape.config file at {{% ref "./vmagent.md#configuration-update" %}} + Interval for checking for changes in -promscrape.config file. By default, the checking is disabled. See how to reload -promscrape.config file at https://docs.victoriametrics.com/vmagent/#configuration-update -promscrape.consul.waitTime duration Wait time used by Consul service discovery. Default value is used if not set -promscrape.consulSDCheckInterval duration - Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#consul_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#consul_sd_configs for details (default 30s) -promscrape.consulagentSDCheckInterval duration - Interval for checking for changes in Consul Agent. This works only if consulagent_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#consulagent_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Consul Agent. This works only if consulagent_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#consulagent_sd_configs for details (default 30s) -promscrape.digitaloceanSDCheckInterval duration - Interval for checking for changes in digital ocean. This works only if digitalocean_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#digitalocean_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in digital ocean. This works only if digitalocean_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#digitalocean_sd_configs for details (default 1m0s) -promscrape.disableCompression Whether to disable sending 'Accept-Encoding: gzip' request headers to all the scrape targets. This may reduce CPU usage on scrape targets at the cost of higher network bandwidth utilization. It is possible to set 'disable_compression: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control -promscrape.disableKeepAlive @@ -2968,33 +2968,33 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -promscrape.discovery.concurrentWaitTime duration The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s) -promscrape.dnsSDCheckInterval duration - Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#dns_sd_configs" %}} for details (default 30s) + Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#dns_sd_configs for details (default 30s) -promscrape.dockerSDCheckInterval duration - Interval for checking for changes in docker. This works only if docker_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#docker_sd_configs" %}} for details (default 30s) + Interval for checking for changes in docker. This works only if docker_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#docker_sd_configs for details (default 30s) -promscrape.dockerswarmSDCheckInterval duration - Interval for checking for changes in dockerswarm. This works only if dockerswarm_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#dockerswarm_sd_configs" %}} for details (default 30s) + Interval for checking for changes in dockerswarm. This works only if dockerswarm_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs for details (default 30s) -promscrape.dropOriginalLabels Whether to drop original labels for scrape targets at /targets and /api/v1/targets pages. This may be needed for reducing memory usage when original labels for big number of scrape targets occupy big amounts of memory. Note that this reduces debuggability for improper per-target relabeling configs -promscrape.ec2SDCheckInterval duration - Interval for checking for changes in ec2. This works only if ec2_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#ec2_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in ec2. This works only if ec2_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs for details (default 1m0s) -promscrape.eurekaSDCheckInterval duration - Interval for checking for changes in eureka. This works only if eureka_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#eureka_sd_configs" %}} for details (default 30s) + Interval for checking for changes in eureka. This works only if eureka_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#eureka_sd_configs for details (default 30s) -promscrape.fileSDCheckInterval duration - Interval for checking for changes in 'file_sd_config'. See {{% ref "./sd_configs.md#file_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in 'file_sd_config'. See https://docs.victoriametrics.com/sd_configs/#file_sd_configs for details (default 1m0s) -promscrape.gceSDCheckInterval duration - Interval for checking for changes in gce. This works only if gce_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#gce_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in gce. This works only if gce_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#gce_sd_configs for details (default 1m0s) -promscrape.hetznerSDCheckInterval duration - Interval for checking for changes in Hetzner API. This works only if hetzner_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#hetzner_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in Hetzner API. This works only if hetzner_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#hetzner_sd_configs for details (default 1m0s) -promscrape.httpSDCheckInterval duration - Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#http_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#http_sd_configs for details (default 1m0s) -promscrape.kubernetes.apiServerTimeout duration How frequently to reload the full state from Kubernetes API server (default 30m0s) -promscrape.kubernetes.attachNodeMetadataAll - Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See {{% ref "./sd_configs.md#kubernetes_sd_configs" %}} + Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs -promscrape.kubernetesSDCheckInterval duration - Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#kubernetes_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs for details (default 30s) -promscrape.kumaSDCheckInterval duration - Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#kuma_sd_configs" %}} for details (default 30s) + Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size @@ -3004,28 +3004,28 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li The maximum size of scrape response in bytes to process from Prometheus targets. Bigger responses are rejected Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 16777216) -promscrape.minResponseSizeForStreamParse size - The minimum target response size for automatic switching to stream parsing mode, which can reduce memory usage. See {{% ref "./vmagent.md#stream-parsing-mode" %}} + The minimum target response size for automatic switching to stream parsing mode, which can reduce memory usage. See https://docs.victoriametrics.com/vmagent/#stream-parsing-mode Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 1000000) -promscrape.noStaleMarkers Whether to disable sending Prometheus stale markers for metrics when scrape target disappears. This option may reduce memory usage if stale markers aren't needed for your setup. This option also disables populating the scrape_series_added metric. See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series -promscrape.nomad.waitTime duration Wait time used by Nomad service discovery. Default value is used if not set -promscrape.nomadSDCheckInterval duration - Interval for checking for changes in Nomad. This works only if nomad_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#nomad_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Nomad. This works only if nomad_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#nomad_sd_configs for details (default 30s) -promscrape.openstackSDCheckInterval duration - Interval for checking for changes in openstack API server. This works only if openstack_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#openstack_sd_configs" %}} for details (default 30s) + Interval for checking for changes in openstack API server. This works only if openstack_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#openstack_sd_configs for details (default 30s) -promscrape.seriesLimitPerTarget int - Optional limit on the number of unique time series a single scrape target can expose. See {{% ref "./vmagent.md#cardinality-limiter" %}} for more info + Optional limit on the number of unique time series a single scrape target can expose. See https://docs.victoriametrics.com/vmagent/#cardinality-limiter for more info -promscrape.streamParse Whether to enable stream parsing for metrics obtained from scrape targets. This may be useful for reducing memory usage when millions of metrics are exposed per each scrape target. It is possible to set 'stream_parse: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control -promscrape.suppressDuplicateScrapeTargetErrors - Whether to suppress 'duplicate scrape target' errors; see {{% ref "./vmagent.md#troubleshooting" %}} for details + Whether to suppress 'duplicate scrape target' errors; see https://docs.victoriametrics.com/vmagent/#troubleshooting for details -promscrape.suppressScrapeErrors Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay -promscrape.suppressScrapeErrorsDelay duration The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors -promscrape.yandexcloudSDCheckInterval duration - Interval for checking for changes in Yandex Cloud API. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#yandexcloud_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Yandex Cloud API. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs for details (default 30s) -pushmetrics.disableCompression Whether to disable request body compression when pushing metrics to every -pushmetrics.url -pushmetrics.extraLabel array @@ -3039,16 +3039,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./README.md#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -relabelConfig string - Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./README.md#relabeling" %}} for details. The config is reloaded on SIGHUP signal + Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal -reloadAuthKey value Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path -retentionFilter array - Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./README.md#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionPeriod value @@ -3061,9 +3061,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.disableAutoCacheReset Whether to disable automatic response cache reset if a sample with timestamp outside -search.cacheTimestampOffset is inserted into VictoriaMetrics -search.disableCache - Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./README.md#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup + Whether to disable response caching. This may be useful when ingesting historical data. See https://docs.victoriametrics.com/#backfilling . See also -search.resetRollupResultCacheOnStartup -search.disableImplicitConversion - Whether to return an error for queries that rely on implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. See also -search.logImplicitConversion + Whether to return an error for queries that rely on implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. See also -search.logImplicitConversion -search.graphiteMaxPointsPerSeries int The maximum number of points per series Graphite render API can return (default 1000000) -search.graphiteStorageStep duration @@ -3073,7 +3073,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.latencyOffset duration The time when data points become visible in query results after the collection. It can be overridden on per-query basis via latency_offset arg. Too small value can result in incomplete last points for query results (default 30s) -search.logImplicitConversion - Whether to log queries with implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. Such conversion can be disabled using -search.disableImplicitConversion + Whether to log queries with implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. Such conversion can be disabled using -search.disableImplicitConversion -search.logQueryMemoryUsage size Log query and increment vm_memory_intensive_queries_total metric each time the query requires more memory than specified by this flag. This may help detecting and optimizing heavy queries. Query logging is disabled by default. See also -search.logSlowQueryDuration and -search.maxMemoryPerQuery Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) @@ -3088,11 +3088,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.maxFederateSeries int The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000) -search.maxGraphiteSeries int - The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./README.md#graphite-render-api-usage" %}} (default 300000) + The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000) -search.maxGraphiteTagKeys int - The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag keys returned from Graphite API, which returns tags. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxGraphiteTagValues int - The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./README.md#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag values returned from Graphite API, which returns tag values. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxLabelsAPIDuration duration The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s) -search.maxLabelsAPISeries int @@ -3156,7 +3156,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path -search.resetRollupResultCacheOnStartup - Whether to reset rollup result cache on startup. See {{% ref "./README.md#rollup-result-cache" %}}. See also -search.disableCache + Whether to reset rollup result cache on startup. See https://docs.victoriametrics.com/#rollup-result-cache . See also -search.disableCache -search.setLookbackToStep Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored -search.treatDotsAsIsInRegexps @@ -3180,52 +3180,52 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -sortLabels Whether to sort labels for incoming samples before writing them to storage. This may be needed for reducing memory usage at storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}. Enabled sorting for labels can slow down ingestion performance a bit -storage.cacheSizeIndexDBDataBlocks size - Overrides max size for indexdb/dataBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBIndexBlocks size - Overrides max size for indexdb/indexBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBTagFilters size - Overrides max size for indexdb/tagFiltersToMetricIDs cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeStorageTSID size - Overrides max size for storage/tsid cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.maxDailySeries int - The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxHourlySeries + The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxHourlySeries -storage.maxHourlySeries int - The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./README.md#cardinality-limiter" %}}. See also -storage.maxDailySeries + The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxDailySeries -storage.minFreeDiskSpaceBytes size The minimum free disk space at -storageDataPath after which the storage stops accepting new data Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 10000000) -storageDataPath string Path to storage data (default "victoria-metrics-data") -streamAggr.config string - Optional path to file with stream aggregation config. See {{% ref "./stream-aggregation.md" %}}. See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval + Optional path to file with stream aggregation config. See https://docs.victoriametrics.com/stream-aggregation/ . See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval -streamAggr.dedupInterval duration - Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and {{% ref "./stream-aggregation.md#deduplication" %}} + Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication -streamAggr.dropInput - Whether to drop all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.keepInput and {{% ref "./stream-aggregation.md" %}} + Whether to drop all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation/ -streamAggr.dropInputLabels array - An optional list of labels to drop from samples before stream de-duplication and aggregation . See {{% ref "./stream-aggregation.md#dropping-unneeded-labels" %}} + An optional list of labels to drop from samples before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -streamAggr.ignoreFirstIntervals int - Number of aggregation intervals to skip after the start. Increase this value if you observe incorrect aggregation results after restarts. It could be caused by receiving unordered delayed data from clients pushing data into the database. See {{% ref "./stream-aggregation.md#ignore-aggregation-intervals-on-start" %}} + Number of aggregation intervals to skip after the start. Increase this value if you observe incorrect aggregation results after restarts. It could be caused by receiving unordered delayed data from clients pushing data into the database. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start -streamAggr.ignoreOldSamples - Whether to ignore input samples with old timestamps outside the current aggregation interval. See {{% ref "./stream-aggregation.md#ignoring-old-samples" %}} + Whether to ignore input samples with old timestamps outside the current aggregation interval. See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples -streamAggr.keepInput - Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.dropInput and {{% ref "./stream-aggregation.md" %}} + Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation/ -tls array Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string - Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string - Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array - Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array diff --git a/docs/Release-Guide.md b/docs/Release-Guide.md index 63315d75a..71684e92d 100644 --- a/docs/Release-Guide.md +++ b/docs/Release-Guide.md @@ -50,7 +50,7 @@ Bumping the limits may significantly improve build speed. 1. Add `(available starting from v1.xx.y)` line to feature docs introduced in the upcoming release. 1. Cut new version in [CHANGELOG.md](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/CHANGELOG.md) and make it merged. See example in this [commit](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b771152039d23b5ccd637a23ea748bc44a9511a7). -1. Cherry-pick bug fixes relevant for [LTS releases](./LTS-releases.md). +1. Cherry-pick bug fixes relevant for [LTS releases](https://docs.victoriametrics.com/lts-releases/). 1. Make sure you get all changes fetched `git fetch --all`. 1. Create the following release tags: * `git tag -s v1.xx.y` in `master` branch diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md index acc38c04e..d8c45d9d2 100644 --- a/docs/Single-server-VictoriaMetrics.md +++ b/docs/Single-server-VictoriaMetrics.md @@ -24,30 +24,30 @@ aliases: VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. -See [case studies for VictoriaMetrics](./CaseStudies.md). +See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/). VictoriaMetrics is available in [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), [Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). -Documentation for the cluster version of VictoriaMetrics is available [here](./Cluster-VictoriaMetrics.md). +Documentation for the cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/cluster-victoriametrics/). -Learn more about [key concepts](./keyConcepts.md) of VictoriaMetrics and follow the -[quick start guide](./Quick-Start.md) for a better experience. +Learn more about [key concepts](https://docs.victoriametrics.com/keyconcepts/) of VictoriaMetrics and follow the +[quick start guide](https://docs.victoriametrics.com/quick-start/) for a better experience. If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://victoriametrics.slack.com/), you can join it via [Slack Inviter](https://slack.victoriametrics.com/). [Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics. -See [features available in enterprise package](./enterprise.md). +See [features available in enterprise package](https://docs.victoriametrics.com/enterprise/). Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/). -VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](./CHANGELOG.md) periodically, +VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/changelog/) periodically, and to perform [regular upgrades](#how-to-upgrade-victoriametrics). -[VictoriaMetrics enterprise](./enterprise.md) provides long-term support lines of releases (LTS releases) - -see [these docs](./LTS-releases.md). +[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) - +see [these docs](https://docs.victoriametrics.com/lts-releases/). VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details. @@ -59,16 +59,16 @@ VictoriaMetrics has the following prominent features: * It can be used as long-term storage for Prometheus. See [these docs](#prometheus-setup) for details. * It can be used as a drop-in replacement for Prometheus in Grafana, because it supports the [Prometheus querying API](#prometheus-querying-api-usage). * It can be used as a drop-in replacement for Graphite in Grafana, because it supports the [Graphite API](#graphite-api-usage). - VictoriaMetrics allows reducing infrastructure costs by more than 10x comparing to Graphite - see [this case study](./CaseStudies.md#grammarly). + VictoriaMetrics allows reducing infrastructure costs by more than 10x comparing to Graphite - see [this case study](https://docs.victoriametrics.com/casestudies/#grammarly). * It is easy to setup and operate: * VictoriaMetrics consists of a single [small executable](https://medium.com/@valyala/stripping-dependency-bloat-in-victoriametrics-docker-image-983fb5912b0d) without external dependencies. * All the configuration is done via explicit command-line flags with reasonable defaults. * All the data is stored in a single directory specified by the `-storageDataPath` command-line flag. * Easy and fast backups from [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) - can be done with [vmbackup](./vmbackup.md) / [vmrestore](./vmrestore.md) tools. + can be done with [vmbackup](https://docs.victoriametrics.com/vmbackup/) / [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883) for more details. -* It implements a PromQL-like query language - [MetricsQL](./MetricsQL.md), which provides improved functionality on top of PromQL. +* It implements a PromQL-like query language - [MetricsQL](https://docs.victoriametrics.com/metricsql/), which provides improved functionality on top of PromQL. * It provides a global query view. Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics. Later this data may be queried via a single query. * It provides high performance and good vertical and horizontal scalability for both [data ingestion](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b) @@ -76,8 +76,8 @@ VictoriaMetrics has the following prominent features: It [outperforms InfluxDB and TimescaleDB by up to 20x](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae). * It [uses 10x less RAM than InfluxDB](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and [up to 7x less RAM than Prometheus, Thanos or Cortex](https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f) - when dealing with millions of unique time series (aka [high cardinality](./FAQ.md#what-is-high-cardinality)). -* It is optimized for time series with [high churn rate](./FAQ.md#what-is-high-churn-rate). + when dealing with millions of unique time series (aka [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality)). +* It is optimized for time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). * It provides high data compression: up to 70x more data points may be stored into limited storage compared with TimescaleDB according to [these benchmarks](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4), and up to 7x less storage space is required compared to Prometheus, Thanos or Cortex. @@ -105,34 +105,34 @@ VictoriaMetrics has the following prominent features: * [DataDog agent or DogStatsD](#how-to-send-data-from-datadog-agent). * [NewRelic infrastructure agent](#how-to-send-data-from-newrelic-agent). * [OpenTelemetry metrics format](#sending-data-via-opentelemetry). -* It supports powerful [stream aggregation](./stream-aggregation.md), which can be used as a [statsd](https://github.com/statsd/statsd) alternative. +* It supports powerful [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/), which can be used as a [statsd](https://github.com/statsd/statsd) alternative. * It supports metrics [relabeling](#relabeling). -* It can deal with [high cardinality issues](./FAQ.md#what-is-high-cardinality) and - [high churn rate](./FAQ.md#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter). +* It can deal with [high cardinality issues](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) and + [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issues via [series limiter](#cardinality-limiter). * It ideally works with big amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data - and various [Enterprise workloads](./enterprise.md). + and various [Enterprise workloads](https://docs.victoriametrics.com/enterprise/). * It has an open source [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster). * It can store data on [NFS-based storages](https://en.wikipedia.org/wiki/Network_File_System) such as [Amazon EFS](https://aws.amazon.com/efs/) and [Google Filestore](https://cloud.google.com/filestore). -See [case studies for VictoriaMetrics](./CaseStudies.md) and [various Articles about VictoriaMetrics](./Articles.md). +See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/casestudies/) and [various Articles about VictoriaMetrics](https://docs.victoriametrics.com/articles/). ## Components -VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](./README.md): +VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](https://docs.victoriametrics.com/): -- [vmagent](./vmagent.md) - lightweight agent for receiving metrics via [pull-based](./vmagent.md#how-to-collect-metrics-in-prometheus-format) - and [push-based](./vmagent.md#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible +- [vmagent](https://docs.victoriametrics.com/vmagent/) - lightweight agent for receiving metrics via [pull-based](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format) + and [push-based](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible remote storage systems such as VictoriaMetrics. -- [vmalert](./vmalert.md) - a service for processing Prometheus-compatible alerting and recording rules. -- [vmalert-tool](./vmalert-tool.md) - a tool for validating alerting and recording rules. -- [vmauth](./vmauth.md) - authorization proxy and load balancer optimized for VictoriaMetrics products. -- [vmgateway](./vmgateway.md) - authorization proxy with per-[tenant](./Cluster-VictoriaMetrics.md#multitenancy) rate limiting capabilities. -- [vmctl](./vmctl.md) - a tool for migrating and copying data between different storage systems for metrics. -- [vmbackup](./vmbackup.md), [vmrestore](./vmrestore.md) and [vmbackupmanager](./vmbackupmanager.md) - +- [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules. +- [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) - a tool for validating alerting and recording rules. +- [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products. +- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - authorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting capabilities. +- [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics. +- [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) - tools for creating backups and restoring from backups for VictoriaMetrics data. -- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md). -- [VictoriaLogs](./VictoriaLogs/README.md) - user-friendly cost-efficient database for logs. +- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). +- [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) - user-friendly cost-efficient database for logs. ## Operation @@ -140,7 +140,7 @@ VictoriaMetrics ecosystem contains the following components additionally to [sin To quickly try VictoriaMetrics, just download the [VictoriaMetrics executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) or [Docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and start it with the desired command-line flags. -See also [QuickStart guide](./Quick-Start.md) for additional information. +See also [QuickStart guide](https://docs.victoriametrics.com/quick-start/) for additional information. VictoriaMetrics can also be installed via these installation methods: @@ -160,9 +160,9 @@ The following command-line flags are used the most: Other flags have good enough default values, so set them only if you really need to. Pass `-help` to see [all the available flags with description and default values](#list-of-command-line-flags). The following docs may be useful during initial VictoriaMetrics setup: -* [How to set up scraping of Prometheus-compatible targets](./#how-to-scrape-prometheus-exporters-such-as-node-exporter) +* [How to set up scraping of Prometheus-compatible targets](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) * [How to ingest data to VictoriaMetrics](#how-to-import-time-series-data) -* [How to set up Prometheus to write data to VictoriaMetrics](./#prometheus-setup) +* [How to set up Prometheus to write data to VictoriaMetrics](https://docs.victoriametrics.com/#prometheus-setup) * [How to query VictoriaMetrics via Grafana](#grafana-setup) * [How to query VictoriaMetrics via Graphite API](#graphite-api-usage) * [How to handle alerts](#alerting) @@ -302,8 +302,8 @@ Read more about tuning remote write for Prometheus [here](https://prometheus.io/ It is recommended upgrading Prometheus to [v2.12.0](https://github.com/prometheus/prometheus/releases/latest) or newer, since previous versions may have issues with `remote_write`. -Take a look also at [vmagent](./vmagent.md) -and [vmalert](./vmalert.md), +Take a look also at [vmagent](https://docs.victoriametrics.com/vmagent/) +and [vmalert](https://docs.victoriametrics.com/vmalert/), which can be used as faster and less resource-hungry alternative to Prometheus. ## Grafana setup @@ -324,10 +324,10 @@ In the "Type and version" section it is recommended to set the type to "Promethe This allows Grafana to use a more efficient API to get label values. Then build graphs and dashboards for the created datasource using [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) -or [MetricsQL](./MetricsQL.md). +or [MetricsQL](https://docs.victoriametrics.com/metricsql/). Alternatively, use VictoriaMetrics [datasource plugin](https://github.com/VictoriaMetrics/victoriametrics-datasource) with support of extra features. -See more in [description](./victoriametrics-datasource.md). +See more in [description](https://docs.victoriametrics.com/victoriametrics-datasource/). Creating a datasource may require [specific permissions](https://grafana.com/docs/grafana/latest/administration/data-source-management/). If you don't see an option to create a data source - try contacting system administrator. @@ -336,7 +336,7 @@ Grafana playground is available for viewing at our [sandbox](https://play-grafan ## How to upgrade VictoriaMetrics -VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking [the CHANGELOG page](./CHANGELOG.md) and performing regular upgrades. +VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking [the CHANGELOG page](https://docs.victoriametrics.com/changelog/) and performing regular upgrades. It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) say otherwise. @@ -350,12 +350,12 @@ The following steps must be performed during the upgrade / downgrade procedure: * Wait until the process stops. This can take a few seconds. * Start the upgraded VictoriaMetrics. -Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](./vmagent.md). +Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent/). ## vmui VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at `http://victoriametrics:8428/vmui` -(or at `http://:8481/select//vmui/` in [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md)). +(or at `http://:8481/select//vmui/` in [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/)). The UI allows exploring query results via graphs and tables. It also provides the following features: - Explore: @@ -369,10 +369,10 @@ The UI allows exploring query results via graphs and tables. It also provides th - [WITH expressions playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/expand-with-exprs) - test how WITH expressions work; - [Metric relabel debugger](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/relabeling) - playground for [relabeling](#relabeling) configs. -VMUI provides auto-completion for [MetricsQL](./MetricsQL.md) functions, metric names, label names and label values. The auto-completion can be enabled +VMUI provides auto-completion for [MetricsQL](https://docs.victoriametrics.com/metricsql/) functions, metric names, label names and label values. The auto-completion can be enabled by checking the `Autocomplete` toggle. When the auto-completion is disabled, it can still be triggered for the current cursor position by pressing `ctrl+space`. -VMUI automatically switches from graph view to heatmap view when the query returns [histogram](./keyConcepts.md#histogram) buckets +VMUI automatically switches from graph view to heatmap view when the query returns [histogram](https://docs.victoriametrics.com/keyconcepts/#histogram) buckets (both [Prometheus histograms](https://prometheus.io/docs/concepts/metric_types/#histogram) and [VictoriaMetrics histograms](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) are supported). Try, for example, [this query](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.expr=sum%28rate%28vm_promscrape_scrape_duration_seconds_bucket%29%29+by+%28vmrange%29&g0.range_input=24h&g0.end_input=2023-04-10T17%3A46%3A12&g0.relative_time=last_24_hours&g0.step_input=31m). @@ -387,8 +387,8 @@ Query history can be navigated by holding `Ctrl` (or `Cmd` on MacOS) and pressin Multi-line queries can be entered by pressing `Shift-Enter` in query input field. -When querying the [backfilled data](./#backfilling) -or during [query troubleshooting](./Troubleshooting.md#unexpected-query-results), +When querying the [backfilled data](https://docs.victoriametrics.com/#backfilling) +or during [query troubleshooting](https://docs.victoriametrics.com/troubleshooting/#unexpected-query-results), it may be useful disabling response cache by clicking `Disable cache` checkbox. VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. @@ -401,7 +401,7 @@ Graphs for a particular query can be temporarily hidden by clicking the `eye` ic When the `eye` icon is clicked while holding the `ctrl` key, then query results for the rest of queries become hidden except of the current query results. -VMUI allows sharing query and [trace](./#query-tracing) results by clicking on +VMUI allows sharing query and [trace](https://docs.victoriametrics.com/#query-tracing) results by clicking on `Export query` button in top right corner of the graph area. The query and trace will be exported as a file that later can be loaded in VMUI via `Query Analyzer` tool. @@ -422,7 +422,7 @@ This information is obtained from the `/api/v1/status/top_queries` HTTP endpoint [VMUI](#vmui) provides `active queries` tab, which shows currently execute queries. It provides the following information per each query: -- The query itself, together with the time range and step args passed to [/api/v1/query_range](./keyConcepts.md#range-query). +- The query itself, together with the time range and step args passed to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). - The duration of the query execution. - The client address, who initiated the query execution. @@ -449,7 +449,7 @@ VictoriaMetrics provides an ability to explore time series cardinality at `Explo - To identify values with the highest number of series for the selected label (aka `focusLabel`). - To identify label=name pairs with the highest number of series. - To identify labels with the highest number of unique values. - Note that [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) + Note that [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) may show lower than expected number of unique label values for labels with small number of unique values. This is because of [implementation limits](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5a6e617b5e41c9170e7c562aecd15ee0c901d489/app/vmselect/netstorage/netstorage.go#L1039-L1045). @@ -464,10 +464,10 @@ See the example of using the cardinality explorer [here](https://victoriametrics ## Cardinality explorer statistic inaccuracy -In [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) each vmstorage tracks the stored time series individually. +In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) each vmstorage tracks the stored time series individually. vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats. This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes -due to [replication](#replication) or [rerouting](./Cluster-VictoriaMetrics.md?highlight=re-routes#cluster-availability). +due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=re-routes#cluster-availability). ## How to apply new config to VictoriaMetrics @@ -477,26 +477,26 @@ VictoriaMetrics is configured via command-line flags, so it must be restarted wh * Wait until the process stops. This can take a few seconds. * Start VictoriaMetrics with the new command-line flags. -Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](./vmagent.md). +Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details. The same applies also to [vmagent](https://docs.victoriametrics.com/vmagent/). ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. -If the provided configuration file contains [unsupported options](./vmagent.md#unsupported-prometheus-config-sections), +If the provided configuration file contains [unsupported options](https://docs.victoriametrics.com/vmagent/#unsupported-prometheus-config-sections), then either delete them from the file or just pass `-promscrape.config.strictParse=false` command-line flag to VictoriaMetrics, so it will ignore unsupported options. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. See also: -- [scrape config examples](./scrape_config_examples.md) -- [the list of supported service discovery types for Prometheus scrape targets](./sd_configs.md) +- [scrape config examples](https://docs.victoriametrics.com/scrape_config_examples/) +- [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs/) VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format). -See also [vmagent](./vmagent.md), which can be used as drop-in replacement for Prometheus. +See also [vmagent](https://docs.victoriametrics.com/vmagent/), which can be used as drop-in replacement for Prometheus. ## How to send data from DataDog agent @@ -519,7 +519,7 @@ DD_DD_URL=http://victoriametrics:8428/datadog ``` -_Choose correct URL for VictoriaMetrics [here](./url-examples.md#datadog)._ +_Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples/#datadog)._ To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33) add the following line: @@ -528,8 +528,8 @@ add the following line: dd_url: http://victoriametrics:8428/datadog ``` -[vmagent](./vmagent.md) also can accept DataDog metrics format. Depending on where vmagent will forward data, -pick [single-node or cluster URL](./url-examples.md#datadog) formats. +[vmagent](https://docs.victoriametrics.com/vmagent/) also can accept DataDog metrics format. Depending on where vmagent will forward data, +pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples/#datadog) formats. ### Sending metrics to DataDog and VictoriaMetrics @@ -544,7 +544,7 @@ Run DataDog using the following ENV variable with VictoriaMetrics as additional DD_ADDITIONAL_ENDPOINTS='{\"http://victoriametrics:8428/datadog\": [\"apikey\"]}' ``` -_Choose correct URL for VictoriaMetrics [here](./url-examples.md#datadog)._ +_Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/url-examples/#datadog)._ To configure DataDog Dual Shipping via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files) @@ -572,9 +572,9 @@ provider: ### Send via cURL -See how to send data to VictoriaMetrics via DataDog "submit metrics" API [here](./url-examples.md#datadogapiv2series). +See how to send data to VictoriaMetrics via DataDog "submit metrics" API [here](https://docs.victoriametrics.com/url-examples/#datadogapiv2series). -The imported data can be read via [export API](./url-examples.md#apiv1export). +The imported data can be read via [export API](https://docs.victoriametrics.com/url-examples/#apiv1export). ### Additional details @@ -588,12 +588,12 @@ For example, `/datadog/api/v2/series?extra_label=foo=bar` would add `{foo="bar"} DataDog agent sends the [configured tags](https://docs.datadoghq.com/getting_started/tagging/) to undocumented endpoint - `/datadog/intake`. This endpoint isn't supported by VictoriaMetrics yet. This prevents from adding the configured tags to DataDog agent data sent into VictoriaMetrics. -The workaround is to run a sidecar [vmagent](./vmagent.md) alongside every DataDog agent, +The workaround is to run a sidecar [vmagent](https://docs.victoriametrics.com/vmagent/) alongside every DataDog agent, which must run with `DD_DD_URL=http://localhost:8429/datadog` environment variable. The sidecar `vmagent` must be configured with the needed tags via `-remoteWrite.label` command-line flag and must forward incoming data with the added tags to a centralized VictoriaMetrics specified via `-remoteWrite.url` command-line flag. -See [these docs](./vmagent.md#adding-labels-to-metrics) for details on how to add labels to metrics at `vmagent`. +See [these docs](https://docs.victoriametrics.com/vmagent/#adding-labels-to-metrics) for details on how to add labels to metrics at `vmagent`. ## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) @@ -611,9 +611,9 @@ and stream plain InfluxDB line protocol data to the configured TCP and/or UDP ad VictoriaMetrics performs the following transformations to the ingested InfluxDB data: * [db query arg](https://docs.influxdata.com/influxdb/v1.7/tools/api/#write-http-endpoint) is mapped into `db` - [label](./keyConcepts.md#labels) value unless `db` tag exists in the InfluxDB line. + [label](https://docs.victoriametrics.com/keyconcepts/#labels) value unless `db` tag exists in the InfluxDB line. The `db` label name can be overridden via `-influxDBLabel` command-line flag. If more strict data isolation is required, - read more about multi-tenancy [here](./keyConcepts.md#multi-tenancy). + read more about multi-tenancy [here](https://docs.victoriametrics.com/keyconcepts/#multi-tenancy). * Field names are mapped to time series names prefixed with `{measurement}{separator}` value, where `{separator}` equals to `_` by default. It can be changed with `-influxMeasurementFieldSeparator` command-line flag. See also `-influxSkipSingleField` command-line flag. If `{measurement}` is empty or if `-influxSkipMeasurement` command-line flag is set, then time series names correspond to field names. * Field values are mapped to time series values. * Tags are mapped to Prometheus labels as-is. @@ -729,7 +729,7 @@ The `/api/v1/export` endpoint should return the following response: {"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]} ``` -[Graphite relabeling](./vmagent.md#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](./MetricsQL.md). +[Graphite relabeling](https://docs.victoriametrics.com/vmagent/#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](https://docs.victoriametrics.com/metricsql/). ## Querying Graphite data @@ -741,7 +741,7 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t ## Selecting Graphite metrics -VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series with Graphite-compatible filters in [MetricsQL](./MetricsQL.md). For example, `{__graphite__="foo.*.bar"}` is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. See [docs for Graphite paths and wildcards](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). VictoriaMetrics also supports [label_graphite_group](./MetricsQL.md#label_graphite_group) function for extracting the given groups from Graphite metric name. +VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series with Graphite-compatible filters in [MetricsQL](https://docs.victoriametrics.com/metricsql/). For example, `{__graphite__="foo.*.bar"}` is equivalent to `{__name__=~"foo[.][^.]*[.]bar"}`, but it works faster and it is easier to use when migrating from Graphite to VictoriaMetrics. See [docs for Graphite paths and wildcards](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). VictoriaMetrics also supports [label_graphite_group](https://docs.victoriametrics.com/metricsql/#label_graphite_group) function for extracting the given groups from Graphite metric name. The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `(value1|...|valueN)`. They are transparently converted to `{value1,...,valueN}` syntax [used in Graphite](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). This allows using [multi-value template variables in Grafana](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, Grafana expands `{__graphite__=~"foo.($bar).baz"}` into `{__graphite__=~"foo.(x|y).baz"}` if `$bar` template variable contains `x` and `y` values. In this case the query is automatically converted into `{__graphite__=~"foo.{x,y}.baz"}` before execution. @@ -837,7 +837,7 @@ For example, `/api/put?extra_label=foo=bar` would add `{foo="bar"}` label to all VictoriaMetrics accepts data from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent) at `/newrelic/infra/v2/metrics/events/bulk` HTTP path. VictoriaMetrics receives [Events](https://docs.newrelic.com/docs/infrastructure/manage-your-data/data-instrumentation/default-infrastructure-monitoring-data/#infrastructure-events) -from NewRelic agent at the given path, transforms them to [raw samples](./keyConcepts.md#raw-samples) +from NewRelic agent at the given path, transforms them to [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) according to [these docs](#newrelic-agent-data-mapping) before storing the raw samples to the database. You need passing `COLLECTOR_URL` and `NRIA_LICENSE_KEY` environment variables to NewRelic infrastructure agent in order to send the collected metrics to VictoriaMetrics. @@ -852,11 +852,11 @@ COLLECTOR_URL="http://localhost:8428/newrelic" NRIA_LICENSE_KEY="NEWRELIC_LICENS ### NewRelic agent data mapping VictoriaMetrics maps [NewRelic Events](https://docs.newrelic.com/docs/infrastructure/manage-your-data/data-instrumentation/default-infrastructure-monitoring-data/#infrastructure-events) -to [raw samples](./keyConcepts.md#raw-samples) in the following way: +to [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in the following way: 1. Every numeric field is converted into a raw sample with the corresponding name. -1. The `eventType` and all the other fields with `string` value type are attached to every raw sample as [metric labels](./keyConcepts.md#labels). -1. The `timestamp` field is used as timestamp for the ingested [raw sample](./keyConcepts.md#raw-samples). +1. The `eventType` and all the other fields with `string` value type are attached to every raw sample as [metric labels](https://docs.victoriametrics.com/keyconcepts/#labels). +1. The `timestamp` field is used as timestamp for the ingested [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples). The `timestamp` field may be specified either in seconds or in milliseconds since the [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time). If the `timestamp` field is missing, then the raw sample is stored with the current timestamp. @@ -909,11 +909,11 @@ curl http://localhost:8428/api/v1/export -d 'match={eventType="SystemSample"}' VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/): -* [/api/v1/query](./keyConcepts.md#instant-query) -* [/api/v1/query_range](./keyConcepts.md#range-query) -* [/api/v1/series](./url-examples.md#apiv1series) -* [/api/v1/labels](./url-examples.md#apiv1labels) -* [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) +* [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) +* [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) +* [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) +* [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) +* [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) * [/api/v1/status/tsdb](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats). See [these docs](#tsdb-stats) for details. * [/api/v1/targets](https://prometheus.io/docs/prometheus/latest/querying/api/#targets) - see [these docs](#how-to-scrape-prometheus-exporters-such-as-node-exporter) for more details. * [/federate](https://prometheus.io/docs/prometheus/latest/federation/) - see [these docs](#federation) for more details. @@ -928,38 +928,38 @@ for enforcing additional label filters for queries. For example, `/api/v1/query_ would automatically add `{user_id="123",group_id="456"}` label filters to the given ``. This functionality can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query args are automatically set by auth proxy sitting in front of VictoriaMetrics. -See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. VictoriaMetrics accepts optional `extra_filters[]=series_selector` query arg, which can be used for enforcing arbitrary label filters for queries. For example, `/api/v1/query_range?extra_filters[]={env=~"prod|staging",user="xyz"}&query=` would automatically add `{env=~"prod|staging",user="xyz"}` label filters to the given ``. This functionality can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_filters[]` query args are automatically set by auth proxy sitting in front of VictoriaMetrics. -See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. VictoriaMetrics accepts multiple formats for `time`, `start` and `end` query args - see [these docs](#timestamp-formats). -VictoriaMetrics accepts `round_digits` query arg for [/api/v1/query](./keyConcepts.md#instant-query) -and [/api/v1/query_range](./keyConcepts.md#range-query) handlers. It can be used for rounding response values +VictoriaMetrics accepts `round_digits` query arg for [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) +and [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) handlers. It can be used for rounding response values to the given number of digits after the decimal point. For example, `/api/v1/query?query=avg_over_time(temperature[1h])&round_digits=2` would round response values to up to two digits after the decimal point. -VictoriaMetrics accepts `limit` query arg for [/api/v1/labels](./url-examples.md#apiv1labels) -and [`/api/v1/label//values`](./url-examples.md#apiv1labelvalues) handlers for limiting the number of returned entries. +VictoriaMetrics accepts `limit` query arg for [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) +and [`/api/v1/label//values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) handlers for limiting the number of returned entries. For example, the query to `/api/v1/labels?limit=5` returns a sample of up to 5 unique labels, while ignoring the rest of labels. If the provided `limit` value exceeds the corresponding `-search.maxTagKeys` / `-search.maxTagValues` command-line flag values, then limits specified in the command-line flags are used. By default, VictoriaMetrics returns time series for the last day starting at 00:00 UTC -from [/api/v1/series](./url-examples.md#apiv1series), -[/api/v1/labels](./url-examples.md#apiv1labels) and -[`/api/v1/label//values`](./url-examples.md#apiv1labelvalues), +from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series), +[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and +[`/api/v1/label//values`](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues), while the Prometheus API defaults to all time. Explicitly set `start` and `end` to select the desired time range. VictoriaMetrics rounds the specified `start..end` time range to day granularity because of performance optimization concerns. If you need the exact set of label names and label values on the given time range, then send queries -to [/api/v1/query](./keyConcepts.md#instant-query) or to [/api/v1/query_range](./keyConcepts.md#range-query). +to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) or to [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). -VictoriaMetrics accepts `limit` query arg at [/api/v1/series](./url-examples.md#apiv1series) +VictoriaMetrics accepts `limit` query arg at [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) for limiting the number of returned entries. For example, the query to `/api/v1/series?limit=5` returns a sample of up to 5 series, while ignoring the rest of series. If the provided `limit` value exceeds the corresponding `-search.maxSeries` command-line flag values, then limits specified in the command-line flags are used. @@ -984,8 +984,8 @@ Additionally, VictoriaMetrics provides the following handlers: ### Timestamp formats VictoriaMetrics accepts the following formats for `time`, `start` and `end` query args -in [query APIs](./#prometheus-querying-api-usage) and -in [export APIs](./#how-to-export-time-series). +in [query APIs](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and +in [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). - Unix timestamps in seconds with optional milliseconds after the point. For example, `1562529662.678`. - Unix timestamps in milliseconds. For example, `1562529662678`. @@ -1007,11 +1007,11 @@ VictoriaMetrics supports the following Graphite querying APIs, which are needed All the Graphite handlers can be pre-pended with `/graphite` prefix. For example, both `/graphite/metrics/find` and `/metrics/find` should work. -VictoriaMetrics accepts optional query args: `extra_label==` and `extra_filters[]=series_selector` query args for all the Graphite APIs. These args can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. See [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md) as examples of such proxies. +VictoriaMetrics accepts optional query args: `extra_label==` and `extra_filters[]=series_selector` query args for all the Graphite APIs. These args can be used for limiting the scope of time series visible to the given tenant. It is expected that the `extra_label` query arg is automatically set by auth proxy sitting in front of VictoriaMetrics. See [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/) as examples of such proxies. [Contact us](mailto:sales@victoriametrics.com) if you need assistance with such a proxy. -VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series with Graphite-compatible filters in [MetricsQL](./MetricsQL.md). See [these docs](#selecting-graphite-metrics). +VictoriaMetrics supports `__graphite__` pseudo-label for filtering time series with Graphite-compatible filters in [MetricsQL](https://docs.victoriametrics.com/metricsql/). See [these docs](#selecting-graphite-metrics). ### Graphite Render API usage @@ -1119,7 +1119,7 @@ Note that `production` builds are not supported via Podman because Podman does n ## Start with docker-compose [Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml) -helps to spin up VictoriaMetrics, [vmagent](./vmagent.md) and Grafana with one command. +helps to spin up VictoriaMetrics, [vmagent](https://docs.victoriametrics.com/vmagent/) and Grafana with one command. More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker). ## Setting up service @@ -1139,7 +1139,7 @@ The page returns the following JSON response on successful creation of snapshot: Snapshots are created under `<-storageDataPath>/snapshots` directory, where `<-storageDataPath>` is the corresponding command-line flag value. Snapshots can be archived to backup storage at any time -with [vmbackup](./vmbackup.md). +with [vmbackup](https://docs.victoriametrics.com/vmbackup/). Snapshots consist of a mix of hard-links and soft-links to various files and directories inside `-storageDataPath`. See [this article](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282) @@ -1148,7 +1148,7 @@ for more details. This adds some restrictions on what can be done with the conte - Do not delete subdirectories inside `<-storageDataPath>/snapshots` with `rm` or similar commands, since this will leave some snapshot data undeleted. Prefer using the `/snapshot/delete` API for deleting snapshot. See below for more details about this API. - Do not copy subdirectories inside `<-storageDataPath>/snapshot` with `cp`, `rsync` or similar commands, since there are high chances - that these commands won't copy some data stored in the snapshot. Prefer using [vmbackup](./vmbackup.md) for making copies of snapshot data. + that these commands won't copy some data stored in the snapshot. Prefer using [vmbackup](https://docs.victoriametrics.com/vmbackup/) for making copies of snapshot data. See also [snapshot troubleshooting](#snapshot-troubleshooting). @@ -1162,7 +1162,7 @@ Navigate to `http://:8428/snapshot/delete_all` in order to ### How to restore from a snapshot 1. Stop VictoriaMetrics with `kill -INT`. -1. Restore snapshot contents from backup with [vmrestore](./vmrestore.md) +1. Restore snapshot contents from backup with [vmrestore](https://docs.victoriametrics.com/vmrestore/) to the directory pointed by `-storageDataPath`. 1. Start VictoriaMetrics. @@ -1540,7 +1540,7 @@ VictoriaMetrics supports data ingestion via [OpenTelemetry protocol for metrics] VictoriaMetrics expects `protobuf`-encoded requests at `/opentelemetry/v1/metrics`. Set HTTP request header `Content-Encoding: gzip` when sending gzip-compressed data to `/opentelemetry/v1/metrics`. -VictoriaMetrics stores the ingested OpenTelemetry [raw samples](./keyConcepts.md#raw-samples) as is without any transformations. +VictoriaMetrics stores the ingested OpenTelemetry [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) as is without any transformations. Pass `-opentelemetry.usePrometheusNaming` command-line flag to VictoriaMetrics for automatic conversion of metric names and labels into Prometheus-compatible format. Using the following exporter configuration in the opentelemetry collector will allow you to send metrics into VictoriaMetrics: @@ -1562,7 +1562,7 @@ service: receivers: - otlp ``` -See [How to use OpenTelemetry metrics with VictoriaMetrics](./guides/getting-started-with-opentelemetry.md). +See [How to use OpenTelemetry metrics with VictoriaMetrics](https://docs.victoriametrics.com/guides/getting-started-with-opentelemetry/). ## JSON line format @@ -1603,8 +1603,8 @@ Too long JSON lines may increase RAM usage at VictoriaMetrics side. [/api/v1/export](#how-to-export-data-in-json-line-format) handler accepts `max_rows_per_line` query arg, which allows limiting the number of samples per each exported line. -It is OK to split [raw samples](./keyConcepts.md#raw-samples) -for the same [time series](./keyConcepts.md#time-series) across multiple lines. +It is OK to split [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +for the same [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) across multiple lines. The number of lines in the request to [/api/v1/import](#how-to-import-data-in-json-line-format) can be arbitrary - they are imported in streaming manner. @@ -1616,7 +1616,7 @@ The `-relabelConfig` also can point to http or https url. For example, `-relabel The following docs can be useful in understanding the relabeling: -* [Cookbook for common relabeling tasks](./relabeling.md). +* [Cookbook for common relabeling tasks](https://docs.victoriametrics.com/relabeling/). * [Relabeling tips and tricks](https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2). The `-relabelConfig` files can contain special placeholders in the form `%{ENV_VAR}`, which are replaced by the corresponding environment variable values. @@ -1635,11 +1635,11 @@ Example contents for `-relabelConfig` file: ``` VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling. -See [these docs](./vmagent.md#relabeling) for more details. +See [these docs](https://docs.victoriametrics.com/vmagent/#relabeling) for more details. The relabeling can be debugged at `http://victoriametrics:8428/metric-relabel-debug` page or at our [public playground](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/relabeling). -See [these docs](./vmagent.md#relabel-debug) for more details. +See [these docs](https://docs.victoriametrics.com/vmagent/#relabel-debug) for more details. ## Federation @@ -1661,9 +1661,9 @@ with scrape intervals exceeding `5m`. ## Capacity planning -VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](./CaseStudies.md). +VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according to [our case studies](https://docs.victoriametrics.com/casestudies/). -VictoriaMetrics capacity scales linearly with the available resources. The needed amounts of CPU and RAM highly depends on the workload - the number of [active time series](./FAQ.md#what-is-an-active-time-series), series [churn rate](./FAQ.md#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics for your production workload and iteratively scaling CPU and RAM resources until it becomes stable according to [troubleshooting docs](#troubleshooting). A single-node VictoriaMetrics works perfectly with the following production workload according to [our case studies](./CaseStudies.md): +VictoriaMetrics capacity scales linearly with the available resources. The needed amounts of CPU and RAM highly depends on the workload - the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series), series [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), query types, query qps, etc. It is recommended setting up a test VictoriaMetrics for your production workload and iteratively scaling CPU and RAM resources until it becomes stable according to [troubleshooting docs](#troubleshooting). A single-node VictoriaMetrics works perfectly with the following production workload according to [our case studies](https://docs.victoriametrics.com/casestudies/): * Ingestion rate: 1.5+ million samples per second * Active time series: 50+ million @@ -1703,57 +1703,57 @@ By default, VictoriaMetrics is tuned for an optimal resource usage under typical of additional memory. So it is better to limit the number of concurrent queries, while pausing additional incoming queries if the concurrency limit is reached. VictoriaMetrics provides `-search.maxQueueDuration` command-line flag for limiting the max wait time for paused queries. See also `-search.maxMemoryPerQuery` command-line flag. - `-search.maxQueueDuration` limits the maximum duration queries may wait for execution when `-search.maxConcurrentRequests` concurrent queries are executed. -- `-search.ignoreExtraFiltersAtLabelsAPI` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](./#prometheus-querying-api-enhancements) - query args at [/api/v1/labels](./url-examples.md#apiv1labels) and - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). +- `-search.ignoreExtraFiltersAtLabelsAPI` enables ignoring of `match[]`, [`extra_filters[]` and `extra_label`](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) + query args at [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels) and + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This may be useful for reducing the load on VictoriaMetrics if the provided extra filters match too many time series. The downside is that the endpoints can return labels and series, which do not match the provided extra filters. - `-search.maxSamplesPerSeries` limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processes raw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memory - and then applies the given [rollup function](./MetricsQL.md#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag + and then applies the given [rollup function](https://docs.victoriametrics.com/metricsql/#rollup-functions). The `-search.maxSamplesPerSeries` command-line flag allows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series. - `-search.maxSamplesPerQuery` limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries. -- `-search.maxResponseSeries` limits the number of time series a single query can return from [`/api/v1/query`](./keyConcepts.md#instant-query) - and [`/api/v1/query_range`](./keyConcepts.md#range-query). +- `-search.maxResponseSeries` limits the number of time series a single query can return from [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) + and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsPerTimeseries` limits the number of calculated points, which can be returned per each matching time series - from [range query](./keyConcepts.md#range-query). + from [range query](https://docs.victoriametrics.com/keyconcepts/#range-query). - `-search.maxPointsSubqueryPerTimeseries` limits the number of calculated points, which can be generated per each matching time series - during [subquery](./MetricsQL.md#subqueries) evaluation. -- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](./MetricsQL.md#aggregate-functions) + during [subquery](https://docs.victoriametrics.com/metricsql/#subqueries) evaluation. +- `-search.maxSeriesPerAggrFunc` limits the number of time series, which can be generated by [MetricsQL aggregate functions](https://docs.victoriametrics.com/metricsql/#aggregate-functions) in a single query. -- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](./url-examples.md#apiv1series). +- `-search.maxSeries` limits the number of time series, which may be returned from [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amounts - of CPU time and memory when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxSeries` to quite low value in order limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](./url-examples.md#apiv1labels). +- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues). +- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.maxLabelsAPISeries`. -- `-search.maxLabelsAPISeries` limits the number of time series, which can be scanned when performing [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series) requests. +- `-search.maxLabelsAPISeries` limits the number of time series, which can be scanned when performing [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series) requests. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPISeries` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPIDuration` and `-search.ignoreExtraFiltersAtLabelsAPI`. -- `-search.maxLabelsAPIDuration` limits the duration for requests to [/api/v1/labels](./url-examples.md#apiv1labels), - [/api/v1/label/.../values](./url-examples.md#apiv1labelvalues) - or [/api/v1/series](./url-examples.md#apiv1series). +- `-search.maxLabelsAPIDuration` limits the duration for requests to [/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels), + [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues) + or [/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series). The limit can be altered for each query by passing `timeout` GET parameter, but can't exceed the limit specified via cmd-line flag. These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memory - when the database contains big number of unique time series because of [high churn rate](./FAQ.md#what-is-high-churn-rate). + when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxLabelsAPIDuration` to quite low value in order to limit CPU and memory usage. See also `-search.maxLabelsAPISeries` and `-search.ignoreExtraFiltersAtLabelsAPI`. - `-search.maxTagValueSuffixesPerSearch` limits the number of entries, which may be returned from `/metrics/find` endpoint. See [Graphite Metrics API usage docs](#graphite-metrics-api-usage). -See also [resource usage limits at VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#resource-usage-limits), +See also [resource usage limits at VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits), [cardinality limiter](#cardinality-limiter) and [capacity planning docs](#capacity-planning). @@ -1762,17 +1762,17 @@ See also [resource usage limits at VictoriaMetrics cluster](./Cluster-VictoriaMe The general approach for achieving high availability is the following: - To run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones); -- To store the collected data simultaneously into these instances via [vmagent](./vmagent.md) or Prometheus. +- To store the collected data simultaneously into these instances via [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus. - To query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable. - This can be done via [vmauth](./vmauth.md) according to [these docs](./vmauth.md#high-availability). + This can be done via [vmauth](https://docs.victoriametrics.com/vmauth/) according to [these docs](https://docs.victoriametrics.com/vmauth/#high-availability). Such a setup guarantees that the collected data isn't lost when one of VictoriaMetrics instance becomes unavailable. The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying. -Both [vmagent](./vmagent.md) and Prometheus buffer the collected data locally if they cannot send it +Both [vmagent](https://docs.victoriametrics.com/vmagent/) and Prometheus buffer the collected data locally if they cannot send it to the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instance after it becomes available. -If you use [vmagent](./vmagent.md) for storing the data into VictoriaMetrics, +If you use [vmagent](https://docs.victoriametrics.com/vmagent/) for storing the data into VictoriaMetrics, then it can be configured with multiple `-remoteWrite.url` command-line flags, where every flag points to the VictoriaMetrics instance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances. For example, the following command instructs `vmagent` to replicate data to `vm-az1` and `vm-az2` instances of VictoriaMetrics: @@ -1793,16 +1793,16 @@ remote_write: - url: http://:8428/api/v1/write ``` -It is recommended to use [vmagent](./vmagent.md) instead of Prometheus for highly loaded setups, +It is recommended to use [vmagent](https://docs.victoriametrics.com/vmagent/) instead of Prometheus for highly loaded setups, since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus. -If you use identically configured [vmagent](./vmagent.md) instances for collecting the same data +If you use identically configured [vmagent](https://docs.victoriametrics.com/vmagent/) instances for collecting the same data and sending it to VictoriaMetrics, then do not forget enabling [deduplication](#deduplication) at VictoriaMetrics side. ## Deduplication -VictoriaMetrics leaves a single [raw sample](./keyConcepts.md#raw-samples) -with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](./keyConcepts.md#time-series) +VictoriaMetrics leaves a single [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) +with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) for each [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) per each `-dedup.minScrapeInterval` discrete interval if `-dedup.minScrapeInterval` is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would leave a single raw sample with the biggest timestamp per each discrete `60s` interval. @@ -1811,12 +1811,12 @@ This aligns with the [staleness rules in Prometheus](https://prometheus.io/docs/ If multiple raw samples have **the same timestamp** on the given `-dedup.minScrapeInterval` discrete interval, then the sample with **the biggest value** is kept. -[Prometheus staleness markers](./vmagent.md#prometheus-staleness-markers) are processed as any other value during de-duplication. +[Prometheus staleness markers](https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers) are processed as any other value during de-duplication. If raw sample with the biggest timestamp on `-dedup.minScrapeInterval` contains a stale marker, then it is kept after the deduplication. This allows properly preserving staleness markers during the de-duplication. -Please note, [labels](./keyConcepts.md#labels) of raw samples should be identical -in order to be deduplicated. For example, this is why [HA pair of vmagents](./vmagent.md#high-availability) +Please note, [labels](https://docs.victoriametrics.com/keyconcepts/#labels) of raw samples should be identical +in order to be deduplicated. For example, this is why [HA pair of vmagents](https://docs.victoriametrics.com/vmagent/#high-availability) needs to be identically configured. The `-dedup.minScrapeInterval=D` is equivalent to `-downsampling.period=0s:D` if [downsampling](#downsampling) is enabled. @@ -1826,21 +1826,21 @@ The recommended value for `-dedup.minScrapeInterval` must equal to `scrape_inter It is recommended to have a single `scrape_interval` across all the scrape targets. See [this article](https://www.robustperception.io/keep-it-simple-scrape_interval-id) for details. -The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](./vmagent.md) +The de-duplication reduces disk space usage if multiple **identically configured** [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus instances in HA pair write data to the same VictoriaMetrics instance. These vmagent or Prometheus instances must have **identical** `external_labels` section in their configs, so they write data to the same time series. -See also [how to set up multiple vmagent instances for scraping the same targets](./vmagent.md#scraping-big-number-of-targets). +See also [how to set up multiple vmagent instances for scraping the same targets](https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets). It is recommended passing different `-promscrape.cluster.name` values to each distinct HA pair of `vmagent` instances, so the de-duplication consistently leaves samples for one `vmagent` instance and removes duplicate samples from other `vmagent` instances. -See [these docs](./vmagent.md#high-availability) for details. +See [these docs](https://docs.victoriametrics.com/vmagent/#high-availability) for details. VictoriaMetrics stores all the ingested samples to disk even if `-dedup.minScrapeInterval` command-line flag is set. The ingested samples are de-duplicated during [background merges](#storage) and during query execution. VictoriaMetrics also supports de-duplication during data ingestion before the data is stored to disk, via `-streamAggr.dedupInterval` command-line flag - -see [these docs](./stream-aggregation.md#deduplication). +see [these docs](https://docs.victoriametrics.com/stream-aggregation/#deduplication). ## Storage @@ -1852,19 +1852,19 @@ can be configured with the `-inmemoryDataFlushInterval` command-line flag (note In-memory parts are persisted to disk into `part` directories under the `<-storageDataPath>/data/small/YYYY_MM/` folder, where `YYYY_MM` is the month partition for the stored data. For example, `2022_11` is the partition for `parts` -with [raw samples](./keyConcepts.md#raw-samples) from `November 2022`. +with [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) from `November 2022`. Each partition directory contains `parts.json` file with the actual list of parts in the partition. Every `part` directory contains `metadata.json` file with the following fields: -- `RowsCount` - the number of [raw samples](./keyConcepts.md#raw-samples) stored in the part +- `RowsCount` - the number of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) stored in the part - `BlocksCount` - the number of blocks stored in the part (see details about blocks below) - `MinTimestamp` and `MaxTimestamp` - minimum and maximum timestamps across raw samples stored in the part - `MinDedupInterval` - the [deduplication interval](#deduplication) applied to the given part. Each `part` consists of `blocks` sorted by internal time series id (aka `TSID`). -Each `block` contains up to 8K [raw samples](./keyConcepts.md#raw-samples), -which belong to a single [time series](./keyConcepts.md#time-series). +Each `block` contains up to 8K [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples), +which belong to a single [time series](https://docs.victoriametrics.com/keyconcepts/#time-series). Raw samples in each block are sorted by `timestamp`. Blocks for the same time series are sorted by the `timestamp` of the first sample. Timestamps and values for all the blocks are stored in [compressed form](https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932) @@ -1893,7 +1893,7 @@ leaving the source `parts` untouched. However, due to hardware issues data on di VictoriaMetrics process. VictoriaMetrics can detect corruption during decompressing, decoding or sanity checking of the data blocks. But **it cannot fix the corrupted data**. Data parts that fail to load on startup need to be deleted or restored from backups. This is why it is recommended performing -[regular backups](./Cluster-VictoriaMetrics.md#backups). +[regular backups](https://docs.victoriametrics.com/cluster-victoriametrics/#backups). VictoriaMetrics doesn't use checksums for stored data blocks. See why [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011). @@ -1938,7 +1938,7 @@ VictoriaMetrics does not support indefinite retention, but you can specify an ar ## Multiple retentions Distinct retentions for distinct time series can be configured via [retention filters](#retention-filters) -in [VictoriaMetrics enterprise](./enterprise.md). +in [VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/). Community version of VictoriaMetrics supports only a single retention, which can be configured via [-retentionPeriod](#retention) command-line flag. If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags: @@ -1947,18 +1947,18 @@ If you need multiple retentions in community version of VictoriaMetrics, then yo * `-storageDataPath`, so the data for each retention period is saved in a separate directory * `-httpListenAddr`, so clients may reach VictoriaMetrics instance with proper retention -Then set up [vmauth](./vmauth.md) in front of VictoriaMetrics instances, +Then set up [vmauth](https://docs.victoriametrics.com/vmauth/) in front of VictoriaMetrics instances, so it could route requests from particular user to VictoriaMetrics with the desired retention. -Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md). -See [these docs](./guides/guide-vmcluster-multiple-retention-setup.md) for multi-retention setup details. +Similar scheme can be applied for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). +See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details. ## Retention filters -[Enterprise version of VictoriaMetrics](./enterprise.md) supports e.g. `retention filters`, -which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](./keyConcepts.md#filtering) +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports e.g. `retention filters`, +which allow configuring multiple retentions for distinct sets of time series matching the configured [series filters](https://docs.victoriametrics.com/keyconcepts/#filtering) via `-retentionFilter` command-line flag. This flag accepts `filter:duration` options, where `filter` must be -a valid [series filter](./keyConcepts.md#filtering), while the `duration` +a valid [series filter](https://docs.victoriametrics.com/keyconcepts/#filtering), while the `duration` must contain valid [retention](#retention) for time series matching the given `filter`. The `duration` of the `-retentionFilter` must be lower or equal to [-retentionPeriod](#retention) flag value. If series doesn't match any configured `-retentionFilter`, then the retention configured via [-retentionPeriod](#retention) @@ -1973,15 +1973,15 @@ For example, the following config sets 3 days retention for time series with `te Important notes: -- The data outside the configured retention isn't deleted instantly - it is deleted eventually during [background merges](./#storage). +- The data outside the configured retention isn't deleted instantly - it is deleted eventually during [background merges](https://docs.victoriametrics.com/#storage). - The `-retentionFilter` doesn't remove old data from `indexdb` (aka inverted index) until the configured [-retentionPeriod](#retention). - So the `indexdb` size can grow big under [high churn rate](./FAQ.md#what-is-high-churn-rate) + So the `indexdb` size can grow big under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) even for small retentions configured via `-retentionFilter`. It is safe updating `-retentionFilter` during VictoriaMetrics restarts - the updated retention filters are applied eventually to historical data. -See [how to configure multiple retentions in VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#retention-filters). +See [how to configure multiple retentions in VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#retention-filters). See also [downsampling](#downsampling). @@ -1990,60 +1990,60 @@ See how to request a free trial license [here](https://victoriametrics.com/produ ## Downsampling -[VictoriaMetrics Enterprise](./enterprise.md) supports multi-level downsampling via `-downsampling.period=offset:interval` command-line flag. -This command-line flag instructs leaving the last sample per each `interval` for [time series](./keyConcepts.md#time-series) -[samples](./keyConcepts.md#raw-samples) older than the `offset`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample +[VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise/) supports multi-level downsampling via `-downsampling.period=offset:interval` command-line flag. +This command-line flag instructs leaving the last sample per each `interval` for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) +[samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) older than the `offset`. For example, `-downsampling.period=30d:5m` instructs leaving the last sample per each 5-minute interval for samples older than 30 days, while the rest of samples are dropped. The `-downsampling.period` command-line flag can be specified multiple times in order to apply different downsampling levels for different time ranges (aka multi-level downsampling). For example, `-downsampling.period=30d:5m,180d:1h` instructs leaving the last sample per each 5-minute interval for samples older than 30 days, while leaving the last sample per each 1-hour interval for samples older than 180 days. -VictoriaMetrics supports configuring independent downsampling per different sets of [time series](./keyConcepts.md#time-series) +VictoriaMetrics supports configuring independent downsampling per different sets of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) via `-downsampling.period=filter:offset:interval` syntax. In this case the given `offset:interval` downsampling is applied only to time series matching the given `filter`. -The `filter` can contain arbitrary [series filter](./keyConcepts.md#filtering). +The `filter` can contain arbitrary [series filter](https://docs.victoriametrics.com/keyconcepts/#filtering). For example, `-downsampling.period='{__name__=~"(node|process)_.*"}:1d:1m` instructs VictoriaMetrics to deduplicate samples older than one day with one minute interval -only for [time series](./keyConcepts.md#time-series) with names starting with `node_` or `process_` prefixes. +only for [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) with names starting with `node_` or `process_` prefixes. The de-duplication for other time series can be configured independently via additional `-downsampling.period` command-line flags. If the time series doesn't match any `filter`, then it isn't downsampled. If the time series matches multiple filters, then the downsampling for the first matching `filter` is applied. For example, `-downsampling.period='{env="prod"}:1d:30s,{__name__=~"node_.*"}:1d:5m'` de-duplicates -samples older than one day with 30 seconds interval across all the time series with `env="prod"` [label](./keyConcepts.md#labels), +samples older than one day with 30 seconds interval across all the time series with `env="prod"` [label](https://docs.victoriametrics.com/keyconcepts/#labels), even if their names start with `node_` prefix. All the other time series with names starting with `node_` prefix are de-duplicated with 5 minutes interval. If downsampling shouldn't be applied to some time series matching the given `filter`, then pass `-downsampling.period=filter:0s:0s` command-line flag to VictoriaMetrics. For example, if series with `env="prod"` label shouldn't be downsampled, then pass `-downsampling.period='{env="prod"}:0s:0s'` command-line flag in front of other `-downsampling.period` flags. -Downsampling is applied independently per each time series and leaves a single [raw sample](./keyConcepts.md#raw-samples) +Downsampling is applied independently per each time series and leaves a single [raw sample](https://docs.victoriametrics.com/keyconcepts/#raw-samples) with the biggest [timestamp](https://en.wikipedia.org/wiki/Unix_time) on the configured interval, in the same way as [deduplication](#deduplication) does. -It works the best for [counters](./keyConcepts.md#counter) and [histograms](./keyConcepts.md#histogram), -as their values are always increasing. Downsampling [gauges](./keyConcepts.md#gauge) -and [summaries](./keyConcepts.md#summary) lose some changes within the downsampling interval, +It works the best for [counters](https://docs.victoriametrics.com/keyconcepts/#counter) and [histograms](https://docs.victoriametrics.com/keyconcepts/#histogram), +as their values are always increasing. Downsampling [gauges](https://docs.victoriametrics.com/keyconcepts/#gauge) +and [summaries](https://docs.victoriametrics.com/keyconcepts/#summary) lose some changes within the downsampling interval, since only the last sample on the given interval is left and the rest of samples are dropped. -You can use [recording rules](./vmalert.md#rules) or [steaming aggregation](./stream-aggregation.md) +You can use [recording rules](https://docs.victoriametrics.com/vmalert/#rules) or [steaming aggregation](https://docs.victoriametrics.com/stream-aggregation/) to apply custom aggregation functions, like min/max/avg etc., in order to make gauges more resilient to downsampling. Downsampling can reduce disk space usage and improve query performance if it is applied to time series with big number of samples per each series. The downsampling doesn't improve query performance and doesn't reduce disk space if the database contains big number of time series with small number of samples per each series, since downsampling doesn't reduce the number of time series. -So there is little sense in applying downsampling to time series with [high churn rate](./FAQ.md#what-is-high-churn-rate). +So there is little sense in applying downsampling to time series with [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). In this case the majority of query time is spent on searching for the matching time series instead of processing the found samples. -It is possible to use [stream aggregation](./stream-aggregation.md) in [vmagent](./vmagent.md) -or [recording rules in vmalert](./vmalert.md#rules) in order to -[reduce the number of time series](./vmalert.md#downsampling-and-aggregation-via-vmalert). +It is possible to use [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/) in [vmagent](https://docs.victoriametrics.com/vmagent/) +or [recording rules in vmalert](https://docs.victoriametrics.com/vmalert/#rules) in order to +[reduce the number of time series](https://docs.victoriametrics.com/vmalert/#downsampling-and-aggregation-via-vmalert). -Downsampling is performed during [background merges](./#storage). -It cannot be performed if there is not enough of free disk space or if vmstorage is in [read-only mode](./Cluster-VictoriaMetrics.md#readonly-mode). +Downsampling is performed during [background merges](https://docs.victoriametrics.com/#storage). +It cannot be performed if there is not enough of free disk space or if vmstorage is in [read-only mode](https://docs.victoriametrics.com/cluster-victoriametrics/#readonly-mode). Please, note that intervals of `-downsampling.period` must be multiples of each other. -In case [deduplication](./#deduplication) is enabled, value of `-dedup.minScrapeInterval` command-line flag must also +In case [deduplication](https://docs.victoriametrics.com/#deduplication) is enabled, value of `-dedup.minScrapeInterval` command-line flag must also be multiple of `-downsampling.period` intervals. This is required to ensure consistency of deduplication and downsampling results. It is safe updating `-downsampling.period` during VictoriaMetrics restarts - the updated downsampling configuration will be -applied eventually to historical data during [background merges](./#storage). +applied eventually to historical data during [background merges](https://docs.victoriametrics.com/#storage). -See [how to configure downsampling in VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#downsampling). +See [how to configure downsampling in VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#downsampling). See also [retention filters](#retention-filters). @@ -2052,7 +2052,7 @@ See [how to request a free trial license](https://victoriametrics.com/products/e ## Multi-tenancy -Single-node VictoriaMetrics doesn't support multi-tenancy. Use the [cluster version](./Cluster-VictoriaMetrics.md#multitenancy) instead. +Single-node VictoriaMetrics doesn't support multi-tenancy. Use the [cluster version](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) instead. ## Scalability and cluster version @@ -2066,7 +2066,7 @@ horizontally scalable long-term remote storage for really large Prometheus deplo ## Alerting -It is recommended using [vmalert](./vmalert.md) for alerting. +It is recommended using [vmalert](https://docs.victoriametrics.com/vmalert/) for alerting. Additionally, alerting can be set up with the following tools: @@ -2077,7 +2077,7 @@ Additionally, alerting can be set up with the following tools: ## mTLS protection By default `VictoriaMetrics` accepts http requests at `8428` port (this port can be changed via `-httpListenAddr` command-line flags). -[Enterprise version of VictoriaMetrics](./enterprise.md) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) +[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports the ability to accept [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `VictoriaMetrics`, which accepts only mTLS requests at port `8428`: ``` @@ -2094,11 +2094,11 @@ See also [security docs](#security). General security recommendations: - All the VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as Internet. - The exception is [vmauth](./vmauth.md) and [vmgateway](./vmgateway.md), + The exception is [vmauth](https://docs.victoriametrics.com/vmauth/) and [vmgateway](https://docs.victoriametrics.com/vmgateway/), which are intended for serving public requests and performing authorization with [TLS termination](https://en.wikipedia.org/wiki/TLS_termination_proxy). -- All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](./vmauth.md) - or [vmgateway](./vmgateway.md). The proxy must be set up with proper authentication and authorization. -- Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](./vmauth.md) +- All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such as [vmauth](https://docs.victoriametrics.com/vmauth/) + or [vmgateway](https://docs.victoriametrics.com/vmgateway/). The proxy must be set up with proper authentication and authorization. +- Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuring [vmauth](https://docs.victoriametrics.com/vmauth/) in front of VictoriaMetrics components. - Set reasonable [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header value to all the components to mitigate [MitM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), for example: `max-age=31536000; includeSubDomains`. See `-http.header.hsts` flag. - Set reasonable [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) header value to mitigate [XSS attacks](https://en.wikipedia.org/wiki/Cross-site_scripting). See `-http.header.csp` flag. @@ -2107,7 +2107,7 @@ General security recommendations: VictoriaMetrics provides the following security-related command-line flags: * `-tls`, `-tlsCertFile` and `-tlsKeyFile` for switching from HTTP to HTTPS at `-httpListenAddr` (TCP port 8428 is listened by default). - [Enterprise version of VictoriaMetrics](./enterprise.md) supports automatic issuing of TLS certificates. + [Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) supports automatic issuing of TLS certificates. See [these docs](#automatic-issuing-of-tls-certificates). * `-mtls` and `-mtlsCAFile` for enabling [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) for requests to `-httpListenAddr`. See [these docs](#mtls-protection). * `-httpAuth.username` and `-httpAuth.password` for protecting all the HTTP endpoints @@ -2128,12 +2128,12 @@ VictoriaMetrics provides the following security-related command-line flags: Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats. For example, substitute `-graphiteListenAddr=:2003` with `-graphiteListenAddr=:2003`. This protects from unexpected requests from untrusted network interfaces. -See also [security recommendation for VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#security) +See also [security recommendation for VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#security) and [the general security page at VictoriaMetrics website](https://victoriametrics.com/security/). ## Automatic issuing of TLS certificates -All the VictoriaMetrics [Enterprise](./enterprise.md) components support automatic issuing of TLS certificates for public HTTPS server running at `-httpListenAddr` +All the VictoriaMetrics [Enterprise](https://docs.victoriametrics.com/enterprise/) components support automatic issuing of TLS certificates for public HTTPS server running at `-httpListenAddr` via [Let's Encrypt service](https://letsencrypt.org/). The following command-line flags must be set in order to enable automatic issuing of TLS certificates: - `-httpListenAddr` must be set for listening TCP port `443`. For example, `-httpListenAddr=:443`. This port must be accessible by the [Let's Encrypt service](https://letsencrypt.org/). @@ -2143,7 +2143,7 @@ via [Let's Encrypt service](https://letsencrypt.org/). The following command-lin - `-tlsAutocertCacheDir` may be set to the directory path for persisting the issued TLS certificates between VictoriaMetrics restarts. If this flag isn't set, then TLS certificates are re-issued on every restart. -This functionality can be evaluated for free according to [these docs](./enterprise.md). +This functionality can be evaluated for free according to [these docs](https://docs.victoriametrics.com/enterprise/). See also [security recommendations](#security). @@ -2170,7 +2170,7 @@ mkfs.ext4 ... -O 64bit,huge_file,extent -T huge ## Monitoring VictoriaMetrics exports internal metrics in Prometheus exposition format at `/metrics` page. -These metrics can be scraped via [vmagent](./vmagent.md) or any other Prometheus-compatible scraper. +These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent/) or any other Prometheus-compatible scraper. If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass `-metrics.exposeMetadata` command-line to them, so they add `TYPE` and `HELP` comments per each exposed metric at `/metrics` page. @@ -2190,14 +2190,14 @@ created by community. Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it. We recommend setting up [alerts](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts) -via [vmalert](./vmalert.md) or via Prometheus. +via [vmalert](https://docs.victoriametrics.com/vmalert/) or via Prometheus. VictoriaMetrics exposes currently running queries and their execution times at [`active queries` page](#active-queries). VictoriaMetrics exposes queries, which take the most time to execute, at [`top queries` page](#top-queries). See also [VictoriaMetrics Monitoring](https://victoriametrics.com/blog/victoriametrics-monitoring/) -and [troubleshooting docs](./Troubleshooting.md). +and [troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/). ## TSDB stats @@ -2209,10 +2209,10 @@ VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way simi * `match[]=SELECTOR` where `SELECTOR` is an arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) for series to take into account during stats calculation. By default all the series are taken into account. * `extra_label=LABEL=VALUE`. See [these docs](#prometheus-querying-api-enhancements) for more details. -In [cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md) each vmstorage tracks the stored time series individually. +In [cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/) each vmstorage tracks the stored time series individually. vmselect requests stats via [/api/v1/status/tsdb](#tsdb-stats) API from each vmstorage node and merges the results by summing per-series stats. This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes -due to [replication](#replication) or [rerouting](./Cluster-VictoriaMetrics.md?highlight=re-routes#cluster-availability). +due to [replication](#replication) or [rerouting](https://docs.victoriametrics.com/cluster-victoriametrics/?highlight=re-routes#cluster-availability). VictoriaMetrics provides an UI on top of `/api/v1/status/tsdb` - see [cardinality explorer docs](#cardinality-explorer). @@ -2288,8 +2288,8 @@ Query tracing is allowed by default. It can be denied by passing `-denyQueryTrac By default VictoriaMetrics doesn't limit the number of stored time series. The limit can be enforced by setting the following command-line flags: -* `-storage.maxHourlySeries` - limits the number of time series that can be added during the last hour. Useful for limiting the number of [active time series](./FAQ.md#what-is-an-active-time-series). -* `-storage.maxDailySeries` - limits the number of time series that can be added during the last day. Useful for limiting daily [churn rate](./FAQ.md#what-is-high-churn-rate). +* `-storage.maxHourlySeries` - limits the number of time series that can be added during the last hour. Useful for limiting the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). +* `-storage.maxDailySeries` - limits the number of time series that can be added during the last day. Useful for limiting daily [churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). Both limits can be set simultaneously. If any of these limits is reached, then incoming samples for new time series are dropped. A sample of dropped series is put in the log with `WARNING` level. @@ -2319,7 +2319,7 @@ The exceeded limits can be [monitored](#monitoring) with the following metrics: These limits are approximate, so VictoriaMetrics can underflow/overflow the limit by a small percentage (usually less than 1%). -See also more advanced [cardinality limiter in vmagent](./vmagent.md#cardinality-limiter) +See also more advanced [cardinality limiter in vmagent](https://docs.victoriametrics.com/vmagent/#cardinality-limiter) and [cardinality explorer docs](#cardinality-explorer). ## Troubleshooting @@ -2342,8 +2342,8 @@ and [cardinality explorer docs](#cardinality-explorer). * If you run VictoriaMetrics on a host with 16 or more CPU cores, then it may be needed to tune the `-search.maxWorkersPerQuery` command-line flag in order to improve query performance. If VictoriaMetrics serves big number of concurrent `select` queries, then try reducing the value for this flag. - If VictoriaMetrics serves heavy queries, which select `>10K` of [time series](./keyConcepts.md#time-series) and/or process `>100M` - of [raw samples](./keyConcepts.md#raw-samples) per query, then try setting the value for this flag to the number of available CPU cores. + If VictoriaMetrics serves heavy queries, which select `>10K` of [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) and/or process `>100M` + of [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) per query, then try setting the value for this flag to the number of available CPU cores. * VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage. This may lead to the following "issues": @@ -2354,7 +2354,7 @@ and [cardinality explorer docs](#cardinality-explorer). See [storage docs](#storage) and [this article](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704) for more details. * If VictoriaMetrics works slowly and eats more than a CPU core per 100K ingested data points per second, - then it is likely you have too many [active time series](./FAQ.md#what-is-an-active-time-series) for the current amount of RAM. + then it is likely you have too many [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) for the current amount of RAM. VictoriaMetrics [exposes](#monitoring) `vm_slow_*` metrics such as `vm_slow_row_inserts_total` and `vm_slow_metric_name_loads_total`, which could be used as an indicator of low amounts of RAM. It is recommended increasing the amount of RAM on the node with VictoriaMetrics in order to improve ingestion and query performance in this case. @@ -2382,8 +2382,8 @@ and [cardinality explorer docs](#cardinality-explorer). This suppresses default gap filling algorithm used by VictoriaMetrics - by default it assumes each time series is continuous instead of discrete, so it fills gaps between real samples with regular intervals. -* Metrics and labels leading to [high cardinality](./FAQ.md#what-is-high-cardinality) - or [high churn rate](./FAQ.md#what-is-high-churn-rate) can be determined +* Metrics and labels leading to [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality) + or [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) can be determined via [cardinality explorer](#cardinality-explorer) and via [/api/v1/status/tsdb](#tsdb-stats) endpoint. * New time series can be logged if `-logNewSeries` command-line flag is passed to VictoriaMetrics. @@ -2401,13 +2401,13 @@ and [cardinality explorer docs](#cardinality-explorer). See also: - [Snapshot troubleshooting](#snapshot-troubleshooting). -- [General troubleshooting docs](./Troubleshooting.md). +- [General troubleshooting docs](https://docs.victoriametrics.com/troubleshooting/). ## Push metrics All the VictoriaMetrics components support pushing their metrics exposed at `/metrics` page to remote storage in Prometheus text exposition format. -This functionality may be used instead of [classic Prometheus-like metrics scraping](./#how-to-scrape-prometheus-exporters-such-as-node-exporter) -if VictoriaMetrics components are located in isolated networks, so they cannot be scraped by local [vmagent](./vmagent.md). +This functionality may be used instead of [classic Prometheus-like metrics scraping](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) +if VictoriaMetrics components are located in isolated networks, so they cannot be scraped by local [vmagent](https://docs.victoriametrics.com/vmagent/). The following command-line flags are related to pushing metrics from VictoriaMetrics components: @@ -2452,7 +2452,7 @@ It is also possible removing [rollup result cache](#rollup-result-cache) on star ## Rollup result cache VictoriaMetrics caches query responses by default. This allows increasing performance for repeated queries -to [`/api/v1/query`](./keyConcepts.md#instant-query) and [`/api/v1/query_range`](./keyConcepts.md#range-query) +to [`/api/v1/query`](https://docs.victoriametrics.com/keyconcepts/#instant-query) and [`/api/v1/query_range`](https://docs.victoriametrics.com/keyconcepts/#range-query) with the increasing `time`, `start` and `end` query args. This cache may work incorrectly when ingesting historical data into VictoriaMetrics. See [these docs](#backfilling) for details. @@ -2507,30 +2507,30 @@ Things to consider when copying data: 1. Copying data folder means complete replacement of the previous data on destination VictoriaMetrics. For more complex scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction -of data - see [vmctl. Migrating data from VictoriaMetrics](./vmctl.md#migrating-data-from-victoriametrics). +of data - see [vmctl. Migrating data from VictoriaMetrics](https://docs.victoriametrics.com/vmctl/#migrating-data-from-victoriametrics). ### From other systems -Use [vmctl](./vmctl.md) for data migration. It supports the following data migration types: +Use [vmctl](https://docs.victoriametrics.com/vmctl/) for data migration. It supports the following data migration types: * From Prometheus to VictoriaMetrics * From InfluxDB to VictoriaMetrics * From VictoriaMetrics to VictoriaMetrics * From OpenTSDB to VictoriaMetrics -See [vmctl docs](./vmctl.md) for more details. +See [vmctl docs](https://docs.victoriametrics.com/vmctl/) for more details. ## Backfilling VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data). -See [how to backfill data with recording rules in vmalert](./vmalert.md#rules-backfilling). +See [how to backfill data with recording rules in vmalert](https://docs.victoriametrics.com/vmalert/#rules-backfilling). Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data. It is recommended disabling [query cache](#rollup-result-cache) with `-search.disableCache` command-line flag when writing historical data with timestamps from the past, since the cache assumes that the data is written with the current timestamps. Query cache can be enabled after the backfilling is complete. -An alternative solution is to query [/internal/resetRollupResultCache](./url-examples.md#internalresetrollupresultcache) +An alternative solution is to query [/internal/resetRollupResultCache](https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache) after the backfilling is complete. This will reset the [query cache](#rollup-result-cache), which could contain incomplete data cached during the backfilling. Yet another solution is to increase `-search.cacheTimestampOffset` flag value in order to disable caching @@ -2547,7 +2547,7 @@ should be used only for one-off updates. It shouldn't be used for frequent updat ## Replication Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead. -See [these docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) for details. +See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details. Storage-level replication may be offloaded to durable persistent storage such as [Google Cloud disks](https://cloud.google.com/compute/docs/disks#pdspecs). @@ -2555,15 +2555,15 @@ See also [high availability docs](#high-availability) and [backup docs](#backups ## Backups -VictoriaMetrics supports backups via [vmbackup](./vmbackup.md) -and [vmrestore](./vmrestore.md) tools. -We also provide [vmbackupmanager](./vmbackupmanager.md) tool for enterprise subscribers. +VictoriaMetrics supports backups via [vmbackup](https://docs.victoriametrics.com/vmbackup/) +and [vmrestore](https://docs.victoriametrics.com/vmrestore/) tools. +We also provide [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) tool for enterprise subscribers. Enterprise binaries can be downloaded and evaluated for free from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/). ## vmalert -A single-node VictoriaMetrics is capable of proxying requests to [vmalert](./vmalert.md) +A single-node VictoriaMetrics is capable of proxying requests to [vmalert](https://docs.victoriametrics.com/vmalert/) when `-vmalert.proxyURL` flag is set. Use this feature for the following cases: * for proxying requests from [Grafana Alerting UI](https://grafana.com/docs/grafana/latest/alerting/); * for accessing vmalerts UI through single-node VictoriaMetrics Web interface. @@ -2578,7 +2578,7 @@ the best parts of their product, while highlighting the worst parts of competing So we encourage users and all independent third parties to conduct their benchmarks for various products they are evaluating in production and publish the results. -As a reference, please see [benchmarks](./Articles.md#benchmarks) conducted by +As a reference, please see [benchmarks](https://docs.victoriametrics.com/articles/#benchmarks) conducted by VictoriaMetrics team. Please also see the [helm chart](https://github.com/VictoriaMetrics/benchmark) for running ingestion benchmarks based on node_exporter metrics. @@ -2613,7 +2613,7 @@ It is safe sharing the collected profiles from security point of view, since the See [this example](https://github.com/go-graphite/carbonapi/blob/main/cmd/carbonapi/carbonapi.example.victoriametrics.yaml). * [netdata](https://github.com/netdata/netdata) can push data into VictoriaMetrics via `Prometheus remote_write API`. See [these docs](https://github.com/netdata/netdata#integrations). -* [vmalert-cli](https://github.com/aorfanos/vmalert-cli) - a CLI application for managing [vmalert](./vmalert.md). +* [vmalert-cli](https://github.com/aorfanos/vmalert-cli) - a CLI application for managing [vmalert](https://docs.victoriametrics.com/vmalert/). ## Third-party contributions @@ -2639,7 +2639,7 @@ Feel free asking any questions regarding VictoriaMetrics: * [Google groups](https://groups.google.com/forum/#!forum/victorametrics-users) * [Mastodon](https://mastodon.social/@victoriametrics/) -If you like VictoriaMetrics and want to contribute, then please [read these docs](./CONTRIBUTING.md). +If you like VictoriaMetrics and want to contribute, then please [read these docs](https://docs.victoriametrics.com/contributing/). ## Reporting bugs @@ -2653,9 +2653,9 @@ and gets automatically updated once changes are merged to [master](https://githu To update the documentation follow the steps below: - [Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks) VictoriaMetrics repo and apply changes to the docs: - - To update [the main page](https://docs.victoriametrics.com/) modify [this file](./README.md). + - To update [the main page](https://docs.victoriametrics.com/) modify [this file](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md). - To update other pages, apply changes to the corresponding file in [docs folder](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs). -- If your changes contain an image then see [images in documentation](./#images-in-documentation). +- If your changes contain an image then see [images in documentation](https://docs.victoriametrics.com/#images-in-documentation). - Once changes are made, execute the command below to finalize and sync the changes: ```sh @@ -2732,7 +2732,7 @@ Files included in each folder: Pass `-help` to VictoriaMetrics in order to see the list of supported command-line flags with their description: -``` +```sh -bigMergeConcurrency int Deprecated: this flag does nothing -blockcache.missesBeforeCaching int @@ -2750,16 +2750,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -datadog.sanitizeMetricName Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true) -dedup.minScrapeInterval duration - Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and ./#deduplication + Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See also -streamAggr.dedupInterval and https://docs.victoriametrics.com/#deduplication -deleteAuthKey value authKey for metrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries Flag value can be read from the given file when using -deleteAuthKey=file:///abs/path/to/file or -deleteAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -deleteAuthKey=http://host/path or -deleteAuthKey=https://host/path -denyQueriesOutsideRetention Whether to deny queries outside the configured -retentionPeriod. When set, then /api/v1/query_range would return '503 Service Unavailable' error for queries with 'from' value outside -retentionPeriod. This may be useful when multiple data sources with distinct retentions are hidden behind query-tee -denyQueryTracing - Whether to disable the ability to trace queries. See {{% ref "./#query-tracing" %}} + Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing -downsampling.period array - Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See {{% ref "./#downsampling" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Comma-separated downsampling periods in the format 'offset:period'. For example, '30d:10m' instructs to leave a single sample per 10 minutes for samples older than 30 days. When setting multiple downsampling periods, it is necessary for the periods to be multiples of each other. See https://docs.victoriametrics.com/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -dryRun @@ -2767,11 +2767,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable - Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See {{% ref "./#environment-variables" %}} for more details + Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details -envflag.prefix string Prefix for environment variables if -envflag.enable is set -eula - Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -finalMergeDelay duration @@ -2788,7 +2788,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -graphite.sanitizeMetricName - Sanitize metric names for the ingested Graphite data. See {{% ref "./#how-to-send-data-from-graphite-compatible-agents-such-as-statsd" %}} + Sanitize metric names for the ingested Graphite data. See https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd -graphiteListenAddr string TCP and UDP address to listen for Graphite plaintext data. Usually :2003 must be set. Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol -graphiteListenAddr.useProxyProtocol @@ -2906,18 +2906,18 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array - Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array - Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -newrelic.maxInsertRequestSize size The maximum size in bytes of a single NewRelic request to /newrelic/infra/v2/metrics/events/bulk Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 67108864) -opentelemetry.usePrometheusNaming - Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see {{% ref "./#sending-data-via-opentelemetry" %}} + Whether to convert metric names and labels into Prometheus-compatible format for the metrics ingested via OpenTelemetry protocol; see https://docs.victoriametrics.com/#sending-data-via-opentelemetry -opentsdbHTTPListenAddr string TCP address to listen for OpenTSDB HTTP put requests. Usually :4242 must be set. Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol -opentsdbHTTPListenAddr.useProxyProtocol @@ -2941,35 +2941,35 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -prevCacheRemovalPercent float Items in the previous caches are removed when the percent of requests it serves becomes lower than this value. Higher values reduce memory usage at the cost of higher CPU usage. See also -cacheExpireDuration (default 0.1) -promscrape.azureSDCheckInterval duration - Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#azure_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in Azure. This works only if azure_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#azure_sd_configs for details (default 1m0s) -promscrape.cluster.memberLabel string - If non-empty, then the label with this name and the -promscrape.cluster.memberNum value is added to all the scraped metrics. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info + If non-empty, then the label with this name and the -promscrape.cluster.memberNum value is added to all the scraped metrics. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info -promscrape.cluster.memberNum string - The number of vmagent instance in the cluster of scrapers. It must be a unique value in the range 0 ... promscrape.cluster.membersCount-1 across scrapers in the cluster. Can be specified as pod name of Kubernetes StatefulSet - pod-name-Num, where Num is a numeric part of pod name. See also -promscrape.cluster.memberLabel . See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default "0") + The number of vmagent instance in the cluster of scrapers. It must be a unique value in the range 0 ... promscrape.cluster.membersCount-1 across scrapers in the cluster. Can be specified as pod name of Kubernetes StatefulSet - pod-name-Num, where Num is a numeric part of pod name. See also -promscrape.cluster.memberLabel . See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default "0") -promscrape.cluster.memberURLTemplate string - An optional template for URL to access vmagent instance with the given -promscrape.cluster.memberNum value. Every %d occurrence in the template is substituted with -promscrape.cluster.memberNum at urls to vmagent instances responsible for scraping the given target at /service-discovery page. For example -promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more details + An optional template for URL to access vmagent instance with the given -promscrape.cluster.memberNum value. Every %d occurrence in the template is substituted with -promscrape.cluster.memberNum at urls to vmagent instances responsible for scraping the given target at /service-discovery page. For example -promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more details -promscrape.cluster.membersCount int - The number of members in a cluster of scrapers. Each member must have a unique -promscrape.cluster.memberNum in the range 0 ... promscrape.cluster.membersCount-1 . Each member then scrapes roughly 1/N of all the targets. By default, cluster scraping is disabled, i.e. a single scraper scrapes all the targets. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default 1) + The number of members in a cluster of scrapers. Each member must have a unique -promscrape.cluster.memberNum in the range 0 ... promscrape.cluster.membersCount-1 . Each member then scrapes roughly 1/N of all the targets. By default, cluster scraping is disabled, i.e. a single scraper scrapes all the targets. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1) -promscrape.cluster.name string - Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info + Optional name of the cluster. If multiple vmagent clusters scrape the same targets, then each cluster must have unique name in order to properly de-duplicate samples received from these clusters. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info -promscrape.cluster.replicationFactor int - The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See {{% ref "./vmagent.md#scraping-big-number-of-targets" %}} for more info (default 1) + The number of members in the cluster, which scrape the same targets. If the replication factor is greater than 1, then the deduplication must be enabled at remote storage side. See https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets for more info (default 1) -promscrape.config string - Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See {{% ref "./#how-to-scrape-prometheus-exporters-such-as-node-exporter" %}} for details + Optional path to Prometheus config file with 'scrape_configs' section containing targets to scrape. The path can point to local file and to http url. See https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details -promscrape.config.dryRun Checks -promscrape.config file for errors and unsupported fields and then exits. Returns non-zero exit code on parsing errors and emits these errors to stderr. See also -promscrape.config.strictParse command-line flag. Pass -loggerLevel=ERROR if you don't need to see info messages in the output. -promscrape.config.strictParse Whether to deny unsupported fields in -promscrape.config . Set to false in order to silently skip unsupported fields (default true) -promscrape.configCheckInterval duration - Interval for checking for changes in -promscrape.config file. By default, the checking is disabled. See how to reload -promscrape.config file at {{% ref "./vmagent.md#configuration-update" %}} + Interval for checking for changes in -promscrape.config file. By default, the checking is disabled. See how to reload -promscrape.config file at https://docs.victoriametrics.com/vmagent/#configuration-update -promscrape.consul.waitTime duration Wait time used by Consul service discovery. Default value is used if not set -promscrape.consulSDCheckInterval duration - Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#consul_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Consul. This works only if consul_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#consul_sd_configs for details (default 30s) -promscrape.consulagentSDCheckInterval duration - Interval for checking for changes in Consul Agent. This works only if consulagent_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#consulagent_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Consul Agent. This works only if consulagent_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#consulagent_sd_configs for details (default 30s) -promscrape.digitaloceanSDCheckInterval duration - Interval for checking for changes in digital ocean. This works only if digitalocean_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#digitalocean_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in digital ocean. This works only if digitalocean_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#digitalocean_sd_configs for details (default 1m0s) -promscrape.disableCompression Whether to disable sending 'Accept-Encoding: gzip' request headers to all the scrape targets. This may reduce CPU usage on scrape targets at the cost of higher network bandwidth utilization. It is possible to set 'disable_compression: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control -promscrape.disableKeepAlive @@ -2979,33 +2979,33 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -promscrape.discovery.concurrentWaitTime duration The maximum duration for waiting to perform API requests if more than -promscrape.discovery.concurrency requests are simultaneously performed (default 1m0s) -promscrape.dnsSDCheckInterval duration - Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#dns_sd_configs" %}} for details (default 30s) + Interval for checking for changes in dns. This works only if dns_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#dns_sd_configs for details (default 30s) -promscrape.dockerSDCheckInterval duration - Interval for checking for changes in docker. This works only if docker_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#docker_sd_configs" %}} for details (default 30s) + Interval for checking for changes in docker. This works only if docker_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#docker_sd_configs for details (default 30s) -promscrape.dockerswarmSDCheckInterval duration - Interval for checking for changes in dockerswarm. This works only if dockerswarm_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#dockerswarm_sd_configs" %}} for details (default 30s) + Interval for checking for changes in dockerswarm. This works only if dockerswarm_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#dockerswarm_sd_configs for details (default 30s) -promscrape.dropOriginalLabels Whether to drop original labels for scrape targets at /targets and /api/v1/targets pages. This may be needed for reducing memory usage when original labels for big number of scrape targets occupy big amounts of memory. Note that this reduces debuggability for improper per-target relabeling configs -promscrape.ec2SDCheckInterval duration - Interval for checking for changes in ec2. This works only if ec2_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#ec2_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in ec2. This works only if ec2_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#ec2_sd_configs for details (default 1m0s) -promscrape.eurekaSDCheckInterval duration - Interval for checking for changes in eureka. This works only if eureka_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#eureka_sd_configs" %}} for details (default 30s) + Interval for checking for changes in eureka. This works only if eureka_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#eureka_sd_configs for details (default 30s) -promscrape.fileSDCheckInterval duration - Interval for checking for changes in 'file_sd_config'. See {{% ref "./sd_configs.md#file_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in 'file_sd_config'. See https://docs.victoriametrics.com/sd_configs/#file_sd_configs for details (default 1m0s) -promscrape.gceSDCheckInterval duration - Interval for checking for changes in gce. This works only if gce_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#gce_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in gce. This works only if gce_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#gce_sd_configs for details (default 1m0s) -promscrape.hetznerSDCheckInterval duration - Interval for checking for changes in Hetzner API. This works only if hetzner_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#hetzner_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in Hetzner API. This works only if hetzner_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#hetzner_sd_configs for details (default 1m0s) -promscrape.httpSDCheckInterval duration - Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#http_sd_configs" %}} for details (default 1m0s) + Interval for checking for changes in http endpoint service discovery. This works only if http_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#http_sd_configs for details (default 1m0s) -promscrape.kubernetes.apiServerTimeout duration How frequently to reload the full state from Kubernetes API server (default 30m0s) -promscrape.kubernetes.attachNodeMetadataAll - Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See {{% ref "./sd_configs.md#kubernetes_sd_configs" %}} + Whether to set attach_metadata.node=true for all the kubernetes_sd_configs at -promscrape.config . It is possible to set attach_metadata.node=false individually per each kubernetes_sd_configs . See https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs -promscrape.kubernetesSDCheckInterval duration - Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#kubernetes_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Kubernetes API server. This works only if kubernetes_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kubernetes_sd_configs for details (default 30s) -promscrape.kumaSDCheckInterval duration - Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#kuma_sd_configs" %}} for details (default 30s) + Interval for checking for changes in kuma service discovery. This works only if kuma_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#kuma_sd_configs for details (default 30s) -promscrape.maxDroppedTargets int The maximum number of droppedTargets to show at /api/v1/targets page. Increase this value if your setup drops more scrape targets during relabeling and you need investigating labels for all the dropped targets. Note that the increased number of tracked dropped targets may result in increased memory usage (default 10000) -promscrape.maxResponseHeadersSize size @@ -3015,28 +3015,28 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li The maximum size of scrape response in bytes to process from Prometheus targets. Bigger responses are rejected Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 16777216) -promscrape.minResponseSizeForStreamParse size - The minimum target response size for automatic switching to stream parsing mode, which can reduce memory usage. See {{% ref "./vmagent.md#stream-parsing-mode" %}} + The minimum target response size for automatic switching to stream parsing mode, which can reduce memory usage. See https://docs.victoriametrics.com/vmagent/#stream-parsing-mode Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 1000000) -promscrape.noStaleMarkers Whether to disable sending Prometheus stale markers for metrics when scrape target disappears. This option may reduce memory usage if stale markers aren't needed for your setup. This option also disables populating the scrape_series_added metric. See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series -promscrape.nomad.waitTime duration Wait time used by Nomad service discovery. Default value is used if not set -promscrape.nomadSDCheckInterval duration - Interval for checking for changes in Nomad. This works only if nomad_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#nomad_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Nomad. This works only if nomad_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#nomad_sd_configs for details (default 30s) -promscrape.openstackSDCheckInterval duration - Interval for checking for changes in openstack API server. This works only if openstack_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#openstack_sd_configs" %}} for details (default 30s) + Interval for checking for changes in openstack API server. This works only if openstack_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#openstack_sd_configs for details (default 30s) -promscrape.seriesLimitPerTarget int - Optional limit on the number of unique time series a single scrape target can expose. See {{% ref "./vmagent.md#cardinality-limiter" %}} for more info + Optional limit on the number of unique time series a single scrape target can expose. See https://docs.victoriametrics.com/vmagent/#cardinality-limiter for more info -promscrape.streamParse Whether to enable stream parsing for metrics obtained from scrape targets. This may be useful for reducing memory usage when millions of metrics are exposed per each scrape target. It is possible to set 'stream_parse: true' individually per each 'scrape_config' section in '-promscrape.config' for fine-grained control -promscrape.suppressDuplicateScrapeTargetErrors - Whether to suppress 'duplicate scrape target' errors; see {{% ref "./vmagent.md#troubleshooting" %}} for details + Whether to suppress 'duplicate scrape target' errors; see https://docs.victoriametrics.com/vmagent/#troubleshooting for details -promscrape.suppressScrapeErrors Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay -promscrape.suppressScrapeErrorsDelay duration The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors -promscrape.yandexcloudSDCheckInterval duration - Interval for checking for changes in Yandex Cloud API. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. See {{% ref "./sd_configs.md#yandexcloud_sd_configs" %}} for details (default 30s) + Interval for checking for changes in Yandex Cloud API. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs/#yandexcloud_sd_configs for details (default 30s) -pushmetrics.disableCompression Whether to disable request body compression when pushing metrics to every -pushmetrics.url -pushmetrics.extraLabel array @@ -3050,16 +3050,16 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -pushmetrics.interval duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array - Optional URL to push metrics exposed at /metrics page. See {{% ref "./#push-metrics" %}}. By default, metrics exposed at /metrics page aren't pushed to any remote storage + Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -relabelConfig string - Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See {{% ref "./#relabeling for details" %}}. The config is reloaded on SIGHUP signal + Optional path to a file with relabeling rules, which are applied to all the ingested metrics. The path can point either to local file or to http url. See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal -reloadAuthKey value Auth key for /-/reload http endpoint. It must be passed via authKey query arg. It overrides -httpAuth.* Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path -retentionFilter array - Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See {{% ref "./#retention-filters" %}} for details. This flag is available only in VictoriaMetrics enterprise. See {{% ref "./enterprise.md" %}} + Retention filter in the format 'filter:retention'. For example, '{env="dev"}:3d' configures the retention for time series with env="dev" label to 3 days. See https://docs.victoriametrics.com/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -retentionPeriod value @@ -3072,9 +3072,9 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.disableAutoCacheReset Whether to disable automatic response cache reset if a sample with timestamp outside -search.cacheTimestampOffset is inserted into VictoriaMetrics -search.disableCache - Whether to disable response caching. This may be useful when ingesting historical data. See {{% ref "./#backfilling" %}}. See also -search.resetRollupResultCacheOnStartup + Whether to disable response caching. This may be useful when ingesting historical data. See https://docs.victoriametrics.com/#backfilling . See also -search.resetRollupResultCacheOnStartup -search.disableImplicitConversion - Whether to return an error for queries that rely on implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. See also -search.logImplicitConversion + Whether to return an error for queries that rely on implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. See also -search.logImplicitConversion -search.graphiteMaxPointsPerSeries int The maximum number of points per series Graphite render API can return (default 1000000) -search.graphiteStorageStep duration @@ -3084,7 +3084,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.latencyOffset duration The time when data points become visible in query results after the collection. It can be overridden on per-query basis via latency_offset arg. Too small value can result in incomplete last points for query results (default 30s) -search.logImplicitConversion - Whether to log queries with implicit subquery conversions, see {{% ref "./MetricsQL.md#subqueries" %}} for details. Such conversion can be disabled using -search.disableImplicitConversion + Whether to log queries with implicit subquery conversions, see https://docs.victoriametrics.com/metricsql/#subqueries for details. Such conversion can be disabled using -search.disableImplicitConversion -search.logQueryMemoryUsage size Log query and increment vm_memory_intensive_queries_total metric each time the query requires more memory than specified by this flag. This may help detecting and optimizing heavy queries. Query logging is disabled by default. See also -search.logSlowQueryDuration and -search.maxMemoryPerQuery Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) @@ -3099,11 +3099,11 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -search.maxFederateSeries int The maximum number of time series, which can be returned from /federate. This option allows limiting memory usage (default 1000000) -search.maxGraphiteSeries int - The maximum number of time series, which can be scanned during queries to Graphite Render API. See {{% ref "./#graphite-render-api-usage" %}} (default 300000) + The maximum number of time series, which can be scanned during queries to Graphite Render API. See https://docs.victoriametrics.com/#graphite-render-api-usage (default 300000) -search.maxGraphiteTagKeys int - The maximum number of tag keys returned from Graphite API, which returns tags. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag keys returned from Graphite API, which returns tags. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxGraphiteTagValues int - The maximum number of tag values returned from Graphite API, which returns tag values. See {{% ref "./#graphite-tags-api-usage" %}} (default 100000) + The maximum number of tag values returned from Graphite API, which returns tag values. See https://docs.victoriametrics.com/#graphite-tags-api-usage (default 100000) -search.maxLabelsAPIDuration duration The maximum duration for /api/v1/labels, /api/v1/label/.../values and /api/v1/series requests. See also -search.maxLabelsAPISeries and -search.ignoreExtraFiltersAtLabelsAPI (default 5s) -search.maxLabelsAPISeries int @@ -3167,7 +3167,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li Optional authKey for resetting rollup cache via /internal/resetRollupResultCache call Flag value can be read from the given file when using -search.resetCacheAuthKey=file:///abs/path/to/file or -search.resetCacheAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -search.resetCacheAuthKey=http://host/path or -search.resetCacheAuthKey=https://host/path -search.resetRollupResultCacheOnStartup - Whether to reset rollup result cache on startup. See {{% ref "./#rollup-result-cache" %}}. See also -search.disableCache + Whether to reset rollup result cache on startup. See https://docs.victoriametrics.com/#rollup-result-cache . See also -search.disableCache -search.setLookbackToStep Whether to fix lookback interval to 'step' query arg value. If set to true, the query model becomes closer to InfluxDB data model. If set to true, then -search.maxLookback and -search.maxStalenessInterval are ignored -search.treatDotsAsIsInRegexps @@ -3191,52 +3191,52 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li -sortLabels Whether to sort labels for incoming samples before writing them to storage. This may be needed for reducing memory usage at storage when the order of labels in incoming samples is random. For example, if m{k1="v1",k2="v2"} may be sent as m{k2="v2",k1="v1"}. Enabled sorting for labels can slow down ingestion performance a bit -storage.cacheSizeIndexDBDataBlocks size - Overrides max size for indexdb/dataBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/dataBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBIndexBlocks size - Overrides max size for indexdb/indexBlocks cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/indexBlocks cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeIndexDBTagFilters size - Overrides max size for indexdb/tagFiltersToMetricIDs cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for indexdb/tagFiltersToMetricIDs cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.cacheSizeStorageTSID size - Overrides max size for storage/tsid cache. See {{% ref "./Single-Server-VictoriaMetrics.md#cache-tuning" %}} + Overrides max size for storage/tsid cache. See https://docs.victoriametrics.com/single-server-victoriametrics/#cache-tuning Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -storage.maxDailySeries int - The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxHourlySeries + The maximum number of unique series can be added to the storage during the last 24 hours. Excess series are logged and dropped. This can be useful for limiting series churn rate. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxHourlySeries -storage.maxHourlySeries int - The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See {{% ref "./#cardinality-limiter" %}}. See also -storage.maxDailySeries + The maximum number of unique series can be added to the storage during the last hour. Excess series are logged and dropped. This can be useful for limiting series cardinality. See https://docs.victoriametrics.com/#cardinality-limiter . See also -storage.maxDailySeries -storage.minFreeDiskSpaceBytes size The minimum free disk space at -storageDataPath after which the storage stops accepting new data Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 10000000) -storageDataPath string Path to storage data (default "victoria-metrics-data") -streamAggr.config string - Optional path to file with stream aggregation config. See {{% ref "./stream-aggregation.md" %}}. See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval + Optional path to file with stream aggregation config. See https://docs.victoriametrics.com/stream-aggregation/ . See also -streamAggr.keepInput, -streamAggr.dropInput and -streamAggr.dedupInterval -streamAggr.dedupInterval duration - Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and {{% ref "./stream-aggregation.md#deduplication" %}} + Input samples are de-duplicated with this interval before optional aggregation with -streamAggr.config . See also -streamAggr.dropInputLabels and -dedup.minScrapeInterval and https://docs.victoriametrics.com/stream-aggregation/#deduplication -streamAggr.dropInput - Whether to drop all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.keepInput and {{% ref "./stream-aggregation.md" %}} + Whether to drop all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.keepInput and https://docs.victoriametrics.com/stream-aggregation/ -streamAggr.dropInputLabels array - An optional list of labels to drop from samples before stream de-duplication and aggregation . See {{% ref "./stream-aggregation.md#dropping-unneeded-labels" %}} + An optional list of labels to drop from samples before stream de-duplication and aggregation . See https://docs.victoriametrics.com/stream-aggregation/#dropping-unneeded-labels Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -streamAggr.ignoreFirstIntervals int - Number of aggregation intervals to skip after the start. Increase this value if you observe incorrect aggregation results after restarts. It could be caused by receiving unordered delayed data from clients pushing data into the database. See {{% ref "./stream-aggregation.md#ignore-aggregation-intervals-on-start" %}} + Number of aggregation intervals to skip after the start. Increase this value if you observe incorrect aggregation results after restarts. It could be caused by receiving unordered delayed data from clients pushing data into the database. See https://docs.victoriametrics.com/stream-aggregation/#ignore-aggregation-intervals-on-start -streamAggr.ignoreOldSamples - Whether to ignore input samples with old timestamps outside the current aggregation interval. See {{% ref "./stream-aggregation.md#ignoring-old-samples" %}} + Whether to ignore input samples with old timestamps outside the current aggregation interval. See https://docs.victoriametrics.com/stream-aggregation/#ignoring-old-samples -streamAggr.keepInput - Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.dropInput and {{% ref "./stream-aggregation.md" %}} + Whether to keep all the input samples after the aggregation with -streamAggr.config. By default, only aggregated samples are dropped, while the remaining samples are stored in the database. See also -streamAggr.dropInput and https://docs.victoriametrics.com/stream-aggregation/ -tls array Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string - Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string - Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array - Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See {{% ref "./enterprise.md" %}} + Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index 702932902..ece608d74 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -26,21 +26,21 @@ If you hit some issue or have some question about VictoriaMetrics components, then please follow the following steps in order to quickly find the solution: 1. Check the version of VictoriaMetrics component, which needs to be troubleshot and compare - it to [the latest available version](./CHANGELOG.md). + it to [the latest available version](https://docs.victoriametrics.com/changelog/). If the used version is lower than the latest available version, then there are high chances - that the issue is already resolved in newer versions. Carefully read [the changelog](./CHANGELOG.md) + that the issue is already resolved in newer versions. Carefully read [the changelog](https://docs.victoriametrics.com/changelog/) between your version and the latest version and check whether the issue is already fixed there. If the issue is already fixed in newer versions, then upgrade to the newer version and verify whether the issue is fixed: - - [How to upgrade single-node VictoriaMetrics](./README.md#how-to-upgrade-victoriametrics) - - [How to upgrade VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md#updating--reconfiguring-cluster-nodes) + - [How to upgrade single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics) + - [How to upgrade VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#updating--reconfiguring-cluster-nodes) Upgrade procedure for other VictoriaMetrics components is as simple as gracefully stopping the component by sending `SIGINT` signal to it and starting the new version of the component. There may be breaking changes between different versions of VictoriaMetrics components in rare cases. - These cases are documented in [the changelog](./CHANGELOG.md). + These cases are documented in [the changelog](https://docs.victoriametrics.com/changelog/). So please read the changelog before the upgrade. 1. Inspect command-line flags passed to VictoriaMetrics components and remove flags which unclear outcomes for your workload. @@ -78,7 +78,7 @@ then please follow the following steps in order to quickly find the solution: before posting your question to VictoriaMetrics Slack chat, since GitHub issues are indexed by Google, while Slack messages aren't indexed by Google. This simplifies searching for the solution to the issue for future VictoriaMetrics users. -1. Pro tip 1: if you see that [VictoriaMetrics docs](./Single-server-VictoriaMetrics.md) contain incomplete or incorrect information, +1. Pro tip 1: if you see that [VictoriaMetrics docs](https://docs.victoriametrics.com/) contain incomplete or incorrect information, then please create a pull request with the relevant changes. This will help VictoriaMetrics community. All the docs published at `https://docs.victoriametrics.com` are located in the [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs) @@ -121,12 +121,12 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try to reduce the number of returned series. Sometimes the query may be improperly constructed, so it returns unexpected results. - It is recommended reading and understanding [MetricsQL docs](./MetricsQL.md), - especially [subqueries](./MetricsQL.md#subqueries) - and [rollup functions](./MetricsQL.md#rollup-functions) sections. + It is recommended reading and understanding [MetricsQL docs](https://docs.victoriametrics.com/metricsql/), + especially [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries) + and [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) sections. 1. If the simplest query continues returning unexpected / unreliable results, then try verifying correctness - of raw unprocessed samples for this query via [/api/v1/export](./README.md#how-to-export-data-in-json-line-format) + of raw unprocessed samples for this query via [/api/v1/export](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format) on the given `[start..end]` time range and check whether they are expected: ```sh @@ -134,16 +134,16 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try cluster: curl http://:8481/select//prometheus/api/v1/export -d 'match[]=http_requests_total' -d 'start=...' -d 'end=...' ``` - Note that responses returned from [/api/v1/query](./keyConcepts.md#instant-query) - and from [/api/v1/query_range](./keyConcepts.md#range-query) contain **evaluated** data + Note that responses returned from [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query) + and from [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query) contain **evaluated** data instead of raw samples stored in VictoriaMetrics. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness) for details. If you migrate from InfluxDB, then pass `-search.setLookbackToStep` command-line flag to single-node VictoriaMetrics - or to `vmselect` in VictoriaMetrics cluster. See also [how to migrate from InfluxDB to VictoriaMetrics](./guides/migrate-from-influx.md). + or to `vmselect` in VictoriaMetrics cluster. See also [how to migrate from InfluxDB to VictoriaMetrics](https://docs.victoriametrics.com/guides/migrate-from-influx.html). 1. Sometimes response caching may lead to unexpected results when samples with older timestamps - are ingested into VictoriaMetrics (aka [backfilling](./README.md#backfilling)). + are ingested into VictoriaMetrics (aka [backfilling](https://docs.victoriametrics.com/#backfilling)). Try disabling response cache and see whether this helps. This can be done in the following ways: - By passing `-search.disableCache` command-line flag to a single-node VictoriaMetrics @@ -153,10 +153,10 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try If you use Grafana, then this query arg can be specified in `Custom Query Parameters` field at Prometheus datasource settings - see [these docs](https://grafana.com/docs/grafana/latest/datasources/prometheus/) for details. - If the problem was in the cache, try resetting it via [resetRollupCache handler](./url-examples.md#internalresetrollupresultcache). + If the problem was in the cache, try resetting it via [resetRollupCache handler](https://docs.victoriametrics.com/url-examples/#internalresetrollupresultcache). 1. If you use cluster version of VictoriaMetrics, then it may return partial responses by default - when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](./Cluster-VictoriaMetrics.md#cluster-availability) + when some of `vmstorage` nodes are temporarily unavailable - see [cluster availability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-availability) for details. If you want to prioritize query consistency over cluster availability, then you can pass `-search.denyPartialResponse` command-line flag to all the `vmselect` nodes. In this case VictoriaMetrics returns an error during querying if at least a single `vmstorage` node is unavailable. @@ -171,20 +171,20 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try 1. If you observe gaps when plotting time series try simplifying your query according to p2 and follow the list. If problem still remains, then it is likely caused by irregular intervals for metrics collection (network delays or targets unavailability on scrapes, irregular pushes, irregular timestamps). - VictoriaMetrics automatically [fills the gaps](./keyConcepts.md#range-query) - based on median interval between [data samples](./keyConcepts.md#raw-samples). + VictoriaMetrics automatically [fills the gaps](https://docs.victoriametrics.com/keyconcepts/#range-query) + based on median interval between [data samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples). This might work incorrect for irregular data as median will be skewed. In this case it is recommended to switch to the static interval for gaps filling by setting `-search.minStalenessInterval=5m` cmd-line flag (`5m` is the static interval used by Prometheus). 1. If you observe recently written data is not immediately visible/queryable, then read more about - [query latency](./keyConcepts.md#query-latency) behavior. + [query latency](https://docs.victoriametrics.com/keyconcepts/#query-latency) behavior. 1. Try upgrading to the [latest available version of VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) and verifying whether the issue is fixed there. 1. Try executing the query with `trace=1` query arg. This enables query tracing, which may contain - useful information on why the query returns unexpected data. See [query tracing docs](./README.md#query-tracing) for details. + useful information on why the query returns unexpected data. See [query tracing docs](https://docs.victoriametrics.com/#query-tracing) for details. 1. Inspect command-line flags passed to VictoriaMetrics components. If you don't understand clearly the purpose or the effect of some flags, then remove them from the list of flags passed to VictoriaMetrics components, @@ -193,15 +193,15 @@ If you see unexpected or unreliable query results from VictoriaMetrics, then try 1. If the steps above didn't help identifying the root cause of unexpected query results, then [file a bugreport](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/new) with details on how to reproduce the issue. - Instead of sharing screenshots in the issue, consider sharing query and [trace](./README.md#query-tracing) - results in [VMUI](./README.md#vmui) by clicking on `Export query` button in top right corner of the graph area. + Instead of sharing screenshots in the issue, consider sharing query and [trace](https://docs.victoriametrics.com/#query-tracing) + results in [VMUI](https://docs.victoriametrics.com/#vmui) by clicking on `Export query` button in top right corner of the graph area. ## Slow data ingestion There are the following most commons reasons for slow data ingestion in VictoriaMetrics: -1. Memory shortage for the given amounts of [active time series](./FAQ.md#what-is-an-active-time-series). +1. Memory shortage for the given amounts of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series). VictoriaMetrics (or `vmstorage` in cluster version of VictoriaMetrics) maintains an in-memory cache for quick search for internal series ids per each incoming metric. @@ -210,10 +210,10 @@ There are the following most commons reasons for slow data ingestion in Victoria for holding all the entries for active time series, then VictoriaMetrics locates the needed data on disk, unpacks it, re-constructs the missing entry and puts it into the cache. This takes additional CPU time and disk read IO. - The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring) + The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) contain `Slow inserts` graph, which shows the cache miss percentage for `storage/tsid` cache during data ingestion. If `slow inserts` graph shows values greater than 5% for more than 10 minutes, - then it is likely the current number of [active time series](./FAQ.md#what-is-an-active-time-series) + then it is likely the current number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series) cannot fit the `storage/tsid` cache. There are the following solutions exist for this issue: @@ -223,28 +223,28 @@ There are the following most commons reasons for slow data ingestion in Victoria memory at `vmstorage` nodes. This can be done in two ways: either to increase the available memory per each existing `vmstorage` node or to add more `vmstorage` nodes to the cluster. - - To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring) + - To reduce the number of active time series. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) contain a graph showing the number of active time series. Recent versions of VictoriaMetrics - provide [cardinality explorer](./README.md#cardinality-explorer), - which can help determining and fixing the source of [high cardinality](./FAQ.md#what-is-high-cardinality). + provide [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer), + which can help determining and fixing the source of [high cardinality](https://docs.victoriametrics.com/faq/#what-is-high-cardinality). -1. [High churn rate](./FAQ.md#what-is-high-churn-rate), +1. [High churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate), e.g. when old time series are substituted with new time series at a high rate. When VictoriaMetrics encounters a sample for new time series, it needs to register the time series in the internal index (aka `indexdb`), so it can be quickly located on subsequent select queries. The process of registering new time series in the internal index is an order of magnitude slower than the process of adding new sample to already registered time series. - So VictoriaMetrics may work slower than expected under [high churn rate](./FAQ.md#what-is-high-churn-rate). + So VictoriaMetrics may work slower than expected under [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). - The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring) + The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) provides `Churn rate` graph, which shows the average number of new time series registered - during the last 24 hours. If this number exceeds the number of [active time series](./FAQ.md#what-is-an-active-time-series), - then you need to identify and fix the source of [high churn rate](./FAQ.md#what-is-high-churn-rate). + during the last 24 hours. If this number exceeds the number of [active time series](https://docs.victoriametrics.com/faq/#what-is-an-active-time-series), + then you need to identify and fix the source of [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate). The most commons source of high churn rate is a label, which frequently changes its value. Try avoiding such labels. - The [cardinality explorer](./README.md#cardinality-explorer) can help identifying + The [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer) can help identifying such labels. -1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](./README.md#monitoring) +1. Resource shortage. The [official Grafana dashboards for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) contain `resource usage` graphs, which show memory usage, CPU usage, disk IO usage and free disk size. Make sure VictoriaMetrics has enough free resources for graceful handling of potential spikes in workload according to the following recommendations: @@ -270,7 +270,7 @@ There are the following most commons reasons for slow data ingestion in Victoria - If free disk space is lower than 20%, then VictoriaMetrics is unable to perform optimal background merge of the incoming data. This leads to increased number of data files on disk, - which, in turn, slows down both data ingestion and querying. See [these docs](./README.md#storage) for details. + which, in turn, slows down both data ingestion and querying. See [these docs](https://docs.victoriametrics.com/#storage) for details. 1. If you run cluster version of VictoriaMetrics, then make sure `vminsert` and `vmstorage` components are located in the same network with small network latency between them. @@ -279,7 +279,7 @@ There are the following most commons reasons for slow data ingestion in Victoria If the network latency between `vminsert` and `vmstorage` is high (for example, if they run in different datacenters), then this may become limiting factor for data ingestion speed. - The [official Grafana dashboard for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring) + The [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) contain `connection saturation` graph for `vminsert` components. If this graph reaches 100% (1s), then it is likely you have issues with network latency between `vminsert` and `vmstorage`. Another possible issue for 100% connection saturation between `vminsert` and `vmstorage` @@ -288,10 +288,10 @@ There are the following most commons reasons for slow data ingestion in Victoria 1. Noisy neighbor. Make sure VictoriaMetrics components run in an environments without other resource-hungry apps. Such apps may steal RAM, CPU, disk IO and network bandwidth, which is needed for VictoriaMetrics components. - Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring) + Issues like this are very hard to catch via [official Grafana dashboard for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) and proper diagnosis would require checking resource usage on the instances where VictoriaMetrics runs. -1. If you see `TooHighSlowInsertsRate` [alert](./README.md#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough +1. If you see `TooHighSlowInsertsRate` [alert](https://docs.victoriametrics.com/#monitoring) when single-node VictoriaMetrics or `vmstorage` has enough free CPU and RAM, then increase `-cacheExpireDuration` command-line flag at single-node VictoriaMetrics or at `vmstorage` to the value, which exceeds the interval between ingested samples for the same time series (aka `scrape_interval`). See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183) for more details. @@ -308,7 +308,7 @@ Some queries may take more time and resources (CPU, RAM, network bandwidth) than VictoriaMetrics logs slow queries if their execution time exceeds the duration passed to `-search.logSlowQueryDuration` command-line flag (5s by default). -VictoriaMetrics provides [`top queries` page at VMUI](./README.md#top-queries), which shows +VictoriaMetrics provides [`top queries` page at VMUI](https://docs.victoriametrics.com/#top-queries), which shows queries that took the most time to execute. There are the following solutions exist for improving performance of slow queries: @@ -318,33 +318,33 @@ There are the following solutions exist for improving performance of slow querie with more CPU and RAM should help improving speed for slow queries. Query performance is always limited by resources of one `vmselect` which processes the query. For example, if 2vCPU cores on `vmselect` isn't enough to process query fast enough, then migrating `vmselect` to a machine with 4vCPU cores should increase heavy query performance by up to 2x. - If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](./Cluster-VictoriaMetrics.md#monitoring) + If the line on `concurrent select` graph form the [official Grafana dashboard for VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#monitoring) is close to the limit, then prefer adding more `vmselect` nodes to the cluster. Sometimes adding more `vmstorage` nodes also can help improving the speed for slow queries. - Rewriting slow queries, so they become faster. Unfortunately it is hard determining whether the given query is slow by just looking at it. - The main source of slow queries in practice is [alerting and recording rules](./vmalert.md#rules) + The main source of slow queries in practice is [alerting and recording rules](https://docs.victoriametrics.com/vmalert/#rules) with long lookbehind windows in square brackets. These queries are frequently used in SLI/SLO calculations such as [Sloth](https://github.com/slok/sloth). For example, `avg_over_time(up[30d]) > 0.99` needs to read and process - all the [raw samples](./keyConcepts.md#raw-samples) - for `up` [time series](./keyConcepts.md#time-series) over the last 30 days + all the [raw samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) + for `up` [time series](https://docs.victoriametrics.com/keyconcepts/#time-series) over the last 30 days each time it executes. If this query is executed frequently, then it can take significant share of CPU, disk read IO, network bandwidth and RAM. Such queries can be optimized in the following ways: - To reduce the lookbehind window in square brackets. For example, `avg_over_time(up[10d])` takes up to 3x less compute resources than `avg_over_time(up[30d])` at VictoriaMetrics. - To increase evaluation interval for alerting and recording rules, so they are executed less frequently. - For example, increasing `-evaluationInterval` command-line flag value at [vmalert](./vmalert.md) + For example, increasing `-evaluationInterval` command-line flag value at [vmalert](https://docs.victoriametrics.com/vmalert/) from `1m` to `2m` should reduce compute resource usage at VictoriaMetrics by 2x. - Another source of slow queries is improper use of [subqueries](./MetricsQL.md#subqueries). + Another source of slow queries is improper use of [subqueries](https://docs.victoriametrics.com/metricsql/#subqueries). It is recommended avoiding subqueries if you don't understand clearly how they work. It is easy to create a subquery without knowing about it. For example, `rate(sum(some_metric))` is implicitly transformed into the following subquery - according to [implicit conversion rules for MetricsQL queries](./MetricsQL.md#implicit-query-conversions): + according to [implicit conversion rules for MetricsQL queries](https://docs.victoriametrics.com/metricsql/#implicit-query-conversions): ```metricsql rate( @@ -357,7 +357,7 @@ There are the following solutions exist for improving performance of slow querie It is likely this query won't return the expected results. Instead, `sum(rate(some_metric))` must be used instead. See [this article](https://www.robustperception.io/rate-then-sum-never-sum-then-rate/) for more details. - VictoriaMetrics provides [query tracing](./README.md#query-tracing) feature, + VictoriaMetrics provides [query tracing](https://docs.victoriametrics.com/#query-tracing) feature, which can help determining the source of slow query. See also [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986), which explains how to determine and optimize slow queries. @@ -374,24 +374,24 @@ There are the following most common sources of out of memory (aka OOM) crashes i VictoriaMetrics is optimized for running with default flag values (e.g. when they aren't set explicitly). For example, it isn't recommended tuning cache sizes in VictoriaMetrics, since it frequently leads to OOM exceptions. - [These docs](./README.md#cache-tuning) refer command-line flags, which aren't + [These docs](https://docs.victoriametrics.com/#cache-tuning) refer command-line flags, which aren't recommended to tune. If you see that VictoriaMetrics needs increasing some cache sizes for the current workload, then it is better migrating to a host with more memory instead of trying to tune cache sizes manually. 1. Unexpected heavy queries. The query is considered as heavy if it needs to select and process millions of unique time series. Such query may lead to OOM exception, since VictoriaMetrics needs to keep some of per-series data in memory. - VictoriaMetrics provides [various settings](./README.md#resource-usage-limits), + VictoriaMetrics provides [various settings](https://docs.victoriametrics.com/#resource-usage-limits), which can help limit resource usage. For more context, see [How to optimize PromQL and MetricsQL queries](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986). - VictoriaMetrics also provides [query tracer](./README.md#query-tracing) + VictoriaMetrics also provides [query tracer](https://docs.victoriametrics.com/#query-tracing) to help identify the source of heavy query. 1. Lack of free memory for processing workload spikes. If VictoriaMetrics components use almost all the available memory under the current workload, then it is recommended migrating to a host with bigger amounts of memory. This would protect from possible OOM crashes on workload spikes. It is recommended to have at least 50% of free memory for graceful handling of possible workload spikes. - See [capacity planning for single-node VictoriaMetrics](./README.md#capacity-planning) - and [capacity planning for cluster version of VictoriaMetrics](./Cluster-VictoriaMetrics.md#capacity-planning). + See [capacity planning for single-node VictoriaMetrics](https://docs.victoriametrics.com/#capacity-planning) + and [capacity planning for cluster version of VictoriaMetrics](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning). ## Cluster instability @@ -405,7 +405,7 @@ The most common sources of cluster instability are: the cluster has no enough free resources for processing the increased workload, then it may become unstable. VictoriaMetrics provides various configuration settings, which can be used for limiting unexpected workload spikes. - See [these docs](./Cluster-VictoriaMetrics.md#resource-usage-limits) for details. + See [these docs](https://docs.victoriametrics.com/cluster-victoriametrics/#resource-usage-limits) for details. - Various maintenance tasks such as rolling upgrades or rolling restarts during configuration changes. For example, if a cluster contains `N=3` `vmstorage` nodes and they are restarted one-by-one (aka rolling restart), @@ -423,10 +423,10 @@ The most common sources of cluster instability are: `N=11` `vmstorage` nodes, then the workload increase during rolling restart of `vmstorage` nodes would be `100%/(N-1)=10%`. It is recommended to have at least 8 `vmstorage` nodes in the cluster. The recommended number of `vmstorage` nodes should be multiplied by `-replicationFactor` if replication is enabled - - see [replication and data safety docs](./Cluster-VictoriaMetrics.md#replication-and-data-safety) + see [replication and data safety docs](https://docs.victoriametrics.com/cluster-victoriametrics/#replication-and-data-safety) for details. -- Time series sharding. Received time series [are consistently sharded](./Cluster-VictoriaMetrics.md#architecture-overview) +- Time series sharding. Received time series [are consistently sharded](https://docs.victoriametrics.com/cluster-victoriametrics/#architecture-overview) by `vminsert` between configured `vmstorage` nodes. As a sharding key `vminsert` is using time series name and labels, respecting their order. If the order of labels in time series is constantly changing, this could cause wrong sharding calculation and result in un-even and sub-optimal time series distribution across available vmstorages. It is expected @@ -436,18 +436,18 @@ The most common sources of cluster instability are: The obvious solution against VictoriaMetrics cluster instability is to make sure cluster components have enough free resources for graceful processing of the increased workload. -See [capacity planning docs](./Cluster-VictoriaMetrics.md#capacity-planning) -and [cluster resizing and scalability docs](./Cluster-VictoriaMetrics.md#cluster-resizing-and-scalability) +See [capacity planning docs](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning) +and [cluster resizing and scalability docs](https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability) for details. ## Too much disk space used -If too much disk space is used by a [single-node VictoriaMetrics](./Single-server-VictoriaMetrics.md) or by `vmstorage` component -at [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md), then please check the following: +If too much disk space is used by a [single-node VictoriaMetrics](https://docs.victoriametrics.com/) or by `vmstorage` component +at [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/), then please check the following: -- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](./README.md#how-to-work-with-snapshots) - and [snapshot troubleshooting](./README.md#snapshot-troubleshooting). +- Make sure that there are no old snapsots, since they can occupy disk space. See [how to work with snapshots](https://docs.victoriametrics.com/#how-to-work-with-snapshots) + and [snapshot troubleshooting](https://docs.victoriametrics.com/#snapshot-troubleshooting). - Under normal conditions the size of `<-storageDataPath>/indexdb` folder must be smaller than the size of `<-storageDataPath>/data` folder, where `-storageDataPath` is the corresponding command-line flag value. This can be checked by the following query if [VictoriaMetrics monitoring](#monitoring) is properly set up: @@ -456,14 +456,14 @@ at [VictoriaMetrics cluster](./Cluster-VictoriaMetrics.md), then please check th / sum(vm_data_size_bytes{type=~"(storage|indexdb)/.+"}) without(type) ``` - If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](./FAQ.md#what-is-high-churn-rate) issue, + If this query returns values bigger than 0.5, then it is likely there is a [high churn rate](https://docs.victoriametrics.com/faq/#what-is-high-churn-rate) issue, which results in excess disk space usage for both `indexdb` and `data` folders under `-storageDataPath` folder. - The solution is to identify and fix the source of high churn rate with [cardinality explorer](./README.md#cardinality-explorer). + The solution is to identify and fix the source of high churn rate with [cardinality explorer](https://docs.victoriametrics.com/#cardinality-explorer). ## Monitoring -Having proper [monitoring](./Single-Server-VictoriaMetrics.md#monitoring) +Having proper [monitoring](https://docs.victoriametrics.com/single-server-victoriametrics/#monitoring) would help identify and prevent most of the issues listed above. [Grafana dashboards](https://grafana.com/orgs/victoriametrics/dashboards) contain panels reflecting the diff --git a/docs/goals.md b/docs/goals.md index d84c9b1e8..64d6a3f1a 100644 --- a/docs/goals.md +++ b/docs/goals.md @@ -9,15 +9,15 @@ menu: --- ## Goals -1. The main goal - **to help users and [clients](./enterprise.md) resolving issues with VictoriaMetrics components, +1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/enterprise/) resolving issues with VictoriaMetrics components, so they could use these components in the most efficient way**. 1. Fixing bugs in the essential functionality of VictoriaMetrics components. Small usability bugs are usually the most annoying, so they **must be fixed first**. 1. Improving [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/docs) for VictoriaMetrics components, so users could find answers to their questions via Google or [Perplexity](https://www.perplexity.ai/) without the need - to ask these questions at our [support channels](./#community-and-contributions). + to ask these questions at our [support channels](https://docs.victoriametrics.com/#community-and-contributions). 1. Simplifying usage of VictoriaMetrics components without breaking backwards compatibility, so users could regularly - upgrade to [the latest available release](./CHANGELOG.md) and remain happy. + upgrade to [the latest available release](https://docs.victoriametrics.com/CHANGELOG.html) and remain happy. 1. Improving usability for the existing functionality of VictoriaMetrics components. 1. Improving the readability and maintainability of the code base by removing unnecessary abstractions and simplifying the code whenever possible. 1. Improving development velocity by optimizing and simplifying CI/CD tasks, so they take less time to execute and debug. @@ -26,7 +26,7 @@ menu: 1. Convincing people to use VictoriaMetrics components when there are better suited solutions exist for their tasks, since users will become angry at VictoriaMetrics after they discover better solutions. -1. Breaking links to [VictoriaMetrics docs](./Single-server-VictoriaMetrics.md), since users will be unhappy seeing 404 page +1. Breaking links to [VictoriaMetrics docs](https://docs.victoriametrics.com/), since users will be unhappy seeing 404 page or unexpected results after they click some old link somewhere on the Internet or in the internal knowledge base. 1. Breaking backwards compatibility in new releases, since users will be unhappy when their working setup breaks after the upgrade. 1. Adding non-trivial features, which require significant changes in the code and the architecture, @@ -50,7 +50,7 @@ menu: with VictoriaMetrics components in the most efficient way. Nobody uses new shiny feature if it isn't documented properly. - **Happy users are more important than the momentary revenue.** Happy users spread the word about VictoriaMetrics, - so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](./enterprise.md). + so more people convert to VictoriaMetrics users. Happy users are eager to become happy [customers](https://docs.victoriametrics.com/enterprise/). This increases long-term revenue. - **Simple solution is better than smart solution.** Simple solution is easier to setup, operate, debug and troubleshoot than the smart solution. diff --git a/docs/guides/migrate-from-influx.md b/docs/guides/migrate-from-influx.md index 441d1f30a..998051edd 100644 --- a/docs/guides/migrate-from-influx.md +++ b/docs/guides/migrate-from-influx.md @@ -18,8 +18,8 @@ sometimes old known solutions just can't keep up with the new expectations. VictoriaMetrics is a high-performance opensource time series database specifically designed to deal with huge volumes of monitoring data while remaining cost-efficient at the same time. Many companies are choosing to migrate from InfluxDB to VictoriaMetrics specifically for performance and scalability reasons. Along them see case studies provided by -[ARNES](./CaseStudies.md#arnes) -and [Brandwatch](./CaseStudies.md#brandwatch). +[ARNES](https://docs.victoriametrics.com/casestudies/#arnes) +and [Brandwatch](https://docs.victoriametrics.com/casestudies/#brandwatch). This guide will cover the differences between two solutions, most commonly asked questions, and approaches for migrating from InfluxDB to VictoriaMetrics. @@ -28,13 +28,13 @@ from InfluxDB to VictoriaMetrics. While readers are likely familiar with [InfluxDB key concepts](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/), the data model of -VictoriaMetrics is something [new to explore](./keyConcepts.md#data-model). Let's start +VictoriaMetrics is something [new to explore](https://docs.victoriametrics.com/keyconcepts/#data-model). Let's start with similarities and differences: * both solutions are **schemaless**, which means there is no need to define metrics or their tags in advance; * multidimensional data support is implemented via [tags](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#tags) - in InfluxDB and via [labels](./keyConcepts.md#structure-of-a-metric) in + in InfluxDB and via [labels](https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric) in VictoriaMetrics. However, labels in VictoriaMetrics are always `strings`, while InfluxDB supports multiple data types; * timestamps are stored with nanosecond resolution in InfluxDB, while in VictoriaMetrics it is **milliseconds**; * in VictoriaMetrics metric value is always `float64`, while InfluxDB supports multiple data types. @@ -47,8 +47,8 @@ with similarities and differences: [buckets](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#bucket) or [organizations](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#organization). All data in VictoriaMetrics is stored in a global namespace or within - a [tenant](./Cluster-VictoriaMetrics.md#multitenancy). - See more about multi-tenancy [here](./keyConcepts.md#multi-tenancy). + a [tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy). + See more about multi-tenancy [here](https://docs.victoriametrics.com/keyconcepts/#multi-tenancy). Let's consider the following [sample data](https://docs.influxdata.com/influxdb/v2.2/reference/key-concepts/data-elements/#sample-data) @@ -78,7 +78,7 @@ VictoriaMetrics, so lookups by names or labels have the same query speed. ## Write data VictoriaMetrics -supports [InfluxDB line protocol](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) +supports [InfluxDB line protocol](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) for data ingestion. For example, to write a measurement to VictoriaMetrics we need to send an HTTP POST request with payload in a line protocol format: @@ -116,7 +116,7 @@ The expected response is the following: ``` Please note, VictoriaMetrics performed additional -[data mapping](./#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) +[data mapping](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) to the data ingested via InfluxDB line protocol. Support of InfluxDB line protocol also means VictoriaMetrics is compatible with @@ -129,20 +129,20 @@ add `http://:8428` URL to Telegraf configs: ``` In addition to InfluxDB line protocol, VictoriaMetrics supports many other ways for -[metrics collection](./keyConcepts.md#write-data). +[metrics collection](https://docs.victoriametrics.com/keyconcepts/#write-data). ## Query data VictoriaMetrics does not have a command-line interface (CLI). Instead, it provides -an [HTTP API](./Single-Server-VictoriaMetrics.md#prometheus-querying-api-usage) +an [HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage) for serving read queries. This API is used in various integrations such as -[Grafana](./Single-Server-VictoriaMetrics.md#grafana-setup). The same API is also used -by [VMUI](./Single-Server-VictoriaMetrics.md#vmui) - a graphical User Interface for +[Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup). The same API is also used +by [VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for querying and visualizing metrics: ![Migrate from Influx](migrate-from-influx_vmui.webp) -See more about [how to query data in VictoriaMetrics](./keyConcepts.md#query-data). +See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyconcepts/#query-data). ### Basic concepts @@ -186,19 +186,19 @@ Having this, let's import the same data sample in VictoriaMetrics and plot it in InfluxQL query might be translated to MetricsQL let's break it into components first: * `SELECT last("bar") FROM "foo"` - all requests - to [instant](./keyConcepts.md#instant-query) - or [range](./keyConcepts.md#range-query) VictoriaMetrics APIs are reads, so no need + to [instant](https://docs.victoriametrics.com/keyconcepts/#instant-query) + or [range](https://docs.victoriametrics.com/keyconcepts/#range-query) VictoriaMetrics APIs are reads, so no need to specify the `SELECT` statement. There are no `measurements` or `fields` in VictoriaMetrics, so the whole expression can be replaced with `foo_bar` in MetricsQL; -* `WHERE ("instance" = 'localhost')`- [filtering by labels](./keyConcepts.md#filtering) +* `WHERE ("instance" = 'localhost')`- [filtering by labels](https://docs.victoriametrics.com/keyconcepts/#filtering) in MetricsQL requires specifying the filter in curly braces next to the metric name. So in MetricsQL filter expression will be translated to `{instance="localhost"}`; * `WHERE $timeFilter` - filtering by time is done via request params sent along with query, so in MetricsQL no need to specify this filter; * `GROUP BY time(1m)` - grouping by time is done by default - in [range](./keyConcepts.md#range-query) API according to specified `step` param. + in [range](https://docs.victoriametrics.com/keyconcepts/#range-query) API according to specified `step` param. This param is also a part of params sent along with request. See how to perform additional - [aggregations and grouping via MetricsQL](./keyConcepts.md#aggregation-and-grouping-functions) + [aggregations and grouping via MetricsQL](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions) . In result, executing the `foo_bar{instance="localhost"}` MetricsQL expression with `step=1m` for the same set of data in @@ -208,13 +208,13 @@ Grafana will have the following form: Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points filling the gaps in the graph. This behavior is described in more -detail [here](./keyConcepts.md#range-query). In InfluxDB, we can achieve a similar +detail [here](https://docs.victoriametrics.com/keyconcepts/#range-query). In InfluxDB, we can achieve a similar behavior by adding `fill(previous)` to the query. VictoriaMetrics fills the gaps on the graph assuming time series are always continuous and not discrete. To limit the interval on which VictoriaMetrics will try to fill the gaps, set `-search.setLookbackToStep` command-line flag. This limits the gap filling to a single `step` interval passed to -[/api/v1/query_range](./keyConcepts.md#range-query). +[/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). This behavior is close to InfluxDB data model. @@ -227,56 +227,56 @@ about 230 PromQL queries in it! But a closer look at those queries shows the fol * ~120 queries are just selecting a metric with label filters, e.g. `node_textfile_scrape_error{instance="$node",job="$job"}`; -* ~80 queries are using [rate](./MetricsQL.md#rate) function for selected metric, +* ~80 queries are using [rate](https://docs.victoriametrics.com/metricsql/#rate) function for selected metric, e.g. `rate(node_netstat_Tcp_InSegs{instance=\"$node\",job=\"$job\"})` * and the rest - are [aggregation functions](./keyConcepts.md#aggregation-and-grouping-functions) - like [sum](./MetricsQL.md#sum) - or [count](./MetricsQL.md#count). + are [aggregation functions](https://docs.victoriametrics.com/keyconcepts/#aggregation-and-grouping-functions) + like [sum](https://docs.victoriametrics.com/metricsql/#sum) + or [count](https://docs.victoriametrics.com/metricsql/#count). To get a better understanding of how MetricsQL works, see the following resources: -* [MetricsQL concepts](./keyConcepts.md#metricsql); -* [MetricsQL functions](./MetricsQL.md); +* [MetricsQL concepts](https://docs.victoriametrics.com/keyconcepts/#metricsql); +* [MetricsQL functions](https://docs.victoriametrics.com/metricsql/); * [PromQL tutorial for beginners](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085). ## How to migrate current data from InfluxDB to VictoriaMetrics Migrating data from other TSDBs to VictoriaMetrics is as simple as importing data via any of -[supported formats](./keyConcepts.md#push-model). +[supported formats](https://docs.victoriametrics.com/keyconcepts/#push-model). -But migration from InfluxDB might get easier when using [vmctl](./vmctl.md) - +But migration from InfluxDB might get easier when using [vmctl](https://docs.victoriametrics.com/vmctl/) - VictoriaMetrics command-line tool. See more about -migrating [from InfluxDB v1.x versions](./vmctl.md#migrating-data-from-influxdb-1x). +migrating [from InfluxDB v1.x versions](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x). Migrating data from InfluxDB v2.x is not supported yet. But there is -useful [3rd party solution](./vmctl.md#migrating-data-from-influxdb-2x) for this. +useful [3rd party solution](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-2x) for this. Please note, that data migration is a backfilling process. So, please -consider [backfilling tips](./Single-Server-VictoriaMetrics.md#backfilling). +consider [backfilling tips](https://docs.victoriametrics.com/single-server-victoriametrics/#backfilling). ## Frequently asked questions * How does VictoriaMetrics compare to InfluxDB? - * _[Answer](./FAQ.md#how-does-victoriametrics-compare-to-influxdb)_ + * _[Answer](https://docs.victoriametrics.com/faq/#how-does-victoriametrics-compare-to-influxdb)_ * Why don't VictoriaMetrics support Remote Read API, so I don't need to learn MetricsQL? - * _[Answer](./FAQ.md#why-doesnt-victoriametrics-support-the-prometheus-remote-read-api)_ + * _[Answer](https://docs.victoriametrics.com/faq/#why-doesnt-victoriametrics-support-the-prometheus-remote-read-api)_ * The PromQL and MetricsQL are often mentioned together - why is that? * _MetricsQL - query language inspired by PromQL. MetricsQL is backward-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work the same after switching from Prometheus to VictoriaMetrics. Both languages mostly share the same concepts with slight differences._ * Query returns more data points than expected - why? * _VictoriaMetrics may return non-existing data points if `step` param is lower than the actual data resolution. See - more about this [here](./keyConcepts.md#range-query)._ + more about this [here](https://docs.victoriametrics.com/keyconcepts/#range-query)._ * How do I get the `real` last data point, not `ephemeral`? - * _[last_over_time](./MetricsQL.md#last_over_time) function can be used for + * _[last_over_time](https://docs.victoriametrics.com/metricsql/#last_over_time) function can be used for limiting the lookbehind window for calculated data. For example, `last_over_time(metric[10s])` would return calculated samples only if the real samples are located closer than 10 seconds to the calculated timestamps according to `start`, `end` and `step` query args passed - to [range query](./keyConcepts.md#range-query)._ + to [range query](https://docs.victoriametrics.com/keyconcepts/#range-query)._ * How do I get raw data points with MetricsQL? * _For getting raw data points specify the interval at which you want them in square brackets and send - as [instant query](./keyConcepts.md#instant-query). For + as [instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query). For example, `GET api/v1/query?query=my_metric[5m]&time=