github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

view documentation locally (#6677)

Browse source 
- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit is contained in:
Andrii Chubatiuk 2024-07-24 11:00:31 +03:00 committed by Aliaksandr Valialkin
parent 4a01709b90
commit 6b97044d8a
No known key found for this signature in database
GPG key ID: 52C003EE2BCDB9EB
94 changed files with 1316 additions and 910 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
Makefile
View file

@ -270,8 +270,9 @@ copy-docs:
fi fi
echo "---" >> ${DST} echo "---" >> ${DST}
cat ${SRC} >> ${DST} cat ${SRC} >> ${DST}
sed -i='.tmp' 's/<img src=\"docs\//<img src=\"/' ${DST} sed -i='.tmp' 's/<img src=\"docs\//<img src=\"\//' ${DST}
sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"/' ${DST} sed -i='.tmp' 's/<source srcset=\"docs\//<source srcset=\"\//' ${DST}
sed -i='.tmp' 's/](docs\//](/' ${DST}
rm -rf docs/*.tmp rm -rf docs/*.tmp
# Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site. # Copies docs for all components and adds the order/weight tag, title, menu position and alias with the backward compatible link for the old site.

4
README.md
View file

@ -1,5 +1,3 @@
# Cluster version
<picture> <picture>
<source srcset="docs/logo_white.webp" media="(prefers-color-scheme: dark)"> <source srcset="docs/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="docs/logo.webp" media="(prefers-color-scheme: light)"> <source srcset="docs/logo.webp" media="(prefers-color-scheme: light)">
@ -50,7 +48,7 @@ Each service may scale independently and may run on the most suitable hardware.
This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture). This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture).
It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling. It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling.
<img src="docs/Cluster-VictoriaMetrics_cluster-scheme.webp"> ![Cluster Scheme](docs/Cluster-VictoriaMetrics_cluster-scheme.webp)
## Multitenancy ## Multitenancy

3
docs/Articles.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /Articles.html - /Articles.html
--- ---
# Articles
See also [case studies](https://docs.victoriametrics.com/casestudies/). See also [case studies](https://docs.victoriametrics.com/casestudies/).
## Third-party articles and slides about VictoriaMetrics ## Third-party articles and slides about VictoriaMetrics

3
docs/BestPractices.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /BestPractices.html - /BestPractices.html
--- ---
# VictoriaMetrics best practices
## Install Recommendation ## Install Recommendation
It is recommended running the latest available release of VictoriaMetrics from [this page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest), It is recommended running the latest available release of VictoriaMetrics from [this page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest),

2
docs/CHANGELOG.md
View file

@ -9,8 +9,6 @@ menu:
aliases: aliases:
- /CHANGELOG.html - /CHANGELOG.html
--- ---
# CHANGELOG
The following `tip` changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs: The following `tip` changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:
* [How to build single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-build-from-sources) * [How to build single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-build-from-sources)

3
docs/CHANGELOG_2020.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CHANGELOG_2020.html - /CHANGELOG_2020.html
--- ---
# CHANGELOG for the year 2020
## [v1.51.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.51.0) ## [v1.51.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.51.0)
Released at 2020-12-27 Released at 2020-12-27

3
docs/CHANGELOG_2021.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CHANGELOG_2021.html - /CHANGELOG_2021.html
--- ---
# CHANGELOG for the year 2021
## [v1.71.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.71.0) ## [v1.71.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.71.0)
Released at 2021-12-20 Released at 2021-12-20

3
docs/CHANGELOG_2022.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CHANGELOG_2022.html - /CHANGELOG_2022.html
--- ---
# CHANGELOG for the year 2022
## [v1.85.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.3) ## [v1.85.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.3)
Released at 2022-12-20 Released at 2022-12-20

3
docs/CHANGELOG_2023.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CHANGELOG_2023.html - /CHANGELOG_2023.html
--- ---
# CHANGELOG for the year 2023
## [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0) ## [v1.96.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0)
Released at 2023-12-13 Released at 2023-12-13

3
docs/CONTRIBUTING.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CONTRIBUTING.html - /CONTRIBUTING.html
--- ---
# Contributing to VictoriaMetrics
If you like VictoriaMetrics and want to contribute, then it would be great: If you like VictoriaMetrics and want to contribute, then it would be great:
- Joining VictoriaMetrics community Slack ([Slack inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/)) - Joining VictoriaMetrics community Slack ([Slack inviter](https://slack.victoriametrics.com/) and [Slack channel](https://victoriametrics.slack.com/))

3
docs/CaseStudies.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /CaseStudies.html - /CaseStudies.html
--- ---
# Case studies and talks
Below please find public case studies and talks from VictoriaMetrics users. You can also join our [community Slack channel](https://slack.victoriametrics.com/) Below please find public case studies and talks from VictoriaMetrics users. You can also join our [community Slack channel](https://slack.victoriametrics.com/)
where you can chat with VictoriaMetrics users to get additional references, reviews and case studies. where you can chat with VictoriaMetrics users to get additional references, reviews and case studies.

10
docs/Cluster-VictoriaMetrics.md
View file

@ -9,12 +9,10 @@ title: Cluster version
aliases: aliases:
- /Cluster-VictoriaMetrics.html - /Cluster-VictoriaMetrics.html
--- ---
# Cluster version
<picture> <picture>
<source srcset="logo_white.webp" media="(prefers-color-scheme: dark)"> <source srcset="/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="logo.webp" media="(prefers-color-scheme: light)"> <source srcset="/logo.webp" media="(prefers-color-scheme: light)">
<img src="logo.webp" width="300" alt="VictoriaMetrics logo"> <img src="/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture> </picture>
VictoriaMetrics is a fast, cost-effective and scalable time series database. It can be used as a long-term remote storage for Prometheus. VictoriaMetrics is a fast, cost-effective and scalable time series database. It can be used as a long-term remote storage for Prometheus.
@ -61,7 +59,7 @@ Each service may scale independently and may run on the most suitable hardware.
This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture). This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture).
It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling. It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling.
<img src="Cluster-VictoriaMetrics_cluster-scheme.webp"> ![Cluster Scheme](Cluster-VictoriaMetrics_cluster-scheme.webp)
## Multitenancy ## Multitenancy

3
docs/FAQ.md
View file

@ -10,9 +10,6 @@ aliases:
- /FAQ.html - /FAQ.html
- /faq.html - /faq.html
--- ---
# FAQ
## What is the main purpose of VictoriaMetrics? ## What is the main purpose of VictoriaMetrics?
To provide the best monitoring solution. To provide the best monitoring solution.

3
docs/LTS-releases.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /LTS-releases.html - /LTS-releases.html
--- ---
# Long-term support releases
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (aka LTS releases). [Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (aka LTS releases).
Every LTS line receives bugfixes and [security fixes](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/SECURITY.md) for 12 months after Every LTS line receives bugfixes and [security fixes](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/SECURITY.md) for 12 months after
the initial release. New LTS lines are published every 6 months, so the latest two LTS lines are supported at any given moment. This gives up to 6 months the initial release. New LTS lines are published every 6 months, so the latest two LTS lines are supported at any given moment. This gives up to 6 months

22
docs/Makefile
View file

@ -1,3 +1,7 @@
ROOTDIR := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))
REPODIR := $(ROOTDIR)/..
WORKDIR := $(REPODIR)/..
# These commands must be run from the VictoriaMetrics repository root # These commands must be run from the VictoriaMetrics repository root
# Converts images at docs folder to webp format # Converts images at docs folder to webp format
@ -14,3 +18,21 @@ docs-images-to-webp-by-extension:
sh -c 'find /docs/ -type f ! -path "/docs/operator/*" ! -path "/docs/_site/*" -name "*.$(IMAGES_EXTENSION)" -print0 | \ sh -c 'find /docs/ -type f ! -path "/docs/operator/*" ! -path "/docs/_site/*" -name "*.$(IMAGES_EXTENSION)" -print0 | \
xargs -0 -P $(MAKE_CONCURRENCY) -I {} sh -c '"'"'cwebp -preset drawing -m 6 -o "$${1%.*}.webp" $$1'"'"' _ {}' xargs -0 -P $(MAKE_CONCURRENCY) -I {} sh -c '"'"'cwebp -preset drawing -m 6 -o "$${1%.*}.webp" $$1'"'"' _ {}'
find docs/ -type f ! -path 'docs/operator/*' ! -path 'docs/_site/*' -name '*.$(IMAGES_EXTENSION)' -print0 | xargs -0 rm -f find docs/ -type f ! -path 'docs/operator/*' ! -path 'docs/_site/*' -name '*.$(IMAGES_EXTENSION)' -print0 | xargs -0 rm -f
docs-debug:
if [ ! -d $(WORKDIR)/vmdocs ]; then \
git clone git@github.com:VictoriaMetrics/vmdocs $(WORKDIR)/vmdocs; \
fi; \
cd $(WORKDIR)/vmdocs && \
git checkout main && \
git pull origin main && \
cd $(REPODIR) && \
docker build \
-t vmdocs \
$(WORKDIR)/vmdocs && \
docker rm -f vmdocs || true && \
docker run \
-d \
--name vmdocs \
-p 1313:1313 \
-v ./docs:/opt/docs/content vmdocs

3
docs/MetricsQL.md
View file

@ -10,9 +10,6 @@ aliases:
- /ExtendedPromQL.html - /ExtendedPromQL.html
- /MetricsQL.html - /MetricsQL.html
--- ---
# MetricsQL
[VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) implements MetricsQL - [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) implements MetricsQL -
query language inspired by [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/). query language inspired by [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/).
MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work MetricsQL is backwards-compatible with PromQL, so Grafana dashboards backed by Prometheus datasource should work

5
docs/PerTenantStatistic.md
View file

@ -9,10 +9,7 @@ menu:
aliases: aliases:
- /PerTenantStatistic.html - /PerTenantStatistic.html
--- ---
![cluster-per-tenant-stat](PerTenantStatistic-stats.webp)
# VictoriaMetrics Cluster Per Tenant Statistic
<img alt="cluster-per-tenant-stat" src="PerTenantStatistic-stats.webp">
***The per-tenant statistic is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). ***The per-tenant statistic is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
To get the license key you can request a [free trial license](https://victoriametrics.com/products/enterprise/trial/).*** To get the license key you can request a [free trial license](https://victoriametrics.com/products/enterprise/trial/).***

3
docs/Quick-Start.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /Quick-Start.html - /Quick-Start.html
--- ---
# Quick start
## How to install ## How to install
VictoriaMetrics is distributed in two forms: VictoriaMetrics is distributed in two forms:

12
docs/README.md
View file

@ -7,9 +7,9 @@
[![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics) [![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics)
<picture> <picture>
<source srcset="logo_white.webp" media="(prefers-color-scheme: dark)"> <source srcset="/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="logo.webp" media="(prefers-color-scheme: light)"> <source srcset="/logo.webp" media="(prefers-color-scheme: light)">
<img src="logo.webp" width="300" alt="VictoriaMetrics logo"> <img src="/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture> </picture>
VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
@ -308,7 +308,7 @@ Substitute `<victoriametrics-addr>` with the hostname or IP address of VictoriaM
In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x": In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x":
<img src="grafana-datasource-prometheus.webp" alt="Grafana datasource" /> ![Datasource](grafana-datasource-prometheus.webp)
This allows Grafana to use a more efficient API to get label values. This allows Grafana to use a more efficient API to get label values.
@ -498,7 +498,7 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL` DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`. or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp"> ![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp)
To configure DataDog agent via ENV variable add the following prefix: To configure DataDog agent via ENV variable add the following prefix:
@ -525,7 +525,7 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples/
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`. sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp"> ![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp)
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver: Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:

5
docs/Release-Guide.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /Release-Guide.html - /Release-Guide.html
--- ---
# Release process guidance
## Pre-reqs ## Pre-reqs
1. Make sure you have enterprise remote configured 1. Make sure you have enterprise remote configured
@ -154,7 +151,7 @@ Once updated, run the following commands:
1. Commit and push changes to `master`. 1. Commit and push changes to `master`.
1. Run "Release" action on Github: 1. Run "Release" action on Github:
<img src="Release-Guide_helm-release.webp" alt="release helm charts"> ![release helm charts](Release-Guide_helm-release.webp)
1. Merge new PRs *"Automatic update CHANGELOGs and READMEs"* and *"Synchronize docs"* after pipelines are complete. 1. Merge new PRs *"Automatic update CHANGELOGs and READMEs"* and *"Synchronize docs"* after pipelines are complete.
## Ansible Roles ## Ansible Roles

14
docs/Single-server-VictoriaMetrics.md
View file

@ -9,8 +9,6 @@ title: VictoriaMetrics
aliases: aliases:
- /Single-server-VictoriaMetrics.html - /Single-server-VictoriaMetrics.html
--- ---
# VictoriaMetrics
[![Latest Release](https://img.shields.io/github/release/VictoriaMetrics/VictoriaMetrics.svg?style=flat-square)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest) [![Latest Release](https://img.shields.io/github/release/VictoriaMetrics/VictoriaMetrics.svg?style=flat-square)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest)
[![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics.svg?maxAge=604800)](https://hub.docker.com/r/victoriametrics/victoria-metrics) [![Docker Pulls](https://img.shields.io/docker/pulls/victoriametrics/victoria-metrics.svg?maxAge=604800)](https://hub.docker.com/r/victoriametrics/victoria-metrics)
[![Slack](https://img.shields.io/badge/join%20slack-%23victoriametrics-brightgreen.svg)](https://slack.victoriametrics.com/) [![Slack](https://img.shields.io/badge/join%20slack-%23victoriametrics-brightgreen.svg)](https://slack.victoriametrics.com/)
@ -20,9 +18,9 @@ aliases:
[![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics) [![codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/branch/master/graph/badge.svg)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics)
<picture> <picture>
<source srcset="logo_white.webp" media="(prefers-color-scheme: dark)"> <source srcset="/logo_white.webp" media="(prefers-color-scheme: dark)">
<source srcset="logo.webp" media="(prefers-color-scheme: light)"> <source srcset="/logo.webp" media="(prefers-color-scheme: light)">
<img src="logo.webp" width="300" alt="VictoriaMetrics logo"> <img src="/logo.webp" width="300" alt="VictoriaMetrics logo">
</picture> </picture>
VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
@ -321,7 +319,7 @@ Substitute `<victoriametrics-addr>` with the hostname or IP address of VictoriaM
In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x": In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x":
<img src="grafana-datasource-prometheus.webp" alt="Grafana datasource" /> ![Grafana datasource](grafana-datasource-prometheus.webp)
This allows Grafana to use a more efficient API to get label values. This allows Grafana to use a more efficient API to get label values.
@ -511,7 +509,7 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL` DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`. or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp"> ![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp)
To configure DataDog agent via ENV variable add the following prefix: To configure DataDog agent via ENV variable add the following prefix:
@ -538,7 +536,7 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples/
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`. sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp"> ![DD to VM](Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp)
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver: Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:

3
docs/Troubleshooting.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /Troubleshooting.html - /Troubleshooting.html
--- ---
# Troubleshooting
This document contains troubleshooting guides for most common issues when working with VictoriaMetrics: This document contains troubleshooting guides for most common issues when working with VictoriaMetrics:
- [General troubleshooting checklist](#general-troubleshooting-checklist) - [General troubleshooting checklist](#general-troubleshooting-checklist)

3
docs/VictoriaLogs/CHANGELOG.md
View file

@ -11,9 +11,6 @@ menu:
aliases: aliases:
- /VictoriaLogs/CHANGELOG.html - /VictoriaLogs/CHANGELOG.html
--- ---
# VictoriaLogs changelog
The following `tip` changes can be tested by building VictoriaLogs from the latest commit of [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/) repository The following `tip` changes can be tested by building VictoriaLogs from the latest commit of [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/) repository
according to [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/#building-from-source-code) according to [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/#building-from-source-code)

3
docs/VictoriaLogs/FAQ.md
View file

@ -12,9 +12,6 @@ aliases:
- /VictoriaLogs/FAQ.html - /VictoriaLogs/FAQ.html
- /VictoriaLogs/faq.html - /VictoriaLogs/faq.html
--- ---
# VictoriaLogs FAQ
## What is the difference between VictoriaLogs and Elasticsearch (OpenSearch)? ## What is the difference between VictoriaLogs and Elasticsearch (OpenSearch)?
Both Elasticsearch and VictoriaLogs allow ingesting structured and unstructured logs Both Elasticsearch and VictoriaLogs allow ingesting structured and unstructured logs

3
docs/VictoriaLogs/LogsQL.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /VictoriaLogs/LogsQL.html - /VictoriaLogs/LogsQL.html
--- ---
# LogsQL
LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/). LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/).
See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [tutorial](#logsql-tutorial) See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [tutorial](#logsql-tutorial)
in order to feel the language. in order to feel the language.

3
docs/VictoriaLogs/QuickStart.md
View file

@ -12,9 +12,6 @@ aliases:
- /victorialogs/quick-start.html - /victorialogs/quick-start.html
- /victorialogs/quick-start/ - /victorialogs/quick-start/
--- ---
# VictoriaLogs Quick Start
It is recommended to read [README](https://docs.victoriametrics.com/victorialogs/) It is recommended to read [README](https://docs.victoriametrics.com/victorialogs/)
and [Key Concepts](https://docs.victoriametrics.com/victorialogs/keyconcepts/) and [Key Concepts](https://docs.victoriametrics.com/victorialogs/keyconcepts/)
before you start working with VictoriaLogs. before you start working with VictoriaLogs.

3
docs/VictoriaLogs/Roadmap.md
View file

@ -11,9 +11,6 @@ menu:
aliases: aliases:
- /VictoriaLogs/Roadmap.html - /VictoriaLogs/Roadmap.html
--- ---
# VictoriaLogs roadmap
The following functionality is available in [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): The following functionality is available in [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/):
- [Data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). - [Data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/).

3
docs/VictoriaLogs/data-ingestion/Filebeat.md
View file

@ -11,9 +11,6 @@ aliases:
- /victorialogs/data-ingestion/Filebeat.html - /victorialogs/data-ingestion/Filebeat.html
- /victorialogs/data-ingestion/filebeat.html - /victorialogs/data-ingestion/filebeat.html
--- ---
# Filebeat setup
Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) section in the `filebeat.yml` Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) section in the `filebeat.yml`
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/):

10
docs/VictoriaLogs/data-ingestion/Fluentbit.md
View file

@ -17,7 +17,7 @@ aliases:
Specify [http output](https://docs.fluentbit.io/manual/pipeline/outputs/http) section in the `fluentbit.conf` Specify [http output](https://docs.fluentbit.io/manual/pipeline/outputs/http) section in the `fluentbit.conf`
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/):
```conf ```fluentbit
[Output] [Output]
Name http Name http
Match * Match *
@ -37,7 +37,7 @@ and uses the correct [stream fields](https://docs.victoriametrics.com/victorialo
This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) in the `uri` This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) in the `uri`
and inspecting VictoriaLogs logs then: and inspecting VictoriaLogs logs then:
```conf ```fluentbit
[Output] [Output]
Name http Name http
Match * Match *
@ -52,7 +52,7 @@ If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#
during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters).
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
```conf ```fluentbit
[Output] [Output]
Name http Name http
Match * Match *
@ -66,7 +66,7 @@ For example, the following config instructs VictoriaLogs to ignore `log.offset`
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option. If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option.
This usually allows saving network bandwidth and costs by up to 5 times: This usually allows saving network bandwidth and costs by up to 5 times:
```conf ```fluentbit
[Output] [Output]
Name http Name http
Match * Match *
@ -82,7 +82,7 @@ By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [te
If you need storing logs in other tenant, then specify the needed tenant via `header` options. If you need storing logs in other tenant, then specify the needed tenant via `header` options.
For example, the following `fluentbit.conf` config instructs Fluentbit to store the data to `(AccountID=12, ProjectID=34)` tenant: For example, the following `fluentbit.conf` config instructs Fluentbit to store the data to `(AccountID=12, ProjectID=34)` tenant:
```conf ```fluentbit
[Output] [Output]
Name http Name http
Match * Match *

13
docs/VictoriaLogs/data-ingestion/Logstash.md
View file

@ -11,13 +11,10 @@ aliases:
- /victorialogs/data-ingestion/logstash.html - /victorialogs/data-ingestion/logstash.html
- /victorialogs/data-ingestion/Logstash.html - /victorialogs/data-ingestion/Logstash.html
--- ---
# Logstash setup
Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) section in the `logstash.conf` file Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) section in the `logstash.conf` file
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/):
```conf ```logstash
output { output {
elasticsearch { elasticsearch {
hosts => ["http://localhost:9428/insert/elasticsearch/"] hosts => ["http://localhost:9428/insert/elasticsearch/"]
@ -39,7 +36,7 @@ and uses the correct [stream fields](https://docs.victoriametrics.com/victorialo
This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters)
and inspecting VictoriaLogs logs then: and inspecting VictoriaLogs logs then:
```conf ```logstash
output { output {
elasticsearch { elasticsearch {
hosts => ["http://localhost:9428/insert/elasticsearch/"] hosts => ["http://localhost:9428/insert/elasticsearch/"]
@ -57,7 +54,7 @@ If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#
during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters).
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
```conf ```logstash
output { output {
elasticsearch { elasticsearch {
hosts => ["http://localhost:9428/insert/elasticsearch/"] hosts => ["http://localhost:9428/insert/elasticsearch/"]
@ -74,7 +71,7 @@ output {
If the Logstash sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `http_compression: true` option. If the Logstash sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `http_compression: true` option.
This usually allows saving network bandwidth and costs by up to 5 times: This usually allows saving network bandwidth and costs by up to 5 times:
```conf ```logstash
output { output {
elasticsearch { elasticsearch {
hosts => ["http://localhost:9428/insert/elasticsearch/"] hosts => ["http://localhost:9428/insert/elasticsearch/"]
@ -92,7 +89,7 @@ By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [te
If you need storing logs in other tenant, then specify the needed tenant via `custom_headers` at `output.elasticsearch` section. If you need storing logs in other tenant, then specify the needed tenant via `custom_headers` at `output.elasticsearch` section.
For example, the following `logstash.conf` config instructs Logstash to store the data to `(AccountID=12, ProjectID=34)` tenant: For example, the following `logstash.conf` config instructs Logstash to store the data to `(AccountID=12, ProjectID=34)` tenant:
```conf ```logstash
output { output {
elasticsearch { elasticsearch {
hosts => ["http://localhost:9428/insert/elasticsearch/"] hosts => ["http://localhost:9428/insert/elasticsearch/"]

2
docs/VictoriaLogs/data-ingestion/Promtail.md
View file

@ -11,8 +11,6 @@ aliases:
- /victorialogs/data-ingestion/Promtail.html - /victorialogs/data-ingestion/Promtail.html
- /victorialogs/data-ingestion/promtail.html - /victorialogs/data-ingestion/promtail.html
--- ---
# Promtail setup
[Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is a default log shipper for Grafana Loki. [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is a default log shipper for Grafana Loki.
Promtail can be configured to send the collected logs to VictoriaLogs according to the following docs. Promtail can be configured to send the collected logs to VictoriaLogs according to the following docs.

3
docs/VictoriaLogs/data-ingestion/Vector.md
View file

@ -11,9 +11,6 @@ aliases:
- /victorialogs/data-ingestion/Vector.html - /victorialogs/data-ingestion/Vector.html
- /victorialogs/data-ingestion/vector.html - /victorialogs/data-ingestion/vector.html
--- ---
# Vector setup
## Elasticsearch sink ## Elasticsearch sink
Specify [Elasticsearch sink type](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) in the `vector.toml` Specify [Elasticsearch sink type](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) in the `vector.toml`

3
docs/VictoriaLogs/data-ingestion/syslog.md
View file

@ -7,9 +7,6 @@ menu:
parent: "victorialogs-data-ingestion" parent: "victorialogs-data-ingestion"
weight: 10 weight: 10
--- ---
# Syslog setup
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) can accept logs in [Syslog formats](https://en.wikipedia.org/wiki/Syslog) at the specified TCP and UDP addresses [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) can accept logs in [Syslog formats](https://en.wikipedia.org/wiki/Syslog) at the specified TCP and UDP addresses
via `-syslog.listenAddr.tcp` and `-syslog.listenAddr.udp` command-line flags. The following syslog formats are supported: via `-syslog.listenAddr.tcp` and `-syslog.listenAddr.udp` command-line flags. The following syslog formats are supported:

3
docs/VictoriaLogs/keyConcepts.md
View file

@ -10,9 +10,6 @@ menu:
aliases: aliases:
- /VictoriaLogs/keyConcepts.html - /VictoriaLogs/keyConcepts.html
--- ---
# VictoriaLogs key concepts
## Data model ## Data model
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) works with both structured and unstructured logs. [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) works with both structured and unstructured logs.

3
docs/VictoriaLogs/logsql-examples.md
View file

@ -7,9 +7,6 @@ menu:
parent: "victorialogs" parent: "victorialogs"
weight: 100 weight: 100
--- ---
# LogsQL examples
## How to select recently ingested logs? ## How to select recently ingested logs?
[Run](https://docs.victoriametrics.com/victorialogs/querying/) the following query: [Run](https://docs.victoriametrics.com/victorialogs/querying/) the following query:

4
docs/VictoriaLogs/querying/README.md
View file

@ -206,14 +206,14 @@ The `<offset>` can contain values in [the format specified here](https://docs.vi
For example, the following command returns per-day number of logs with `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) For example, the following command returns per-day number of logs with `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word)
over the last week in New York time zone (`-4h`): over the last week in New York time zone (`-4h`):
```logsql ```sh
curl http://localhost:9428/select/logsql/hits -d 'query=error' -d 'start=1w' -d 'step=1d' -d 'offset=-4h' curl http://localhost:9428/select/logsql/hits -d 'query=error' -d 'start=1w' -d 'step=1d' -d 'offset=-4h'
``` ```
Additionally, any number of `field=<field_name>` args can be passed to `/select/logsql/hits` for grouping hits buckets by the mentioned `<field_name>` fields. Additionally, any number of `field=<field_name>` args can be passed to `/select/logsql/hits` for grouping hits buckets by the mentioned `<field_name>` fields.
For example, the following query groups hits by `level` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) additionally to the provided `step`: For example, the following query groups hits by `level` [field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) additionally to the provided `step`:
```logsql ```sh
curl http://localhost:9428/select/logsql/hits -d 'query=*' -d 'start=3h' -d 'step=1h' -d 'field=level' curl http://localhost:9428/select/logsql/hits -d 'query=*' -d 'start=3h' -d 'step=1h' -d 'field=level'
``` ```

7
docs/VictoriaLogs/victorialogs-datasource.md
View file

@ -9,10 +9,6 @@ menu:
aliases: aliases:
- /victorialogs/victorialogs-datasource.html - /victorialogs/victorialogs-datasource.html
--- ---
# VictoriaLogs datasource for Grafana
The VictoriaLogs datasource plugin allows you to query and visualize The VictoriaLogs datasource plugin allows you to query and visualize
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) data in [Grafana](https://grafana.com). [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) data in [Grafana](https://grafana.com).
@ -94,7 +90,8 @@ docker-compose -f docker-compose.yaml up
After Grafana starts successfully, datasource should be available in the datasources tab After Grafana starts successfully, datasource should be available in the datasources tab
<img src="provision_datasources.png" width="800" alt="Configuration"> ![Configuration](provision_datasources.png)
{width="800"}
### Install in Kubernetes ### Install in Kubernetes

75
docs/anomaly-detection/CHANGELOG.md
View file

@ -10,73 +10,70 @@ menu:
aliases: aliases:
- /anomaly-detection/CHANGELOG.html - /anomaly-detection/CHANGELOG.html
--- ---
# CHANGELOG
Please find the changelog for VictoriaMetrics Anomaly Detection below. Please find the changelog for VictoriaMetrics Anomaly Detection below.
> **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or newer for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](/anomaly-detection/components/scheduler/#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100) or newer.** > **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or newer for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](./components/scheduler.md#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100) or newer.**
## v1.13.3 ## v1.13.3
Released: 2024-07-17 Released: 2024-07-17
- FIX: now validation of `args` argument for [`HoltWinters`](/anomaly-detection/components/models#holt-winters) model works properly. - FIX: now validation of `args` argument for [`HoltWinters`](./components/models.md#holt-winters) model works properly.
## v1.13.2 ## v1.13.2
Released: 2024-07-15 Released: 2024-07-15
- IMPROVEMENT: update `node-exporter` [preset](/anomaly-detection/presets/#node-exporter) to reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/index.html#false-positive) - IMPROVEMENT: update `node-exporter` [preset](./Presets.md#node-exporter) to reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/index.html#false-positive)
- FIX: add `verify_tls` arg for [`push`](/anomaly-detection/components/monitoring/#push-config-parameters) monitoring section. Also, `verify_tls` is now correctly used in [VmWriter](/anomaly-detection/components/writer/#vm-writer). - FIX: add `verify_tls` arg for [`push`](./components/monitoring.md#push-config-parameters) monitoring section. Also, `verify_tls` is now correctly used in [VmWriter](./components/writer.md#vm-writer).
- FIX: now [`AutoTuned`](/anomaly-detection/components/models/#autotuned) model wrapper works correctly in [on-disk model storage mode](/anomaly-detection/faq/#resource-consumption-of-vmanomaly). - FIX: now [`AutoTuned`](./components/models.md#autotuned) model wrapper works correctly in [on-disk model storage mode](./FAQ.md#resource-consumption-of-vmanomaly).
- FIX: now [rolling models](/anomaly-detection/components/models/#rolling-models), like [`RollingQuantile`](/anomaly-detection/components/models/#rolling-quantile) are properly handled in [One-off scheduler](/anomaly-detection/components/scheduler/#oneoff-scheduler), when wrapped in [`AutoTuned`](/anomaly-detection/components/models/#autotuned) - FIX: now [rolling models](./components/models.md#rolling-models), like [`RollingQuantile`](./components/models.md#rolling-quantile) are properly handled in [One-off scheduler](./components/scheduler.md#oneoff-scheduler), when wrapped in [`AutoTuned`](./components/models.md#autotuned)
## v1.13.0 ## v1.13.0
Released: 2024-06-11 Released: 2024-06-11
- FEATURE: Introduced `preset` [mode to run vmanomaly service](/anomaly-detection/presets) with minimal user input and on widely-known metrics, like those produced by [`node_exporter`](/anomaly-detection/presets#node-exporter). - FEATURE: Introduced `preset` [mode to run vmanomaly service](./Presets.md) with minimal user input and on widely-known metrics, like those produced by [`node_exporter`](./Presets.md#node-exporter).
- FEATURE: Introduced `min_dev_from_expected` [model common arg](/anomaly-detection/components/models/#minimal-deviation-from-expected), aimed at **reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the real value `y` and the expected value `yhat` are **relatively** high and may cause models to generate high [anomaly scores](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score). However, these deviations are not significant enough in **absolute values** to be considered anomalies based on domain knowledge. - FEATURE: Introduced `min_dev_from_expected` [model common arg](./components/models.md#minimal-deviation-from-expected), aimed at **reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the real value `y` and the expected value `yhat` are **relatively** high and may cause models to generate high [anomaly scores](./FAQ.md#what-is-anomaly-score). However, these deviations are not significant enough in **absolute values** to be considered anomalies based on domain knowledge.
- FEATURE: Introduced `detection_direction` [model common arg](/anomaly-detection/components/models/#detection-direction), enabling domain-driven anomaly detection strategies. Configure models to identify anomalies occurring *above, below, or in both directions* relative to the expected values. - FEATURE: Introduced `detection_direction` [model common arg](./components/models.md#detection-direction), enabling domain-driven anomaly detection strategies. Configure models to identify anomalies occurring *above, below, or in both directions* relative to the expected values.
- FEATURE: add `n_jobs` arg to [`BacktestingScheduler`](/anomaly-detection/components/scheduler/#backtesting-scheduler) to allow *proportionally faster (yet more resource-intensive)* evaluations of a config on historical data. Default value is 1, that implies *sequential* execution. - FEATURE: add `n_jobs` arg to [`BacktestingScheduler`](./components/scheduler.md#backtesting-scheduler) to allow *proportionally faster (yet more resource-intensive)* evaluations of a config on historical data. Default value is 1, that implies *sequential* execution.
- FEATURE: allow anomaly detection models to be dumped to a host filesystem after `fit` stage (instead of in-memory). Resource-intensive setups (many models, many metrics, bigger [`fit_window` arg](/anomaly-detection/components/scheduler/#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](/anomaly-detection/components/models/index.html#prophet) or [HoltWinters](/anomaly-detection/components/models/index.html#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. Please find how to enable it [here](/anomaly-detection/faq/#resource-consumption-of-vmanomaly) - FEATURE: allow anomaly detection models to be dumped to a host filesystem after `fit` stage (instead of in-memory). Resource-intensive setups (many models, many metrics, bigger [`fit_window` arg](./components/scheduler.md#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](./components/models.md#prophet) or [HoltWinters](./components/models.md#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. Please find how to enable it [here](./FAQ.md#resource-consumption-of-vmanomaly)
- IMPROVEMENT: Reduced the resource used for each fitted [`ProphetModel`](/anomaly-detection/components/models/index.html#prophet) by up to 6 times. This includes both RAM for in-memory models and disk space for on-disk models storage. For more details, refer to [this discussion on Facebook's Prophet](https://github.com/facebook/prophet/issues/1159#issuecomment-537415637). - IMPROVEMENT: Reduced the resource used for each fitted [`ProphetModel`](./components/models.md#prophet) by up to 6 times. This includes both RAM for in-memory models and disk space for on-disk models storage. For more details, refer to [this discussion on Facebook's Prophet](https://github.com/facebook/prophet/issues/1159#issuecomment-537415637).
- IMPROVEMENT: now config [components](/anomaly-detection/components/index.html) class can be referenced by a short alias instead of a full class path - i.e. `model.zscore.ZscoreModel` becomes `zscore`, `reader.vm.VmReader` becomes `vm`, `scheduler.periodic.PeriodicScheduler` becomes `periodic`, etc. - IMPROVEMENT: now config [components](./components/README.md) class can be referenced by a short alias instead of a full class path - i.e. `model.zscore.ZscoreModel` becomes `zscore`, `reader.vm.VmReader` becomes `vm`, `scheduler.periodic.PeriodicScheduler` becomes `periodic`, etc.
- FIX: if using multi-scheduler setup (introduced in [v1.11.0](/anomaly-detection/changelog/#v1110)), prevent schedulers (and correspondent services) that are not attached to any model (so neither found in ['schedulers' arg](/anomaly-detection/components/models/index.html#schedulers) nor left blank in `model` section) from being spawn, causing resource overhead and slight interference with existing ones. - FIX: if using multi-scheduler setup (introduced in [v1.11.0](./CHANGELOG.md#v1110)), prevent schedulers (and correspondent services) that are not attached to any model (so neither found in ['schedulers' arg](./components/models.md#schedulers) nor left blank in `model` section) from being spawn, causing resource overhead and slight interference with existing ones.
- FIX: set random seed for [ProphetModel](/anomaly-detection/components/models#prophet) to assure uncertainty estimates (like `yhat_lower`, `yhat_upper`) and dependant series (like `anomaly_score`), produced during `.infer()` calls are always deterministic given the same input. See [initial issue](https://github.com/facebook/prophet/issues/1124) for the details. - FIX: set random seed for [ProphetModel](./components/models.md#prophet) to assure uncertainty estimates (like `yhat_lower`, `yhat_upper`) and dependant series (like `anomaly_score`), produced during `.infer()` calls are always deterministic given the same input. See [initial issue](https://github.com/facebook/prophet/issues/1124) for the details.
- FIX: prevent *orphan* queries (that are not attached to any model or scheduler) found in `queries` arg of [Reader config section](/anomaly-detection/components/reader/index.html#vm-reader) to be fetched from VictoriaMetrics TSDB, avoiding redundant data processing. A warning will be logged, if such queries exist in a parsed config. - FIX: prevent *orphan* queries (that are not attached to any model or scheduler) found in `queries` arg of [Reader config section](./components/reader.md#vm-reader) to be fetched from VictoriaMetrics TSDB, avoiding redundant data processing. A warning will be logged, if such queries exist in a parsed config.
## v1.12.0 ## v1.12.0
Released: 2024-03-31 Released: 2024-03-31
- FEATURE: Introduction of `AutoTunedModel` model class to optimize any [built-in model](/anomaly-detection/components/models/#built-in-models) on data during `fit` phase. Specify as little as `anomaly_percentage` param from `(0, 0.5)` interval and `tuned_model_class` (i.e. [`model.zscore.ZscoreModel`](/anomaly-detection/components/models/#z-score)) to get it working with best settings that match your data. See details [here](/anomaly-detection/components/models/#autotuned). - FEATURE: Introduction of `AutoTunedModel` model class to optimize any [built-in model](./components/models.md#built-in-models) on data during `fit` phase. Specify as little as `anomaly_percentage` param from `(0, 0.5)` interval and `tuned_model_class` (i.e. [`model.zscore.ZscoreModel`](./components/models.md#z-score)) to get it working with best settings that match your data. See details [here](./components/models.md#autotuned).
<!-- <!--
- FEATURE: Preset support enablement. From now users will be able to specify only a few parameters (like `datasource_url`) + a new (backward-compatible) `preset: preset_name` field in a config file and get a service run with **predefined queries, scheduling and models**. Also, now preset assets (guide, configs, dashboards) will be available at `:8490/presets` endpoint. - FEATURE: Preset support enablement. From now users will be able to specify only a few parameters (like `datasource_url`) + a new (backward-compatible) `preset: preset_name` field in a config file and get a service run with **predefined queries, scheduling and models**. Also, now preset assets (guide, configs, dashboards) will be available at `:8490/presets` endpoint.
--> -->
- IMPROVEMENT: Better logging of model lifecycle (fit/infer stages). - IMPROVEMENT: Better logging of model lifecycle (fit/infer stages).
- IMPROVEMENT: Introduce `provide_series` arg to all the [built-in models](/anomaly-detection/components/models/#built-in-models) to define what output fields to generate for writing (i.e. `provide_series: ['anomaly_score']` means only scores are being produced) - IMPROVEMENT: Introduce `provide_series` arg to all the [built-in models](./components/models.md#built-in-models) to define what output fields to generate for writing (i.e. `provide_series: ['anomaly_score']` means only scores are being produced)
- FIX: [Self-monitoring metrics](anomaly-detection/components/monitoring/#models-behaviour-metrics) are now aggregated to `queries` aliases level (not to label sets of individual timeseries) and aligned with [reader, writer and model sections](/anomaly-detection/components/monitoring/#metrics-generated-by-vmanomaly) description , so `/metrics` endpoint holds only necessary information for scraping. - FIX: [Self-monitoring metrics](./components/monitoring.md#models-behaviour-metrics) are now aggregated to `queries` aliases level (not to label sets of individual timeseries) and aligned with [reader, writer and model sections](./components/monitoring.md#metrics-generated-by-vmanomaly) description , so `/metrics` endpoint holds only necessary information for scraping.
- FIX: Self-monitoring metric `vmanomaly_models_active` now has additional labels `model_alias`, `scheduler_alias`, `preset` to align with model-centric [self-monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#models-behaviour-metrics). - FIX: Self-monitoring metric `vmanomaly_models_active` now has additional labels `model_alias`, `scheduler_alias`, `preset` to align with model-centric [self-monitoring](./components/monitoring.md#models-behaviour-metrics).
- IMPROVEMENT: Add possibility to use temporal information in [IsolationForest models](/anomaly-detection/components/models/#isolation-forest-multivariate) via [cyclical encoding](https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca). This is particularly helpful to detect multivariate [seasonality](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#seasonality)-dependant anomalies. - IMPROVEMENT: Add possibility to use temporal information in [IsolationForest models](./components/models.md#isolation-forest-multivariate) via [cyclical encoding](https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca). This is particularly helpful to detect multivariate [seasonality](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#seasonality)-dependant anomalies.
- BREAKING CHANGE: **ARIMA** model is removed from [built-in models](/anomaly-detection/components/models/#built-in-models). For affected users, it is suggested to replace ARIMA by [Prophet](/anomaly-detection/components/models/#prophet) or [Holt-Winters](/anomaly-detection/components/models/#holt-winters). - BREAKING CHANGE: **ARIMA** model is removed from [built-in models](./components/models.md#built-in-models). For affected users, it is suggested to replace ARIMA by [Prophet](./components/models.md#prophet) or [Holt-Winters](./components/models.md#holt-winters).
## v1.11.0 ## v1.11.0
Released: 2024-02-22 Released: 2024-02-22
- FEATURE: Multi-scheduler support. Now users can use multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), each spec can be run with its own (even multiple) [schedulers](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/). - FEATURE: Multi-scheduler support. Now users can use multiple [model specs](./components/models.md) in a single config (via aliasing), each spec can be run with its own (even multiple) [schedulers](./components/scheduler.md).
- Introduction of `schedulers` arg in model spec: - Introduction of `schedulers` arg in model spec:
- It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved. - It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved.
- Passing an empty list or not specifying this param implies that each model is run in **all** the schedulers, which is a backward-compatible behavior. - Passing an empty list or not specifying this param implies that each model is run in **all** the schedulers, which is a backward-compatible behavior.
- Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#schedulers) - Please find more details in docs on [Model section](./components/models.md#schedulers)
- DEPRECATION: slight refactor of a scheduler config section - DEPRECATION: slight refactor of a scheduler config section
- Now schedulers are passed as a mapping of `scheduler_alias: scheduler_spec` under [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) sections. Using old format (< [1.11.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1110)) will produce warnings for now and will be removed in future versions. - Now schedulers are passed as a mapping of `scheduler_alias: scheduler_spec` under [scheduler](./components/scheduler.md) sections. Using old format (< [1.11.0](./CHANGELOG.md#v1110)) will produce warnings for now and will be removed in future versions.
- DEPRECATION: The `--watch` CLI option for config file reloads is deprecated and will be ignored in the future. - DEPRECATION: The `--watch` CLI option for config file reloads is deprecated and will be ignored in the future.
## v1.10.0 ## v1.10.0
Released: 2024-02-15 Released: 2024-02-15
- FEATURE: Multi-model support. Now users can specify multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), as well as to reference what [queries from VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=queries#config-parameters) it should be run on. - FEATURE: Multi-model support. Now users can specify multiple [model specs](./components/models.md) in a single config (via aliasing), as well as to reference what [queries from VmReader](./components/reader.md#config-parameters) it should be run on.
- Introduction of `queries` arg in model spec: - Introduction of `queries` arg in model spec:
- It allows the model to be executed only on a particular query subset from `reader` section. - It allows the model to be executed only on a particular query subset from `reader` section.
- Passing an empty list or not specifying this param implies that each model is run on results from **all** queries, which is a backward-compatible behavior. - Passing an empty list or not specifying this param implies that each model is run on results from **all** queries, which is a backward-compatible behavior.
- Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#queries) - Please find more details in docs on [Model section](./components/models.md#queries)
- DEPRECATION: slight refactor of a model config section - DEPRECATION: slight refactor of a model config section
- Now models are passed as a mapping of `model_alias: model_spec` under [model](https://docs.victoriametrics.com/anomaly-detection/components/models/) sections. Using old format (<= [1.9.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v192)) will produce warnings for now and will be removed in future versions. - Now models are passed as a mapping of `model_alias: model_spec` under [model](./components/models.md) sections. Using old format (<= [1.9.2](./CHANGELOG.md#v192)) will produce warnings for now and will be removed in future versions.
- Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/) - Please find more details in docs on [Model section](./components/models.md)
- IMPROVEMENT: now logs from [`monitoring.pull`](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#monitoring-section-config-example) GET requests to `/metrics` endpoint are shown only in DEBUG mode - IMPROVEMENT: now logs from [`monitoring.pull`](./components/monitoring.md#monitoring-section-config-example) GET requests to `/metrics` endpoint are shown only in DEBUG mode
- IMPROVEMENT: labelset for multivariate models is deduplicated and cleaned, resulting in better UX - IMPROVEMENT: labelset for multivariate models is deduplicated and cleaned, resulting in better UX
> **Note**: These updates support more flexible setup and effective resource management in service, as now it's not longer needed to spawn several instances of `vmanomaly` to split queries/models context across. > **Note**: These updates support more flexible setup and effective resource management in service, as now it's not longer needed to spawn several instances of `vmanomaly` to split queries/models context across.
@ -84,7 +81,7 @@ Released: 2024-02-15
## v1.9.2 ## v1.9.2
Released: 2024-01-29 Released: 2024-01-29
- BUGFIX: now multivariate models (like [`IsolationForestMultivariateModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#isolation-foresthttpsenwikipediaorgwikiisolation_forest-multivariate)) are properly handled throughout fit/infer phases. - BUGFIX: now multivariate models (like [`IsolationForestMultivariateModel`](./components/models.md#isolation-foresthttpsenwikipediaorgwikiisolation_forest-multivariate)) are properly handled throughout fit/infer phases.
## v1.9.1 ## v1.9.1
@ -95,17 +92,17 @@ Released: 2024-01-27
## v1.9.0 ## v1.9.0
Released: 2024-01-26 Released: 2024-01-26
- BUGFIX: The `query_from_last_seen_timestamp` internal logic in [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader.html#vm-reader), first introduced in [v1.5.1](https://docs.victoriametrics.com/anomaly-detection/CHANGELOG.html#v151), now functions correctly. This fix ensures that the input data shape remains consistent for subsequent `fit`-based model calls in the service. - BUGFIX: The `query_from_last_seen_timestamp` internal logic in [VmReader](./components/reader.md#vm-reader), first introduced in [v1.5.1](#v151), now functions correctly. This fix ensures that the input data shape remains consistent for subsequent `fit`-based model calls in the service.
- BREAKING CHANGE: The `sampling_period` parameter is now mandatory in [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader.html#vm-reader). This change aims to clarify and standardize the frequency of input/output in `vmanomaly`, thereby reducing uncertainty and aligning with user expectations. - BREAKING CHANGE: The `sampling_period` parameter is now mandatory in [VmReader](./components/reader.md#vm-reader). This change aims to clarify and standardize the frequency of input/output in `vmanomaly`, thereby reducing uncertainty and aligning with user expectations.
> **Note**: The majority of users, who have been proactively specifying the `sampling_period` parameter in their configurations, will experience no disruption from this update. This transition formalizes a practice that was already prevalent and expected among our user base. > **Note**: The majority of users, who have been proactively specifying the `sampling_period` parameter in their configurations, will experience no disruption from this update. This transition formalizes a practice that was already prevalent and expected among our user base.
## v1.8.0 ## v1.8.0
Released: 2024-01-15 Released: 2024-01-15
- FEATURE: Added Univariate [MAD (median absolute deviation)](/anomaly-detection/components/models.html#mad-median-absolute-deviation) model support. - FEATURE: Added Univariate [MAD (median absolute deviation)](./components/models.md#mad-median-absolute-deviation) model support.
- IMPROVEMENT: Update Python to 3.12.1 and all the dependencies. - IMPROVEMENT: Update Python to 3.12.1 and all the dependencies.
- IMPROVEMENT: Don't check /health endpoint, check the real /query_range or /import endpoints directly. Users kept getting problems with /health. - IMPROVEMENT: Don't check /health endpoint, check the real /query_range or /import endpoints directly. Users kept getting problems with /health.
- DEPRECATION: "health_path" param is deprecated and doesn't do anything in config ([reader](/anomaly-detection/components/reader.html#vm-reader), [writer](/anomaly-detection/components/writer.html#vm-writer), [monitoring.push](/anomaly-detection/components/monitoring.html#push-config-parameters)). - DEPRECATION: "health_path" param is deprecated and doesn't do anything in config ([reader](./components/reader.md#vm-reader), [writer](./components/writer.md#vm-writer), [monitoring.push](./components/monitoring.md#push-config-parameters)).
## v1.7.2 ## v1.7.2
@ -203,4 +200,4 @@ Released: 2023-01-06
## v1.0.0-beta ## v1.0.0-beta
Released: 2022-12-08 Released: 2022-12-08
- First public release is available - First public release is available

65
docs/anomaly-detection/FAQ.md
View file

@ -10,68 +10,65 @@ menu:
aliases: aliases:
- /anomaly-detection/FAQ.html - /anomaly-detection/FAQ.html
--- ---
# FAQ - VictoriaMetrics Anomaly Detection
## What is VictoriaMetrics Anomaly Detection (vmanomaly)? ## What is VictoriaMetrics Anomaly Detection (vmanomaly)?
VictoriaMetrics Anomaly Detection, also known as `vmanomaly`, is a service for detecting unexpected changes in time series data. Utilizing machine learning models, it computes and pushes back an ["anomaly score"](/anomaly-detection/components/models.html#vmanomaly-output) for user-specified metrics. This hands-off approach to anomaly detection reduces the need for manual alert setup and can adapt to various metrics, improving your observability experience. VictoriaMetrics Anomaly Detection, also known as `vmanomaly`, is a service for detecting unexpected changes in time series data. Utilizing machine learning models, it computes and pushes back an ["anomaly score"](./components/models.md#vmanomaly-output) for user-specified metrics. This hands-off approach to anomaly detection reduces the need for manual alert setup and can adapt to various metrics, improving your observability experience.
Please refer to [our guide section](/anomaly-detection/#practical-guides-and-installation) to find out more. Please refer to [our guide section](./README.md#practical-guides-and-installation) to find out more.
> **Note: `vmanomaly` is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.** > **Note: `vmanomaly` is a part of [enterprise package](../enterprise.md). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.**
## What is anomaly score? ## What is anomaly score?
Among the metrics produced by `vmanomaly` (as detailed in [vmanomaly output metrics](/anomaly-detection/components/models.html#vmanomaly-output)), `anomaly_score` is a pivotal one. It is **a continuous score > 0**, calculated in such a way that **scores ranging from 0.0 to 1.0 usually represent normal data**, while **scores exceeding 1.0 are typically classified as anomalous**. However, it's important to note that the threshold for anomaly detection can be customized in the alert configuration settings. Among the metrics produced by `vmanomaly` (as detailed in [vmanomaly output metrics](./components/models.md#vmanomaly-output)), `anomaly_score` is a pivotal one. It is **a continuous score > 0**, calculated in such a way that **scores ranging from 0.0 to 1.0 usually represent normal data**, while **scores exceeding 1.0 are typically classified as anomalous**. However, it's important to note that the threshold for anomaly detection can be customized in the alert configuration settings.
The decision to set the changepoint at `1.0` is made to ensure consistency across various models and alerting configurations, such that a score above `1.0` consistently signifies an anomaly, thus, alerting rules are maintained more easily. The decision to set the changepoint at `1.0` is made to ensure consistency across various models and alerting configurations, such that a score above `1.0` consistently signifies an anomaly, thus, alerting rules are maintained more easily.
> Note: `anomaly_score` is a metric itself, which preserves all labels found in input data and (optionally) appends [custom labels, specified in writer](/anomaly-detection/components/writer.html#metrics-formatting) - follow the link for detailed output example. > Note: `anomaly_score` is a metric itself, which preserves all labels found in input data and (optionally) appends [custom labels, specified in writer](./components/writer.md#metrics-formatting) - follow the link for detailed output example.
## How is anomaly score calculated? ## How is anomaly score calculated?
For most of the [univariate models](/anomaly-detection/components/models/#univariate-models) that can generate `yhat`, `yhat_lower`, and `yhat_upper` time series in [their output](/anomaly-detection/components/models/#vmanomaly-output) (such as [Prophet](/anomaly-detection/components/models/#prophet) or [Z-score](/anomaly-detection/components/models/#z-score)), the anomaly score is calculated as follows: For most of the [univariate models](./components/models.md#univariate-models) that can generate `yhat`, `yhat_lower`, and `yhat_upper` time series in [their output](./components/models.md#vmanomaly-output) (such as [Prophet](./components/models.md#prophet) or [Z-score](./components/models.md#z-score)), the anomaly score is calculated as follows:
- If `yhat` (expected series behavior) equals `y` (actual value observed), then the anomaly score is 0. - If `yhat` (expected series behavior) equals `y` (actual value observed), then the anomaly score is 0.
- If `y` (actual value observed) falls within the `[yhat_lower, yhat_upper]` confidence interval, the anomaly score will gradually approach 1, the closer `y` is to the boundary. - If `y` (actual value observed) falls within the `[yhat_lower, yhat_upper]` confidence interval, the anomaly score will gradually approach 1, the closer `y` is to the boundary.
- If `y` (actual value observed) strictly exceeds the `[yhat_lower, yhat_upper]` interval, the anomaly score will be greater than 1, increasing as the margin between the actual value and the expected range grows. - If `y` (actual value observed) strictly exceeds the `[yhat_lower, yhat_upper]` interval, the anomaly score will be greater than 1, increasing as the margin between the actual value and the expected range grows.
Please see example graph illustrating this logic below: Please see example graph illustrating this logic below:
<img alt="anomaly-score-calculation-example" src="vmanomaly-prophet-example.webp"> ![anomaly-score-calculation-example](vmanomaly-prophet-example.webp)
## How does vmanomaly work? ## How does vmanomaly work?
`vmanomaly` applies built-in (or custom) [anomaly detection algorithms](/anomaly-detection/components/models.html), specified in a config file. Although a single config file supports one model, running multiple instances of `vmanomaly` with different configs is possible and encouraged for parallel processing or better support for your use case (i.e. simpler model for simple metrics, more sophisticated one for metrics with trends and seasonalities). `vmanomaly` applies built-in (or custom) [anomaly detection algorithms](./components/models.md), specified in a config file. Although a single config file supports one model, running multiple instances of `vmanomaly` with different configs is possible and encouraged for parallel processing or better support for your use case (i.e. simpler model for simple metrics, more sophisticated one for metrics with trends and seasonalities).
1. For more detailed information, please visit the [overview section](/anomaly-detection/Overview.html#about). 1. For more detailed information, please visit the [overview section](./Overview.md#about).
2. To view a diagram illustrating the interaction of components, please explore the [components section](/anomaly-detection/components/). 2. To view a diagram illustrating the interaction of components, please explore the [components section](./components/README.md).
## What data does vmanomaly operate on? ## What data does vmanomaly operate on?
`vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](https://docs.victoriametrics.com/metricsql/) for data selection, sampling, and processing. Users can also [apply global filters](https://docs.victoriametrics.com/#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility. `vmanomaly` operates on data fetched from VictoriaMetrics, where you can leverage full power of [MetricsQL](../MetricsQL.md) for data selection, sampling, and processing. Users can also [apply global filters](../README.md#prometheus-querying-api-enhancements) for more targeted data analysis, enhancing scope limitation and tenant visibility.
Respective config is defined in a [`reader`](/anomaly-detection/components/reader.html#vm-reader) section. Respective config is defined in a [`reader`](./components/reader.md#vm-reader) section.
## Handling noisy input data ## Handling noisy input data
`vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy. `vmanomaly` operates on data fetched from VictoriaMetrics using [MetricsQL](../MetricsQL.md) queries, so the initial data quality can be fine-tuned with aggregation, grouping, and filtering to reduce noise and improve anomaly detection accuracy.
## Output produced by vmanomaly ## Output produced by vmanomaly
`vmanomaly` models generate [metrics](/anomaly-detection/components/models.html#vmanomaly-output) like `anomaly_score`, `yhat`, `yhat_lower`, `yhat_upper`, and `y`. These metrics provide a comprehensive view of the detected anomalies. The service also produces [health check metrics](/anomaly-detection/components/monitoring.html#metrics-generated-by-vmanomaly) for monitoring its performance. `vmanomaly` models generate [metrics](./components/models.md#vmanomaly-output) like `anomaly_score`, `yhat`, `yhat_lower`, `yhat_upper`, and `y`. These metrics provide a comprehensive view of the detected anomalies. The service also produces [health check metrics](./components/monitoring.md#metrics-generated-by-vmanomaly) for monitoring its performance.
## Choosing the right model for vmanomaly ## Choosing the right model for vmanomaly
Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect. For instance, [Z-score](anomaly-detection/components/models.html#z-score) is suitable for data without trends or seasonality, while more complex patterns might require models like [Prophet](anomaly-detection/components/models.html#prophet). Selecting the best model for `vmanomaly` depends on the data's nature and the [types of anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/#categories-of-anomalies) to detect. For instance, [Z-score](./components/models.md#z-score) is suitable for data without trends or seasonality, while more complex patterns might require models like [Prophet](./components/models.md#prophet).
Also, starting from [v1.12.0](/anomaly-detection/changelog/#v1120) it's possible to auto-tune the most important params of selected model class, find [the details here](https://docs.victoriametrics.com/anomaly-detection/components/models/#autotuned). Also, starting from [v1.12.0](./CHANGELOG.md#v1120) it's possible to auto-tune the most important params of selected model class, find [the details here](./components/models.md#autotuned).
Please refer to [respective blogpost on anomaly types and alerting heuristics](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/) for more details. Please refer to [respective blogpost on anomaly types and alerting heuristics](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/) for more details.
Still not 100% sure what to use? We are [here to help](/anomaly-detection/#get-in-touch). Still not 100% sure what to use? We are [here to help](./README.md#get-in-touch).
## Alert generation in vmanomaly ## Alert generation in vmanomaly
While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where an external alerting tool, like [`vmalert`](/vmalert.html), can be used to create alerts based on these scores for integrating it with your alerting management system. While `vmanomaly` detects anomalies and produces scores, it *does not directly generate alerts*. The anomaly scores are written back to VictoriaMetrics, where an external alerting tool, like [`vmalert`](../vmalert.md), can be used to create alerts based on these scores for integrating it with your alerting management system.
## Preventing alert fatigue ## Preventing alert fatigue
Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](/vmalert.html) can use [MetricsQL](https://docs.victoriametrics.com/metricsql/) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive). Produced anomaly scores are designed in such a way that values from 0.0 to 1.0 indicate non-anomalous data, while a value greater than 1.0 is generally classified as an anomaly. However, there are no perfect models for anomaly detection, that's why reasonable defaults expressions like `anomaly_score > 1` may not work 100% of the time. However, anomaly scores, produced by `vmanomaly` are written back as metrics to VictoriaMetrics, where tools like [`vmalert`](../vmalert.md) can use [MetricsQL](../MetricsQL.md) expressions to fine-tune alerting thresholds and conditions, balancing between avoiding [false negatives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-negative) and reducing [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive).
## How to backtest particular configuration on historical data? ## How to backtest particular configuration on historical data?
Starting from [v1.7.2](/anomaly-detection/changelog/#v172) you can produce (and write back to VictoriaMetrics TSDB) anomaly scores for historical (backtesting) period, using `BacktestingScheduler` [component](/anomaly-detection/components/scheduler/#backtesting-scheduler) to imitate consecutive "production runs" of `PeriodicScheduler` [component](/anomaly-detection/components/scheduler/#periodic-scheduler). Please find an example config below: Starting from [v1.7.2](./CHANGELOG.md#v172) you can produce (and write back to VictoriaMetrics TSDB) anomaly scores for historical (backtesting) period, using `BacktestingScheduler` [component](./components/scheduler.md#backtesting-scheduler) to imitate consecutive "production runs" of `PeriodicScheduler` [component](./components/scheduler.md#periodic-scheduler). Please find an example config below:
```yaml ```yaml
schedulers: schedulers:
@ -90,7 +87,7 @@ models:
# ... # ...
schedulers: ['scheduler_alias'] # if omitted, all the defined schedulers will be attached schedulers: ['scheduler_alias'] # if omitted, all the defined schedulers will be attached
queries: ['query_alias1'] # if omitted, all the defined queries will be attached queries: ['query_alias1'] # if omitted, all the defined queries will be attached
# https://docs.victoriametrics.com/anomaly-detection/components/models/#provide-series # {{% ref "./components/models.md#provide-series" %}}
provide_series: ['anomaly_score'] provide_series: ['anomaly_score']
# ... other models # ... other models
@ -100,23 +97,23 @@ reader:
query_alias1: 'some_metricsql_query' query_alias1: 'some_metricsql_query'
sampling_frequency: '1m' # change to whatever you need in data granularity sampling_frequency: '1m' # change to whatever you need in data granularity
# other params if needed # other params if needed
# https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader # {{% ref "./components/reader.md#vm-reader" %}}
writer: writer:
datasource_url: 'some_url_to_write_produced_data_to' datasource_url: 'some_url_to_write_produced_data_to'
# other params if needed # other params if needed
# https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer # {{% ref "./components/writer.md#vm-writer" %}}
# optional monitoring section if needed # optional monitoring section if needed
# https://docs.victoriametrics.com/anomaly-detection/components/monitoring/ # {{% ref "./components/monitoring.md" %}}
``` ```
Configuration above will produce N intervals of full length (`fit_window`=14d + `fit_every`=1h) until `to_iso` timestamp is reached to run N consecutive `fit` calls to train models; Then these models will be used to produce `M = [fit_every / sampling_frequency]` infer datapoints for `fit_every` range at the end of each such interval, imitating M consecutive calls of `infer_every` in `PeriodicScheduler` [config](/anomaly-detection/components/scheduler/#periodic-scheduler). These datapoints then will be written back to VictoriaMetrics TSDB, defined in `writer` [section](/anomaly-detection/components/writer/#vm-writer) for further visualization (i.e. in VMUI or Grafana) Configuration above will produce N intervals of full length (`fit_window`=14d + `fit_every`=1h) until `to_iso` timestamp is reached to run N consecutive `fit` calls to train models; Then these models will be used to produce `M = [fit_every / sampling_frequency]` infer datapoints for `fit_every` range at the end of each such interval, imitating M consecutive calls of `infer_every` in `PeriodicScheduler` [config](./components/scheduler.md#periodic-scheduler). These datapoints then will be written back to VictoriaMetrics TSDB, defined in `writer` [section](./components/writer.md#vm-writer) for further visualization (i.e. in VMUI or Grafana)
## Resource consumption of vmanomaly ## Resource consumption of vmanomaly
`vmanomaly` itself is a lightweight service, resource usage is primarily dependent on [scheduling](/anomaly-detection/components/scheduler.html) (how often and on what data to fit/infer your models), [# and size of timeseries returned by your queries](/anomaly-detection/components/reader.html#vm-reader), and the complexity of the employed [models](anomaly-detection/components/models.html). Its resource usage is directly related to these factors, making it adaptable to various operational scales. `vmanomaly` itself is a lightweight service, resource usage is primarily dependent on [scheduling](./components/scheduler.md) (how often and on what data to fit/infer your models), [# and size of timeseries returned by your queries](./components/reader.md#vm-reader), and the complexity of the employed [models](./components/models.md). Its resource usage is directly related to these factors, making it adaptable to various operational scales.
> **Note**: Starting from [v1.13.0](/anomaly-detection/changelog/#v1130), there is a mode to save anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default). **Resource-intensive setups** (many models, many metrics, bigger [`fit_window` arg](/anomaly-detection/components/scheduler/#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](/anomaly-detection/components/models/index.html#prophet) or [HoltWinters](/anomaly-detection/components/models/index.html#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. To enable it, you need to set environment variable `VMANOMALY_MODEL_DUMPS_DIR` to desired location. [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) are being updated accordingly ([`StatefulSet`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for persistent storage starting from chart version `1.3.0`). > **Note**: Starting from [v1.13.0](./CHANGELOG.md#v1130), there is a mode to save anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default). **Resource-intensive setups** (many models, many metrics, bigger [`fit_window` arg](./components/scheduler.md#periodic-scheduler-config-example)) and/or 3rd-party models that store fit data (like [ProphetModel](./components/models.md#prophet) or [HoltWinters](./components/models.md#holt-winters)) will have RAM consumption greatly reduced at a cost of slightly slower `infer` stage. To enable it, you need to set environment variable `VMANOMALY_MODEL_DUMPS_DIR` to desired location. [Helm charts](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/README.md) are being updated accordingly ([`StatefulSet`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for persistent storage starting from chart version `1.3.0`).
Here's an example of how to set it up in docker-compose using volumes: Here's an example of how to set it up in docker-compose using volumes:
```yaml ```yaml
@ -150,13 +147,13 @@ volumes:
## Scaling vmanomaly ## Scaling vmanomaly
> **Note:** As of latest release we don't support cluster or auto-scaled version yet (though, it's in our roadmap for - better backends, more parallelization, etc.), so proposed workarounds should be addressed manually. > **Note:** As of latest release we don't support cluster or auto-scaled version yet (though, it's in our roadmap for - better backends, more parallelization, etc.), so proposed workarounds should be addressed manually.
`vmanomaly` can be scaled horizontally by launching multiple independent instances, each with its own [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries and [configurations](/anomaly-detection/components/): `vmanomaly` can be scaled horizontally by launching multiple independent instances, each with its own [MetricsQL](../MetricsQL.md) queries and [configurations](./components/README.md):
- By splitting **queries**, [defined in reader section](/anomaly-detection/components/reader/?highlight=queries#vm-reader) and spawn separate service around it. Also in case you have *only 1 query returning huge amount of timeseries*, you can further split it by applying MetricsQL filters, i.e. using "extra_filters" [param in reader](/anomaly-detection/components/reader/?highlight=extra_filters#vm-reader) - By splitting **queries**, [defined in reader section](./components/reader.md#vm-reader) and spawn separate service around it. Also in case you have *only 1 query returning huge amount of timeseries*, you can further split it by applying MetricsQL filters, i.e. using "extra_filters" [param in reader](./components/reader.md#vm-reader)
- or **models** (in case you decide to run several models for each timeseries received i.e. for averaging anomaly scores in your alerting rules of `vmalert` or using a vote approach to reduce false positives) - see `queries` arg in [model config](/anomaly-detection/components/models/#queries) - or **models** (in case you decide to run several models for each timeseries received i.e. for averaging anomaly scores in your alerting rules of `vmalert` or using a vote approach to reduce false positives) - see `queries` arg in [model config](./components/models.md#queries)
- or **schedulers** (in case you want the same models to be trained under several schedules) - see `schedulers` arg [model section](/anomaly-detection/components/models/#schedulers) and `scheduler` [component itself](/anomaly-detection/components/scheduler/) - or **schedulers** (in case you want the same models to be trained under several schedules) - see `schedulers` arg [model section](./components/models.md#schedulers) and `scheduler` [component itself](./components/scheduler.md)
Here's an example of how to split on `extra_filters` param Here's an example of how to split on `extra_filters` param

56
docs/anomaly-detection/Overview.md
View file

@ -11,10 +11,6 @@ aliases:
- /anomaly-detection.html - /anomaly-detection.html
- /anomaly-detection/overview.html - /anomaly-detection/overview.html
--- ---
# Overview
## About ## About
**VictoriaMetrics Anomaly Detection** (or shortly, `vmanomaly`) is a service that continuously scans VictoriaMetrics time **VictoriaMetrics Anomaly Detection** (or shortly, `vmanomaly`) is a service that continuously scans VictoriaMetrics time
@ -37,7 +33,7 @@ metrics.
`vmanomaly` can be used as a helper to set up your own alerting. You can rely on the spikes you see in anomaly scores to form the metric queries for alerting rules. `vmanomaly` can be used as a helper to set up your own alerting. You can rely on the spikes you see in anomaly scores to form the metric queries for alerting rules.
> **Note: `vmanomaly` is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.** > **Note: `vmanomaly` is a part of [enterprise package](../enterprise.md). You need to get a [free trial license](https://victoriametrics.com/products/enterprise/trial/) for evaluation.**
## How? ## How?
@ -53,9 +49,9 @@ processes in parallel, each using its own config.
## Models ## Models
Currently, vmanomaly ships with a set of built-in models: Currently, vmanomaly ships with a set of built-in models:
> For a detailed overview, see [model section](/anomaly-detection/components/models.html) > For a detailed overview, see [model section](./components/models.md)
1. [**ZScore**](/anomaly-detection/components/models.html#z-score) 1. [**ZScore**](./components/models.md#z-score)
_(useful for testing)_ _(useful for testing)_
@ -63,7 +59,7 @@ Currently, vmanomaly ships with a set of built-in models:
from time-series mean (straight line). Keeps only two model parameters internally: from time-series mean (straight line). Keeps only two model parameters internally:
`mean` and `std` (standard deviation). `mean` and `std` (standard deviation).
1. [**Prophet**](/anomaly-detection/components/models.html#prophet) 1. [**Prophet**](./components/models.md#prophet)
_(simplest in configuration, recommended for getting started)_ _(simplest in configuration, recommended for getting started)_
@ -77,32 +73,32 @@ Currently, vmanomaly ships with a set of built-in models:
See [Prophet documentation](https://facebook.github.io/prophet/) See [Prophet documentation](https://facebook.github.io/prophet/)
1. [**Holt-Winters**](/anomaly-detection/components/models.html#holt-winters) 1. [**Holt-Winters**](./components/models.md#holt-winters)
Very popular forecasting algorithm. See [statsmodels.org documentation]( Very popular forecasting algorithm. See [statsmodels.org documentation](
https://www.statsmodels.org/stable/generated/statsmodels.tsa.holtwinters.ExponentialSmoothing.html) https://www.statsmodels.org/stable/generated/statsmodels.tsa.holtwinters.ExponentialSmoothing.html)
for Holt-Winters exponential smoothing. for Holt-Winters exponential smoothing.
1. [**Seasonal-Trend Decomposition**](/anomaly-detection/components/models.html#seasonal-trend-decomposition) 1. [**Seasonal-Trend Decomposition**](./components/models.md#seasonal-trend-decomposition)
Extracts three components: season, trend, and residual, that can be plotted individually for Extracts three components: season, trend, and residual, that can be plotted individually for
easier debugging. Uses LOESS (locally estimated scatterplot smoothing). easier debugging. Uses LOESS (locally estimated scatterplot smoothing).
See [statsmodels.org documentation](https://www.statsmodels.org/dev/examples/notebooks/generated/stl_decomposition.html) See [statsmodels.org documentation](https://www.statsmodels.org/dev/examples/notebooks/generated/stl_decomposition.html)
for LOESS STD. for LOESS STD.
1. [**Rolling Quantile**](/anomaly-detection/components/models.html#rolling-quantile) 1. [**Rolling Quantile**](./components/models.md#rolling-quantile)
A simple moving window of quantiles. Easy to use, easy to understand, but not as powerful as A simple moving window of quantiles. Easy to use, easy to understand, but not as powerful as
other models. other models.
1. [**Isolation Forest**](/anomaly-detection/components/models.html#isolation-forest-multivariate) 1. [**Isolation Forest**](./components/models.md#isolation-forest-multivariate)
Detects anomalies using binary trees. It works for both univariate and multivariate data. Be aware of [the curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality) in the case of multivariate data - we advise against using a single model when handling multiple time series *if the number of these series significantly exceeds their average length (# of data points)*. Detects anomalies using binary trees. It works for both univariate and multivariate data. Be aware of [the curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality) in the case of multivariate data - we advise against using a single model when handling multiple time series *if the number of these series significantly exceeds their average length (# of data points)*.
The algorithm has a linear time complexity and a low memory requirement, which works well with high-volume data. See [scikit-learn.org documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html) for Isolation Forest. The algorithm has a linear time complexity and a low memory requirement, which works well with high-volume data. See [scikit-learn.org documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html) for Isolation Forest.
1. [**MAD (Median Absolute Deviation)**](anomaly-detection/components/models.html#mad-median-absolute-deviation) 1. [**MAD (Median Absolute Deviation)**](./components/models.md#mad-median-absolute-deviation)
A robust method for anomaly detection that is less sensitive to outliers in data compared to standard deviation-based models. It considers a point as an anomaly if the absolute deviation from the median is significantly large. A robust method for anomaly detection that is less sensitive to outliers in data compared to standard deviation-based models. It considers a point as an anomaly if the absolute deviation from the median is significantly large.
@ -111,14 +107,14 @@ Currently, vmanomaly ships with a set of built-in models:
For example, here’s how Prophet predictions could look like on a real-data example For example, here’s how Prophet predictions could look like on a real-data example
(Prophet auto-detected seasonality interval): (Prophet auto-detected seasonality interval):
<img alt="propher-example" src="vmanomaly-prophet-example.webp"> ![propher-example](vmanomaly-prophet-example.webp)
And here’s what Holt-Winters predictions real-world data could look like (seasonality manually And here’s what Holt-Winters predictions real-world data could look like (seasonality manually
set to 1 week). Notice that it predicts anomalies in set to 1 week). Notice that it predicts anomalies in
different places than Prophet because the model noticed there are usually spikes on Friday different places than Prophet because the model noticed there are usually spikes on Friday
morning, so it accounted for that: morning, so it accounted for that:
<img alt="holtwinters-example" src="vmanomaly-holtwinters-example.webp"> ![holtwinters-example](vmanomaly-holtwinters-example.webp)
## Process ## Process
Upon starting, vmanomaly queries the initial range of data, and trains its model (“fit” by convention). Upon starting, vmanomaly queries the initial range of data, and trains its model (“fit” by convention).
@ -132,26 +128,26 @@ optionally preserving labels).
## Usage ## Usage
> Starting from [v1.5.0](/anomaly-detection/CHANGELOG.html#v150), vmanomaly requires a license key to run. You can obtain a trial license key [here](https://victoriametrics.com/products/enterprise/trial/). > Starting from [v1.5.0](./CHANGELOG.md#v150), vmanomaly requires a license key to run. You can obtain a trial license key [here](https://victoriametrics.com/products/enterprise/trial/).
> See [Quickstart](/anomaly-detection/QuickStart.html). > See [Quickstart](./QuickStart.md).
> See [Integration guide: vmanomaly and vmalert](/anomaly-detection/guides/guide-vmanomaly-vmalert.html). > See [Integration guide: vmanomaly and vmalert](./guides/guide-vmanomaly-vmalert/README.md).
### Config file ### Config file
There are 4 required sections in config file: There are 4 required sections in config file:
* [`schedulers`](/anomaly-detection/components/scheduler.html) - defines how often to run and make inferences, as well as what timerange to use to train the model. * [`schedulers`](./components/scheduler.md) - defines how often to run and make inferences, as well as what timerange to use to train the model.
* [`models`](/anomaly-detection/components/models.html) - specific model parameters and configurations. * [`models`](./components/models.md) - specific model parameters and configurations.
* [`reader`](/anomaly-detection/components/reader.html) - how to read data and where it is located * [`reader`](./components/reader.md) - how to read data and where it is located
* [`writer`](/anomaly-detection/components/writer.html) - where and how to write the generated output. * [`writer`](./components/writer.md) - where and how to write the generated output.
[`monitoring`](#monitoring) - defines how to monitor work of *vmanomaly* service. This config section is *optional*. [`monitoring`](#monitoring) - defines how to monitor work of *vmanomaly* service. This config section is *optional*.
> For a detailed description, see [config sections](/anomaly-detection/components) > For a detailed description, see [config sections](./components/README.md)
#### Config example #### Config example
Here is an example of config file that will run [Facebook's Prophet model](/anomaly-detection/components/models.html#prophet), that will be retrained every 2 hours on 14 days of previous data. It will generate inference results (including `anomaly_score` metric) every 1 minute. Here is an example of config file that will run [Facebook's Prophet model](./components/models.md#prophet), that will be retrained every 2 hours on 14 days of previous data. It will generate inference results (including `anomaly_score` metric) every 1 minute.
You need to specify your datasource urls to use it: You need to specify your datasource urls to use it:
@ -183,7 +179,7 @@ writer:
*vmanomaly* can be monitored by using push or pull approach. *vmanomaly* can be monitored by using push or pull approach.
It can push metrics to VictoriaMetrics or expose metrics in Prometheus exposition format. It can push metrics to VictoriaMetrics or expose metrics in Prometheus exposition format.
> For a detailed description, see [monitoring section](/anomaly-detection/components/monitoring.html) > For a detailed description, see [monitoring section](./components/monitoring.md)
#### Push approach #### Push approach
@ -201,7 +197,7 @@ monitoring:
#### Pull approach #### Pull approach
*vmanomaly* can export internal metrics in Prometheus exposition format at `/metrics` page. *vmanomaly* can export internal metrics in Prometheus exposition format at `/metrics` page.
These metrics can be scraped via [vmagent](https://docs.victoriametrics.com/vmagent/) or Prometheus. These metrics can be scraped via [vmagent](../vmagent.md) or Prometheus.
In order to enable pull approach, specify `pull` section in config file: In order to enable pull approach, specify `pull` section in config file:
@ -222,9 +218,9 @@ To use *vmanomaly* you need to pull docker image:
docker pull victoriametrics/vmanomaly:latest docker pull victoriametrics/vmanomaly:latest
``` ```
> Note: please check what is latest release in [CHANGELOG](/anomaly-detection/CHANGELOG.html) > Note: please check what is latest release in [CHANGELOG](./CHANGELOG.md)
> Note: `us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial` is deprecated since [v1.6.0](/anomaly-detection/CHANGELOG.html#v160). Use [DockerHub repo](https://hub.docker.com/r/victoriametrics/vmanomaly/tags) instead > Note: `us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial` is deprecated since [v1.6.0](./CHANGELOG.md#v160). Use [DockerHub repo](https://hub.docker.com/r/victoriametrics/vmanomaly/tags) instead
You can put a tag on it for your convenience: You can put a tag on it for your convenience:
@ -263,7 +259,7 @@ The license key can be passed via the following command-line flags:
In order to make it easier to monitor the license expiration date, the following metrics are exposed(see In order to make it easier to monitor the license expiration date, the following metrics are exposed(see
[Monitoring](#monitoring) section for details on how to scrape them): [Monitoring](#monitoring) section for details on how to scrape them):
``` ```promtextmetric
# HELP vm_license_expires_at When the license expires as a Unix timestamp in seconds # HELP vm_license_expires_at When the license expires as a Unix timestamp in seconds
# TYPE vm_license_expires_at gauge # TYPE vm_license_expires_at gauge
vm_license_expires_at 1.6963776e+09 vm_license_expires_at 1.6963776e+09
@ -272,7 +268,7 @@ vm_license_expires_at 1.6963776e+09
vm_license_expires_in_seconds 4.886608e+06 vm_license_expires_in_seconds 4.886608e+06
``` ```
Example alerts for [vmalert](https://docs.victoriametrics.com/vmalert/): Example alerts for [vmalert](../vmalert.md):
```yaml ```yaml
groups: groups:

133
docs/anomaly-detection/Presets.md
View file

@ -8,17 +8,16 @@ menu:
weight: 1 weight: 1
title: Presets title: Presets
--- ---
# Anomaly detection presets > Please check the [Quick Start Guide](./QuickStart.md) to install and run `vmanomaly`
> Please check the [Quick Start Guide](/anomaly-detection/quickstart/) to install and run `vmanomaly`
> Presets are available starting from [v1.13.0](/anomaly-detection/CHANGELOG/#v1130) > Presets are available starting from [v1.13.0](./CHANGELOG.md#v1130)
**Preset** mode allows for simpler configuration and anomaly detection with `vmanomaly` on widely-recognized metrics, such as those generated by [node_exporter](https://github.com/prometheus/node_exporter), which are typically challenging to monitor using static threshold-based alerting rules. **Preset** mode allows for simpler configuration and anomaly detection with `vmanomaly` on widely-recognized metrics, such as those generated by [node_exporter](https://github.com/prometheus/node_exporter), which are typically challenging to monitor using static threshold-based alerting rules.
This approach represents a paradigm shift from traditional [static threshold-based alerting rules](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#rule-based-alerting), focused on *raw metric values*, to *static* rules based on [`anomaly_scores`](/anomaly-detection/faq/#what-is-anomaly-score). These scores offer a consistent, default threshold that remains stable over time, being adjusted for trends, seasonality, data scale, thus, reducing the engineering effort required for maintenance. Anomaly scores are produced by [machine learning models](/anomaly-detection/components/models), which are regularly retrained on varying time frames, ensuring alerts remain current and responsive to evolving data patterns. This approach represents a paradigm shift from traditional [static threshold-based alerting rules](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#rule-based-alerting), focused on *raw metric values*, to *static* rules based on [`anomaly_scores`](./FAQ.md#what-is-anomaly-score). These scores offer a consistent, default threshold that remains stable over time, being adjusted for trends, seasonality, data scale, thus, reducing the engineering effort required for maintenance. Anomaly scores are produced by [machine learning models](./components/models.md), which are regularly retrained on varying time frames, ensuring alerts remain current and responsive to evolving data patterns.
Additionally, **preset mode** minimizes user input needed to run the service. You can configure `vmanomaly` by specifying only the preset name and data sources in the [`reader`](/anomaly-detection/components/reader/) and [`writer`](/anomaly-detection/components/writer/) sections of the configuration file. All other parameters are already preconfigured. Additionally, **preset mode** minimizes user input needed to run the service. You can configure `vmanomaly` by specifying only the preset name and data sources in the [`reader`](./components/reader.md) and [`writer`](./components/writer.md) sections of the configuration file. All other parameters are already preconfigured.
Available presets: Available presets:
@ -35,11 +34,12 @@ writer:
datasource_url: "http://victoriametrics:8428/" # your datasource url datasource_url: "http://victoriametrics:8428/" # your datasource url
# tenant_id: '0:0' # specify for cluster version # tenant_id: '0:0' # specify for cluster version
``` ```
Run a service using config file with one of the [available options](/anomaly-detection/quickstart/#how-to-install-and-run-vmanomaly). Run a service using config file with one of the [available options](./QuickStart.md#how-to-install-and-run-vmanomaly).
After you run `vmanomaly` with `preset` arg specified, available assets can be viewed, copied and downloaded at `http://localhost:8490/presets/` endpoint. After you run `vmanomaly` with `preset` arg specified, available assets can be viewed, copied and downloaded at `http://localhost:8490/presets/` endpoint.
<img alt="preset-localhost" src="presets-localhost.webp" width="800px"/> ![preset-localhost](presets-localhost.webp)
{width="800px"}
## Node-Exporter ## Node-Exporter
@ -51,12 +51,12 @@ preset: "node-exporter"
``` ```
### Generated anomaly scores ### Generated anomaly scores
Machine learning models will be fit for each timeseries, returned by underlying [MetricsQL](https://docs.victoriametrics.com/metricsql/) queries. Machine learning models will be fit for each timeseries, returned by underlying [MetricsQL](../MetricsQL.md) queries.
Anomaly score metric labels will also contain [model classes](/anomaly-detection/components/models/) and [schedulers](/anomaly-detection/components/scheduler/) for labelset uniqueness. Anomaly score metric labels will also contain [model classes](./components/models.md) and [schedulers](./components/scheduler.md) for labelset uniqueness.
Here's an example of produced metrics: Here's an example of produced metrics:
```shell ```promtextmetric
anomaly_score{for="cpu_seconds_total", instance="node-exporter:9100", preset="node-exporter", mode="system", model_alias="prophet", scheduler_alias="1d_1m"} 0.23451242720277776 anomaly_score{for="cpu_seconds_total", instance="node-exporter:9100", preset="node-exporter", mode="system", model_alias="prophet", scheduler_alias="1d_1m"} 0.23451242720277776
anomaly_score{for="cpu_seconds_total", instance="node-exporter:9100", preset="node-exporter", mode="user", model_alias="prophet", scheduler_alias="1d_1m"} 0.2637952255694444 anomaly_score{for="cpu_seconds_total", instance="node-exporter:9100", preset="node-exporter", mode="user", model_alias="prophet", scheduler_alias="1d_1m"} 0.2637952255694444
anomaly_score{for="page_faults", instance="node-exporter:9100", job="node-exporter", preset="node-exporter", model_alias="prophet", scheduler_alias="1d_1m"} 0.00593712535 anomaly_score{for="page_faults", instance="node-exporter:9100", job="node-exporter", preset="node-exporter", model_alias="prophet", scheduler_alias="1d_1m"} 0.00593712535
@ -101,7 +101,7 @@ Grafana dashboard `.json` file can be found [here](https://github.com/VictoriaMe
The produced anomaly scores will have a label `for` containing the name of corresponding indicator. The produced anomaly scores will have a label `for` containing the name of corresponding indicator.
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Indicator</th> <th>Indicator</th>
@ -111,34 +111,94 @@ The produced anomaly scores will have a label `for` containing the name of corre
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>page_faults</code></td> <td>
<td><code>node_vmstat_pgmajfault</code></td>
<td>Number of major faults that have occurred since the last update. Major faults occur when a process tries to access a page in memory that is not currently mapped in the process's address space, and it requires loading data from the disk.</td> `page_faults`
</td>
<td>
`node_vmstat_pgmajfault`
</td>
<td>
Number of major faults that have occurred since the last update. Major faults occur when a process tries to access a page in memory that is not currently mapped in the process's address space, and it requires loading data from the disk.
</td>
</tr> </tr>
<tr> <tr>
<td><code>context_switch</code></td> <td>
<td><code>node_context_switches_total</code></td>
<td>This metric represents the total number of context switches across all CPUs.</td> `context_switch`
</td>
<td>
`node_context_switches_total`
</td>
<td>
This metric represents the total number of context switches across all CPUs.
</td>
</tr> </tr>
<tr> <tr>
<td><code>cpu_seconds_total</code></td> <td>
<td><code>node_cpu_seconds_total</code></td>
<td>Total amount of CPU time consumed by the system in seconds by CPU processing mode (e.g., user, system, idle).</td> `cpu_seconds_total`
</td>
<td>
`node_cpu_seconds_total`
</td>
<td>
Total amount of CPU time consumed by the system in seconds by CPU processing mode (e.g., user, system, idle).
</td>
</tr> </tr>
<tr> <tr>
<td><code>host_network_receive_errors</code> & <code>host_network_transmit_errors</code></td> <td>
<td><code>node_network_receive_errs_total</code>, <code>node_network_receive_packets_total</code>, <code>node_network_transmit_errs_total</code>, <code>node_network_transmit_packets_total</code>
<td>Total number of errors encountered while receiving/transmitting packets on the network interfaces of a node.</td> `host_network_receive_errors` & `host_network_transmit_errors`
</td>
<td>
`node_network_receive_errs_total`,
`node_network_receive_packets_total`,
`node_network_transmit_errs_total`,
`node_network_transmit_packets_total`
<td>
Total number of errors encountered while receiving/transmitting packets on the network interfaces of a node.
</td>
</tr> </tr>
<tr> <tr>
<td><code>receive_bytes</code> & <code>transmit_bytes</code></td> <td>
<td><code>node_network_receive_bytes_total</code>, <code>node_network_transmit_bytes_total</code></td>
<td>Total number of bytes received/transmitted on network interfaces of a node.</td> `receive_bytes` & `transmit_bytes`
</td>
<td>
`node_network_receive_bytes_total`,
`node_network_transmit_bytes_total`
</td>
<td>
Total number of bytes received/transmitted on network interfaces of a node.
</td>
</tr> </tr>
<tr> <tr>
<td><code>read_latency</code> & <code>write_latency</code></td> <td>
<td><code>node_disk_read_time_seconds_total</code>, <code>node_disk_reads_completed_total</code>, <code>node_disk_write_time_seconds_total</code>, <code>node_disk_writes_completed_total</code></td>
<td>Disk latency. The total read/write time spent in seconds. / The total number of reads/writes completed successfully.</td> `read_latency` & `write_latency`
</td>
<td>
`node_disk_read_time_seconds_total`,
`node_disk_reads_completed_total`,
`node_disk_write_time_seconds_total`,
`node_disk_writes_completed_total`
</td>
<td>
Disk latency. The total read/write time spent in seconds. / The total number of reads/writes completed successfully.
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -149,21 +209,26 @@ Here's how attached [Grafana dashboard](https://github.com/VictoriaMetrics/Victo
On the (global) graph **'Percentage of Anomalies'**, you can see a spike 8.75% of anomalies at the timestamp '2024-06-03 10:35:00'. The (global) graph **'Anomalies per Indicator'** shows the indicators that were anomalous at the corresponding time. On the (global) graph **'Percentage of Anomalies'**, you can see a spike 8.75% of anomalies at the timestamp '2024-06-03 10:35:00'. The (global) graph **'Anomalies per Indicator'** shows the indicators that were anomalous at the corresponding time.
<img alt="global" src="presets_global_percentage.webp" width="800px"/> ![global](presets_global_percentage.webp)
{width="800px"}
At this timestamp on the **'Number of Anomalous Indicators by Node'** graph we can identify the node that had the most anomalies: `10.142.0.27` At this timestamp on the **'Number of Anomalous Indicators by Node'** graph we can identify the node that had the most anomalies: `10.142.0.27`
<img alt="by_node" src="presets_anomalies_by_node.webp" width="800px"/> ![by_node](presets_anomalies_by_node.webp)
{width="800px"}
Now you can select anomalous node to drill down further (local): Now you can select anomalous node to drill down further (local):
<img alt="anomalous_node_selection" src="presets_anomalous_node_selection.webp" width="800px"/> ![anomalous_node_selection](presets_anomalous_node_selection.webp)
{width="800px"}
For this node from the timestamp `2024-06-03 10:35:00` CPU time spent handling software interrupts started to grow. For this node from the timestamp `2024-06-03 10:35:00` CPU time spent handling software interrupts started to grow.
(`cpu_seconds_total{mode="softirq"}`) (`cpu_seconds_total{mode="softirq"}`)
<img alt="irq" src="presets_cpu_seconds_softirq.webp" width="800px"/> ![irq](presets_cpu_seconds_softirq.webp)
{width="800px"}
At the same time `cpu_seconds_total` for `steal` mode started to grow as well. At the same time `cpu_seconds_total` for `steal` mode started to grow as well.
<img alt="steal" src="presets_cpu_seconds_steal.webp" width="800px"/> ![steal](presets_cpu_seconds_steal.webp)
{width="800px"}

43
docs/anomaly-detection/QuickStart.md
View file

@ -10,11 +10,8 @@ menu:
aliases: aliases:
- /anomaly-detection/QuickStart.html - /anomaly-detection/QuickStart.html
--- ---
For service introduction visit [README](./README.md) page
# VictoriaMetrics Anomaly Detection Quick Start and [Overview](./Overview.md) of how `vmanomaly` works.
For service introduction visit [README](/anomaly-detection/) page
and [Overview](/anomaly-detection/overview.html) of how `vmanomaly` works.
## How to install and run vmanomaly ## How to install and run vmanomaly
@ -25,7 +22,7 @@ The following options are available:
- [To run Docker image](#docker) - [To run Docker image](#docker)
- [To run in Kubernetes with Helm charts](#kubernetes-with-helm-charts) - [To run in Kubernetes with Helm charts](#kubernetes-with-helm-charts)
> **Note**: Starting from [v1.13.0](/anomaly-detection/changelog/#v1130) there is a mode to keep anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default); This may lead to **noticeable reduction of RAM used** on bigger setups. See instructions [here](/anomaly-detection/faq/#resource-consumption-of-vmanomaly). > **Note**: Starting from [v1.13.0](./CHANGELOG.md#v1130) there is a mode to keep anomaly detection models on host filesystem after `fit` stage (instead of keeping them in-memory by default); This may lead to **noticeable reduction of RAM used** on bigger setups. See instructions [here](./FAQ.md#resource-consumption-of-vmanomaly).
### Docker ### Docker
@ -84,13 +81,13 @@ services:
# ... # ...
``` ```
For a complete docker-compose example please refer to [our alerting guide](/anomaly-detection/guides/guide-vmanomaly-vmalert/), chapter [docker-compose](/anomaly-detection/guides/guide-vmanomaly-vmalert/#docker-compose) For a complete docker-compose example please refer to [our alerting guide](./guides/guide-vmanomaly-vmalert/README.md), chapter [docker-compose](./guides/guide-vmanomaly-vmalert/README.md#docker-compose)
See also: See also:
- Verify the license online OR offline. See the details [here](/anomaly-detection/overview/#licensing). - Verify the license online OR offline. See the details [here](./Overview.md#licensing).
- [How to configure `vmanomaly`](#how-to-configure-vmanomaly) - [How to configure `vmanomaly`](#how-to-configure-vmanomaly)
### Kubernetes with Helm charts ### Kubernetes with Helm charts
@ -110,47 +107,47 @@ Here is an example of config file that will run [Facebook Prophet](https://faceb
```yaml ```yaml
schedulers: schedulers:
2h_1m: 2h_1m:
# https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler # {{% ref "./components/scheduler.md#periodic-scheduler" %}}
class: 'periodic' class: 'periodic'
infer_every: '1m' infer_every: '1m'
fit_every: '2h' fit_every: '2h'
fit_window: '2w' fit_window: '2w'
models: models:
# https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet # {{% ref "./components/models.md#prophet" %}}
prophet_model: prophet_model:
class: "prophet" # or "model.prophet.ProphetModel" until v1.13.0 class: "prophet" # or "model.prophet.ProphetModel" until v1.13.0
args: args:
interval_width: 0.98 interval_width: 0.98
reader: reader:
# https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader # {{% ref "./components/reader.md#vm-reader" %}}
datasource_url: "http://victoriametrics:8428/" # [YOUR_DATASOURCE_URL] datasource_url: "http://victoriametrics:8428/" # [YOUR_DATASOURCE_URL]
sampling_period: "1m" sampling_period: "1m"
queries: queries:
# define your queries with MetricsQL - https://docs.victoriametrics.com/metricsql/ # define your queries with MetricsQL - {{% ref "../../MetricsQL.md" %}}
cache: "sum(rate(vm_cache_entries))" cache: "sum(rate(vm_cache_entries))"
writer: writer:
# https://docs.victoriametrics.com/anomaly-detection/components/writer/#vm-writer # {{% ref "./components/writer.md#vm-writer" %}}
datasource_url: "http://victoriametrics:8428/" # [YOUR_DATASOURCE_URL] datasource_url: "http://victoriametrics:8428/" # [YOUR_DATASOURCE_URL]
``` ```
Next steps: Next steps:
- Define how often to run and make inferences in the [scheduler](/anomaly-detection/components/scheduler/) section of a config file. - Define how often to run and make inferences in the [scheduler](./components/scheduler.md) section of a config file.
- Setup the datasource to read data from in the [reader](/anomaly-detection/components/reader/) section. - Setup the datasource to read data from in the [reader](./components/reader.md) section.
- Specify where and how to store anomaly detection metrics in the [writer](/anomaly-detection/components/writer/) section. - Specify where and how to store anomaly detection metrics in the [writer](./components/writer.md) section.
- Configure built-in models parameters according to your needs in the [models](/anomaly-detection/components/models/) section. - Configure built-in models parameters according to your needs in the [models](./components/models.md) section.
- Integrate your [custom models](/anomaly-detection/components/models/#custom-model-guide) with `vmanomaly`. - Integrate your [custom models](./components/models.md#custom-model-guide) with `vmanomaly`.
- Define queries for input data using [MetricsQL](https://docs.victoriametrics.com/metricsql/). - Define queries for input data using [MetricsQL](../../MetricsQL.md).
## Check also ## Check also
Here are other materials that you might find useful: Here are other materials that you might find useful:
- [Guide: Anomaly Detection and Alerting Setup](/anomaly-detection/guides/guide-vmanomaly-vmalert/) - [Guide: Anomaly Detection and Alerting Setup](./guides/guide-vmanomaly-vmalert/README.md)
- [FAQ](/anomaly-detection/faq/) - [FAQ](./FAQ.md)
- [Changelog](/anomaly-detection/changelog/) - [Changelog](./CHANGELOG.md)
- [Anomaly Detection Blog](https://victoriametrics.com/blog/tags/anomaly-detection/) - [Anomaly Detection Blog](https://victoriametrics.com/blog/tags/anomaly-detection/)

28
docs/anomaly-detection/README.md
View file

@ -1,28 +1,28 @@
In the dynamic and complex world of system monitoring, VictoriaMetrics Anomaly Detection, being a part of our [Enterprise offering](https://victoriametrics.com/products/enterprise/), stands as a pivotal tool for achieving advanced observability. It empowers SREs and DevOps teams by automating the intricate task of identifying abnormal behavior in time-series data. It goes beyond traditional threshold-based alerting, utilizing machine learning techniques to not only detect anomalies but also minimize false positives, thus reducing alert fatigue. By providing simplified alerting mechanisms atop of [unified anomaly scores](/anomaly-detection/components/models.html#vmanomaly-output), it enables teams to spot and address potential issues faster, ensuring system reliability and operational efficiency. In the dynamic and complex world of system monitoring, VictoriaMetrics Anomaly Detection, being a part of our [Enterprise offering](https://victoriametrics.com/products/enterprise/), stands as a pivotal tool for achieving advanced observability. It empowers SREs and DevOps teams by automating the intricate task of identifying abnormal behavior in time-series data. It goes beyond traditional threshold-based alerting, utilizing machine learning techniques to not only detect anomalies but also minimize false positives, thus reducing alert fatigue. By providing simplified alerting mechanisms atop of [unified anomaly scores](./components/models.md#vmanomaly-output), it enables teams to spot and address potential issues faster, ensuring system reliability and operational efficiency.
## Practical Guides and Installation ## Practical Guides and Installation
Begin your VictoriaMetrics Anomaly Detection journey with ease using our guides and installation instructions: Begin your VictoriaMetrics Anomaly Detection journey with ease using our guides and installation instructions:
- **Quickstart**: Check out how to get `vmanomaly` up and running [here](/anomaly-detection/QuickStart.html). - **Quickstart**: Check out how to get `vmanomaly` up and running [here](./QuickStart.md).
- **Overview**: Find out how `vmanomaly` service operates [here](/anomaly-detection/Overview.html) - **Overview**: Find out how `vmanomaly` service operates [here](./Overview.md)
- **Integration**: Integrate anomaly detection into your observability ecosystem. Get started [here](/anomaly-detection/guides/guide-vmanomaly-vmalert.html). - **Integration**: Integrate anomaly detection into your observability ecosystem. Get started [here](./guides/guide-vmanomaly-vmalert/README.md).
- **Anomaly Detection Presets**: Enable anomaly detection on predefined set of indicators, that require frequently changing static thresholds for alerting. Find more information [here](/anomaly-detection/presets/). - **Anomaly Detection Presets**: Enable anomaly detection on predefined set of indicators, that require frequently changing static thresholds for alerting. Find more information [here](./Presets.md).
- **Installation Options**: Select the method that aligns with your technical requirements: - **Installation Options**: Select the method that aligns with your technical requirements:
- **Docker Installation**: Suitable for containerized environments. See [Docker guide](/anomaly-detection/Overview.html#run-vmanomaly-docker-container). - **Docker Installation**: Suitable for containerized environments. See [Docker guide](./Overview.md#run-vmanomaly-docker-container).
- **Helm Chart Installation**: Appropriate for those using Kubernetes. See our [Helm charts](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-anomaly). - **Helm Chart Installation**: Appropriate for those using Kubernetes. See our [Helm charts](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-anomaly).
> **Note**: starting from [v1.5.0](./CHANGELOG.md#v150) `vmanomaly` requires a [license key](/anomaly-detection/Overview.html#licensing) to run. You can obtain a trial license key [**here**](https://victoriametrics.com/products/enterprise/trial/). > **Note**: starting from [v1.5.0](./CHANGELOG.md#v150) `vmanomaly` requires a [license key](./Overview.md#licensing) to run. You can obtain a trial license key [**here**](https://victoriametrics.com/products/enterprise/trial/).
## Key Components ## Key Components
Explore the integral components that configure VictoriaMetrics Anomaly Detection: Explore the integral components that configure VictoriaMetrics Anomaly Detection:
* [Explore components and their interation](/anomaly-detection/components) * [Explore components and their interation](./components/README.md)
- [Models](/anomaly-detection/components/models) - [Models](./components/models.md)
- [Reader](/anomaly-detection/components/reader) - [Reader](./components/reader.md)
- [Scheduler](/anomaly-detection/components/scheduler) - [Scheduler](./components/scheduler.md)
- [Writer](/anomaly-detection/components/writer) - [Writer](./components/writer.md)
- [Monitoring](/anomaly-detection/components/monitoring) - [Monitoring](./components/monitoring.md)
## Deep Dive into Anomaly Detection ## Deep Dive into Anomaly Detection
Enhance your knowledge with our handbook on Anomaly Detection & Root Cause Analysis and stay updated: Enhance your knowledge with our handbook on Anomaly Detection & Root Cause Analysis and stay updated:
@ -35,7 +35,7 @@ Enhance your knowledge with our handbook on Anomaly Detection & Root Cause Analy
## Frequently Asked Questions (FAQ) ## Frequently Asked Questions (FAQ)
Got questions about VictoriaMetrics Anomaly Detection? Chances are, we've got the answers ready for you. Got questions about VictoriaMetrics Anomaly Detection? Chances are, we've got the answers ready for you.
Dive into [our FAQ section](/anomaly-detection/FAQ) to find responses to common questions. Dive into [our FAQ section](./FAQ.md) to find responses to common questions.
## Get in Touch ## Get in Touch
We are eager to connect with you and adapt our solutions to your specific needs. Here's how you can engage with us: We are eager to connect with you and adapt our solutions to your specific needs. Here's how you can engage with us:

36
docs/anomaly-detection/components/README.md
View file

@ -1,28 +1,28 @@
This chapter describes different components, that correspond to respective sections of a config to launch VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](/anomaly-detection/overview.html)) service: This chapter describes different components, that correspond to respective sections of a config to launch VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](../Overview.md)) service:
- [Model(s) section](models.html) - Required - [Model(s) section](./models.md) - Required
- [Reader section](reader.html) - Required - [Reader section](./reader.md) - Required
- [Scheduler(s) section](scheduler.html) - Required - [Scheduler(s) section](./scheduler.md) - Required
- [Writer section](writer.html) - Required - [Writer section](./writer.md) - Required
- [Monitoring section](monitoring.html) - Optional - [Monitoring section](./monitoring.md) - Optional
> **Note**: starting from [v1.7.0](/anomaly-detection/CHANGELOG#v172), once the service starts, automated config validation is performed. Please see container logs for errors that need to be fixed to create fully valid config, visiting sections above for examples and documentation. > **Note**: starting from [v1.7.0](../CHANGELOG.md#v172), once the service starts, automated config validation is performed. Please see container logs for errors that need to be fixed to create fully valid config, visiting sections above for examples and documentation.
> **Note**: starting from [v1.13.0](/anomaly-detection/CHANGELOG#v1130), components' class can be referenced by a short alias instead of a full class path - i.e. `model.zscore.ZscoreModel` becomes `zscore`, `reader.vm.VmReader` becomes `vm`, `scheduler.periodic.PeriodicScheduler` becomes `periodic`, etc. Please see according sections for the details. > **Note**: starting from [v1.13.0](../CHANGELOG.md#v1130), components' class can be referenced by a short alias instead of a full class path - i.e. `model.zscore.ZscoreModel` becomes `zscore`, `reader.vm.VmReader` becomes `vm`, `scheduler.periodic.PeriodicScheduler` becomes `periodic`, etc. Please see according sections for the details.
> **Note:** Starting from [v1.13.0](/anomaly-detection/CHANGELOG#v1130) `preset` modes are available for `vmanomaly`. Please find the guide [here](/anomaly-detection/presets/). > **Note:** Starting from [v1.13.0](../CHANGELOG.md#v1130) `preset` modes are available for `vmanomaly`. Please find the guide [here](../Presets.md).
Below, you will find an example illustrating how the components of `vmanomaly` interact with each other and with a single-node VictoriaMetrics setup. Below, you will find an example illustrating how the components of `vmanomaly` interact with each other and with a single-node VictoriaMetrics setup.
> **Note**: [Reader](/anomaly-detection/components/reader.html#vm-reader) and [Writer](/anomaly-detection/components/writer.html#vm-writer) also support [multitenancy](/Cluster-VictoriaMetrics.html#multitenancy), so you can read/write from/to different locations - see `tenant_id` param description. > **Note**: [Reader](./reader.md#vm-reader) and [Writer](./writer.md#vm-writer) also support [multitenancy](../../Cluster-VictoriaMetrics.md#multitenancy), so you can read/write from/to different locations - see `tenant_id` param description.
<img alt="vmanomaly-components" src="vmanomaly-components.webp" width="800px"/> ![vmanomaly-components](vmanomaly-components.webp)
{width="800px"}
Here's a minimalistic full config example, demonstrating many-to-many configuration (actual for [latest version](/anomaly-detection/CHANGELOG/)): Here's a minimalistic full config example, demonstrating many-to-many configuration (actual for [latest version](../CHANGELOG.md)):
```yaml ```yaml
# how and when to run the models is defined by schedulers {{% ref "./scheduler.md" %}}
# https://docs.victoriametrics.com/anomaly-detection/components/scheduler/
schedulers: schedulers:
periodic_1d: # alias periodic_1d: # alias
class: 'periodic' # scheduler class class: 'periodic' # scheduler class
@ -36,7 +36,7 @@ schedulers:
fit_window: "7d" fit_window: "7d"
# what model types and with what hyperparams to run on your data # what model types and with what hyperparams to run on your data
# https://docs.victoriametrics.com/anomaly-detection/components/models/ # {{% ref "./models.md" %}}
models: models:
zscore: # alias zscore: # alias
class: 'zscore' # model class class: 'zscore' # model class
@ -53,7 +53,7 @@ models:
interval_width: 0.98 interval_width: 0.98
# where to read data from # where to read data from
# https://docs.victoriametrics.com/anomaly-detection/components/reader/ # {{% ref "./reader.md" %}}
reader: reader:
datasource_url: "http://victoriametrics:8428/" datasource_url: "http://victoriametrics:8428/"
tenant_id: "0:0" tenant_id: "0:0"
@ -64,12 +64,12 @@ reader:
host_network_receive_errors: 'rate(node_network_receive_errs_total[3m]) / rate(node_network_receive_packets_total[3m])' host_network_receive_errors: 'rate(node_network_receive_errs_total[3m]) / rate(node_network_receive_packets_total[3m])'
# where to write data to # where to write data to
# https://docs.victoriametrics.com/anomaly-detection/components/writer/ # {{% ref "./writer.md" %}}
writer: writer:
datasource_url: "http://victoriametrics:8428/" datasource_url: "http://victoriametrics:8428/"
# enable self-monitoring in pull and/or push mode # enable self-monitoring in pull and/or push mode
# https://docs.victoriametrics.com/anomaly-detection/components/monitoring/ # {{% ref "./monitoring.md" %}}
monitoring: monitoring:
pull: # Enable /metrics endpoint. pull: # Enable /metrics endpoint.
addr: "0.0.0.0" addr: "0.0.0.0"

98
docs/anomaly-detection/components/models.md
View file

@ -13,15 +13,12 @@ aliases:
- /anomaly-detection/components/models/custom_model.html - /anomaly-detection/components/models/custom_model.html
- /anomaly-detection/components/models/models.html - /anomaly-detection/components/models/models.html
--- ---
This section describes `Models` component of VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](../Overview.md)) and the guide of how to define a respective section of a config to launch the service.
# Models
This section describes `Models` component of VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](/anomaly-detection/overview/)) and the guide of how to define a respective section of a config to launch the service.
- `vmanomaly` includes various [built-in models](#built-in-models). - `vmanomaly` includes various [built-in models](#built-in-models).
- you can also integrate your custom model - see [custom model](#custom-model-guide). - you can also integrate your custom model - see [custom model](#custom-model-guide).
> **Note: Starting from [v1.10.0](/anomaly-detection/changelog#v1100) model section in config supports multiple models via aliasing. <br>Also, `vmanomaly` expects model section to be named `models`. Using old (flat) format with `model` key is deprecated and will be removed in future versions. Having `model` and `models` sections simultaneously in a config will result in only `models` being used:** > **Note: Starting from [v1.10.0](../CHANGELOG.md#v1100) model section in config supports multiple models via aliasing. <br>Also, `vmanomaly` expects model section to be named `models`. Using old (flat) format with `model` key is deprecated and will be removed in future versions. Having `model` and `models` sections simultaneously in a config will result in only `models` being used:**
```yaml ```yaml
models: models:
@ -40,7 +37,7 @@ models:
# ... # ...
``` ```
Old-style configs (< [1.10.0](/anomaly-detection/changelog#v1100)) Old-style configs (< [1.10.0](../CHANGELOG.md#v1100))
```yaml ```yaml
model: model:
@ -65,11 +62,11 @@ models:
## Common args ## Common args
From [1.10.0](/anomaly-detection/changelog#1100), **common args**, supported by *every model (and model type)* were introduced. From [1.10.0](../CHANGELOG.md#1100), **common args**, supported by *every model (and model type)* were introduced.
### Queries ### Queries
Introduced in [1.10.0](/anomaly-detection/changelog#1100), as a part to support multi-model configs, `queries` arg is meant to define [queries from VmReader](/anomaly-detection/components/reader/?highlight=queries#config-parameters) particular model should be run on (meaning, all the series returned by each of these queries will be used in such model for fitting and inferencing). Introduced in [1.10.0](../CHANGELOG.md#1100), as a part to support multi-model configs, `queries` arg is meant to define [queries from VmReader](./reader.md#config-parameters) particular model should be run on (meaning, all the series returned by each of these queries will be used in such model for fitting and inferencing).
`queries` arg is supported for all [the built-in](#built-in-models) (as well as for [custom](#custom-model-guide)) models. `queries` arg is supported for all [the built-in](#built-in-models) (as well as for [custom](#custom-model-guide)) models.
@ -94,7 +91,7 @@ models:
### Schedulers ### Schedulers
Introduced in [1.11.0](/anomaly-detection/changelog#1110), as a part to support multi-scheduler configs, `schedulers` arg is meant to define [schedulers](/anomaly-detection/components/scheduler) particular model should be attached to. Introduced in [1.11.0](../CHANGELOG.md#1110), as a part to support multi-scheduler configs, `schedulers` arg is meant to define [schedulers](./scheduler.md) particular model should be attached to.
`schedulers` arg is supported for all [the built-in](#built-in-models) (as well as for [custom](#custom-model-guide)) models. `schedulers` arg is supported for all [the built-in](#built-in-models) (as well as for [custom](#custom-model-guide)) models.
@ -119,7 +116,7 @@ models:
### Provide series ### Provide series
Introduced in [1.12.0](/anomaly-detection/changelog#1120), `provide_series` arg limit the [output generated](#vmanomaly-output) by `vmanomaly` for writing. I.e. if the model produces default output series `['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']` by specifying `provide_series` section as below, you limit the data being written to only `['anomaly_score']` for each metric received as a subject to anomaly detection. Introduced in [1.12.0](../CHANGELOG.md#1120), `provide_series` arg limit the [output generated](#vmanomaly-output) by `vmanomaly` for writing. I.e. if the model produces default output series `['anomaly_score', 'yhat', 'yhat_lower', 'yhat_upper']` by specifying `provide_series` section as below, you limit the data being written to only `['anomaly_score']` for each metric received as a subject to anomaly detection.
```yaml ```yaml
models: models:
@ -131,23 +128,26 @@ models:
**Note** If `provide_series` is not specified in model config, the model will produce its default [model-dependent output](#vmanomaly-output). The output can't be less than `['anomaly_score']`. Even if `timestamp` column is omitted, it will be implicitly added to `provide_series` list, as it's required for metrics to be properly written. **Note** If `provide_series` is not specified in model config, the model will produce its default [model-dependent output](#vmanomaly-output). The output can't be less than `['anomaly_score']`. Even if `timestamp` column is omitted, it will be implicitly added to `provide_series` list, as it's required for metrics to be properly written.
### Detection direction ### Detection direction
Introduced in [1.13.0](/anomaly-detection/CHANGELOG/#1130), `detection_direction` arg can help in reducing the number of [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/index.html#false-positive) and increasing the accuracy, when domain knowledge suggest to identify anomalies occurring when actual values (`y`) are *above, below, or in both directions* relative to the expected values (`yhat`). Available choices are: `both`, `above_expected`, `below_expected`. Introduced in [1.13.0](../CHANGELOG.md#1130), `detection_direction` arg can help in reducing the number of [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/index.html#false-positive) and increasing the accuracy, when domain knowledge suggest to identify anomalies occurring when actual values (`y`) are *above, below, or in both directions* relative to the expected values (`yhat`). Available choices are: `both`, `above_expected`, `below_expected`.
Here's how default (backward-compatible) behavior looks like - anomalies will be tracked in `both` directions (`y > yhat` or `y < yhat`). This is useful when there is no domain expertise to filter the required direction. Here's how default (backward-compatible) behavior looks like - anomalies will be tracked in `both` directions (`y > yhat` or `y < yhat`). This is useful when there is no domain expertise to filter the required direction.
<img src="/anomaly-detection/components/schema_detection_direction=both.webp" width="800px" alt="schema_detection_direction=both"/> ![schema_detection_direction=both](schema_detection_direction=both.webp)
{width="800px"}
When set to `above_expected`, anomalies are tracked only when `y > yhat`. When set to `above_expected`, anomalies are tracked only when `y > yhat`.
*Example metrics*: Error rate, response time, page load time, number of failed transactions - metrics where *lower values are better*, so **higher** values are typically tracked. *Example metrics*: Error rate, response time, page load time, number of failed transactions - metrics where *lower values are better*, so **higher** values are typically tracked.
<img src="/anomaly-detection/components/schema_detection_direction=above_expected.webp" width="800px" alt="schema_detection_direction=above_expected"/> ![schema_detection_direction=above_expected](schema_detection_direction=above_expected.webp)
{width="800px"}
When set to `below_expected`, anomalies are tracked only when `y < yhat`. When set to `below_expected`, anomalies are tracked only when `y < yhat`.
*Example metrics*: Service Level Agreement (SLA) compliance, conversion rate, Customer Satisfaction Score (CSAT) - metrics where *higher values are better*, so **lower** values are typically tracked. *Example metrics*: Service Level Agreement (SLA) compliance, conversion rate, Customer Satisfaction Score (CSAT) - metrics where *higher values are better*, so **lower** values are typically tracked.
<img src="/anomaly-detection/components/schema_detection_direction=below_expected.webp" width="800px" alt="schema_detection_direction=below_expected"/> ![schema_detection_direction=below_expected](schema_detection_direction=below_expected.webp)
{width="800px"}
Config with a split example: Config with a split example:
@ -185,7 +185,7 @@ reader:
### Minimal deviation from expected ### Minimal deviation from expected
Introduced in [v1.13.0](/anomaly-detection/CHANGELOG/#1130), the `min_dev_from_expected` argument is designed to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the actual value (`y`) and the expected value (`yhat`) are **relatively** high. Such deviations can cause models to generate high [anomaly scores](/anomaly-detection/faq/#what-is-anomaly-score). However, these deviations may not be significant enough in **absolute values** from a business perspective to be considered anomalies. This parameter ensures that anomaly scores for data points where `|y - yhat| < min_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it behaves as `min_dev_from_expected=0` to maintain backward compatibility. Introduced in [v1.13.0](../CHANGELOG.md#1130), the `min_dev_from_expected` argument is designed to **reduce [false positives](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#false-positive)** in scenarios where deviations between the actual value (`y`) and the expected value (`yhat`) are **relatively** high. Such deviations can cause models to generate high [anomaly scores](../FAQ.md#what-is-anomaly-score). However, these deviations may not be significant enough in **absolute values** from a business perspective to be considered anomalies. This parameter ensures that anomaly scores for data points where `|y - yhat| < min_dev_from_expected` are explicitly set to 0. By default, if this parameter is not set, it behaves as `min_dev_from_expected=0` to maintain backward compatibility.
> **Note**: `min_dev_from_expected` must be >= 0. The higher the value of `min_dev_from_expected`, the fewer data points will be available for anomaly detection, and vice versa. > **Note**: `min_dev_from_expected` must be >= 0. The higher the value of `min_dev_from_expected`, the fewer data points will be available for anomaly detection, and vice versa.
@ -193,11 +193,14 @@ Introduced in [v1.13.0](/anomaly-detection/CHANGELOG/#1130), the `min_dev_from_e
Visualizations below demonstrate this concept; the green zone defined as the `[yhat - min_dev_from_expected, yhat + min_dev_from_expected]` range excludes actual data points (`y`) from generating anomaly scores if they fall within that range. Visualizations below demonstrate this concept; the green zone defined as the `[yhat - min_dev_from_expected, yhat + min_dev_from_expected]` range excludes actual data points (`y`) from generating anomaly scores if they fall within that range.
<img src="/anomaly-detection/components/schema_min_dev_from_expected=0.webp" width="800px" alt="min_dev_from_expected-default"/> ![min_dev_from_expected-default](schema_min_dev_from_expected=0.webp)
{width="800px"}
<img src="/anomaly-detection/components/schema_min_dev_from_expected=1.0.webp" width="800px" alt="min_dev_from_expected-small"/> ![min_dev_from_expected-small](schema_min_dev_from_expected=1.0.webp)
{width="800px"}
<img src="/anomaly-detection/components/schema_min_dev_from_expected=5.0.webp" width="800px" alt="min_dev_from_expected-big"/> ![min_dev_from_expected-big](schema_min_dev_from_expected=5.0.webp)
{width="800px}
Example config of how to use this param based on query results: Example config of how to use this param based on query results:
@ -238,7 +241,7 @@ Each of these models can also be
For a univariate type, **one separate model** is fit/used for inference per **each time series**, defined in its [queries](#queries) arg. For a univariate type, **one separate model** is fit/used for inference per **each time series**, defined in its [queries](#queries) arg.
For example, if you have some **univariate** model, defined to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be 3*5=15 models created in total. Each such model produce **individual [output](#vmanomaly-output)** for each of time series. For example, if you have some **univariate** model, defined to use 3 [MetricQL queries](../../MetricsQL.md), each returning 5 time series, there will be 3*5=15 models created in total. Each such model produce **individual [output](#vmanomaly-output)** for each of time series.
If during an inference, you got a series having **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step. If during an inference, you got a series having **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step.
@ -247,13 +250,14 @@ If during an inference, you got a series having **new labelset** (not present in
**Examples:** [Prophet](#prophet), [Holt-Winters](#holt-winters) **Examples:** [Prophet](#prophet), [Holt-Winters](#holt-winters)
<p></p> <p></p>
<img alt="vmanomaly-model-type-univariate" src="/anomaly-detection/components/model-lifecycle-univariate.webp" width="800px"/> ![vmanomaly-model-type-univariate](model-lifecycle-univariate.webp)
{width="800px"}
### Multivariate Models ### Multivariate Models
For a multivariate type, **one shared model** is fit/used for inference on **all time series** simultaneously, defined in its [queries](#queries) arg. For a multivariate type, **one shared model** is fit/used for inference on **all time series** simultaneously, defined in its [queries](#queries) arg.
For example, if you have some **multivariate** model to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**. For example, if you have some **multivariate** model to use 3 [MetricQL queries](../../MetricsQL.md), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**.
If during an inference, you got a **different amount of series** or some series having a **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step. If during an inference, you got a **different amount of series** or some series having a **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step.
@ -262,7 +266,8 @@ If during an inference, you got a **different amount of series** or some series
**Examples:** [IsolationForest](#isolation-forest-multivariate) **Examples:** [IsolationForest](#isolation-forest-multivariate)
<p></p> <p></p>
<img alt="vmanomaly-model-type-multivariate" src="/anomaly-detection/components/model-lifecycle-multivariate.webp" width="800px"/> ![vmanomaly-model-type-multivariate](model-lifecycle-multivariate.webp)
{width="800px"}
### Rolling Models ### Rolling Models
@ -270,16 +275,17 @@ A rolling model is a model that, once trained, **cannot be (naturally) used to m
An instance of rolling model is **simultaneously fit and used for inference** during its `infer` method call. An instance of rolling model is **simultaneously fit and used for inference** during its `infer` method call.
As a result, such model instances are **not stored** between consecutive re-fit calls (defined by `fit_every` [arg](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler) in `PeriodicScheduler`), leading to **lower RAM** consumption. As a result, such model instances are **not stored** between consecutive re-fit calls (defined by `fit_every` [arg](./scheduler.md#periodic-scheduler) in `PeriodicScheduler`), leading to **lower RAM** consumption.
Such models put **more pressure** on your reader's source, i.e. if your model should be fit on large amount of data (say, 14 days with 1-minute resolution) and at the same time you have **frequent inference** (say, once per minute) on new chunks of data - that's because such models require (fit + infer) window of data to be fit first to be used later in each inference call. Such models put **more pressure** on your reader's source, i.e. if your model should be fit on large amount of data (say, 14 days with 1-minute resolution) and at the same time you have **frequent inference** (say, once per minute) on new chunks of data - that's because such models require (fit + infer) window of data to be fit first to be used later in each inference call.
> **Note**: Rolling models require `fit_every` to be set equal to `infer_every` in your [PeriodicScheduler](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler). > **Note**: Rolling models require `fit_every` to be set equal to `infer_every` in your [PeriodicScheduler](./scheduler.md#periodic-scheduler).
**Examples:** [RollingQuantile](#rolling-quantile) **Examples:** [RollingQuantile](#rolling-quantile)
<p></p> <p></p>
<img alt="vmanomaly-model-type-rolling" src="/anomaly-detection/components/model-type-rolling.webp" width="800px"/> ![vmanomaly-model-type-rolling](model-type-rolling.webp)
{width="800px"}
### Non-Rolling Models ### Non-Rolling Models
@ -291,12 +297,13 @@ Such models put **less pressure** on your reader's source, i.e. if you fit on la
> **Note**: However, it's still highly recommended, to keep your model up-to-date with tendencies found in your data as it evolves in time. > **Note**: However, it's still highly recommended, to keep your model up-to-date with tendencies found in your data as it evolves in time.
Produced model instances are **stored in-memory** between consecutive re-fit calls (defined by `fit_every` [arg](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler) in `PeriodicScheduler`), leading to **higher RAM** consumption. Produced model instances are **stored in-memory** between consecutive re-fit calls (defined by `fit_every` [arg](./scheduler#periodic-scheduler) in `PeriodicScheduler`), leading to **higher RAM** consumption.
**Examples:** [Prophet](#prophet) **Examples:** [Prophet](#prophet)
<p></p> <p></p>
<img alt="vmanomaly-model-type-non-rolling" src="/anomaly-detection/components/model-type-non-rolling.webp" width="800px"/> ![vmanomaly-model-type-non-rolling](model-type-non-rolling.webp)
{width="800px}
## Built-in Models ## Built-in Models
@ -322,12 +329,12 @@ VM Anomaly Detection (`vmanomaly` hereinafter) models support 2 groups of parame
### AutoTuned ### AutoTuned
Tuning hyperparameters of a model can be tricky and often requires in-depth knowledge of Machine Learning. `AutoTunedModel` is designed specifically to take the cognitive load off the user - specify as little as `anomaly_percentage` param from `(0, 0.5)` interval and `tuned_model_class` (i.e. [`model.zscore.ZscoreModel`](/anomaly-detection/components/models/#z-score)) to get it working with best settings that match your data. Tuning hyperparameters of a model can be tricky and often requires in-depth knowledge of Machine Learning. `AutoTunedModel` is designed specifically to take the cognitive load off the user - specify as little as `anomaly_percentage` param from `(0, 0.5)` interval and `tuned_model_class` (i.e. [`model.zscore.ZscoreModel`](./models.md#z-score)) to get it working with best settings that match your data.
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.auto.AutoTunedModel"` (or `auto` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.auto.AutoTunedModel"` (or `auto` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `tuned_class_name` (string) - Built-in model class to tune, i.e. `model.zscore.ZscoreModel` (or `zscore` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support). * `tuned_class_name` (string) - Built-in model class to tune, i.e. `model.zscore.ZscoreModel` (or `zscore` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support).
* `optimization_params` (dict) - Optimization parameters for unsupervised model tuning. Control % of found anomalies, as well as a tradeoff between time spent and the accuracy. The more `timeout` and `n_trials` are, the better model configuration can be found for `tuned_class_name`, but the longer it takes and vice versa. Set `n_jobs` to `-1` to use all the CPUs available, it makes sense if only you have a big dataset to train on during `fit` calls, otherwise overhead isn't worth it. * `optimization_params` (dict) - Optimization parameters for unsupervised model tuning. Control % of found anomalies, as well as a tradeoff between time spent and the accuracy. The more `timeout` and `n_trials` are, the better model configuration can be found for `tuned_class_name`, but the longer it takes and vice versa. Set `n_jobs` to `-1` to use all the CPUs available, it makes sense if only you have a big dataset to train on during `fit` calls, otherwise overhead isn't worth it.
- `anomaly_percentage` (float) - expected percentage of anomalies that can be seen in training data, from (0, 0.5) interval. - `anomaly_percentage` (float) - expected percentage of anomalies that can be seen in training data, from (0, 0.5) interval.
- `seed` (int) - Random seed for reproducibility and deterministic nature of underlying optimizations. - `seed` (int) - Random seed for reproducibility and deterministic nature of underlying optimizations.
@ -335,7 +342,8 @@ Tuning hyperparameters of a model can be tricky and often requires in-depth know
- `n_trials` (int) - How many trials to sample from hyperparameter search space. The higher, the longer it takes but the better the results can be. Defaults to 128. - `n_trials` (int) - How many trials to sample from hyperparameter search space. The higher, the longer it takes but the better the results can be. Defaults to 128.
- `timeout` (float) - How many seconds in total can be spent on each model to tune hyperparameters. The higher, the longer it takes, allowing to test more trials out of defined `n_trials`, but the better the results can be. - `timeout` (float) - How many seconds in total can be spent on each model to tune hyperparameters. The higher, the longer it takes, allowing to test more trials out of defined `n_trials`, but the better the results can be.
<img alt="vmanomaly-autotune-schema" src="/anomaly-detection/components/autotune.webp" width="800px"/> ![vmanomaly-autotune-schema](autotune.webp)
{width="800px"}
```yaml ```yaml
# ... # ...
@ -356,7 +364,7 @@ models:
> **Note**: There are some expected limitations of Autotune mode: > **Note**: There are some expected limitations of Autotune mode:
> - It can't be made on your [custom model](#custom-model-guide). > - It can't be made on your [custom model](#custom-model-guide).
> - It can't be applied to itself (like `tuned_class_name: 'model.auto.AutoTunedModel'`) > - It can't be applied to itself (like `tuned_class_name: 'model.auto.AutoTunedModel'`)
> - `AutoTunedModel` can't be used on [rolling models](/anomaly-detection/components/models/#rolling-models) like [`RollingQuantile`](/anomaly-detection/components/models/#rolling-quantile) in combination with [on-disk model storage mode](/anomaly-detection/faq/#resource-consumption-of-vmanomaly), as the rolling models exists only during `infer` calls and aren't persisted neither in RAM, nor on disk. > - `AutoTunedModel` can't be used on [rolling models](./models.md#rolling-models) like [`RollingQuantile`](./models.md#rolling-quantile) in combination with [on-disk model storage mode](../FAQ.md#resource-consumption-of-vmanomaly), as the rolling models exists only during `infer` calls and aren't persisted neither in RAM, nor on disk.
### [Prophet](https://facebook.github.io/prophet/) ### [Prophet](https://facebook.github.io/prophet/)
@ -364,7 +372,7 @@ Here we utilize the Facebook Prophet implementation, as detailed in their [libra
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.prophet.ProphetModel"` (or `prophet` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.prophet.ProphetModel"` (or `prophet` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `seasonalities` (list[dict], optional) - Extra seasonalities to pass to Prophet. See [`add_seasonality()`](https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#modeling-holidays-and-special-events:~:text=modeling%20the%20cycle-,Specifying,-Custom%20Seasonalities) Prophet param. * `seasonalities` (list[dict], optional) - Extra seasonalities to pass to Prophet. See [`add_seasonality()`](https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#modeling-holidays-and-special-events:~:text=modeling%20the%20cycle-,Specifying,-Custom%20Seasonalities) Prophet param.
**Note**: Apart from standard `vmanomaly` output, Prophet model can provide [additional metrics](#additional-output-metrics-produced-by-fb-prophet). **Note**: Apart from standard `vmanomaly` output, Prophet model can provide [additional metrics](#additional-output-metrics-produced-by-fb-prophet).
@ -404,7 +412,7 @@ Resulting metrics of the model are described [here](#vmanomaly-output)
### [Z-score](https://en.wikipedia.org/wiki/Standard_score) ### [Z-score](https://en.wikipedia.org/wiki/Standard_score)
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.zscore.ZscoreModel"` (or `zscore` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.zscore.ZscoreModel"` (or `zscore` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `z_threshold` (float, optional) - [standard score](https://en.wikipedia.org/wiki/Standard_score) for calculation boundaries and anomaly score. Defaults to `2.5`. * `z_threshold` (float, optional) - [standard score](https://en.wikipedia.org/wiki/Standard_score) for calculation boundaries and anomaly score. Defaults to `2.5`.
*Config Example* *Config Example*
@ -425,7 +433,7 @@ Here we use Holt-Winters Exponential Smoothing implementation from `statsmodels`
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.holtwinters.HoltWinters"` (or `holtwinters` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.holtwinters.HoltWinters"` (or `holtwinters` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `frequency` (string) - Must be set equal to sampling_period. Model needs to know expected data-points frequency (e.g. '10m'). If omitted, frequency is guessed during fitting as **the median of intervals between fitting data timestamps**. During inference, if incoming data doesn't have the same frequency, then it will be interpolated. E.g. data comes at 15 seconds resolution, and our resample_freq is '1m'. Then fitting data will be downsampled to '1m' and internal model is trained at '1m' intervals. So, during inference, prediction data would be produced at '1m' intervals, but interpolated to "15s" to match with expected output, as output data must have the same timestamps. As accepted by pandas.Timedelta (e.g. '5m'). * `frequency` (string) - Must be set equal to sampling_period. Model needs to know expected data-points frequency (e.g. '10m'). If omitted, frequency is guessed during fitting as **the median of intervals between fitting data timestamps**. During inference, if incoming data doesn't have the same frequency, then it will be interpolated. E.g. data comes at 15 seconds resolution, and our resample_freq is '1m'. Then fitting data will be downsampled to '1m' and internal model is trained at '1m' intervals. So, during inference, prediction data would be produced at '1m' intervals, but interpolated to "15s" to match with expected output, as output data must have the same timestamps. As accepted by pandas.Timedelta (e.g. '5m').
@ -467,7 +475,7 @@ The MAD model is a robust method for anomaly detection that is *less sensitive*
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.mad.MADModel"` (or `mad` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.mad.MADModel"` (or `mad` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `threshold` (float, optional) - The threshold multiplier for the MAD to determine anomalies. Defaults to `2.5`. Higher values will identify fewer points as anomalies. * `threshold` (float, optional) - The threshold multiplier for the MAD to determine anomalies. Defaults to `2.5`. Higher values will identify fewer points as anomalies.
*Config Example* *Config Example*
@ -487,7 +495,7 @@ Resulting metrics of the model are described [here](#vmanomaly-output).
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.rolling_quantile.RollingQuantileModel"` (or `rolling_quantile` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.rolling_quantile.RollingQuantileModel"` (or `rolling_quantile` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `quantile` (float) - quantile value, from 0.5 to 1.0. This constraint is implied by 2-sided confidence interval. * `quantile` (float) - quantile value, from 0.5 to 1.0. This constraint is implied by 2-sided confidence interval.
* `window_steps` (integer) - size of the moving window. (see 'sampling_period') * `window_steps` (integer) - size of the moving window. (see 'sampling_period')
@ -508,7 +516,7 @@ Here we use Seasonal Decompose implementation from `statsmodels` [library](https
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.std.StdModel"` (or `std` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.std.StdModel"` (or `std` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `period` (integer) - Number of datapoints in one season. * `period` (integer) - Number of datapoints in one season.
* `z_threshold` (float, optional) - [standard score](https://en.wikipedia.org/wiki/Standard_score) for calculating boundaries to define anomaly score. Defaults to `2.5`. * `z_threshold` (float, optional) - [standard score](https://en.wikipedia.org/wiki/Standard_score) for calculating boundaries to define anomaly score. Defaults to `2.5`.
@ -541,11 +549,11 @@ Here we use Isolation Forest implementation from `scikit-learn` [library](https:
*Parameters specific for vmanomaly*: *Parameters specific for vmanomaly*:
* `class` (string) - model class name `"model.isolation_forest.IsolationForestMultivariateModel"` (or `isolation_forest_multivariate` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) * `class` (string) - model class name `"model.isolation_forest.IsolationForestMultivariateModel"` (or `isolation_forest_multivariate` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
* `contamination` (float or string, optional) - The amount of contamination of the data set, i.e. the proportion of outliers in the data set. Used when fitting to define the threshold on the scores of the samples. Default value - "auto". Should be either `"auto"` or be in the range (0.0, 0.5]. * `contamination` (float or string, optional) - The amount of contamination of the data set, i.e. the proportion of outliers in the data set. Used when fitting to define the threshold on the scores of the samples. Default value - "auto". Should be either `"auto"` or be in the range (0.0, 0.5].
* `seasonal_features` (list of string) - List of seasonality to encode through [cyclical encoding](https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca), i.e. `dow` (day of week). **Introduced in [1.12.0](/anomaly-detection/CHANGELOG/#v1120)**. * `seasonal_features` (list of string) - List of seasonality to encode through [cyclical encoding](https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca), i.e. `dow` (day of week). **Introduced in [1.12.0](../CHANGELOG.md#v1120)**.
- Empty by default for backward compatibility. - Empty by default for backward compatibility.
- Example: `seasonal_features: ['dow', 'hod']`. - Example: `seasonal_features: ['dow', 'hod']`.
- Supported seasonalities: - Supported seasonalities:
@ -601,12 +609,12 @@ The default metrics produced by `vmanomaly` include:
## vmanomaly monitoring metrics ## vmanomaly monitoring metrics
Each model exposes [several monitoring metrics](/anomaly-detection/components/monitoring/#models-behaviour-metrics) to its `health_path` endpoint: Each model exposes [several monitoring metrics](./monitoring.md#models-behaviour-metrics) to its `health_path` endpoint:
## Custom Model Guide ## Custom Model Guide
Apart from `vmanomaly` [built-in models](/anomaly-detection/components/models/#built-in-models), users can create their own custom models for anomaly detection. Apart from `vmanomaly` [built-in models](./models.md#built-in-models), users can create their own custom models for anomaly detection.
Here in this guide, we will Here in this guide, we will
- Make a file containing our custom model definition - Make a file containing our custom model definition
@ -698,8 +706,8 @@ class CustomModel(Model):
### 2. Configuration file ### 2. Configuration file
Next, we need to create `config.yaml` file with `vmanomaly` configuration and model input parameters. Next, we need to create `config.yaml` file with `vmanomaly` configuration and model input parameters.
In the config file's `models` section we need to set our model class to `model.custom.CustomModel` (or `custom` starting from [v1.13.0](/anomaly-detection/CHANGELOG/#1130) with class alias support) and define all parameters used in `__init__` method. In the config file's `models` section we need to set our model class to `model.custom.CustomModel` (or `custom` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support) and define all parameters used in `__init__` method.
You can find out more about configuration parameters in `vmanomaly` [config docs](/anomaly-detection/components/). You can find out more about configuration parameters in `vmanomaly` [config docs](./README.md).
```yaml ```yaml
schedulers: schedulers:
@ -762,7 +770,7 @@ victoriametrics/vmanomaly:latest /config.yaml \
--license-file=/license --license-file=/license
``` ```
Please find more detailed instructions (license, etc.) [here](/anomaly-detection/overview.html#run-vmanomaly-docker-container) Please find more detailed instructions (license, etc.) [here](../Overview.md#run-vmanomaly-docker-container)
### Output ### Output

318
docs/anomaly-detection/components/monitoring.md
View file

@ -8,16 +8,13 @@ menu:
weight: 5 weight: 5
identifier: "vmanomaly-monitoring" identifier: "vmanomaly-monitoring"
aliases: aliases:
- /anomaly-detection/components/monitoring.html - ./monitoring.html
--- ---
There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push](../../keyConcepts.md#push-model) and [pull](../../keyConcepts.md#pull-model). Parameters for each of them should be specified in the config file, `monitoring` section.
# Monitoring
There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [pull](https://docs.victoriametrics.com/keyconcepts/#pull-model). Parameters for each of them should be specified in the config file, `monitoring` section.
## Pull Model Config parameters ## Pull Model Config parameters
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -27,13 +24,25 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>addr</code></td> <td>
<td><code>"0.0.0.0"</code></td>
`addr`
</td>
<td>
`"0.0.0.0"`
</td>
<td>Server IP Address</td> <td>Server IP Address</td>
</tr> </tr>
<tr> <tr>
<td><code>port</code></td> <td>
<td><code>8080</code></td>
`port`
</td>
<td>
`8080`
</td>
<td>Port</td> <td>Port</td>
</tr> </tr>
</tbody> </tbody>
@ -41,7 +50,7 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
## Push Config parameters ## Push Config parameters
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -51,42 +60,84 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>url</code></td> <td>
`url`
</td>
<td></td> <td></td>
<td>Link where to push metrics to. Example: <code>"http://localhost:8480/"</code></td> <td>
Link where to push metrics to. Example: `"http://localhost:8480/"`
</td>
</tr> </tr>
<tr> <tr>
<td><code>tenant_id</code></td> <td>
`tenant_id`
</td>
<td></td> <td></td>
<td>Tenant ID for cluster version. Example: <code>"0:0"</code></td> <td>
Tenant ID for cluster version. Example: `"0:0"`
</td>
</tr> </tr>
<tr> <tr>
<td><code>health_path</code></td> <td>
<td><code>"health"</code></td>
<td>Deprecated since <a href="https://docs.victoriametrics.com/anomaly-detection/CHANGELOG.html#v180">v1.8.0</a>. Absolute, to override <code>/health</code> path</td> `health_path`
</td>
<td>
`"health"`
</td>
<td>
Deprecated since [v1.8.0](../CHANGELOG.md#v180). Absolute, to override `/health` path
</td>
</tr> </tr>
<tr> <tr>
<td><code>user</code></td> <td>
`user`
</td>
<td></td> <td></td>
<td>BasicAuth username</td> <td>BasicAuth username</td>
</tr> </tr>
<tr> <tr>
<td><code>password</code></td> <td>
`password`
</td>
<td></td> <td></td>
<td>BasicAuth password</td> <td>BasicAuth password</td>
</tr> </tr>
<tr> <tr>
<td><code>verify_tls</code></td> <td>
<td><code>False</code></td>
`verify_tls`
</td>
<td>
`False`
</td>
<td>Allows disabling TLS verification of the remote certificate.</td> <td>Allows disabling TLS verification of the remote certificate.</td>
</tr> </tr>
<tr> <tr>
<td><code>timeout</code></td> <td>
<td><code>"5s"</code></td>
`timeout`
</td>
<td>
`"5s"`
</td>
<td>Stop waiting for a response after a given number of seconds.</td> <td>Stop waiting for a response after a given number of seconds.</td>
</tr> </tr>
<tr> <tr>
<td><code>extra_labels</code></td> <td>
`extra_labels`
</td>
<td></td> <td></td>
<td>Section for custom labels specified by user.</td> <td>Section for custom labels specified by user.</td>
</tr> </tr>
@ -114,7 +165,7 @@ monitoring:
## Metrics generated by vmanomaly ## Metrics generated by vmanomaly
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Metric</th> <th>Metric</th>
@ -124,7 +175,10 @@ monitoring:
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>vmanomaly_start_time_seconds</code></td> <td>
`vmanomaly_start_time_seconds`
</td>
<td>Gauge</td> <td>Gauge</td>
<td>vmanomaly start time in UNIX time</td> <td>vmanomaly start time in UNIX time</td>
</tr> </tr>
@ -134,9 +188,9 @@ monitoring:
### Models Behaviour Metrics ### Models Behaviour Metrics
Label names [description](#labelnames) Label names [description](#labelnames)
> **Note**: There is a new label key `model_alias` introduced in multi-model support [v1.10.0](/anomaly-detection/changelog/#v1100). This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics. > **Note**: There is a new label key `model_alias` introduced in multi-model support [v1.10.0](../CHANGELOG.md#v1100). This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Metric</th> <th>Metric</th>
@ -147,40 +201,76 @@ Label names [description](#labelnames)
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>vmanomaly_model_runs</code></td> <td>
`vmanomaly_model_runs`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many times models ran (per model)</td> <td>How many times models ran (per model)</td>
<td><code>stage, query_key, model_alias, scheduler_alias, preset</code></td> <td>
`stage, query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_model_run_duration_seconds</code></td> <td>
`vmanomaly_model_run_duration_seconds`
</td>
<td>Summary</td> <td>Summary</td>
<td>How much time (in seconds) model invocations took</td> <td>How much time (in seconds) model invocations took</td>
<td><code>stage, query_key, model_alias, scheduler_alias, preset</code></td> <td>
`stage, query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_model_datapoints_accepted</code></td> <td>
`vmanomaly_model_datapoints_accepted`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many datapoints did models accept</td> <td>How many datapoints did models accept</td>
<td><code>stage, query_key, model_alias, scheduler_alias, preset</code></td> <td>
`stage, query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_model_datapoints_produced</code></td> <td>
`vmanomaly_model_datapoints_produced`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many datapoints were generated by models</td> <td>How many datapoints were generated by models</td>
<td><code>stage, query_key, model_alias, scheduler_alias, preset</code></td> <td>
`stage, query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_models_active</code></td> <td>
`vmanomaly_models_active`
</td>
<td>Gauge</td> <td>Gauge</td>
<td>How many models are currently inferring</td> <td>How many models are currently inferring</td>
<td><code>query_key, model_alias, scheduler_alias, preset</code></td> <td>
`query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_model_runs_skipped</code></td> <td>
`vmanomaly_model_runs_skipped`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many times a run was skipped (per model)</td> <td>How many times a run was skipped (per model)</td>
<td><code>stage, query_key, model_alias, scheduler_alias, preset</code></td> <td>
`stage, query_key, model_alias, scheduler_alias, preset`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -188,7 +278,7 @@ Label names [description](#labelnames)
### Writer Behaviour Metrics ### Writer Behaviour Metrics
Label names [description](#labelnames) Label names [description](#labelnames)
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Metric</th> <th>Metric</th>
@ -199,40 +289,76 @@ Label names [description](#labelnames)
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>vmanomaly_writer_request_duration_seconds</code></td> <td>
`vmanomaly_writer_request_duration_seconds`
</td>
<td>Summary</td> <td>Summary</td>
<td>How much time (in seconds) did requests to VictoriaMetrics take</td> <td>How much time (in seconds) did requests to VictoriaMetrics take</td>
<td><code>url, query_key</code></td> <td>
`url, query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_writer_response_count</code></td> <td>
`vmanomaly_writer_response_count`
</td>
<td>Counter</td> <td>Counter</td>
<td>Response code counts we got from VictoriaMetrics</td> <td>Response code counts we got from VictoriaMetrics</td>
<td><code>url, query_key, code</code></td> <td>
`url, query_key, code`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_writer_sent_bytes</code></td> <td>
`vmanomaly_writer_sent_bytes`
</td>
<td>Counter</td> <td>Counter</td>
<td>How much bytes were sent to VictoriaMetrics</td> <td>How much bytes were sent to VictoriaMetrics</td>
<td><code>url, query_key</code></td> <td>
`url, query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_writer_request_serialize_seconds</code></td> <td>
`vmanomaly_writer_request_serialize_seconds`
</td>
<td>Summary</td> <td>Summary</td>
<td>How much time (in seconds) did serializing take</td> <td>How much time (in seconds) did serializing take</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_writer_datapoints_sent</code></td> <td>
`vmanomaly_writer_datapoints_sent`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many datapoints were sent to VictoriaMetrics</td> <td>How many datapoints were sent to VictoriaMetrics</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_writer_timeseries_sent</code></td> <td>
`vmanomaly_writer_timeseries_sent`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many timeseries were sent to VictoriaMetrics</td> <td>How many timeseries were sent to VictoriaMetrics</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -240,7 +366,7 @@ Label names [description](#labelnames)
### Reader Behaviour Metrics ### Reader Behaviour Metrics
Label names [description](#labelnames) Label names [description](#labelnames)
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Metric</th> <th>Metric</th>
@ -251,57 +377,87 @@ Label names [description](#labelnames)
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>vmanomaly_reader_request_duration_seconds</code></td> <td>
`vmanomaly_reader_request_duration_seconds`
</td>
<td>Summary</td> <td>Summary</td>
<td>How much time (in seconds) did queries to VictoriaMetrics take</td> <td>How much time (in seconds) did queries to VictoriaMetrics take</td>
<td><code>url, query_key</code></td> <td>
`url, query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_reader_response_count</code></td> <td>
`vmanomaly_reader_response_count`
</td>
<td>Counter</td> <td>Counter</td>
<td>Response code counts we got from VictoriaMetrics</td> <td>Response code counts we got from VictoriaMetrics</td>
<td><code>url, query_key, code</code></td> <td>
`url, query_key, code`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_reader_received_bytes</code></td> <td>
`vmanomaly_reader_received_bytes`
</td>
<td>Counter</td> <td>Counter</td>
<td>How much bytes were received in responses</td> <td>How much bytes were received in responses</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_reader_response_parsing_seconds</code></td> <td>
`vmanomaly_reader_response_parsing_seconds`
</td>
<td>Summary</td> <td>Summary</td>
<td>How much time (in seconds) did parsing take for each step</td> <td>How much time (in seconds) did parsing take for each step</td>
<td><code>step</code></td> <td>
`step`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_reader_timeseries_received</code></td> <td>
`vmanomaly_reader_timeseries_received`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many timeseries were received from VictoriaMetrics</td> <td>How many timeseries were received from VictoriaMetrics</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
<tr> <tr>
<td><code>vmanomaly_reader_datapoints_received</code></td> <td>
`vmanomaly_reader_datapoints_received`
</td>
<td>Counter</td> <td>Counter</td>
<td>How many rows were received from VictoriaMetrics</td> <td>How many rows were received from VictoriaMetrics</td>
<td><code>query_key</code></td> <td>
`query_key`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
### Labelnames ### Labelnames
<code>stage</code> - stage of model - 'fit', 'infer' or 'fit_infer' for models that do it simultaneously, see [model types](/anomaly-detection/components/models/#model-types).
<code>query_key</code> - query alias from [`reader`](/anomaly-detection/components/reader.html) config section. * `stage` - stage of model - 'fit', 'infer' or 'fit_infer' for models that do it simultaneously, see [model types](./models/#model-types).
* `query_key` - query alias from [`reader`](./reader.md) config section.
<code>model_alias</code> - model alias from [`models`](/anomaly-detection/components/models.html) config section. **Introduced in [v1.10.0](/anomaly-detection/changelog/#v1100).** * `model_alias` - model alias from [`models`](./models.md) config section. **Introduced in [v1.10.0](../CHANGELOG.md#v1100).**
* `scheduler_alias` - scheduler alias from [`schedulers`](./scheduler.md) config section. **Introduced in [v1.11.0](../CHANGELOG.md#v1110).**
<code>scheduler_alias</code> - scheduler alias from [`schedulers`](anomaly-detection/components/scheduler/) config section. **Introduced in [v1.11.0](/anomaly-detection/changelog/#v1110).** * `preset` - preset alias for forthcoming `preset` section compatibility. **Introduced in [v1.12.0](../CHANGELOG.md#v1120).**
* `url` - writer or reader url endpoint.
<code>preset</code> - preset alias for forthcoming `preset` section compatibility. **Introduced in [v1.12.0](/anomaly-detection/changelog/#v1120).** * `code` - response status code or `connection_error`, `timeout`.
* `step` - json or dataframe reading step.
<code>url</code> - writer or reader url endpoint.
<code>code</code> - response status code or `connection_error`, `timeout`.
<code>step</code> - json or dataframe reading step.

204
docs/anomaly-detection/components/reader.md
View file

@ -9,14 +9,11 @@ menu:
aliases: aliases:
- /anomaly-detection/components/reader.html - /anomaly-detection/components/reader.html
--- ---
# Reader
<!-- <!--
There are 4 sources available to read data into VM Anomaly Detection from: VictoriaMetrics, (ND)JSON file, QueryRange, or CSV file. Depending on the data source, different parameters should be specified in the config file in the `reader` section. There are 4 sources available to read data into VM Anomaly Detection from: VictoriaMetrics, (ND)JSON file, QueryRange, or CSV file. Depending on the data source, different parameters should be specified in the config file in the `reader` section.
--> -->
VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily uses [VmReader](#vm-reader) to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful [MetricsQL](https://docs.victoriametrics.com/metricsql/) expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling. VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily uses [VmReader](#vm-reader) to ingest data. This reader focuses on fetching time-series data directly from VictoriaMetrics with the help of powerful [MetricsQL](../../MetricsQL.md) expressions for aggregating, filtering and grouping your data, ensuring seamless integration and efficient data handling.
Future updates will introduce additional readers, expanding the range of data sources `vmanomaly` can work with. Future updates will introduce additional readers, expanding the range of data sources `vmanomaly` can work with.
@ -25,7 +22,7 @@ Future updates will introduce additional readers, expanding the range of data so
### Config parameters ### Config parameters
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -35,69 +32,186 @@ Future updates will introduce additional readers, expanding the range of data so
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>class</code></td> <td>
<td><code>"reader.vm.VmReader" (or "vm" starting from <a href="https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130">v1.13.0</a>)</code></td>
<td>Name of the class needed to enable reading from VictoriaMetrics or Prometheus. VmReader is the default option, if not specified.</td> `class`
</td>
<td>
`reader.vm.VmReader` (or `vm` starting from [v1.13.0](../CHANGELOG.md#v1130))
</td>
<td>
Name of the class needed to enable reading from VictoriaMetrics or Prometheus. VmReader is the default option, if not specified.
</td>
</tr> </tr>
<tr> <tr>
<td><code>queries</code></td> <td>
<td><code>"ingestion_rate: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'"</code></td>
<td>PromQL/MetricsQL query to select data in format: <code>QUERY_ALIAS: "QUERY"</code>. As accepted by <code>"/query_range?query=%s"</code>.</td> `queries`
</td>
<td>
`ingestion_rate: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'`
</td>
<td>
PromQL/MetricsQL query to select data in format: `QUERY_ALIAS: "QUERY"`. As accepted by `/query_range?query=%s`.
</td>
</tr> </tr>
<tr> <tr>
<td><code>datasource_url</code></td> <td>
<td><code>"http://localhost:8481/"</code></td>
<td>Datasource URL address</td> `datasource_url`
</td>
<td>
`http://localhost:8481/`
</td>
<td>
Datasource URL address
</td>
</tr> </tr>
<tr> <tr>
<td><code>tenant_id</code></td> <td>
<td><code>"0:0"</code></td>
<td>For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster <a href="https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy">multitenancy docs</a></td> `tenant_id`
</td>
<td>
`0:0`
</td>
<td>
For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster [multitenancy docs](../../Cluster-VictoriaMetrics.md#multitenancy)
</td>
</tr> </tr>
<tr> <tr>
<td><code>sampling_period</code></td> <td>
<td><code>"1h"</code></td>
<td>Frequency of the points returned. Will be converted to <code>"/query_range?step=%s"</code> param (in seconds). <b>Required</b> since <a href="https://docs.victoriametrics.com/anomaly-detection/changelog/#v190">v1.9.0</a>.</td> `sampling_period`
</td>
<td>
`1h`
</td>
<td>
Frequency of the points returned. Will be converted to `/query_range?step=%s` param (in seconds). **Required** since [v1.9.0](../CHANGELOG.md#v190).
</td>
</tr> </tr>
<tr> <tr>
<td><code>query_range_path</code></td> <td>
<td><code>"api/v1/query_range"</code></td>
<td>Performs PromQL/MetricsQL range query. Default <code>"api/v1/query_range"</code></td> `query_range_path`
</td>
<td>
`/api/v1/query_range`
</td>
<td>
Performs PromQL/MetricsQL range query
</td>
</tr> </tr>
<tr> <tr>
<td><code>health_path</code></td> <td>
<td><code>"health"</code></td>
<td>Absolute or relative URL address where to check availability of the datasource. Default is <code>"health"</code>.</td> `health_path`
</td>
<td>
`health`
</td>
<td>
Absolute or relative URL address where to check availability of the datasource.
</td>
</tr> </tr>
<tr> <tr>
<td><code>user</code></td> <td>
<td><code>"USERNAME"</code></td>
<td>BasicAuth username</td> `user`
</td>
<td>
`USERNAME`
</td>
<td>
BasicAuth username
</td>
</tr> </tr>
<tr> <tr>
<td><code>password</code></td> <td>
<td><code>"PASSWORD"</code></td>
<td>BasicAuth password</td> `password`
</td>
<td>
`PASSWORD`
</td>
<td>
BasicAuth password
</td>
</tr> </tr>
<tr> <tr>
<td><code>timeout</code></td> <td>
<td><code>"30s"</code></td>
<td>Timeout for the requests, passed as a string. Defaults to "30s"</td> `timeout`
</td>
<td>
`30s`
</td>
<td>
Timeout for the requests, passed as a string
</td>
</tr> </tr>
<tr> <tr>
<td><code>verify_tls</code></td> <td>
<td><code>"false"</code></td>
<td>Allows disabling TLS verification of the remote certificate.</td> `verify_tls`
</td>
<td>
`false`
</td>
<td>
Allows disabling TLS verification of the remote certificate.
</td>
</tr> </tr>
<tr> <tr>
<td><code>bearer_token</code></td> <td>
<td><code>"token"</code></td>
<td>Token is passed in the standard format with header: "Authorization: bearer {token}"</td> `bearer_token`
</td>
<td>
`token`
</td>
<td>
Token is passed in the standard format with header: `Authorization: bearer {token}`
</td>
</tr> </tr>
<tr> <tr>
<td><code>extra_filters</code></td> <td>
<td><code>"[]"</code></td>
<td>List of strings with series selector. See: <a href="https://docs.victoriametrics.com/#prometheus-querying-api-enhancements">Prometheus querying API enhancements</a></td> `extra_filters`
</td>
<td>
`[]`
</td>
<td>
List of strings with series selector. See: [Prometheus querying API enhancements](../../README.md##prometheus-querying-api-enhancements)
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -116,4 +230,4 @@ reader:
### Healthcheck metrics ### Healthcheck metrics
`VmReader` exposes [several healthchecks metrics](./monitoring.html#reader-behaviour-metrics). `VmReader` exposes [several healthchecks metrics](./monitoring.md#reader-behaviour-metrics).

212
docs/anomaly-detection/components/scheduler.md
View file

@ -9,13 +9,10 @@ menu:
aliases: aliases:
- /anomaly-detection/components/scheduler.html - /anomaly-detection/components/scheduler.html
--- ---
# Scheduler
Scheduler defines how often to run and make inferences, as well as what timerange to use to train the model. Scheduler defines how often to run and make inferences, as well as what timerange to use to train the model.
Is specified in `scheduler` section of a config for VictoriaMetrics Anomaly Detection. Is specified in `scheduler` section of a config for VictoriaMetrics Anomaly Detection.
> **Note: Starting from [v1.11.0](/anomaly-detection/changelog#v1110) scheduler section in config supports multiple schedulers via aliasing. <br>Also, `vmanomaly` expects scheduler section to be named `schedulers`. Using old (flat) format with `scheduler` key is deprecated and will be removed in future versions.** > **Note: Starting from [v1.11.0](../CHANGELOG.md#v1110) scheduler section in config supports multiple schedulers via aliasing. <br>Also, `vmanomaly` expects scheduler section to be named `schedulers`. Using old (flat) format with `scheduler` key is deprecated and will be removed in future versions.**
```yaml ```yaml
schedulers: schedulers:
@ -32,7 +29,7 @@ schedulers:
... ...
``` ```
Old-style configs (< [1.11.0](/anomaly-detection/changelog#v1110)) Old-style configs (< [1.11.0](../CHANGELOG.md#v1110))
```yaml ```yaml
scheduler: scheduler:
@ -64,7 +61,7 @@ options={`"scheduler.periodic.PeriodicScheduler"`, `"scheduler.oneoff.OneoffSche
- `"scheduler.oneoff.OneoffScheduler"`: runs the process once and exits. Useful for testing. - `"scheduler.oneoff.OneoffScheduler"`: runs the process once and exits. Useful for testing.
- `"scheduler.backtesting.BacktestingScheduler"`: imitates consecutive backtesting runs of OneoffScheduler. Runs the process once and exits. Use to get more granular control over testing on historical data. - `"scheduler.backtesting.BacktestingScheduler"`: imitates consecutive backtesting runs of OneoffScheduler. Runs the process once and exits. Use to get more granular control over testing on historical data.
> **Note**: starting from [v.1.13.0](/anomaly-detection/CHANGELOG/#v1130), class aliases are supported, so `"scheduler.periodic.PeriodicScheduler"` can be substituted to `"periodic"`, `"scheduler.oneoff.OneoffScheduler"` - to `"oneoff"`, `"scheduler.backtesting.BacktestingScheduler"` - to `"backtesting"` > **Note**: starting from [v1.13.0](../CHANGELOG.md#v1130), class aliases are supported, so `"scheduler.periodic.PeriodicScheduler"` can be substituted to `"periodic"`, `"scheduler.oneoff.OneoffScheduler"` - to `"oneoff"`, `"scheduler.backtesting.BacktestingScheduler"` - to `"backtesting"`
**Depending on selected class, different parameters should be used** **Depending on selected class, different parameters should be used**
@ -76,7 +73,7 @@ For periodic scheduler parameters are defined as differences in times, expressed
Examples: `"50s"`, `"4m"`, `"3h"`, `"2d"`, `"1w"`. Examples: `"50s"`, `"4m"`, `"3h"`, `"2d"`, `"1w"`.
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th></th> <th></th>
@ -107,7 +104,7 @@ Examples: `"50s"`, `"4m"`, `"3h"`, `"2d"`, `"1w"`.
</tbody> </tbody>
</table> </table>
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -118,22 +115,43 @@ Examples: `"50s"`, `"4m"`, `"3h"`, `"2d"`, `"1w"`.
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>fit_window</code></td> <td>
`fit_window`
</td>
<td>str</td> <td>str</td>
<td><code>"14d"</code></td> <td>
`"14d"`
</td>
<td>What time range to use for training the models. Must be at least 1 second.</td> <td>What time range to use for training the models. Must be at least 1 second.</td>
</tr> </tr>
<tr> <tr>
<td><code>infer_every</code></td> <td>
`infer_every`
</td>
<td>str</td> <td>str</td>
<td><code>"1m"</code></td> <td>
`"1m"`
</td>
<td>How often a model will write its conclusions on newly added data. Must be at least 1 second.</td> <td>How often a model will write its conclusions on newly added data. Must be at least 1 second.</td>
</tr> </tr>
<tr> <tr>
<td><code>fit_every</code></td> <td>
`fit_every`
</td>
<td>str, Optional</td> <td>str, Optional</td>
<td><code>"1h"</code></td> <td>
<td>How often to completely retrain the models. If missing value of <code>infer_every</code> is used and retrain on every inference run.</td>
`"1h"`
</td>
<td>
How often to completely retrain the models. If missing value of `infer_every` is used and retrain on every inference run.
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -165,7 +183,7 @@ ISO format supported time zone offset formats are:
If a time zone is omitted, a timezone-naive datetime is used. If a time zone is omitted, a timezone-naive datetime is used.
### Defining fitting timeframe ### Defining fitting timeframe
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Format</th> <th>Format</th>
@ -178,27 +196,48 @@ If a time zone is omitted, a timezone-naive datetime is used.
<tbody> <tbody>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>fit_start_iso</code></td> <td>
`fit_start_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-01T00:00:00Z", "2022-04-01T00:00:00+01:00", "2022-04-01T00:00:00+0100", "2022-04-01T00:00:00+01"</code></td> <td>
`"2022-04-01T00:00:00Z", "2022-04-01T00:00:00+01:00", "2022-04-01T00:00:00+0100", "2022-04-01T00:00:00+01"`
</td>
<td rowspan=2>Start datetime to use for training a model. ISO string or UNIX time in seconds.</td> <td rowspan=2>Start datetime to use for training a model. ISO string or UNIX time in seconds.</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>fit_start_s</code></td> <td>
`fit_start_s`
</td>
<td>float</td> <td>float</td>
<td>1648771200</td> <td>1648771200</td>
</tr> </tr>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>fit_end_iso</code></td> <td>
`fit_end_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-10T00:00:00Z", "2022-04-10T00:00:00+01:00", "2022-04-10T00:00:00+0100", "2022-04-10T00:00:00+01"</code></td> <td>
<td rowspan=2>End datetime to use for training a model. Must be greater than <code>fit_start_*</code>. ISO string or UNIX time in seconds.</td>
`"2022-04-10T00:00:00Z", "2022-04-10T00:00:00+01:00", "2022-04-10T00:00:00+0100", "2022-04-10T00:00:00+01"`
</td>
<td rowspan=2>End datetime to use for training a model. Must be greater than
`fit_start_*`
. ISO string or UNIX time in seconds.</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>fit_end_s</code></td> <td>
`fit_end_s`
</td>
<td>float</td> <td>float</td>
<td>1649548800</td> <td>1649548800</td>
</tr> </tr>
@ -206,7 +245,7 @@ If a time zone is omitted, a timezone-naive datetime is used.
</table> </table>
### Defining inference timeframe ### Defining inference timeframe
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Format</th> <th>Format</th>
@ -219,27 +258,48 @@ If a time zone is omitted, a timezone-naive datetime is used.
<tbody> <tbody>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>infer_start_iso</code></td> <td>
`infer_start_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-11T00:00:00Z", "2022-04-11T00:00:00+01:00", "2022-04-11T00:00:00+0100", "2022-04-11T00:00:00+01"</code></td> <td>
`"2022-04-11T00:00:00Z", "2022-04-11T00:00:00+01:00", "2022-04-11T00:00:00+0100", "2022-04-11T00:00:00+01"`
</td>
<td rowspan=2>Start datetime to use for a model inference. ISO string or UNIX time in seconds.</td> <td rowspan=2>Start datetime to use for a model inference. ISO string or UNIX time in seconds.</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>infer_start_s</code></td> <td>
`infer_start_s`
</td>
<td>float</td> <td>float</td>
<td>1649635200</td> <td>1649635200</td>
</tr> </tr>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>infer_end_iso</code></td> <td>
`infer_end_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-14T00:00:00Z", "2022-04-14T00:00:00+01:00", "2022-04-14T00:00:00+0100", "2022-04-14T00:00:00+01"</code></td> <td>
<td rowspan=2>End datetime to use for a model inference. Must be greater than <code>infer_start_*</code>. ISO string or UNIX time in seconds.</td>
`"2022-04-14T00:00:00Z", "2022-04-14T00:00:00+01:00", "2022-04-14T00:00:00+0100", "2022-04-14T00:00:00+01"`
</td>
<td rowspan=2>End datetime to use for a model inference. Must be greater than
`infer_start_*`
. ISO string or UNIX time in seconds.</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>infer_end_s</code></td> <td>
`infer_end_s`
</td>
<td>float</td> <td>float</td>
<td>1649894400</td> <td>1649894400</td>
</tr> </tr>
@ -284,7 +344,7 @@ ISO format supported time zone offset formats are:
If a time zone is omitted, a timezone-naive datetime is used. If a time zone is omitted, a timezone-naive datetime is used.
### Parallelization ### Parallelization
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -295,10 +355,19 @@ If a time zone is omitted, a timezone-naive datetime is used.
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>n_jobs</code></td> <td>
`n_jobs`
</td>
<td>int</td> <td>int</td>
<td><code>1</code></td> <td>
<td>Allows <i>proportionally faster (yet more resource-intensive)</i> evaluations of a config on historical data. Default value is 1, that implies <i>sequential</i> execution. Introduced in <a href="https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130">v1.13.0</a></td>
`1`
</td>
<td>
Allows *proportionally faster (yet more resource-intensive)* evaluations of a config on historical data. Default value is 1, that implies *sequential* execution. Introduced in [v1.13.0](../CHANGELOG.md#v1130)
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -306,7 +375,7 @@ If a time zone is omitted, a timezone-naive datetime is used.
### Defining overall timeframe ### Defining overall timeframe
This timeframe will be used for slicing on intervals `(fit_window, infer_window == fit_every)`, starting from the *latest available* time point, which is `to_*` and going back, until no full `fit_window + infer_window` interval exists within the provided timeframe. This timeframe will be used for slicing on intervals `(fit_window, infer_window == fit_every)`, starting from the *latest available* time point, which is `to_*` and going back, until no full `fit_window + infer_window` interval exists within the provided timeframe.
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Format</th> <th>Format</th>
@ -319,27 +388,48 @@ This timeframe will be used for slicing on intervals `(fit_window, infer_window
<tbody> <tbody>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>from_iso</code></td> <td>
`from_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-01T00:00:00Z", "2022-04-01T00:00:00+01:00", "2022-04-01T00:00:00+0100", "2022-04-01T00:00:00+01"</code></td> <td>
`"2022-04-01T00:00:00Z", "2022-04-01T00:00:00+01:00", "2022-04-01T00:00:00+0100", "2022-04-01T00:00:00+01"`
</td>
<td rowspan=2>Start datetime to use for backtesting.</td> <td rowspan=2>Start datetime to use for backtesting.</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>from_s</code></td> <td>
`from_s`
</td>
<td>float</td> <td>float</td>
<td>1648771200</td> <td>1648771200</td>
</tr> </tr>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td><code>to_iso</code></td> <td>
`to_iso`
</td>
<td>str</td> <td>str</td>
<td><code>"2022-04-10T00:00:00Z", "2022-04-10T00:00:00+01:00", "2022-04-10T00:00:00+0100", "2022-04-10T00:00:00+01"</code></td> <td>
<td rowspan=2>End datetime to use for backtesting. Must be greater than <code>from_start_*</code>.</td>
`"2022-04-10T00:00:00Z", "2022-04-10T00:00:00+01:00", "2022-04-10T00:00:00+0100", "2022-04-10T00:00:00+01"`
</td>
<td rowspan=2>End datetime to use for backtesting. Must be greater than
`from_start_*`
</td>
</tr> </tr>
<tr> <tr>
<td>UNIX time</td> <td>UNIX time</td>
<td><code>to_s</code></td> <td>
`to_s`
</td>
<td>float</td> <td>float</td>
<td>1649548800</td> <td>1649548800</td>
</tr> </tr>
@ -348,7 +438,7 @@ This timeframe will be used for slicing on intervals `(fit_window, infer_window
### Defining training timeframe ### Defining training timeframe
The same *explicit* logic as in [Periodic scheduler](#periodic-scheduler) The same *explicit* logic as in [Periodic scheduler](#periodic-scheduler)
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Format</th> <th>Format</th>
@ -361,21 +451,30 @@ The same *explicit* logic as in [Periodic scheduler](#periodic-scheduler)
<tbody> <tbody>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td rowspan=2><code>fit_window</code></td> <td rowspan=2>
`fit_window`
</td>
<td rowspan=2>str</td> <td rowspan=2>str</td>
<td><code>"PT1M", "P1H"</code></td> <td>
`"PT1M", "P1H"`
</td>
<td rowspan=2>What time range to use for training the models. Must be at least 1 second.</td> <td rowspan=2>What time range to use for training the models. Must be at least 1 second.</td>
</tr> </tr>
<tr> <tr>
<td>Prometheus-compatible</td> <td>Prometheus-compatible</td>
<td><code>"1m", "1h"</code></td> <td>
`"1m", "1h"`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
### Defining inference timeframe ### Defining inference timeframe
In `BacktestingScheduler`, the inference window is *implicitly* defined as a period between 2 consecutive model `fit_every` runs. The *latest* inference window starts from `to_s` - `fit_every` and ends on the *latest available* time point, which is `to_s`. The previous periods for fit/infer are defined the same way, by shifting `fit_every` seconds backwards until we get the last full fit period of `fit_window` size, which start is >= `from_s`. In `BacktestingScheduler`, the inference window is *implicitly* defined as a period between 2 consecutive model `fit_every` runs. The *latest* inference window starts from `to_s` - `fit_every` and ends on the *latest available* time point, which is `to_s`. The previous periods for fit/infer are defined the same way, by shifting `fit_every` seconds backwards until we get the last full fit period of `fit_window` size, which start is >= `from_s`.
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Format</th> <th>Format</th>
@ -388,14 +487,23 @@ In `BacktestingScheduler`, the inference window is *implicitly* defined as a per
<tbody> <tbody>
<tr> <tr>
<td>ISO 8601</td> <td>ISO 8601</td>
<td rowspan=2><code>fit_every</code></td> <td rowspan=2>
`fit_every`
</td>
<td rowspan=2>str</td> <td rowspan=2>str</td>
<td><code>"PT1M", "P1H"</code></td> <td>
`"PT1M", "P1H"`
</td>
<td rowspan=2>What time range to use previously trained model to infer on new data until next retrain happens.</td> <td rowspan=2>What time range to use previously trained model to infer on new data until next retrain happens.</td>
</tr> </tr>
<tr> <tr>
<td>Prometheus-compatible</td> <td>Prometheus-compatible</td>
<td><code>"1m", "1h"</code></td> <td>
`"1m", "1h"`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -424,4 +532,4 @@ schedulers:
fit_window: '14d' fit_window: '14d'
fit_every: '1h' fit_every: '1h'
n_jobs: 1 # default = 1 (sequential), set it up to # of CPUs for parallel execution n_jobs: 1 # default = 1 (sequential), set it up to # of CPUs for parallel execution
``` ```

212
docs/anomaly-detection/components/writer.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /anomaly-detection/components/writer.html - /anomaly-detection/components/writer.html
--- ---
# Writer
For exporting data, VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily employs the [VmWriter](#vm-writer), which writes produced anomaly scores **(preserving initial labelset and optionally applying additional ones)** back to VictoriaMetrics. This writer is tailored for smooth data export within the VictoriaMetrics ecosystem. For exporting data, VictoriaMetrics Anomaly Detection (`vmanomaly`) primarily employs the [VmWriter](#vm-writer), which writes produced anomaly scores **(preserving initial labelset and optionally applying additional ones)** back to VictoriaMetrics. This writer is tailored for smooth data export within the VictoriaMetrics ecosystem.
Future updates will introduce additional export methods, offering users more flexibility in data handling and integration. Future updates will introduce additional export methods, offering users more flexibility in data handling and integration.
@ -20,7 +17,7 @@ Future updates will introduce additional export methods, offering users more fle
### Config parameters ### Config parameters
<table> <table class="params">
<thead> <thead>
<tr> <tr>
<th>Parameter</th> <th>Parameter</th>
@ -30,71 +27,190 @@ Future updates will introduce additional export methods, offering users more fle
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>class</code></td> <td>
<td><code>"writer.vm.VmWriter" (or "vm" starting from <a href="https://docs.victoriametrics.com/anomaly-detection/changelog/#v1130">v1.13.0</a>)</code></td>
<td>Name of the class needed to enable writing to VictoriaMetrics or Prometheus. VmWriter is the default option, if not specified.</td> `class`
</td>
<td>
`writer.vm.VmWriter` or `vm` starting from [`v1.13.0`](../CHANGELOG.md#v1130)
</td>
<td>
Name of the class needed to enable writing to VictoriaMetrics or Prometheus. VmWriter is the default option, if not specified.
</td>
</tr> </tr>
<tr> <tr>
<td><code>datasource_url</code></td> <td>
<td><code>"http://localhost:8481/"</code></td>
<td>Datasource URL address</td> `datasource_url`
</td>
<td>
`http://localhost:8481/`
</td>
<td>
Datasource URL address
</td>
</tr> </tr>
<tr> <tr>
<td><code>tenant_id</code></td> <td>
<td><code>"0:0"</code></td>
<td>For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster <a href="https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy">multitenancy docs</a></td> `tenant_id`
</td>
<td>
`0:0`
</td>
<td>
For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster [multitenancy docs](../../Cluster-VictoriaMetrics.md#multitenancy)
</td>
</tr> </tr>
<!-- Additional rows for metric_format --> <!-- Additional rows for metric_format -->
<tr> <tr>
<td rowspan="4"><code>metric_format</code></td> <td rowspan="4">
<td><code>__name__: "vmanomaly_$VAR"</code></td>
<td rowspan="4">Metrics to save the output (in metric names or labels). Must have <code>__name__</code> key. Must have a value with <code>$VAR</code> placeholder in it to distinguish between resulting metrics. Supported placeholders: `metric_format`
</td>
<td>
`__name__: "vmanomaly_$VAR"`
</td>
<td rowspan="4">
Metrics to save the output (in metric names or labels). Must have `__name__` key. Must have a value with `$VAR` placeholder in it to distinguish between resulting metrics. Supported placeholders:
<ul> <ul>
<li><code>$VAR</code> -- Variables that model provides, all models provide the following set: {"anomaly_score", "y", "yhat", "yhat_lower", "yhat_upper"}. Description of standard output is <a href="/anomaly-detection/components/models.html#vmanomaly-output">here</a>. Depending on <a href="/anomaly-detection/components/models.html">model type</a> it can provide more metrics, like "trend", "seasonality" etc.</li> <li>
<li><code>$QUERY_KEY</code> -- E.g. "ingestion_rate".</li>
`$VAR` -- Variables that model provides, all models provide the following set: {"anomaly_score", "y", "yhat", "yhat_lower", "yhat_upper"}. Description of standard output is [here](./models.md#vmanomaly-output). Depending on [model type](./models.md) it can provide more metrics, like "trend", "seasonality" etc.
</li>
<li>
`$QUERY_KEY` -- E.g. "ingestion_rate".
</li>
</ul> </ul>
Other keys are supposed to be configured by the user to help identify generated metrics, e.g., specific config file name etc. Other keys are supposed to be configured by the user to help identify generated metrics, e.g., specific config file name etc.
More details on metric formatting are <a href="#metrics-formatting">here</a>. More details on metric formatting are [here](#metrics-formatting).
</td> </td>
</tr> </tr>
<tr><td><code>for: "$QUERY_KEY"</code></td></tr> <tr>
<tr><td><code>run: "test_metric_format"</code></td></tr> <td>
<tr><td><code>config: "io_vm_single.yaml"</code></td></tr>
`for: "$QUERY_KEY"`
</td>
</tr>
<tr>
<td>
`run: "test_metric_format"`
</td>
</tr>
<tr>
<td>
`config: "io_vm_single.yaml"`
</td>
</tr>
<!-- End of additional rows --> <!-- End of additional rows -->
<tr> <tr>
<td><code>import_json_path</code></td> <td>
<td><code>"/api/v1/import"</code></td>
<td>Optional, to override the default import path</td> `import_json_path`
</td>
<td>
`/api/v1/import`
</td>
<td>
Optional, to override the default import path
</td>
</tr> </tr>
<tr> <tr>
<td><code>health_path</code></td> <td>
<td><code>"health"</code></td>
<td>Absolute or relative URL address where to check the availability of the datasource. Optional, to override the default <code>"/health"</code> path.</td> `health_path`
</td>
<td>
`/health`
</td>
<td>
Absolute or relative URL address where to check the availability of the datasource. Optional, to override the default `/health` path.
</td>
</tr> </tr>
<tr> <tr>
<td><code>user</code></td> <td>
<td><code>"USERNAME"</code></td>
<td>BasicAuth username</td> `user`
</td>
<td>
`USERNAME`
</td>
<td>
BasicAuth username
</td>
</tr> </tr>
<tr> <tr>
<td><code>password</code></td> <td>
<td><code>"PASSWORD"</code></td>
<td>BasicAuth password</td> `password`
</td>
<td>
`PASSWORD`
</td>
<td>
BasicAuth password
</td>
</tr> </tr>
<tr> <tr>
<td><code>timeout</code></td> <td>
<td><code>"5s"</code></td>
<td>Timeout for the requests, passed as a string. Defaults to "5s"</td> `timeout`
</td>
<td>
`5s`
</td>
<td>
Timeout for the requests, passed as a string
</td>
</tr> </tr>
<tr> <tr>
<td><code>verify_tls</code></td> <td>
<td><code>"false"</code></td>
<td>Allows disabling TLS verification of the remote certificate.</td> `verify_tls`
</td>
<td>
`false`
</td>
<td>
Allows disabling TLS verification of the remote certificate.
</td>
</tr> </tr>
<tr> <tr>
<td><code>bearer_token</code></td> <td>
<td><code>"token"</code></td>
<td>Token is passed in the standard format with the header: "Authorization: bearer {token}"</td> `bearer_token`
</td>
<td>
`token`
</td>
<td>
Token is passed in the standard format with the header: `Authorization: bearer {token}`
</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
@ -119,7 +235,7 @@ writer:
### Healthcheck metrics ### Healthcheck metrics
`VmWriter` exposes [several healthchecks metrics](./monitoring.html#writer-behaviour-metrics). `VmWriter` exposes [several healthchecks metrics](./monitoring.md#writer-behaviour-metrics).
### Metrics formatting ### Metrics formatting
@ -130,8 +246,8 @@ __name__: PREFIX1_$VAR
for: PREFIX2_$QUERY_KEY for: PREFIX2_$QUERY_KEY
``` ```
* for `__name__` parameter it will name metrics returned by models as `PREFIX1_anomaly_score`, `PREFIX1_yhat_lower`, etc. Vmanomaly output metrics names described [here](/anomaly-detection/components/models.html#vmanomaly-output) * for `__name__` parameter it will name metrics returned by models as `PREFIX1_anomaly_score`, `PREFIX1_yhat_lower`, etc. Vmanomaly output metrics names described [here](./models.md#vmanomaly-output)
* for `for` parameter will add labels `PREFIX2_query_name_1`, `PREFIX2_query_name_2`, etc. Query names are set as aliases in config `reader` section in [`queries`](anomaly-detection/components/reader.html#config-parameters) parameter. * for `for` parameter will add labels `PREFIX2_query_name_1`, `PREFIX2_query_name_2`, etc. Query names are set as aliases in config `reader` section in [`queries`](./reader.md#config-parameters) parameter.
It is possible to specify other custom label names needed. It is possible to specify other custom label names needed.
For example: For example:
@ -141,7 +257,7 @@ custom_label_1: label_name_1
custom_label_2: label_name_2 custom_label_2: label_name_2
``` ```
Apart from specified labels, output metrics will return **labels inherited from input metrics returned by [queries](/anomaly-detection/components/reader.html#config-parameters)**. Apart from specified labels, output metrics will return **labels inherited from input metrics returned by [queries](./reader.md#config-parameters)**.
For example if input data contains labels such as `cpu=1, device=eth0, instance=node-exporter:9100` all these labels will be present in vmanomaly output metrics. For example if input data contains labels such as `cpu=1, device=eth0, instance=node-exporter:9100` all these labels will be present in vmanomaly output metrics.
So if metric_format section was set up like this: So if metric_format section was set up like this:
@ -156,7 +272,7 @@ metric_format:
It will return metrics that will look like: It will return metrics that will look like:
```yaml ```promtextmetric
{__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"} {__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}
{__name__="PREFIX1_yhat_lower", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"} {__name__="PREFIX1_yhat_lower", for="PREFIX2_query_name_1", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}
{__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_2", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"} {__name__="PREFIX1_anomaly_score", for="PREFIX2_query_name_2", custom_label_1="label_name_1", custom_label_2="label_name_2", cpu=1, device="eth0", instance="node-exporter:9100"}

4
docs/anomaly-detection/guides/README.md
View file

@ -1,5 +1,5 @@
This section holds guides of how to set up and use VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](/anomaly-detection/Overview.html)) service, integrating it with different observability components. This section holds guides of how to set up and use VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](../Overview.md)) service, integrating it with different observability components.
Guides: Guides:
* [Anomaly Detection & Alerting Setup](/anomaly-detection/guides/guide-vmanomaly-vmalert.html) * [Anomaly Detection & Alerting Setup](./guide-vmanomaly-vmalert/README.md)

77
docs/anomaly-detection/guides/guide-vmanomaly-vmalert.md → docs/anomaly-detection/guides/guide-vmanomaly-vmalert/README.md
View file

@ -1,46 +1,29 @@
---
weight: 1
sort: 1
title: Anomaly Detection and Alerting Setup
menu:
docs:
parent: "anomaly-detection-guides"
weight: 1
aliases:
- /anomaly-detection/guides/guide-vmanomaly-vmalert.html
- /guides/guide-vmanomaly-vmalert.html
- /guides/vmanomaly-integration.html
- /guides/anomaly-detection-and-alerting-setup.html
---
# Anomaly Detection and Alerting Setup
**Prerequisites**: **Prerequisites**:
- To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use. - To use *vmanomaly*, part of the enterprise package, a license key is required. Obtain your key [here](https://victoriametrics.com/products/enterprise/trial/) for this tutorial or for enterprise use.
- In the tutorial, we'll be using the following VictoriaMetrics components: - In the tutorial, we'll be using the following VictoriaMetrics components:
- [VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/) (v1.102.0) - [VictoriaMetrics Single-Node](../../../Single-Server-VictoriaMetrics.md) (v1.102.0)
- [vmalert](https://docs.victoriametrics.com/vmalert/) (v1.102.0) - [vmalert](../../../vmalert.md) (v1.102.0)
- [vmagent](https://docs.victoriametrics.com/vmagent/) (v1.102.0) - [vmagent](../../../vmagent.md) (v1.102.0)
- [Grafana](https://grafana.com/) (v.10.2.1) - [Grafana](https://grafana.com/) (v.10.2.1)
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/) - [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.7.0) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.27.0) - [Node exporter](https://github.com/prometheus/node_exporter#node-exporter) (v1.7.0) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) (v0.27.0)
<img src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp" alt="vmanomaly typical setup diagram"> ![typical setup diagram](guide-vmanomaly-vmalert_overview.webp)
> **Note: Configurations used throughout this guide can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)** > **Note: Configurations used throughout this guide can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)**
> **Note:** Starting from [v1.13.0](/anomaly-detection/CHANGELOG#v1130) `node-exporter` observability preset is available for `vmanomaly`. Please find the guide [here](/anomaly-detection/presets/#node-exporter). > **Note:** Starting from [v1.13.0](../../CHANGELOG.md#v1130) `node-exporter` observability preset is available for `vmanomaly`. Please find the guide [here](../../Presets.md#node-exporter).
## 1. What is vmanomaly? ## 1. What is vmanomaly?
*VictoriaMetrics Anomaly Detection* ([vmanomaly](/anomaly-detection/overview/)) is a service that continuously scans time series stored in VictoriaMetrics and detects unexpected changes within data patterns in real-time. It does so by utilizing user-configurable machine learning models. *VictoriaMetrics Anomaly Detection* ([vmanomaly](../../Overview.md)) is a service that continuously scans time series stored in VictoriaMetrics and detects unexpected changes within data patterns in real-time. It does so by utilizing user-configurable machine learning models.
All the service parameters are defined in a config file. All the service parameters are defined in a config file.
> **Note**: Starting from [1.10.0](/anomaly-detection/changelog/#v1100), each `vmanomaly` configuration file can support more that one model type. To utilize *different models* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/components/models/) config section for more details. > **Note**: Starting from [1.10.0](../../CHANGELOG.md#v1100), each `vmanomaly` configuration file can support more that one model type. To utilize *different models* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](../..//components/models.md) config section for more details.
> **Note**: Starting from [1.11.0](/anomaly-detection/changelog/#v1110), each `vmanomaly` configuration file can support more that one model type, each attached to one (or more) schedulers. To utilize *different models* with *different schedulers* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/components/models/#schedulers) and [scheduler](/anomaly-detection/components/scheduler/) config sections for more details. > **Note**: Starting from [1.11.0](../../CHANGELOG.md#v1110), each `vmanomaly` configuration file can support more that one model type, each attached to one (or more) schedulers. To utilize *different models* with *different schedulers* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](../../components/models.md#schedulers) and [scheduler](../../components/scheduler.md) config sections for more details.
**vmanomaly** does the following: **vmanomaly** does the following:
@ -61,9 +44,9 @@ Then, users can enable alerting rules based on the **anomaly score** with [vmale
## 2. What is vmalert? ## 2. What is vmalert?
[vmalert](https://docs.victoriametrics.com/vmalert/) is an alerting tool for VictoriaMetrics. It executes a list of the given alerting or recording rules against configured `-datasource.url`. [vmalert](../../../vmalert.md) is an alerting tool for VictoriaMetrics. It executes a list of the given alerting or recording rules against configured `-datasource.url`.
[Alerting rules](https://docs.victoriametrics.com/vmalert/#alerting-rules) allow you to define conditions that, when met, will notify the user. The alerting condition is defined in a form of a query expression via [MetricsQL query language](https://docs.victoriametrics.com/metricsql/). For example, in our case, the expression `anomaly_score > 1.0` will notify a user when the calculated anomaly score exceeds a threshold of `1.0`. [Alerting rules](../../../vmalert.md#alerting-rules) allow you to define conditions that, when met, will notify the user. The alerting condition is defined in a form of a query expression via [MetricsQL query language](../../../MetricsQL.md). For example, in our case, the expression `anomaly_score > 1.0` will notify a user when the calculated anomaly score exceeds a threshold of `1.0`.
## 3. How does vmanomaly works with vmalert? ## 3. How does vmanomaly works with vmalert?
@ -72,9 +55,9 @@ Compared to classical alerting rules, anomaly detection is more "hands-off" and
Practical use case is to put anomaly score generated by vmanomaly into alerting rules with some threshold. Practical use case is to put anomaly score generated by vmanomaly into alerting rules with some threshold.
**In this tutorial we are going to:** **In this tutorial we are going to:**
- Configure docker-compose file with all needed services ([VictoriaMetrics Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/), [vmalert](https://docs.victoriametrics.com/vmalert/), [vmagent](https://docs.victoriametrics.com/vmagent/), [Grafana](https://grafana.com/), [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) and [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/overview/) ). - Configure docker-compose file with all needed services ([VictoriaMetrics Single-Node](../../../Single-Server-VictoriaMetrics.md), [vmalert](../../../vmalert.md), [vmagent](../../../vmagent.md), [Grafana](https://grafana.com/), [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) and [vmanomaly](../../Overview.md) ).
- Explore configuration files for [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/overview/) and [vmalert](https://docs.victoriametrics.com/vmalert/). - Explore configuration files for [vmanomaly](../../Overview.md) and [vmalert](../../../vmalert.md).
- Run our own [VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) database with data scraped from [Node Exporter](https://prometheus.io/docs/guides/node-exporter/). - Run our own [VictoriaMetrics](../../../Single-Server-VictoriaMetrics.md) database with data scraped from [Node Exporter](https://prometheus.io/docs/guides/node-exporter/).
- Explore data for analysis in [Grafana](https://grafana.com/). - Explore data for analysis in [Grafana](https://grafana.com/).
- Explore `vmanomaly` results. - Explore `vmanomaly` results.
- Explore `vmalert` alerts - Explore `vmalert` alerts
@ -105,10 +88,10 @@ node_cpu_seconds_total{cpu="1",mode="iowait"} 51.22
In this context, the metric `node_cpu_seconds_total` provides a comprehensive breakdown of the time each CPU core has spent in various operational modes. These modes include: _user_, _system_, _iowait_, _idle_, _irq&softirq_, _guest_, and _steal_. Each of these eight modes is mutually exclusive, offering distinct insights into CPU activity. For instance, a predominant _iowait_ suggests disk or network bottlenecks, while elevated levels in _user_ or _system_ indicate significant CPU utilization. In this context, the metric `node_cpu_seconds_total` provides a comprehensive breakdown of the time each CPU core has spent in various operational modes. These modes include: _user_, _system_, _iowait_, _idle_, _irq&softirq_, _guest_, and _steal_. Each of these eight modes is mutually exclusive, offering distinct insights into CPU activity. For instance, a predominant _iowait_ suggests disk or network bottlenecks, while elevated levels in _user_ or _system_ indicate significant CPU utilization.
The `node_cpu_seconds_total` metric is classified as a [counter](https://docs.victoriametrics.com/keyconcepts/#counter) type. To analyze the duration each CPU core spends in these modes, it is necessary to compute the rate of change per second using the [rate function](https://docs.victoriametrics.com/metricsql/#rate): `rate(node_cpu_seconds_total)`. For a more refined and smoother aggregation of data by mode, we apply the sum function. The resulting query is formulated as follows: `sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)`. The `node_cpu_seconds_total` metric is classified as a [counter](../../../keyConcepts.md#counter) type. To analyze the duration each CPU core spends in these modes, it is necessary to compute the rate of change per second using the [rate function](../../../MetricsQL.md#rate): `rate(node_cpu_seconds_total)`. For a more refined and smoother aggregation of data by mode, we apply the sum function. The resulting query is formulated as follows: `sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)`.
Below is an illustrative example of how this query might be visualized in Grafana: Below is an illustrative example of how this query might be visualized in Grafana:
<img alt="node_cpu_rate_graph" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert-query.webp"> ![node_cpu_rate_graph](guide-vmanomaly-vmalert-query.webp)
This query will yield a total of eight time series, each corresponding to a CPU mode. The number of series is unaffected by the number of CPU cores, due to the `by` aggregation applied. These series serve as the input for `vmanomaly`, where the service independently fits a separate instance of the configured model type to each of time series. This query will yield a total of eight time series, each corresponding to a CPU mode. The number of series is unaffected by the number of CPU cores, due to the `by` aggregation applied. These series serve as the input for `vmanomaly`, where the service independently fits a separate instance of the configured model type to each of time series.
@ -119,25 +102,25 @@ This query will yield a total of eight time series, each corresponding to a CPU
**Parameter description**: **Parameter description**:
The configuration file for `vmanomaly` comprises 4 essential sections: The configuration file for `vmanomaly` comprises 4 essential sections:
1. [`scheduler`](/anomaly-detection/components/scheduler.html) - This section determines the frequency of model inferences and training, including the time range for model training. Starting from [v1.11.0](/anomaly-detection/changelog/#v1110), multiple individual schedulers are supported for each model type in a single config. 1. [`scheduler`](../../components/scheduler.md) - This section determines the frequency of model inferences and training, including the time range for model training. Starting from [v1.11.0](../../CHANGELOG.md#v1110), multiple individual schedulers are supported for each model type in a single config.
2. [`models`](/anomaly-detection/components/models.html) - Here, you define specific parameters and configurations for the models being used for anomaly detection. Starting from [v1.10.0](/anomaly-detection/changelog/#v1100), multiple model configurations are supported in a single config. 2. [`models`](../../components/models.md) - Here, you define specific parameters and configurations for the models being used for anomaly detection. Starting from [v1.10.0](../../CHANGELOG.md#v1100), multiple model configurations are supported in a single config.
3. [`reader`](/anomaly-detection/components/reader.html) - This section outlines the methodology for data reading, including the data source location. 3. [`reader`](../../components/reader.md) - This section outlines the methodology for data reading, including the data source location.
4. [`writer`](/anomaly-detection/components/writer.html) - Specifies the destination and method for writing the generated output. 4. [`writer`](../../components/writer.md) - Specifies the destination and method for writing the generated output.
5. [`monitoring`](/anomaly-detection/components/monitoring.html) (optional) - Describes how to monitor and expose health check metrics of `vmanomaly`. 5. [`monitoring`](../../components/monitoring.md) (optional) - Describes how to monitor and expose health check metrics of `vmanomaly`.
Detailed parameters in each section: Detailed parameters in each section:
* `schedulers` ([PeriodicScheduler](/anomaly-detection/components/scheduler/#periodic-scheduler) is used here) * `schedulers` ([PeriodicScheduler](../../components/scheduler.md#periodic-scheduler) is used here)
* `infer_every` - Specifies the frequency at which the trained models perform inferences on new data, essentially determining how often new anomaly score data points are generated. Format examples: 30s, 4m, 2h, 1d (time units: 's' for seconds, 'm' for minutes, 'h' for hours, 'd' for days). This parameter essentially asks, at regular intervals (e.g., every 1 minute), whether the latest data points appear abnormal based on historical data. * `infer_every` - Specifies the frequency at which the trained models perform inferences on new data, essentially determining how often new anomaly score data points are generated. Format examples: 30s, 4m, 2h, 1d (time units: 's' for seconds, 'm' for minutes, 'h' for hours, 'd' for days). This parameter essentially asks, at regular intervals (e.g., every 1 minute), whether the latest data points appear abnormal based on historical data.
* `fit_every` - Sets the frequency for retraining the models. A higher frequency ensures more updated models but requires more CPU resources. If omitted, models are retrained in each `infer_every` cycle. Format is similar to `infer_every`. * `fit_every` - Sets the frequency for retraining the models. A higher frequency ensures more updated models but requires more CPU resources. If omitted, models are retrained in each `infer_every` cycle. Format is similar to `infer_every`.
* `fit_window` - Defines the data interval for training the models. Longer intervals allow for capturing extensive historical behavior and better seasonal pattern detection but may slow down the model's response to permanent metric changes and increase resource consumption. A minimum of two full seasonal cycles is recommended. Example format: 3h for three hours of data. * `fit_window` - Defines the data interval for training the models. Longer intervals allow for capturing extensive historical behavior and better seasonal pattern detection but may slow down the model's response to permanent metric changes and increase resource consumption. A minimum of two full seasonal cycles is recommended. Example format: 3h for three hours of data.
* `models` * `models`
* `class` - Specifies the model to be used. Options include custom models ([guide here](/anomaly-detection/components/models.html#custom-model-guide)) or a selection from [built-in models](/anomaly-detection/components/models.html#built-in-models), such as the [Facebook Prophet](/anomaly-detection/components/models.html#prophet) (`model.prophet.ProphetModel`). * `class` - Specifies the model to be used. Options include custom models ([guide here](../../components/models.md#custom-model-guide)) or a selection from [built-in models](../../components/models.md#built-in-models), such as the [Facebook Prophet](../../components/models.md#prophet) (`model.prophet.ProphetModel`).
* `args` - Model-specific parameters, formatted as a YAML dictionary in the `key: value` structure. Parameters available in [FB Prophet](https://facebook.github.io/prophet/docs/quick_start.html) can be used as an example. * `args` - Model-specific parameters, formatted as a YAML dictionary in the `key: value` structure. Parameters available in [FB Prophet](https://facebook.github.io/prophet/docs/quick_start.html) can be used as an example.
* `reader` * `reader`
@ -210,7 +193,7 @@ groups:
summary: Anomaly Score exceeded 1.0. `sum(rate(node_cpu_seconds_total))` is showing abnormal behavior. summary: Anomaly Score exceeded 1.0. `sum(rate(node_cpu_seconds_total))` is showing abnormal behavior.
``` ```
In the query expression `expr`, it's crucial to establish a criterion based on the generated anomaly scores. Typically, an [anomaly score](/anomaly-detection/faq/#what-is-anomaly-score) ranging from 0.0 to 1.0 indicates that the analyzed value falls within normal behavior. Scores exceeding 1.0 signal increasing confidence from our model that the observed value is anomalous. In the query expression `expr`, it's crucial to establish a criterion based on the generated anomaly scores. Typically, an [anomaly score](../../FAQ.md#what-is-anomaly-score) ranging from 0.0 to 1.0 indicates that the analyzed value falls within normal behavior. Scores exceeding 1.0 signal increasing confidence from our model that the observed value is anomalous.
Selecting an appropriate threshold for the anomaly score depends on your specific requirements and the context of the data. One effective method for determining this threshold is through visual analysis. By plotting the `anomaly_score` metric in conjunction with the predicted 'expected' range, delineated by `yhat_lower` and `yhat_upper`, you can make a more informed decision. Later in this tutorial, we will demonstrate this process with a practical example. Selecting an appropriate threshold for the anomaly score depends on your specific requirements and the context of the data. One effective method for determining this threshold is through visual analysis. By plotting the `anomaly_score` metric in conjunction with the predicted 'expected' range, delineated by `yhat_lower` and `yhat_upper`, you can make a more informed decision. Later in this tutorial, we will demonstrate this process with a practical example.
@ -306,7 +289,7 @@ scrape_configs:
We will utilize the license key stored locally in the file `vmanomaly_license`. We will utilize the license key stored locally in the file `vmanomaly_license`.
For additional licensing options, please refer to the [VictoriaMetrics Anomaly Detection documentation on licensing](https://docs.victoriametrics.com/anomaly-detection/overview/#licensing). For additional licensing options, please refer to the [VictoriaMetrics Anomaly Detection documentation on licensing](../../Overview.md#licensing).
### Alertmanager setup ### Alertmanager setup
@ -452,7 +435,7 @@ networks:
Before running our docker-compose make sure that your directory contains all required files: Before running our docker-compose make sure that your directory contains all required files:
<img src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_files.webp" alt="all files"> ![all files](guide-vmanomaly-vmalert_files.webp)
This docker-compose file will pull docker images, set up each service and run them all together with the command: This docker-compose file will pull docker images, set up each service and run them all together with the command:
@ -489,7 +472,7 @@ Each of these metrics will contain same labels our query `sum(rate(node_cpu_seco
### Anomaly scores for each metric with its according labels. ### Anomaly scores for each metric with its according labels.
Query: `anomaly_score` Query: `anomaly_score`
<img alt="Anomaly score graph" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_anomaly-score.webp"> ![Anomaly score graph](guide-vmanomaly-vmalert_anomaly-score.webp)
<br>Check out if the anomaly score is high for datapoints you think are anomalies. If not, you can try other parameters in the config file or try other model type. <br>Check out if the anomaly score is high for datapoints you think are anomalies. If not, you can try other parameters in the config file or try other model type.
@ -500,7 +483,7 @@ As you may notice a lot of data shows anomaly score greater than 1. It is expect
Queries: `yhat_lower`, `yhat_upper` and `yhat` Queries: `yhat_lower`, `yhat_upper` and `yhat`
<img alt="yhat lower and yhat upper" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert-boundaries.webp"> ![yhat lower and yhat upper](guide-vmanomaly-vmalert-boundaries.webp)
Boundaries of 'normal' metric values according to model inference. Boundaries of 'normal' metric values according to model inference.
@ -508,10 +491,10 @@ Boundaries of 'normal' metric values according to model inference.
On the page `http://localhost:8880/vmalert/groups` you can find our configured Alerting rule: On the page `http://localhost:8880/vmalert/groups` you can find our configured Alerting rule:
<img alt="alert rule" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_alert-rule.webp"> ![alert rule](guide-vmanomaly-vmalert_alert-rule.webp)
According to the rule configured for vmalert we will see Alert when anomaly score exceed 1. You will see an alert on Alert tab. `http://localhost:8880/vmalert/alerts`: According to the rule configured for vmalert we will see Alert when anomaly score exceed 1. You will see an alert on Alert tab. `http://localhost:8880/vmalert/alerts`:
<img alt="alerts firing" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_alerts-firing.webp"> ![alerts firing](guide-vmanomaly-vmalert_alerts-firing.webp)
## 10. Conclusion ## 10. Conclusion
@ -529,4 +512,4 @@ Key takeaways include:
As you continue to use VictoriaMetrics Anomaly Detection and `vmalert`, remember that the effectiveness of anomaly detection largely depends on the appropriateness of the model chosen, the accuracy of configurations and the data patterns observed. This guide serves as a starting point, and we encourage you to experiment with different configurations and models to best suit your specific data needs and use cases. In case you need a helping hand - [contact us](https://victoriametrics.com/contact-us/). As you continue to use VictoriaMetrics Anomaly Detection and `vmalert`, remember that the effectiveness of anomaly detection largely depends on the appropriateness of the model chosen, the accuracy of configurations and the data patterns observed. This guide serves as a starting point, and we encourage you to experiment with different configurations and models to best suit your specific data needs and use cases. In case you need a helping hand - [contact us](https://victoriametrics.com/contact-us/).
> **Note:** Starting from [v1.13.0](/anomaly-detection/CHANGELOG#v1130) `node-exporter` observability preset is available for `vmanomaly`. Please find the guide [here](/anomaly-detection/presets/#node-exporter). > **Note:** Starting from [v1.13.0](../../CHANGELOG.md#v1130) `node-exporter` observability preset is available for `vmanomaly`. Please find the guide [here](../../Presets.md#node-exporter).

15
docs/anomaly-detection/guides/guide-vmanomaly-vmalert/_index.md Normal file
View file

@ -0,0 +1,15 @@
---
weight: 1
sort: 1
title: Anomaly Detection and Alerting Setup
menu:
docs:
parent: "anomaly-detection-guides"
weight: 1
aliases:
- /anomaly-detection/guides/guide-vmanomaly-vmalert.html
- /guides/guide-vmanomaly-vmalert.html
- /guides/vmanomaly-integration.html
- /guides/anomaly-detection-and-alerting-setup.html
---
{{% content "README.md" %}}

10
docs/data-ingestion/Proxmox.md
View file

@ -12,8 +12,6 @@ aliases:
- /data-ingestion/proxmox.html - /data-ingestion/proxmox.html
- /data-ingestion/Proxmox.html - /data-ingestion/Proxmox.html
--- ---
# Proxmox Data Ingestion
Since Proxmox Virtual Environment(PVE) and Proxmox Backup Server(PBS) support sending data using the InfluxDB We can use the InfluxDB write support built into VictoriaMetrics Since Proxmox Virtual Environment(PVE) and Proxmox Backup Server(PBS) support sending data using the InfluxDB We can use the InfluxDB write support built into VictoriaMetrics
Currently PVE and PBS only support using an Authorization Token for authentication and does not support basic auth or a username and password. Currently PVE and PBS only support using an Authorization Token for authentication and does not support basic auth or a username and password.
@ -23,7 +21,7 @@ If want help Sending your data to Managed VictoriaMetrics check out [our blog](h
1. Login to PVE as an administrator 1. Login to PVE as an administrator
2. Go to DataCenter > MetricServer > Add > InfluxDB 2. Go to DataCenter > MetricServer > Add > InfluxDB
<img alt="PVE Metric Navigation" src="pve-nav.webp"> ![PVE Metric Navigation](pve-nav.webp)
3. Set the parameters as follows: 3. Set the parameters as follows:
- Name: VictoriaMetrics (can be changed to any string) - Name: VictoriaMetrics (can be changed to any string)
@ -36,7 +34,7 @@ If want help Sending your data to Managed VictoriaMetrics check out [our blog](h
- If you need to ignore TLS/SSL errors check the advanced box and uncheck the verify certificate box - If you need to ignore TLS/SSL errors check the advanced box and uncheck the verify certificate box
4. Click the `Create` button 4. Click the `Create` button
<img alt="PVE Metric Form" src="pve-form.webp"> ![PVE Metric Form](pve-form.webp)
5. Run `system_uptime{object="nodes"}` in vmui or in the explore view in Grafana to verify metrics from PVE are being sent to VictoriaMetrics. 5. Run `system_uptime{object="nodes"}` in vmui or in the explore view in Grafana to verify metrics from PVE are being sent to VictoriaMetrics.
You should see 1 time series per node in your PVE cluster. You should see 1 time series per node in your PVE cluster.
@ -46,7 +44,7 @@ You should see 1 time series per node in your PVE cluster.
2. Go to Configuration > Metrics Server > Add > InfluxDB 2. Go to Configuration > Metrics Server > Add > InfluxDB
<img alt="PBS Metric Navigation" src="pbs-nav.webp"> ![PBS Metric Navigation](pbs-nav.webp)
3. Set the parameters as follows: 3. Set the parameters as follows:
@ -60,7 +58,7 @@ You should see 1 time series per node in your PVE cluster.
4. Click the `Create` button 4. Click the `Create` button
<img alt="PBS Metric Form" src="pbs-form.webp"> ![PBS Metric Form](pbs-form.webp)
5. Run `cpustat_idle{object="host"}` in vmui or in the explore view in Grafana to verify metrics from PBS are being to VictoriaMetrics. 5. Run `cpustat_idle{object="host"}` in vmui or in the explore view in Grafana to verify metrics from PBS are being to VictoriaMetrics.

1
docs/data-ingestion/Telegraf.md
View file

@ -12,7 +12,6 @@ aliases:
- /data-ingestion/telegraf.html - /data-ingestion/telegraf.html
- /data-ingestion/Telegraf.html - /data-ingestion/Telegraf.html
--- ---
# Telegraf Setup
You will need to add the following output section to a Telegraf configuration file and reload Telegraf to enable shipping data from Telegraf to VictoriaMetrics. You will need to add the following output section to a Telegraf configuration file and reload Telegraf to enable shipping data from Telegraf to VictoriaMetrics.
All the options examples below can be combined to fit your use case All the options examples below can be combined to fit your use case

1
docs/data-ingestion/Vector.md
View file

@ -12,7 +12,6 @@ aliases:
- /data-ingestion/Vector.html - /data-ingestion/Vector.html
- /data-ingestion/vector.html - /data-ingestion/vector.html
--- ---
# Vector
To Send data to Vector you need to configure with a Prometheus remote write sink and forward metrics to that sink from at least 1 source. To Send data to Vector you need to configure with a Prometheus remote write sink and forward metrics to that sink from at least 1 source.
You will need to replace the values in `<>` with your to match your setup. You will need to replace the values in `<>` with your to match your setup.

3
docs/enterprise.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /enterprise.html - /enterprise.html
--- ---
# VictoriaMetrics Enterprise
VictoriaMetrics components are provided in two kinds - [Community edition](https://victoriametrics.com/products/open-source/) VictoriaMetrics components are provided in two kinds - [Community edition](https://victoriametrics.com/products/open-source/)
and [Enterprise edition](https://victoriametrics.com/products/enterprise/). and [Enterprise edition](https://victoriametrics.com/products/enterprise/).

3
docs/goals.md
View file

@ -7,9 +7,6 @@ menu:
parent: 'victoriametrics' parent: 'victoriametrics'
weight: 500 weight: 500
--- ---
# VictoriaMetrics development goals
## Goals ## Goals
1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/enterprise/) resolving issues with VictoriaMetrics components, 1. The main goal - **to help users and [clients](https://docs.victoriametrics.com/enterprise/) resolving issues with VictoriaMetrics components,

2
docs/guides/README.md
View file

@ -8,4 +8,4 @@
1. [How to delete or replace metrics in VictoriaMetrics](guide-delete-or-replace-metrics.html) 1. [How to delete or replace metrics in VictoriaMetrics](guide-delete-or-replace-metrics.html)
1. [How to monitor kubernetes cluster using Managed VictoriaMetrics](/managed-victoriametrics/how-to-monitor-k8s.html) 1. [How to monitor kubernetes cluster using Managed VictoriaMetrics](/managed-victoriametrics/how-to-monitor-k8s.html)
1. [How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect](grafana-vmgateway-openid-configuration.html) 1. [How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect](grafana-vmgateway-openid-configuration.html)
1. [How to setup vmanomaly together with vmalert](/anomaly-detection/guides/guide-vmanomaly-vmalert.html) 1. [How to setup vmanomaly together with vmalert](/anomaly-detection/guides/guide-vmanomaly-vmalert/README.md)

1
docs/guides/_index.md
View file

@ -1,6 +1,7 @@
--- ---
sort: 26 sort: 26
weight: 01 weight: 01
title: Guides
disableToc: true disableToc: true
menu: menu:
docs: docs:

9
docs/guides/getting-started-with-opentelemetry.md
View file

@ -6,7 +6,6 @@ menu:
parent: "guides" parent: "guides"
weight: 5 weight: 5
--- ---
VictoriaMetrics supports metrics ingestion with [OpenTelemetry metrics format](https://opentelemetry.io/docs/specs/otel/metrics/). VictoriaMetrics supports metrics ingestion with [OpenTelemetry metrics format](https://opentelemetry.io/docs/specs/otel/metrics/).
This guide covers data ingestion via [opentelemetry-collector](https://opentelemetry.io/docs/collector/) and direct metrics push from application. This guide covers data ingestion via [opentelemetry-collector](https://opentelemetry.io/docs/collector/) and direct metrics push from application.
@ -45,7 +44,7 @@ Read Data:
## Using opentelemetry-collector with VictoriaMetrics ## Using opentelemetry-collector with VictoriaMetrics
<img src="/guides/getting-started-with-opentelemetry-collector.webp"> ![OTEL Collector](getting-started-with-opentelemetry-collector.webp)
### Deploy opentelemetry-collector and configure metrics forwarding ### Deploy opentelemetry-collector and configure metrics forwarding
@ -164,14 +163,14 @@ for i in `seq 1 20`; do curl http://localhost:8080/rolldice; done
``` ```
After a few seconds you should start to see metrics sent over to the vmui interface by visiting `http://localhost:8428/vmui/#/?g0.expr=dice.rolls` in your browser or by querying the metric `dice.rolls` in the vmui interface. After a few seconds you should start to see metrics sent over to the vmui interface by visiting `http://localhost:8428/vmui/#/?g0.expr=dice.rolls` in your browser or by querying the metric `dice.rolls` in the vmui interface.
<img src="/guides/vmui-dice-roll.webp"> ![Dice roll](vmui-dice-roll.webp)
## Direct metrics push ## Direct metrics push
Metrics could be ingested into VictoriaMetrics directly with HTTP requests. You can use any compatible OpenTelemetry Metrics could be ingested into VictoriaMetrics directly with HTTP requests. You can use any compatible OpenTelemetry
instrumentation [clients](https://opentelemetry.io/docs/languages/). instrumentation [clients](https://opentelemetry.io/docs/languages/).
In our example, we'll create a WEB server in [Golang](https://go.dev/) and instrument it with metrics. In our example, we'll create a WEB server in [Golang](https://go.dev/) and instrument it with metrics.
<img src="/guides/getting-started-with-opentelemetry-direct.webp"> ![OTEL direct](getting-started-with-opentelemetry-direct.webp)
### Building the Go application instrumented with metrics ### Building the Go application instrumented with metrics
@ -350,7 +349,7 @@ curl http://localhost:8081/api/slow
Open [vmui](https://docs.victoriametrics.com/#vmui) and query `http_requests_total` or `http_active_requests` Open [vmui](https://docs.victoriametrics.com/#vmui) and query `http_requests_total` or `http_active_requests`
with [metricsql](https://docs.victoriametrics.com/metricsql/). with [metricsql](https://docs.victoriametrics.com/metricsql/).
<img src= "/guides/getting-started-with-opentelemetry-vmui.webp"> ![OTEL VMUI](getting-started-with-opentelemetry-vmui.webp)
## Limitations ## Limitations

8
docs/guides/getting-started-with-vm-operator.md
View file

@ -8,8 +8,6 @@ menu:
aliases: aliases:
- /guides/getting-started-with-vm-operator.html - /guides/getting-started-with-vm-operator.html
--- ---
# Getting started with VM Operator
**The guide covers:** **The guide covers:**
* The setup of a [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator) via Helm in [Kubernetes](https://kubernetes.io/) with Helm charts. * The setup of a [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator) via Helm in [Kubernetes](https://kubernetes.io/) with Helm charts.
@ -215,7 +213,7 @@ Forwarding from [::1]:8429 -> 8429
To check that `VMAgent` collects metrics from the k8s cluster open in the browser [http://127.0.0.1:8429/targets](http://127.0.0.1:8429/targets) . To check that `VMAgent` collects metrics from the k8s cluster open in the browser [http://127.0.0.1:8429/targets](http://127.0.0.1:8429/targets) .
You will see something like this: You will see something like this:
<img src="getting-started-with-vm-operator_vmcluster.webp"> ![VMCluster](getting-started-with-vm-operator_vmcluster.webp)
`VMAgent` connects to [kubernetes service discovery](https://kubernetes.io/docs/concepts/services-networking/service/) and gets targets which needs to be scraped. This service discovery is controlled by [VictoriaMetrics Operator](https://github.com/VictoriaMetrics/operator) `VMAgent` connects to [kubernetes service discovery](https://kubernetes.io/docs/concepts/services-networking/service/) and gets targets which needs to be scraped. This service discovery is controlled by [VictoriaMetrics Operator](https://github.com/VictoriaMetrics/operator)
@ -289,11 +287,11 @@ EOF
To check that [VictoriaMetrics](https://victoriametrics.com) collecting metrics from the k8s cluster open in your browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `VictoriaMetrics - cluster` dashboard. Use `admin` for login and the `password` that you previously got from kubectl. To check that [VictoriaMetrics](https://victoriametrics.com) collecting metrics from the k8s cluster open in your browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `VictoriaMetrics - cluster` dashboard. Use `admin` for login and the `password` that you previously got from kubectl.
<img src="getting-started-with-vm-operator_vmcluster-grafana1.webp" alt="grafana dashboards"> ![Dashboards 1](getting-started-with-vm-operator_vmcluster-grafana1.webp)
The expected output is: The expected output is:
<img src="getting-started-with-vm-operator_vmcluster-grafana2.webp" alt="grafana dashboards"> ![Dashboards 2](getting-started-with-vm-operator_vmcluster-grafana2.webp)
## 6. Summary ## 6. Summary

42
docs/guides/grafana-vmgateway-openid-configuration.md → docs/guides/grafana-vmgateway-openid-configuration/README.md
View file

@ -1,15 +1,3 @@
---
weight: 5
title: How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect
menu:
docs:
parent: "guides"
weight: 5
aliases:
- /guides/grafana-vmgateway-openid-configuration.html
---
# How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect
Using [Grafana](https://grafana.com/) with [vmgateway](https://docs.victoriametrics.com/vmgateway/) is a great way to provide [multi-tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) access to your metrics. Using [Grafana](https://grafana.com/) with [vmgateway](https://docs.victoriametrics.com/vmgateway/) is a great way to provide [multi-tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) access to your metrics.
vmgateway provides a way to authenticate users using [JWT tokens](https://en.wikipedia.org/wiki/JSON_Web_Token) issued by an external identity provider. vmgateway provides a way to authenticate users using [JWT tokens](https://en.wikipedia.org/wiki/JSON_Web_Token) issued by an external identity provider.
Those tokens can include information about the user and the tenant they belong to, which can be used Those tokens can include information about the user and the tenant they belong to, which can be used
@ -48,21 +36,21 @@ See details about all supported options in the [vmgateway documentation](https:/
Use `OpenID Connect` as `Client Type`.<br> Use `OpenID Connect` as `Client Type`.<br>
Specify `grafana` as `Client ID`.<br> Specify `grafana` as `Client ID`.<br>
Click `Next`.<br> Click `Next`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-1.webp" > ![Create client 1](create-client-1.webp)
1. Enable `Client authentication`.<br> 1. Enable `Client authentication`.<br>
Enable `Authorization`.<br> Enable `Authorization`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-2.webp" ><br> ![Create client 2](create-client-2.webp)
Click `Next`.<br> Click `Next`.<br>
1. Add Grafana URL as `Root URL`. For example, `http://localhost:3000/`.<br> 1. Add Grafana URL as `Root URL`. For example, `http://localhost:3000/`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-3.webp" ><br> ![Create client 3](create-client-3.webp)
Click `Save`.<br> Click `Save`.<br>
1. Go to `Clients` -> `grafana` -> `Credentials`.<br> 1. Go to `Clients` -> `grafana` -> `Credentials`.<br>
<img src="grafana-vmgateway-openid-configuration/client-secret.webp" ><br> ![Client secret](client-secret.webp)
Copy the value of `Client secret`. It will be used later in Grafana configuration.<br> Copy the value of `Client secret`. It will be used later in Grafana configuration.<br>
1. Go to `Clients` -> `grafana` -> `Client scopes`.<br> 1. Go to `Clients` -> `grafana` -> `Client scopes`.<br>
Click at `grafana-dedicated` -> `Add mapper` -> `By configuration` -> `User attribute`.<br> Click at `grafana-dedicated` -> `Add mapper` -> `By configuration` -> `User attribute`.<br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-1.webp" ><br> ![Create mapper 1](create-mapper-1.webp)
<img src="grafana-vmgateway-openid-configuration/create-mapper-2.webp" ><br> ![Create mapper 2](create-mapper-2.webp)
Configure the mapper as follows<br> Configure the mapper as follows<br>
- `Name` as `vm_access`. - `Name` as `vm_access`.
- `Token Claim Name` as `vm_access`. - `Token Claim Name` as `vm_access`.
@ -70,7 +58,7 @@ See details about all supported options in the [vmgateway documentation](https:/
- `Claim JSON Type` as `JSON`. - `Claim JSON Type` as `JSON`.
Enable `Add to ID token` and `Add to access token`.<br> Enable `Add to ID token` and `Add to access token`.<br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-3.webp" ><br> ![Create mapper 3](create-mapper-3.webp)
Click `Save`.<br> Click `Save`.<br>
1. Go to `Users` -> select user to configure claims -> `Attributes`.<br> 1. Go to `Users` -> select user to configure claims -> `Attributes`.<br>
Specify `vm_access` as `Key`.<br> Specify `vm_access` as `Key`.<br>
@ -78,7 +66,7 @@ See details about all supported options in the [vmgateway documentation](https:/
- for the first user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 0 },"extra_labels":{ "team": "admin" }}` as `Value`. - for the first user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 0 },"extra_labels":{ "team": "admin" }}` as `Value`.
- for the second user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 1 },"extra_labels":{ "team": "dev" }}` as `Value`. - for the second user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 1 },"extra_labels":{ "team": "dev" }}` as `Value`.
<br> <br>
<img src="grafana-vmgateway-openid-configuration/user-attributes.webp" ><br> ![User attributes](user-attributes.webp)
Click `Save`. Click `Save`.
## Configure grafana ## Configure grafana
@ -199,7 +187,7 @@ URL should point to the vmgateway instance.
In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x": In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x":
<img src="grafana-vmgateway-openid-configuration/grafana-datasource-prometheus.webp" alt="Grafana datasource" /> ![Prometheus datasource](grafana-datasource-prometheus.webp)
This allows Grafana to use a more efficient API to get label values. This allows Grafana to use a more efficient API to get label values.
@ -207,7 +195,7 @@ You can also use VictoriaMetrics [Grafana datasource](https://github.com/Victori
See installation instructions [here](https://docs.victoriametrics.com/victoriametrics-datasource/#installation). See installation instructions [here](https://docs.victoriametrics.com/victoriametrics-datasource/#installation).
Enable `Forward OAuth identity` flag.<br> Enable `Forward OAuth identity` flag.<br>
<img src="grafana-vmgateway-openid-configuration/grafana-ds.webp" > ![Oauth identity](grafana-ds.webp)
Now you can use Grafana to query metrics from the specified tenant. Now you can use Grafana to query metrics from the specified tenant.
Users with `vm_access` claim will be able to query metrics from the specified tenant. Users with `vm_access` claim will be able to query metrics from the specified tenant.
@ -330,22 +318,22 @@ vmagent will write data into VictoriaMetrics single-node and cluster(with tenant
Grafana datasources configuration will be the following: Grafana datasources configuration will be the following:
<img src="grafana-vmgateway-openid-configuration/grafana-test-datasources.webp" > [Test datasources](grafana-vmgateway-openid-configuration/grafana-test-datasources.webp)
Let's login as user with `team=dev` labels limitation set via claims. Let's login as user with `team=dev` labels limitation set via claims.
Using `vmgateway-cluster` results into `No data` response as proxied request will go to tenant `0:1`. Using `vmgateway-cluster` results into `No data` response as proxied request will go to tenant `0:1`.
Since vmagent is only configured to write to `0:0` `No data` is an expected response. Since vmagent is only configured to write to `0:0` `No data` is an expected response.
<img src="grafana-vmgateway-openid-configuration/dev-cluster-nodata.webp" > [Dev cluster nodata](grafana-vmgateway-openid-configuration/dev-cluster-nodata.webp)
Switching to `vmgateway-single` does have data. Note that it is limited to metrics with `team=dev` label. Switching to `vmgateway-single` does have data. Note that it is limited to metrics with `team=dev` label.
<img src="grafana-vmgateway-openid-configuration/dev-single-data.webp" > [Dev single data](grafana-vmgateway-openid-configuration/dev-single-data.webp)
Now lets login as user with `team=admin`. Now lets login as user with `team=admin`.
Both cluster and single node datasources now return metrics for `team=admin`. Both cluster and single node datasources now return metrics for `team=admin`.
<img src="grafana-vmgateway-openid-configuration/admin-cluster-data.webp" > [Admin cluster data](grafana-vmgateway-openid-configuration/admin-cluster-data.webp)
<img src="grafana-vmgateway-openid-configuration/admin-single-data.webp" > [Admin single data](grafana-vmgateway-openid-configuration/admin-single-data.webp)

11
docs/guides/grafana-vmgateway-openid-configuration/_index.md Normal file
View file

@ -0,0 +1,11 @@
---
weight: 5
title: How to configure vmgateway for multi-tenant access using Grafana and OpenID Connect
menu:
docs:
parent: "guides"
weight: 5
aliases:
- /guides/grafana-vmgateway-openid-configuration.html
---
{{% content "README.md" %}}

16
docs/guides/guide-delete-or-replace-metrics.md
View file

@ -8,8 +8,6 @@ menu:
aliases: aliases:
- /guides/guide-delete-or-replace-metrics.html - /guides/guide-delete-or-replace-metrics.html
--- ---
# How to delete or replace metrics in VictoriaMetrics
Data deletion is an operation people expect a database to have. [VictoriaMetrics](https://victoriametrics.com) supports Data deletion is an operation people expect a database to have. [VictoriaMetrics](https://victoriametrics.com) supports
[delete operation](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series) but to a limited extent. Due to implementation details, VictoriaMetrics remains an [append-only database](https://en.wikipedia.org/wiki/Append-only), which perfectly fits the case for storing time series data. But the drawback of such architecture is that it is extremely expensive to mutate the data. Hence, `delete` or `update` operations support is very limited. In this guide, we'll walk through the possible workarounds for deleting or changing already written data in VictoriaMetrics. [delete operation](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-delete-time-series) but to a limited extent. Due to implementation details, VictoriaMetrics remains an [append-only database](https://en.wikipedia.org/wiki/Append-only), which perfectly fits the case for storing time series data. But the drawback of such architecture is that it is extremely expensive to mutate the data. Hence, `delete` or `update` operations support is very limited. In this guide, we'll walk through the possible workarounds for deleting or changing already written data in VictoriaMetrics.
@ -31,7 +29,7 @@ To check that metrics are present in **VictoriaMetrics Cluster** run the followi
_Warning: response can return many metrics, so be careful with series selector._ _Warning: response can return many metrics, so be careful with series selector._
```curl ```sh
curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available' | jq curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available' | jq
``` ```
@ -81,7 +79,7 @@ The expected output:
When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example: When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example:
```curl ```sh
curl -s 'http://vmselect:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=process_cpu_cores_available' curl -s 'http://vmselect:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=process_cpu_cores_available'
``` ```
@ -91,7 +89,7 @@ If operation was successful, the deleted series will stop being [queryable](http
To trigger [forced merge](https://docs.victoriametrics.com/single-server-victoriametrics/#forced-merge) on VictoriaMetrics Cluster run the following command: To trigger [forced merge](https://docs.victoriametrics.com/single-server-victoriametrics/#forced-merge) on VictoriaMetrics Cluster run the following command:
```curl ```sh
curl -v -X POST http://vmstorage:8482/internal/force_merge curl -v -X POST http://vmstorage:8482/internal/force_merge
``` ```
@ -111,7 +109,7 @@ By default, VictoriaMetrics doesn't provide a mechanism for replacing or updatin
For example, let's export metric for `node_memory_MemTotal_bytes` with labels `instance="node-exporter:9100"` and `job="hostname.com"`: For example, let's export metric for `node_memory_MemTotal_bytes` with labels `instance="node-exporter:9100"` and `job="hostname.com"`:
```curl ```sh
curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match[]=node_memory_MemTotal_bytes{instance="node-exporter:9100", job="hostname.com"}' > data.jsonl curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match[]=node_memory_MemTotal_bytes{instance="node-exporter:9100", job="hostname.com"}' > data.jsonl
``` ```
@ -119,7 +117,7 @@ curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match
To check that exported file contains time series we can use [cat](https://man7.org/linux/man-pages/man1/cat.1.html) and [jq](https://stedolan.github.io/jq/download/) To check that exported file contains time series we can use [cat](https://man7.org/linux/man-pages/man1/cat.1.html) and [jq](https://stedolan.github.io/jq/download/)
```curl ```sh
cat data.jsonl | jq cat data.jsonl | jq
``` ```
@ -196,14 +194,14 @@ Victoriametrics supports a lot of [ingestion protocols](https://docs.victoriamet
The next command will import metrics from `data.jsonl` to VictoriaMetrics: The next command will import metrics from `data.jsonl` to VictoriaMetrics:
```curl ```sh
curl -v -X POST http://vminsert:8480/insert/0/prometheus/api/v1/import -T data.jsonl curl -v -X POST http://vminsert:8480/insert/0/prometheus/api/v1/import -T data.jsonl
``` ```
### Check imported metrics ### Check imported metrics
```curl ```sh
curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d match[]=node_memory_MemTotal_bytes curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d match[]=node_memory_MemTotal_bytes
``` ```

4
docs/guides/guide-vmcluster-multiple-retention-setup.md
View file

@ -8,8 +8,6 @@ menu:
aliases: aliases:
- /guides/guide-vmcluster-multiple-retention-setup.html - /guides/guide-vmcluster-multiple-retention-setup.html
--- ---
# Multi Retention Setup within VictoriaMetrics Cluster
**Objective** **Objective**
Setup Victoria Metrics Cluster with support of multiple retention periods within one installation. Setup Victoria Metrics Cluster with support of multiple retention periods within one installation.
@ -34,7 +32,7 @@ The [-retentionPeriod](https://docs.victoriametrics.com/#retention) sets how lon
The diagram below shows a proposed solution The diagram below shows a proposed solution
<img src="guide-vmcluster-multiple-retention-setup.webp"> ![Setup](guide-vmcluster-multiple-retention-setup.webp)
**Implementation Details** **Implementation Details**

15
docs/guides/k8s-ha-monitoring-via-vm-cluster.md
View file

@ -8,9 +8,6 @@ menu:
aliases: aliases:
- /guides/k8s-ha-monitoring-via-vm-cluster.html - /guides/k8s-ha-monitoring-via-vm-cluster.html
--- ---
# HA monitoring setup in Kubernetes via VictoriaMetrics Cluster
**The guide covers:** **The guide covers:**
* High availability monitoring via [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) with Helm charts * High availability monitoring via [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) with Helm charts
@ -300,7 +297,7 @@ kubectl port-forward svc/vmcluster-victoria-metrics-cluster-vmselect 8481:8481
Execute the following command to get metrics via `curl`: Execute the following command to get metrics via `curl`:
```curl ```sh
curl -sg 'http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range?query=count(up{kubernetes_pod_name=~".*vmselect.*"})&start=-10m&step=1m' | jq curl -sg 'http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range?query=count(up{kubernetes_pod_name=~".*vmselect.*"})&start=-10m&step=1m' | jq
``` ```
@ -362,15 +359,15 @@ The expected result of the query `count(up{kubernetes_pod_name=~".*vmselect.*"})
To test via Grafana, we need to install it first. [Install and connect Grafana to VictoriaMetrics](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster.html#4-install-and-connect-grafana-to-victoriametrics-with-helm), login into Grafana and open the metrics [Explore](http://127.0.0.1:3000/explore) page. To test via Grafana, we need to install it first. [Install and connect Grafana to VictoriaMetrics](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster.html#4-install-and-connect-grafana-to-victoriametrics-with-helm), login into Grafana and open the metrics [Explore](http://127.0.0.1:3000/explore) page.
<img src="k8s-ha-monitoring-via-vm-cluster_explore.webp" alt="grafana explore"> ![Explore](k8s-ha-monitoring-via-vm-cluster_explore.webp)
Choose `victoriametrics` from the list of datasources and enter `count(up{kubernetes_pod_name=~".*vmselect.*"})` to the **Metric browser** field as shown on the screenshot, then press **Run query** button: Choose `victoriametrics` from the list of datasources and enter `count(up{kubernetes_pod_name=~".*vmselect.*"})` to the **Metric browser** field as shown on the screenshot, then press **Run query** button:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up.webp"> ![Explore count up](k8s-ha-monitoring-via-vm-cluster_explore-count-up.webp)
The expected output is: The expected output is:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp"> ![Explore count up graph](k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp)
## 5. High Availability ## 5. High Availability
@ -398,13 +395,13 @@ Return to Grafana Explore and press the **Run query** button again.
The expected output is: The expected output is:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp" > ![Explore count up graph](k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp)
As you can see, after we scaled down the `vmstorage` replicas number from three to two pods, metrics are still available and correct. The response is not partial as it was before scaling. Also we see that query `count(up{kubernetes_pod_name=~".*vmselect.*"})` returns the same value as before. As you can see, after we scaled down the `vmstorage` replicas number from three to two pods, metrics are still available and correct. The response is not partial as it was before scaling. Also we see that query `count(up{kubernetes_pod_name=~".*vmselect.*"})` returns the same value as before.
To confirm that the number of `vmstorage` pods is equivalent to two, execute the following request in Grafana Explore: To confirm that the number of `vmstorage` pods is equivalent to two, execute the following request in Grafana Explore:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph2.webp" > ![Explore count up graph 2](k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph2.webp)
## 6. Final thoughts ## 6. Final thoughts

13
docs/guides/k8s-monitoring-via-vm-cluster.md
View file

@ -8,9 +8,6 @@ menu:
aliases: aliases:
- /guides/k8s-monitoring-via-vm-cluster.html - /guides/k8s-monitoring-via-vm-cluster.html
--- ---
# Kubernetes monitoring with VictoriaMetrics Cluster
**This guide covers:** **This guide covers:**
* The setup of a [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts * The setup of a [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
@ -26,7 +23,7 @@ We will use:
* [Helm 3 ](https://helm.sh/docs/intro/install) * [Helm 3 ](https://helm.sh/docs/intro/install)
* [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl) * [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl)
<img src="k8s-monitoring-via-vm-cluster_scheme.webp" alt="VictoriaMetrics Cluster on Kubernetes cluster"> ![VMCluster on K8s](k8s-monitoring-via-vm-cluster_scheme.webp)
## 1. VictoriaMetrics Helm repository ## 1. VictoriaMetrics Helm repository
@ -506,19 +503,19 @@ kubectl --namespace default port-forward $POD_NAME 3000
To check that [VictoriaMetrics](https://victoriametrics.com) collects metrics from k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously got from kubectl. To check that [VictoriaMetrics](https://victoriametrics.com) collects metrics from k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously got from kubectl.
<img src="k8s-monitoring-via-vm-cluster_dashes-agent.webp" alt="grafana dashboards"> ![Dashboards](k8s-monitoring-via-vm-cluster_dashes-agent.webp)
You will see something like this: You will see something like this:
<img src="k8s-monitoring-via-vm-cluster_dashboard.webp" alt="Kubernetes metrics provided by vmcluster"> ![VMCluster metrics](k8s-monitoring-via-vm-cluster_dashboard.webp)
The VictoriaMetrics dashboard is also available to use: The VictoriaMetrics dashboard is also available to use:
<img src="k8s-monitoring-via-vm-cluster_grafana-dash.webp" alt="VictoriaMetrics cluster dashboard"> ![VMCluster dashboard](k8s-monitoring-via-vm-cluster_grafana-dash.webp)
vmagent has its own dashboard: vmagent has its own dashboard:
<img src="k8s-monitoring-via-vm-cluster_vmagent-grafana-dash.webp" alt="vmagent dashboard"> ![VMAgent dashboard](k8s-monitoring-via-vm-cluster_vmagent-grafana-dash.webp)
## 6. Final thoughts ## 6. Final thoughts

11
docs/guides/k8s-monitoring-via-vm-single.md
View file

@ -8,9 +8,6 @@ menu:
aliases: aliases:
- /guides/k8s-monitoring-via-vm-single.html - /guides/k8s-monitoring-via-vm-single.html
--- ---
# Kubernetes monitoring via VictoriaMetrics Single
**This guide covers:** **This guide covers:**
* The setup of a [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts * The setup of a [VictoriaMetrics Single](https://docs.victoriametrics.com/single-server-victoriametrics/) in [Kubernetes](https://kubernetes.io/) via Helm charts
@ -26,7 +23,7 @@ We will use:
* [Helm 3 ](https://helm.sh/docs/intro/install) * [Helm 3 ](https://helm.sh/docs/intro/install)
* [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl) * [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl)
<img src="k8s-monitoring-via-vm-single_k8s-scheme.webp" alt="VictoriaMetrics Single on Kubernetes cluster"> ![VictoriaMetrics Single on Kubernetes cluster](k8s-monitoring-via-vm-single_k8s-scheme.webp)
## 1. VictoriaMetrics Helm repository ## 1. VictoriaMetrics Helm repository
@ -308,15 +305,15 @@ Now Grafana should be accessible on the [http://127.0.0.1:3000](http://127.0.0.1
To check that VictoriaMetrics has collects metrics from the k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously obtained from kubectl. To check that VictoriaMetrics has collects metrics from the k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously obtained from kubectl.
<img src="k8s-monitoring-via-vm-single_grafana-dashboards.webp" alt=""> ![single dashboards](k8s-monitoring-via-vm-single_grafana-dashboards.webp)
You will see something like this: You will see something like this:
<img src="k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp" alt=""> ![k8s dashboards](k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp)
VictoriaMetrics dashboard also available to use: VictoriaMetrics dashboard also available to use:
<img src="k8s-monitoring-via-vm-single_grafana.webp" alt=""> ![single](k8s-monitoring-via-vm-single_grafana.webp)
## 5. Final thoughts ## 5. Final thoughts

8
docs/guides/migrate-from-influx.md
View file

@ -8,8 +8,6 @@ menu:
aliases: aliases:
- /guides/migrate-from-influx.html - /guides/migrate-from-influx.html
--- ---
# Migrate from InfluxDB to VictoriaMetrics
InfluxDB is a well-known time series database built for InfluxDB is a well-known time series database built for
[IoT](https://en.wikipedia.org/wiki/Internet_of_things) monitoring, Application Performance Monitoring (APM) and [IoT](https://en.wikipedia.org/wiki/Internet_of_things) monitoring, Application Performance Monitoring (APM) and
analytics. It has its query language, unique data model, and rich tooling for collecting and processing metrics. analytics. It has its query language, unique data model, and rich tooling for collecting and processing metrics.
@ -142,7 +140,7 @@ for serving read queries. This API is used in various integrations such as
by [VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for by [VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui) - a graphical User Interface for
querying and visualizing metrics: querying and visualizing metrics:
<img src="migrate-from-influx_vmui.webp"> ![Migrate from Influx](migrate-from-influx_vmui.webp)
See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyconcepts/#query-data). See more about [how to query data in VictoriaMetrics](https://docs.victoriametrics.com/keyconcepts/#query-data).
@ -171,7 +169,7 @@ The data sample consists data points for a measurement `foo`
and a field `bar` with additional tag `instance=localhost`. If we would like plot this data as a time series in Grafana and a field `bar` with additional tag `instance=localhost`. If we would like plot this data as a time series in Grafana
it might have the following look: it might have the following look:
<img src="migrate-from-influx_data-sample-in-influx.webp"> ![Data sample in Influx](migrate-from-influx_data-sample-in-influx.webp)
The query used for this panel is written in The query used for this panel is written in
[InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/): [InfluxQL](https://docs.influxdata.com/influxdb/v1.8/query_language/):
@ -206,7 +204,7 @@ InfluxQL query might be translated to MetricsQL let's break it into components f
In result, executing the `foo_bar{instance="localhost"}` MetricsQL expression with `step=1m` for the same set of data in In result, executing the `foo_bar{instance="localhost"}` MetricsQL expression with `step=1m` for the same set of data in
Grafana will have the following form: Grafana will have the following form:
<img src="migrate-from-influx_data-sample-in-vm.webp"> ![Data sample in VM](migrate-from-influx_data-sample-in-vm.webp)
Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points Visualizations from both databases are a bit different - VictoriaMetrics shows some extra points
filling the gaps in the graph. This behavior is described in more filling the gaps in the graph. This behavior is described in more

4
docs/guides/multi-regional-setup-dedicated-regions.md
View file

@ -8,15 +8,13 @@ menu:
aliases: aliases:
- /guides/multi-regional-setup-dedicated-regions.html - /guides/multi-regional-setup-dedicated-regions.html
--- ---
# Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring
### Scenario ### Scenario
Let's cover the case. You have multiple regions with workloads and want to collect metrics. Let's cover the case. You have multiple regions with workloads and want to collect metrics.
The monitoring setup is in the dedicated regions as shown below: The monitoring setup is in the dedicated regions as shown below:
<img src="multi-regional-setup-dedicated-regions.webp" alt="Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring"> ![Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring](multi-regional-setup-dedicated-regions.webp)
Every workload region (Earth, Mars, Venus) has a vmagent that sends data to multiple regions with a monitoring setup. Every workload region (Earth, Mars, Venus) has a vmagent that sends data to multiple regions with a monitoring setup.
The monitoring setup (Ground Control 1,2) contains VictoriaMetrics Time Series Database(TSDB) cluster or single. The monitoring setup (Ground Control 1,2) contains VictoriaMetrics Time Series Database(TSDB) cluster or single.

2
docs/guides/understand-your-setup-size.md
View file

@ -8,8 +8,6 @@ menu:
aliases: aliases:
- /guides/understand-your-setup-size.html - /guides/understand-your-setup-size.html
--- ---
# Understand Your Setup Size
The docs provide a simple and high-level overview of Ingestion Rate, Active Time Series, and Query per Second. These terms are a part of capacity planning ([Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning), [Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)) and [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/) pricing. The docs provide a simple and high-level overview of Ingestion Rate, Active Time Series, and Query per Second. These terms are a part of capacity planning ([Single-Node](https://docs.victoriametrics.com/single-server-victoriametrics/#capacity-planning), [Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/#capacity-planning)) and [Managed VictoriaMetrics](https://docs.victoriametrics.com/managed-victoriametrics/) pricing.
## Terminology ## Terminology

36
docs/keyConcepts.md
View file

@ -10,9 +10,6 @@ aliases:
- /keyConcepts.html - /keyConcepts.html
- /keyсoncepts.html - /keyсoncepts.html
--- ---
# Key concepts
## Data model ## Data model
### What is a metric ### What is a metric
@ -138,7 +135,7 @@ So, the `counter` metric shows the number of observed events since the service s
In programming, `counter` is a variable that you **increment** each time something happens. In programming, `counter` is a variable that you **increment** each time something happens.
<img src="keyConcepts_counter.webp"> ![counter](keyConcepts_counter.webp)
`vm_http_requests_total` is a typical example of a counter. The interpretation of a graph `vm_http_requests_total` is a typical example of a counter. The interpretation of a graph
above is that time series `vm_http_requests_total{instance="localhost:8428", job="victoriametrics", path="api/v1/query_range"}` above is that time series `vm_http_requests_total{instance="localhost:8428", job="victoriametrics", path="api/v1/query_range"}`
@ -164,7 +161,7 @@ by humans from other metric types.
Gauge is used for measuring a value that can go up and down: Gauge is used for measuring a value that can go up and down:
<img src="keyConcepts_gauge.webp"> ![gauge](keyConcepts_gauge.webp)
The metric `process_resident_memory_anon_bytes` on the graph shows the memory usage of the application at every given time. The metric `process_resident_memory_anon_bytes` on the graph shows the memory usage of the application at every given time.
It is changing frequently, going up and down showing how the process allocates and frees the memory. It is changing frequently, going up and down showing how the process allocates and frees the memory.
@ -262,7 +259,7 @@ Such a combination of `counter` metrics allows
plotting [Heatmaps in Grafana](https://grafana.com/docs/grafana/latest/visualizations/heatmap/) plotting [Heatmaps in Grafana](https://grafana.com/docs/grafana/latest/visualizations/heatmap/)
and calculating [quantiles](https://prometheus.io/docs/practices/histograms/#quantiles): and calculating [quantiles](https://prometheus.io/docs/practices/histograms/#quantiles):
<img src="keyConcepts_histogram.webp"> ![histogram](keyConcepts_histogram.webp)
Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/metricsql/#prometheus_buckets) Grafana doesn't understand buckets with `vmrange` labels, so the [prometheus_buckets](https://docs.victoriametrics.com/metricsql/#prometheus_buckets)
function must be used for converting buckets with `vmrange` labels to buckets with `le` labels before building heatmaps in Grafana. function must be used for converting buckets with `vmrange` labels to buckets with `le` labels before building heatmaps in Grafana.
@ -304,7 +301,7 @@ go_gc_duration_seconds_count 83
The visualization of summaries is pretty straightforward: The visualization of summaries is pretty straightforward:
<img src="keyConcepts_summary.webp"> ![summary](keyConcepts_summary.webp)
Such an approach makes summaries easier to use but also puts significant limitations compared to [histograms](#histogram): Such an approach makes summaries easier to use but also puts significant limitations compared to [histograms](#histogram):
@ -375,7 +372,7 @@ VictoriaMetrics supports both models used in modern monitoring applications: [pu
Client regularly sends the collected metrics to the server in the push model: Client regularly sends the collected metrics to the server in the push model:
<img src="keyConcepts_push_model.webp"> ![push model](keyConcepts_push_model.webp)
The client (application) decides when and where to send its metrics. VictoriaMetrics supports many protocols The client (application) decides when and where to send its metrics. VictoriaMetrics supports many protocols
for data ingestion (aka `push protocols`) - see [the full list here](https://docs.victoriametrics.com/#how-to-import-time-series-data). for data ingestion (aka `push protocols`) - see [the full list here](https://docs.victoriametrics.com/#how-to-import-time-series-data).
@ -420,7 +417,7 @@ The cons of push protocol:
Pull model is an approach popularized by [Prometheus](https://prometheus.io/), where the monitoring system decides when Pull model is an approach popularized by [Prometheus](https://prometheus.io/), where the monitoring system decides when
and where to pull metrics from: and where to pull metrics from:
<img src="keyConcepts_pull_model.webp"> ![pull model](keyConcepts_pull_model.webp)
In pull model, the monitoring system needs to be aware of all the applications it needs to monitor. The metrics are In pull model, the monitoring system needs to be aware of all the applications it needs to monitor. The metrics are
scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`). scraped (pulled) from the known applications (aka `scrape targets`) via HTTP protocol on a regular basis (aka `scrape_interval`).
@ -451,7 +448,7 @@ models for data collection. Many installations use exclusively one of these mode
The most common approach for data collection is using both models: The most common approach for data collection is using both models:
<img src="keyConcepts_data_collection.webp"> ![data collection](keyConcepts_data_collection.webp)
In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent/). Vmagent is In this approach the additional component is used - [vmagent](https://docs.victoriametrics.com/vmagent/). Vmagent is
a lightweight agent whose main purpose is to collect, filter, relabel and deliver metrics to VictoriaMetrics. a lightweight agent whose main purpose is to collect, filter, relabel and deliver metrics to VictoriaMetrics.
@ -466,7 +463,7 @@ installation for querying collected data.
VictoriaMetrics components allow building more advanced topologies. For example, vmagents can push metrics from separate datacenters to the central VictoriaMetrics: VictoriaMetrics components allow building more advanced topologies. For example, vmagents can push metrics from separate datacenters to the central VictoriaMetrics:
<img src="keyConcepts_two_dcs.webp"> ![two dcs](keyConcepts_two_dcs.webp)
VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) VictoriaMetrics in this example may be either [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). Vmagent also allows or [VictoriaMetrics Cluster](https://docs.victoriametrics.com/cluster-victoriametrics/). Vmagent also allows
@ -530,7 +527,8 @@ foo_bar 4.00 1652170560000 # 2022-05-10 10:16:00
The data above contains a list of samples for the `foo_bar` time series with time intervals between samples The data above contains a list of samples for the `foo_bar` time series with time intervals between samples
ranging from 1m to 3m. If we plot this data sample on the graph, it will have the following form: ranging from 1m to 3m. If we plot this data sample on the graph, it will have the following form:
<img src="keyConcepts_data_samples.webp" width="500"> ![data samples](keyConcepts_data_samples.webp)
{width="500"}
To get the value of the `foo_bar` series at some specific moment of time, for example `2022-05-10 10:03:00`, in To get the value of the `foo_bar` series at some specific moment of time, for example `2022-05-10 10:03:00`, in
VictoriaMetrics we need to issue an **instant query**: VictoriaMetrics we need to issue an **instant query**:
@ -564,7 +562,8 @@ In response, VictoriaMetrics returns a single sample-timestamp pair with a value
we'll see that there is no raw sample at `2022-05-10 10:03`. When there is no raw sample at the we'll see that there is no raw sample at `2022-05-10 10:03`. When there is no raw sample at the
requested timestamp, VictoriaMetrics will try to locate the closest sample before the requested timestamp: requested timestamp, VictoriaMetrics will try to locate the closest sample before the requested timestamp:
<img src="keyConcepts_instant_query.webp" width="500"> ![instant query](keyConcepts_instant_query.webp)
{width="500"}
The time range in which VictoriaMetrics will try to locate a replacement for a missing data sample is equal to `5m` The time range in which VictoriaMetrics will try to locate a replacement for a missing data sample is equal to `5m`
by default and can be overridden via the `step` parameter. by default and can be overridden via the `step` parameter.
@ -704,7 +703,8 @@ see that it contains only 13 raw samples. What happens here is that the range qu
an [instant query](#instant-query) executed `1 + (start-end)/step` times on the time range from `start` to `end`. If we plot an [instant query](#instant-query) executed `1 + (start-end)/step` times on the time range from `start` to `end`. If we plot
this request in VictoriaMetrics the graph will be shown as the following: this request in VictoriaMetrics the graph will be shown as the following:
<img src="keyConcepts_range_query.webp" width="500"> ![range query](keyConcepts_range_query.webp)
{width="500"}
The blue dotted lines in the figure are the moments when the instant query was executed. Since the instant query retains the The blue dotted lines in the figure are the moments when the instant query was executed. Since the instant query retains the
ability to return replacements for missing points, the graph contains two types of data points: `real` and `ephemeral`. ability to return replacements for missing points, the graph contains two types of data points: `real` and `ephemeral`.
@ -745,12 +745,14 @@ This flag prevents from non-consistent results due to the fact that only part of
Here is an illustration of a potential problem when `-search.latencyOffset` is set to zero: Here is an illustration of a potential problem when `-search.latencyOffset` is set to zero:
<img src="keyConcepts_without_latencyOffset.webp" width="1000"> ![without latency offset](keyConcepts_without_latencyOffset.webp)
{width="1000"}
When this flag is set, the VM will return the last metric value collected before the `-search.latencyOffset` When this flag is set, the VM will return the last metric value collected before the `-search.latencyOffset`
duration throughout the `-search.latencyOffset` duration: duration throughout the `-search.latencyOffset` duration:
<img src="keyConcepts_with_latencyOffset.webp" width="1000"> ![with latency offset](keyConcepts_with_latencyOffset.webp)
{width="1000"}
It can be overridden on per-query basis via `latency_offset` query arg. It can be overridden on per-query basis via `latency_offset` query arg.
@ -948,7 +950,7 @@ VictoriaMetrics has a built-in graphical User Interface for querying and visuali
[VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui). [VMUI](https://docs.victoriametrics.com/single-server-victoriametrics/#vmui).
Open `http://victoriametrics:8428/vmui` page, type the query and see the results: Open `http://victoriametrics:8428/vmui` page, type the query and see the results:
<img src="keyConcepts_vmui.webp"> ![vmui](keyConcepts_vmui.webp)
VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage) VictoriaMetrics supports [Prometheus HTTP API](https://docs.victoriametrics.com/single-server-victoriametrics/#prometheus-querying-api-usage)
which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup) which makes it possible to [query it with Grafana](https://docs.victoriametrics.com/single-server-victoriametrics/#grafana-setup)

14
docs/managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.md
View file

@ -9,11 +9,9 @@ menu:
aliases: aliases:
- /managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.html - /managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.html
--- ---
# Alerting with vmalert and Managed VictoriaMetrics
This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics
<img src="alerting-vmalert-managed-victoria-metrics_setup.webp"> ![Metrics setup](alerting-vmalert-managed-victoria-metrics_setup.webp)
## Preconditions ## Preconditions
@ -55,14 +53,14 @@ For instructions on how to create tokens, please refer to this section of the [d
#### Single-Node #### Single-Node
<img src="alerting-vmalert-managed-victoria-metrics_token_created_single.webp"> ![Token created single](alerting-vmalert-managed-victoria-metrics_token_created_single.webp)
<img src="alerting-vmalert-managed-victoria-metrics_copy_datasource_single.webp"> ![Copy datasource single](alerting-vmalert-managed-victoria-metrics_copy_datasource_single.webp)
#### Cluster #### Cluster
<img src="alerting-vmalert-managed-victoria-metrics_token_created_cluster.webp"> ![Token created cluster](alerting-vmalert-managed-victoria-metrics_token_created_cluster.webp)
<img src="alerting-vmalert-managed-victoria-metrics_copy_reading_datasource_cluster.webp"> ![Reading datasource cluster](alerting-vmalert-managed-victoria-metrics_copy_reading_datasource_cluster.webp)
<img src="alerting-vmalert-managed-victoria-metrics_copy_writing_datasource_cluster.webp"> ![Writing atasource cluster](alerting-vmalert-managed-victoria-metrics_copy_writing_datasource_cluster.webp)
### vmalert configuration ### vmalert configuration

19
docs/managed-victoriametrics/alertmanager-setup-for-deployment.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/alertmanager-setup-for-deployment.html - /managed-victoriametrics/alertmanager-setup-for-deployment.html
--- ---
# Alerting stack configuration and Managed VictoriaMetrics
Managed VictoriaMetrics supports configuring alerting rules, powered by vmalert, and sending notifications with hosted Alertmanager. Managed VictoriaMetrics supports configuring alerting rules, powered by vmalert, and sending notifications with hosted Alertmanager.
## Configure Alertmanager ## Configure Alertmanager
@ -19,9 +16,9 @@ Managed VictoriaMetrics supports configuring alerting rules, powered by vmalert,
You have two options to configure Cloud Alertmanager: You have two options to configure Cloud Alertmanager:
1. From integrations section: Menu **"Integrations" `->` "Cloud Alertmanager" `->` "New configuration"**: 1. From integrations section: Menu **"Integrations" `->` "Cloud Alertmanager" `->` "New configuration"**:
<img src="alertmanager-setup-for-deployment_integrations.webp"> ![Setup for deployment integrations](alertmanager-setup-for-deployment_integrations.webp)
2. From deployment page: **"Deployment page" `->` "Rules" tab `->` "Settings" `->` "Connect notifier" `/` "New notifier"**: 2. From deployment page: **"Deployment page" `->` "Rules" tab `->` "Settings" `->` "Connect notifier" `/` "New notifier"**:
<img src="alertmanager-setup-for-deployment_connect_notifier.webp"> ![Setup for deployment connect notifier](alertmanager-setup-for-deployment_connect_notifier.webp)
For creating a new configuration, you need to provide the following parameters: For creating a new configuration, you need to provide the following parameters:
@ -34,7 +31,7 @@ After creating the configuration, you can connect it to one or multiple deployme
In order to do this you need to go to the "Deployment page" `->` "Rules" tab `->` "Settings" `, In order to do this you need to go to the "Deployment page" `->` "Rules" tab `->` "Settings" `,
select the created notifier and confirm the action: select the created notifier and confirm the action:
<img src="alertmanager-setup-for-deployment_select_notifier.webp"> ![Select notifier](alertmanager-setup-for-deployment_select_notifier.webp)
Alertmanager is now set up for your deployment, and you be able to get notifications from it. Alertmanager is now set up for your deployment, and you be able to get notifications from it.
@ -129,7 +126,7 @@ If for some reason Cloud Alertmanager is not suitable for you, you can use Manag
For that select Custom Alertmanager instead of Cloud Alertmanager when [creating the Alertmanager](#configure-alertmanager): For that select Custom Alertmanager instead of Cloud Alertmanager when [creating the Alertmanager](#configure-alertmanager):
<img src="alertmanager-setup-for-deployment_custom_am.webp"> ![Custom AlertManager](alertmanager-setup-for-deployment_custom_am.webp)
Limitations for the Custom Alertmanager: Limitations for the Custom Alertmanager:
@ -143,7 +140,7 @@ You can test the connection to your custom Alertmanager by clicking the "Test co
Alerting and recording rules could be uploaded on **"Deployment page" `->` "Rules" tab `->` "Settings"**: Alerting and recording rules could be uploaded on **"Deployment page" `->` "Rules" tab `->` "Settings"**:
<img src="alertmanager-setup-for-deployment_upload_rules.webp"> ![Upload rules](alertmanager-setup-for-deployment_upload_rules.webp)
You can click on the upload area or drag and drop the files with rules there. You can click on the upload area or drag and drop the files with rules there.
@ -161,7 +158,7 @@ You can also use API for uploading rules. Switch to **"Upload with API"** on the
- Choose the API key for uploading rules - Choose the API key for uploading rules
- After that you can copy the curl command for uploading rules and execute it in your terminal - After that you can copy the curl command for uploading rules and execute it in your terminal
<img src="alertmanager-setup-for-deployment_upload_with_api.webp"> ![Upload with API](alertmanager-setup-for-deployment_upload_with_api.webp)
You can use the following API endpoints for the automation with rules: You can use the following API endpoints for the automation with rules:
@ -202,13 +199,13 @@ groups:
The state of created rules is located in the `Rules` section of your deployment: The state of created rules is located in the `Rules` section of your deployment:
<img src="alertmanager-setup-for-deployment_rules_state.webp"> ![Rules state](alertmanager-setup-for-deployment_rules_state.webp)
### Debug ### Debug
It's possible to debug the alerting stack with logs for vmalert and Alertmanager, which are accessible in the `Logs` section of the deployment. It's possible to debug the alerting stack with logs for vmalert and Alertmanager, which are accessible in the `Logs` section of the deployment.
<img src="alertmanager-setup-for-deployment_troubleshoot_logs.webp"> ![Troubleshoot logs](alertmanager-setup-for-deployment_troubleshoot_logs.webp)
### Monitoring ### Monitoring

6
docs/managed-victoriametrics/how-to-monitor-k8s.md
View file

@ -9,8 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/how-to-monitor-k8s.html - /managed-victoriametrics/how-to-monitor-k8s.html
--- ---
# Kubernetes Monitoring with Managed VictoriaMetrics
Monitoring kubernetes cluster is necessary to build SLO/SLI, to analyze performance and cost-efficiency of your workloads. Monitoring kubernetes cluster is necessary to build SLO/SLI, to analyze performance and cost-efficiency of your workloads.
To enable kubernetes cluster monitoring, we will be collecting metrics about cluster performance and utilization from kubernetes components like `kube-api-server`, `kube-controller-manager`, `kube-scheduler`, `kube-state-metrics`, `etcd`, `core-dns`, `kubelet` and `kube-proxy`. We will also install some recording rules, alert rules and dashboards to provide visibility of cluster performance, as well as alerting for cluster metrics. To enable kubernetes cluster monitoring, we will be collecting metrics about cluster performance and utilization from kubernetes components like `kube-api-server`, `kube-controller-manager`, `kube-scheduler`, `kube-state-metrics`, `etcd`, `core-dns`, `kubelet` and `kube-proxy`. We will also install some recording rules, alert rules and dashboards to provide visibility of cluster performance, as well as alerting for cluster metrics.
@ -47,7 +45,7 @@ Install the Helm chart in a custom namespace
``` ```
You can find your access token on the "Access" tab of your deployment You can find your access token on the "Access" tab of your deployment
<img src="kubernetes_monitoring.webp"> ![K8s Monitoring](kubernetes_monitoring.webp)
1. Set up a Helm repository using the following commands: 1. Set up a Helm repository using the following commands:
```shell ```shell
@ -129,7 +127,7 @@ Connect to grafana and create your datasource
Choose VictoriaMetrics or Prometheus as datasource type. Make sure you made this datasource as default for dashboards to work. Choose VictoriaMetrics or Prometheus as datasource type. Make sure you made this datasource as default for dashboards to work.
> You can find token and URL in your deployment, on Access tab > You can find token and URL in your deployment, on Access tab
<img src="how-to-monitor-k8s_datasource.webp"> ![K8s datasource](how-to-monitor-k8s_datasource.webp)
## Test it ## Test it

3
docs/managed-victoriametrics/overview.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/overview.html - /managed-victoriametrics/overview.html
--- ---
# Overview of Managed VictoriaMetrics
VictoriaMetrics is a fast and easy-to-use monitoring solution and time series database. VictoriaMetrics is a fast and easy-to-use monitoring solution and time series database.
It integrates well with existing monitoring systems such as Grafana, Prometheus, Graphite, It integrates well with existing monitoring systems such as Grafana, Prometheus, Graphite,
InfluxDB, OpenTSDB and DataDog - see [these docs](https://docs.victoriametrics.com/#how-to-import-time-series-data) for details. InfluxDB, OpenTSDB and DataDog - see [these docs](https://docs.victoriametrics.com/#how-to-import-time-series-data) for details.

80
docs/managed-victoriametrics/quickstart.md
View file

@ -9,8 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/quickstart.html - /managed-victoriametrics/quickstart.html
--- ---
# Quick Start
Managed VictoriaMetrics is a hosted monitoring platform, where users can run the VictoriaMetrics Managed VictoriaMetrics is a hosted monitoring platform, where users can run the VictoriaMetrics
that they know and love on AWS without the need to perform typical DevOps tasks such as proper configuration, that they know and love on AWS without the need to perform typical DevOps tasks such as proper configuration,
monitoring, logs collection, access protection, software updates, backups, etc. monitoring, logs collection, access protection, software updates, backups, etc.
@ -34,66 +32,66 @@ There are two different methods to create an account:
### Create an account via Google Auth service: ### Create an account via Google Auth service:
1. Click `Continue with Google` button on the [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart) 1. Click `Continue with Google` button on the [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart)
<img src="quick_start_signup_google_click.webp" > ![Signup Google click](quick_start_signup_google_click.webp)
1. Choose Google account you want to use for registration 1. Choose Google account you want to use for registration
<img src="quick_start_signup_choose_google_account.webp" > ![Signup Google choose account](quick_start_signup_choose_google_account.webp)
1. You will be automatically redirected to the main page of the Managed VictoriaMetrics 1. You will be automatically redirected to the main page of the Managed VictoriaMetrics
<img src="quick_start_signup_success.webp" > ![Signup success](quick_start_signup_success.webp)
### Create an account by filling in a registration form: ### Create an account by filling in a registration form:
1. Fill in your email, password and password confirmation on [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart). 1. Fill in your email, password and password confirmation on [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart).
<img src="quick_start_signup.webp" > ![Signup](quick_start_signup.webp)
1.All fields are required. Any errors will be shown in the interface, so it is easy to understand what should be adjusted. 1.All fields are required. Any errors will be shown in the interface, so it is easy to understand what should be adjusted.
<img src="quick_start_signup_errors.webp" > ![Signup errors](quick_start_signup_errors.webp)
1. Press `Create account` button when all fields are filled in. 1. Press `Create account` button when all fields are filled in.
<img src="quick_start_signup_create_account_click.webp" > ![Signup create account](quick_start_signup_create_account_click.webp)
You will be redirected to the main page with a notification message to confirm your email. You will be redirected to the main page with a notification message to confirm your email.
<img src="quick_start_signup_success.webp" > ![Signup success](quick_start_signup_success.webp)
You will also receive an email with a confirmation link as shown on the picture below: You will also receive an email with a confirmation link as shown on the picture below:
<img src="quick_start_signup_email_confirm.webp" > ![Signup email confirm](quick_start_signup_email_confirm.webp)
It is necessary to confirm your email address. Otherwise, you won't be able to create a deployment. It is necessary to confirm your email address. Otherwise, you won't be able to create a deployment.
After successful confirmation of your email address, you'll be able to [create your first deployment](#creating-deployments) or [add a payment method](#adding-a-payment-method). After successful confirmation of your email address, you'll be able to [create your first deployment](#creating-deployments) or [add a payment method](#adding-a-payment-method).
<img src="quick_start_signup_email_confirmed.webp" > ![Signup email confirmed](quick_start_signup_email_confirmed.webp)
## Adding a payment method ## Adding a payment method
1. Navigate to a [Billing](https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart) page or click on `Upgrade` button as shown below: 1. Navigate to a [Billing](https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart) page or click on `Upgrade` button as shown below:
<img src="how_to_add_payment_method_upgrade.webp" > ![Add payment method](how_to_add_payment_method_upgrade.webp)
1. Choose a payment method 1. Choose a payment method
<img src="how_to_add_payment_method_choose_method.webp" > ![Add payment method choose](how_to_add_payment_method_choose_method.webp)
### Pay with a card ### Pay with a card
1. Click on an `Add card` panel and fill in all the fields in the form and press `Add card` button 1. Click on an `Add card` panel and fill in all the fields in the form and press `Add card` button
<img src="how_to_add_payment_method_add_card.webp" > ![Add payment method add card](how_to_add_payment_method_add_card.webp)
1. An error message will appear if a card us invalid 1. An error message will appear if a card us invalid
<img src="how_to_add_payment_method_invalid_card.webp" > ![Add payment method invalid card](how_to_add_payment_method_invalid_card.webp)
1. Successfully added card will be shown on the page as follows: 1. Successfully added card will be shown on the page as follows:
<img src="how_to_add_payment_method_card_added.webp" > ![Add payment method card added](how_to_add_payment_method_card_added.webp)
### Link your AWS billing account via AWS Marketplace ### Link your AWS billing account via AWS Marketplace
@ -102,28 +100,28 @@ When you need to unify your AWS billing, you can start a subscription on AWS Mar
1. Click on the `Buy on AWS Marketplace` panel: 1. Click on the `Buy on AWS Marketplace` panel:
<img src="how_to_add_payment_method_aws_click.webp" > ![Add payment method AWS click](how_to_add_payment_method_aws_click.webp)
1. You will be redirected to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc" target="_blank">Managed VictoriaMetrics</a> product page. 1. You will be redirected to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc" target="_blank">Managed VictoriaMetrics</a> product page.
1. Click on `View purchase option` button, and you will be redirected to an AWS login page or to a subscribe page on AWS Marketplace. 1. Click on `View purchase option` button, and you will be redirected to an AWS login page or to a subscribe page on AWS Marketplace.
<img src="quickstart_aws-purchase-click.webp" > ![AWS purchase click](quickstart_aws-purchase-click.webp)
1. Go to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc">Managed VictoriaMetrics</a> product page and click `Continue to Subscribe` button: 1. Go to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc">Managed VictoriaMetrics</a> product page and click `Continue to Subscribe` button:
<img src="quickstart_continue-subscribe.webp" > ![Continue subscribe](quickstart_continue-subscribe.webp)
1. Press the `Subscribe` button: 1. Press the `Subscribe` button:
<img src="quickstart_subscribe.webp" > ![Subscribe](quickstart_subscribe.webp)
1. After that you will see a success message where you should click `Set up your account` button: 1. After that you will see a success message where you should click `Set up your account` button:
<img src="quickstart_setup-your-account.webp" > ![Setup your account](quickstart_setup-your-account.webp)
1. You'll be redirected back to Managed VictoriaMetrics <a href="https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart" target="_blank">billing page</a>: 1. You'll be redirected back to Managed VictoriaMetrics <a href="https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart" target="_blank">billing page</a>:
<img src="how_to_add_payment_method_aws_finish.webp" > ![Add payment method finish](how_to_add_payment_method_aws_finish.webp)
### Switching between payment methods ### Switching between payment methods
@ -132,17 +130,17 @@ If both payment methods are added, it is possible to easily switch between them.
Click on the radio button like on the picture below and confirm the change: Click on the radio button like on the picture below and confirm the change:
<img src="change_payment_method.webp" > ![Change payment method](change_payment_method.webp)
<img src="change_payment_confirmation.webp" > ![Change payment method confirmation](change_payment_confirmation.webp)
If the payment method was changed successfully, the following message will appear: If the payment method was changed successfully, the following message will appear:
<img src="change_payment_method_success.webp" > ![Change payment method success](change_payment_method_success.webp)
## Password restoration ## Password restoration
@ -151,31 +149,31 @@ If you forgot your password, it can be restored in the following way:
1. Click `Forgot password?` link on the [Sign In](https://cloud.victoriametrics.com/signIn?utm_source=website&utm_campaign=docs_quickstart) page: 1. Click `Forgot password?` link on the [Sign In](https://cloud.victoriametrics.com/signIn?utm_source=website&utm_campaign=docs_quickstart) page:
<img src="quick_start_restore_password.webp" > ![Restore password](quick_start_restore_password.webp)
1. Enter your email and click `Reset password` button: 1. Enter your email and click `Reset password` button:
<img src="quick_start_restore_password_email_field.webp" > ![Restore password email](quick_start_restore_password_email_field.webp)
<img src="quick_start_restore_password_message.webp" > ![Restore password message](quick_start_restore_password_message.webp)
1. Follow the instructions sent to your email in order to get access to your Managed VictoriaMetrics account: 1. Follow the instructions sent to your email in order to get access to your Managed VictoriaMetrics account:
<img src="quick_start_restore_password_email.webp" > ![Restore password email](quick_start_restore_password_email.webp)
1. Navigate to the Profile page by clicking the corresponding link in the top right corner: 1. Navigate to the Profile page by clicking the corresponding link in the top right corner:
<img src="quick_start_restore_password_profile_click.webp" > ![Restore password profile click](quick_start_restore_password_profile_click.webp)
1. Enter a new password on the Profile page and press `Save`: 1. Enter a new password on the Profile page and press `Save`:
<img src="quick_start_restore_password_profile_fields.webp" > ![Restpre password profile fields](quick_start_restore_password_profile_fields.webp)
## Creating deployments ## Creating deployments
@ -186,11 +184,11 @@ will see a list of your existing deployments and will be able to manage them.
To create a deployment click on the button `Start sending metrics` button: To create a deployment click on the button `Start sending metrics` button:
<img src="create_deployment_start.webp" > ![Create deployment start](create_deployment_start.webp)
When you already have at least one deployment you can create a new one by clicking on the `Create deployment` button: When you already have at least one deployment you can create a new one by clicking on the `Create deployment` button:
<img src="create_deployment_continue.webp"> ![Create deployment continue](create_deployment_continue.webp)
On the opened screen, choose parameters of your new deployment: On the opened screen, choose parameters of your new deployment:
@ -204,7 +202,7 @@ On the opened screen, choose parameters of your new deployment:
* `Size` of your deployment [based on your needs](https://docs.victoriametrics.com/guides/understand-your-setup-size.html) * `Size` of your deployment [based on your needs](https://docs.victoriametrics.com/guides/understand-your-setup-size.html)
<img src="create_deployment_form.webp" > ![Create deployment form](create_deployment_form.webp)
When all parameters are configured, click on the `Create` button, and deployment will be created. When all parameters are configured, click on the `Create` button, and deployment will be created.
@ -214,11 +212,11 @@ while the hardware spins-up, just wait for a couple of minutes and reload the pa
You'll also be notified via email once your deployment is ready to use: You'll also be notified via email once your deployment is ready to use:
<img src="create_deployment_created.webp" > ![Create deployment created](create_deployment_created.webp)
<img src="create_deployment_active_email.webp" > ![Create deployment active email](create_deployment_active_email.webp)
## Start writing and reading data ## Start writing and reading data
@ -229,7 +227,7 @@ and is ready to accept write and read requests.
Click on deployment name and navigate to the Access tab to get the access token: Click on deployment name and navigate to the Access tab to get the access token:
<img src="deployment_access.webp" > ![Deployment access](deployment_access.webp)
Access tokens are used in token-based authentication to allow an application to access the VictoriaMetrics API. Access tokens are used in token-based authentication to allow an application to access the VictoriaMetrics API.
@ -237,11 +235,11 @@ Supported token types are `Read-Only`, `Write-Only` and `Read-Write`. Click on t
to see usage examples: to see usage examples:
<img src="deployment_access_write_example.webp" > ![Deployment access write example](deployment_access_write_example.webp)
<img src="deployment_access_read_example.webp" > ![Deployment access read example](deployment_access_read_example.webp)
Follow usage examples in order to configure access to VictoriaMetrics for your Prometheus, Follow usage examples in order to configure access to VictoriaMetrics for your Prometheus,
@ -254,13 +252,13 @@ deployment's page.
It is important to know that downgrade for cluster is currently not available. It is important to know that downgrade for cluster is currently not available.
<img src="modify_deployment.webp" > ![Modify deployment](modify_deployment.webp)
To discover additional configuration options click on `Advanced Settings` button, so you should see the following: To discover additional configuration options click on `Advanced Settings` button, so you should see the following:
<img src="modify_deployment_additional_settings.webp" > ![Modify deployment additional settings](modify_deployment_additional_settings.webp)
In that section, additional params can be set: In that section, additional params can be set:

22
docs/managed-victoriametrics/setup-notifications.md
View file

@ -9,8 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/setup-notifications.html - /managed-victoriametrics/setup-notifications.html
--- ---
# Notifications in Managed VictoriaMetrics
The guide covers how to enable email and Slack notifications. The guide covers how to enable email and Slack notifications.
Table of content: Table of content:
@ -21,7 +19,7 @@ Table of content:
When you enter the notification section, you will be able to fill in the channels in which you When you enter the notification section, you will be able to fill in the channels in which you
want to receive notifications want to receive notifications
<img src="notifications_view.webp" > ![Notifications view](notifications_view.webp)
## Setup Slack notifications ## Setup Slack notifications
@ -29,22 +27,22 @@ want to receive notifications
How to do this is indicated on the following link How to do this is indicated on the following link
<a href="https://api.slack.com/messaging/webhooks" target="_blank">https://api.slack.com/messaging/webhooks</a> <a href="https://api.slack.com/messaging/webhooks" target="_blank">https://api.slack.com/messaging/webhooks</a>
<img src="notifications_view.webp" > ![Notifications view](notifications_view.webp)
1. Specify Slack channels 1. Specify Slack channels
Enter one or more channels into input and press enter or choose it after each input. Enter one or more channels into input and press enter or choose it after each input.
<img src="notifications_setup_slack.webp" > ![Slack setup](notifications_setup_slack.webp)
<img src="notifications_setup_slack_enter_channel.webp"> ![Slack enter channel](notifications_setup_slack_enter_channel.webp)
## Setup emails notifications ## Setup emails notifications
You can specify one or multiple emails for notifications in the input field. By default, You can specify one or multiple emails for notifications in the input field. By default,
email notifications are enabled for the account owner email notifications are enabled for the account owner
<img src="notifications_setup_emails.webp" > ![Setup emails](notifications_setup_emails.webp)
<img src="notifications_setup_emails_input.webp" > ![Emails input](notifications_setup_emails_input.webp)
## Send test notification ## Send test notification
@ -55,18 +53,18 @@ If only Slack channels and webhook are specified correctly, you will receive the
If only the emails are specified, you will receive notifications to those emails. If only the emails are specified, you will receive notifications to those emails.
When both notifications are specified, all notifications will be sent to Slack channels and emails. When both notifications are specified, all notifications will be sent to Slack channels and emails.
<img src="notifications_save_and_test.webp" > ![Save and test](notifications_save_and_test.webp)
If the Save button is pressed, then entered channels will be only saved, and you get a success message. If the Save button is pressed, then entered channels will be only saved, and you get a success message.
If the Save and Test button is pressed, then all entered information will be saved, If the Save and Test button is pressed, then all entered information will be saved,
and test notifications will be sent to the entered channels and test notifications will be sent to the entered channels
<img src="notifications_save_success.webp" > ![Save success](notifications_save_success.webp)
Examples of the test notification messages: Examples of the test notification messages:
<img src="notifications_slack_test.webp" > ![Slack test](notifications_slack_test.webp)
<img src="notifications_email_test.webp" > ![Email test](notifications_email_test.webp)

32
docs/managed-victoriametrics/user-managment.md
View file

@ -9,8 +9,6 @@ menu:
aliases: aliases:
- /managed-victoriametrics/user-management.html - /managed-victoriametrics/user-management.html
--- ---
# User Management in Managed VictoriaMetrics
The user management system enables admins to control user access and onboard and offboard users to the Managed VictoriaMetrics. It organizes users according to their needs and role. The user management system enables admins to control user access and onboard and offboard users to the Managed VictoriaMetrics. It organizes users according to their needs and role.
The document covers the following topics The document covers the following topics
@ -29,7 +27,7 @@ You assign the role to the user during the user creation procedure. You can chan
### Roles definition ### Roles definition
<table> <table class="params">
<tr> <tr>
<td><strong>User Role</strong></td> <td><strong>User Role</strong></td>
<td><strong>Categories</strong></td> <td><strong>Categories</strong></td>
@ -85,7 +83,7 @@ You assign the role to the user during the user creation procedure. You can chan
### User statuses ### User statuses
<table> <table class="params">
<tr> <tr>
<td class="highlight"><strong class="sr">Active</strong></td> <td class="highlight"><strong class="sr">Active</strong></td>
<td>The user can log in and use Managed VictoriaMetrics. The user role defines the access level.</td> <td>The user can log in and use Managed VictoriaMetrics. The user role defines the access level.</td>
@ -105,12 +103,12 @@ You assign the role to the user during the user creation procedure. You can chan
It shows all users with different roles, and you can apply provided actions with them. It shows all users with different roles, and you can apply provided actions with them.
<img src="user_management_list.webp"> ![User Management list](user_management_list.webp)
In the table, there is additional information about the users: In the table, there is additional information about the users:
<table> <table class="params">
<tr> <tr>
<td>Email:</td> <td>Email:</td>
<td>Registration user email</td> <td>Registration user email</td>
@ -138,7 +136,7 @@ In the table, there is additional information about the users:
Click on `Invite user` button Click on `Invite user` button
the user invitation button and fill out the form in the modal, which will appear after you click. All fields are mandatory. the user invitation button and fill out the form in the modal, which will appear after you click. All fields are mandatory.
<img src="user_management_invite_user.webp" > ![Invite user](user_management_invite_user.webp)
After filling out the form, click on the `Invite` button. After filling out the form, click on the `Invite` button.
The user will be saved, and an invitation email to the provided email address will be sent. As a confirmation, you will see the success message. The user will be saved, and an invitation email to the provided email address will be sent. As a confirmation, you will see the success message.
@ -148,7 +146,7 @@ The user will be saved, and an invitation email to the provided email address wi
The user will be at Pending Invitation status. After accepting the invitation user status changes to Active. The user will be at Pending Invitation status. After accepting the invitation user status changes to Active.
<img src="user_management_invite_success.webp" > ![Invite success](user_management_invite_success.webp)
## How to Update User ## How to Update User
@ -156,17 +154,17 @@ The user will be at Pending Invitation status. After accepting the invitation us
To edit the user role and details, activate or deactivate a user, and click on To edit the user role and details, activate or deactivate a user, and click on
<img src="user_management_update_user.webp" > ![Update user](user_management_update_user.webp)
or `user email` to edit the user. or `user email` to edit the user.
User editing form: User editing form:
<img src="user_management_user_update_form.webp" > ![Update form](user_management_user_update_form.webp)
To save changes, click the `Save changes` button. If changes are saved successfully, you will see a message at the top of the page. To save changes, click the `Save changes` button. If changes are saved successfully, you will see a message at the top of the page.
<img src="user_management_user_updated_success.webp" > ![Updated success](user_management_user_updated_success.webp)
## How to Delete User ## How to Delete User
@ -174,15 +172,15 @@ You can delete a user from your account. However, you will need to invite them a
Click on `Delete` button to delete the user. Click on `Delete` button to delete the user.
<img src="user_management_user_delete.webp" > ![User delete](user_management_user_delete.webp)
To confirm the deletion of a user, you will need to re-enter their email address and press the **Delete** button To confirm the deletion of a user, you will need to re-enter their email address and press the **Delete** button
<img src="user_management_delete_user_form.webp" > ![User delete form](user_management_delete_user_form.webp)
You will be redirected to the main page with a success or error message You will be redirected to the main page with a success or error message
<img src="user_management_delete_success.webp" > ![Delete success](user_management_delete_success.webp)
## How to resend invitation ## How to resend invitation
@ -190,12 +188,12 @@ If the invitation is expired, it is possible to resend email to the user
Click `Resend invitation` button Click `Resend invitation` button
<img src="user_management_resend_invitation.webp" > ![Resend invitation](user_management_resend_invitation.webp)
Confirm resend invitation by clicking `Resend` button in the modal dialog Confirm resend invitation by clicking `Resend` button in the modal dialog
<img src="user_management_confirm_resend_invitation.webp" > ![Confirm resend invitation](user_management_confirm_resend_invitation.webp)
If invitation successfully resented to the user success message will appear If invitation successfully resented to the user success message will appear
<img src="user_management_resend_success.webp" > ![Resend success](user_management_resend_success.webp)

3
docs/relabeling.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /relabeling.html - /relabeling.html
--- ---
# Relabeling cookbook
VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) support VictoriaMetrics and [vmagent](https://docs.victoriametrics.com/vmagent/) support
[Prometheus-compatible relabeling](https://docs.victoriametrics.com/vmagent/#relabeling) [Prometheus-compatible relabeling](https://docs.victoriametrics.com/vmagent/#relabeling)
with [additional enhancements](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements). with [additional enhancements](https://docs.victoriametrics.com/vmagent/#relabeling-enhancements).

3
docs/scrape_config_examples.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /scrape_config_examples.html - /scrape_config_examples.html
--- ---
# Scrape config examples
- [Static configs](#static-configs) - [Static configs](#static-configs)
- [File-based target discovery](#file-based-target-discovery) - [File-based target discovery](#file-based-target-discovery)
- [HTTP-based target discovery](#http-based-target-discovery) - [HTTP-based target discovery](#http-based-target-discovery)

3
docs/sd_configs.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /sd_configs.html - /sd_configs.html
--- ---
# Prometheus service discovery
# Supported service discovery configs # Supported service discovery configs
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) [vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter)

19
docs/stream-aggregation.md
View file

@ -9,9 +9,6 @@ menu:
aliases: aliases:
- /stream-aggregation.html - /stream-aggregation.html
--- ---
# Streaming aggregation
[vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/) [vmagent](https://docs.victoriametrics.com/vmagent/) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage can aggregate incoming [samples](https://docs.victoriametrics.com/keyconcepts/#raw-samples) in streaming mode by time and by labels before data is written to remote storage
(or local storage for single-node VictoriaMetrics). (or local storage for single-node VictoriaMetrics).
@ -589,7 +586,7 @@ sum(sum_over_time(some_metric[interval])) / sum(count_over_time(some_metric[inte
For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query:
<img alt="avg aggregation" src="stream-aggregation-check-avg.webp"> ![avg aggregation](stream-aggregation-check-avg.webp)
See also: See also:
@ -669,7 +666,7 @@ then take a look at [increase_prometheus](#increase_prometheus).
For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query:
<img alt="increase aggregation" src="stream-aggregation-check-increase.webp"> ![increase aggregation](stream-aggregation-check-increase.webp)
Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/) Aggregating irregular and sporadic metrics (received from [Lambdas](https://aws.amazon.com/lambda/)
or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option. or [Cloud Functions](https://cloud.google.com/functions)) can be controlled via [staleness_interval](#staleness) option.
@ -735,7 +732,7 @@ max(max_over_time(some_metric[interval]))
For example, see below time series produced by config with aggregation interval `1m` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and the regular query:
<img alt="total aggregation" src="stream-aggregation-check-max.webp"> ![total aggregation](stream-aggregation-check-max.webp)
See also: See also:
@ -756,7 +753,7 @@ min(min_over_time(some_metric[interval]))
For example, see below time series produced by config with aggregation interval `1m` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and the regular query:
<img alt="min aggregation" src="stream-aggregation-check-min.webp"> ![min aggregation](stream-aggregation-check-min.webp)
See also: See also:
@ -831,7 +828,7 @@ histogram_stdvar(sum(histogram_over_time(some_metric[interval])) by (vmrange))
For example, see below time series produced by config with aggregation interval `1m` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and the regular query:
<img alt="stdvar aggregation" src="stream-aggregation-check-stdvar.webp"> ![stdvar aggregation](stream-aggregation-check-stdvar.webp)
See also: See also:
@ -852,7 +849,7 @@ sum(sum_over_time(some_metric[interval]))
For example, see below time series produced by config with aggregation interval `1m` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and the regular query:
<img alt="sum_samples aggregation" src="stream-aggregation-check-sum-samples.webp"> ![sum_samples aggregation](stream-aggregation-check-sum-samples.webp)
See also: See also:
@ -876,7 +873,7 @@ then take a look at [total_prometheus](#total_prometheus).
For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query: For example, see below time series produced by config with aggregation interval `1m` and `by: ["instance"]` and the regular query:
<img alt="total aggregation" src="stream-aggregation-check-total.webp"> ![total aggregation](stream-aggregation-check-total.webp)
`total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) - `total` is not affected by [counter resets](https://docs.victoriametrics.com/keyconcepts/#counter) -
it continues to increase monotonically with respect to the previous value. it continues to increase monotonically with respect to the previous value.
@ -884,7 +881,7 @@ The counters are most often reset when the application is restarted.
For example: For example:
<img alt="total aggregation counter reset" src="stream-aggregation-check-total-reset.webp"> ![total aggregation counter reset](stream-aggregation-check-total-reset.webp)
The same behavior occurs when creating or deleting new series in an aggregation group - The same behavior occurs when creating or deleting new series in an aggregation group -
`total` output increases monotonically considering the values of the series set. `total` output increases monotonically considering the values of the series set.

3
docs/url-examples.md
View file

@ -10,9 +10,6 @@ menu:
aliases: aliases:
- /url-examples.html - /url-examples.html
--- ---
# VictoriaMetrics API examples
### /api/v1/admin/tsdb/delete_series ### /api/v1/admin/tsdb/delete_series
**Deletes time series from VictoriaMetrics** **Deletes time series from VictoriaMetrics**

6
docs/victoriametrics-datasource.md
View file

@ -13,9 +13,6 @@ aliases:
- /grafana-datasource/ - /grafana-datasource/
- /grafana-datasource.html - /grafana-datasource.html
--- ---
# VictoriaMetrics datasource for Grafana
The [VictoriaMetrics datasource plugin](https://github.com/VictoriaMetrics/victoriametrics-datasource) The [VictoriaMetrics datasource plugin](https://github.com/VictoriaMetrics/victoriametrics-datasource)
allows to query and visualize data from VictoriaMetrics in Grafana. allows to query and visualize data from VictoriaMetrics in Grafana.
@ -140,7 +137,8 @@ docker-compose -f docker-compose.yaml up
When Grafana starts successfully datasources should be present on the datasources tab When Grafana starts successfully datasources should be present on the datasources tab
<img src="provision_datasources.webp" width="800" alt="Configuration"> ![Configuration](provision_datasources.webp)
{width="800"}
### Install in Kubernetes ### Install in Kubernetes

4
docs/vmagent.md
View file

@ -9,8 +9,6 @@ title: vmagent
aliases: aliases:
- /vmagent.html - /vmagent.html
--- ---
# vmagent
`vmagent` is a tiny agent which helps you collect metrics from various sources, `vmagent` is a tiny agent which helps you collect metrics from various sources,
[relabel and filter the collected metrics](#relabeling) [relabel and filter the collected metrics](#relabeling)
and store them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) and store them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics)
@ -19,7 +17,7 @@ or via [VictoriaMetrics `remote_write` protocol](#victoriametrics-remote-write-p
See [Quick Start](#quick-start) for details. See [Quick Start](#quick-start) for details.
<img alt="vmagent" src="vmagent.webp"> ![vmagent](vmagent.webp)
## Motivation ## Motivation

3
docs/vmalert-tool.md
View file

@ -9,9 +9,6 @@ title: vmalert-tool
aliases: aliases:
- /vmalert-tool.html - /vmalert-tool.html
--- ---
# vmalert-tool
VMAlert command-line tool VMAlert command-line tool
## Unit testing for rules ## Unit testing for rules

22
docs/vmalert.md
View file

@ -9,8 +9,6 @@ title: vmalert
aliases: aliases:
- /vmalert.html - /vmalert.html
--- ---
# vmalert
`vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) `vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
rules against configured `-datasource.url` compatible with Prometheus HTTP API. For sending alerting notifications rules against configured `-datasource.url` compatible with Prometheus HTTP API. For sending alerting notifications
@ -541,7 +539,8 @@ rules execution, storing recording rules results and alerts state.
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
``` ```
<img alt="vmalert single" width="500" src="vmalert_single.webp"> ![vmalert single](vmalert_single.webp)
{width="500"}
#### Cluster VictoriaMetrics #### Cluster VictoriaMetrics
@ -561,7 +560,7 @@ Cluster mode could have multiple `vminsert` and `vmselect` components.
-notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger -notifier.url=http://alertmanager:9093 # AlertManager addr to send alerts when they trigger
``` ```
<img alt="vmalert cluster" src="vmalert_cluster.webp"> ![vmalert cluster](vmalert_cluster.webp)
In case when you want to spread the load on these components - add balancers before them and configure In case when you want to spread the load on these components - add balancers before them and configure
`vmalert` with balancer addresses. Please, see more about VM's cluster architecture `vmalert` with balancer addresses. Please, see more about VM's cluster architecture
@ -585,7 +584,8 @@ Alertmanagers.
-notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers -notifier.url=http://alertmanagerN:9093 # The same alert will be sent to all configured notifiers
``` ```
<img alt="vmalert ha" width="800px" src="vmalert_ha.webp"> ![vmalert ha](vmalert_ha.webp)
{width="800px"}
To avoid recording rules results and alerts state duplication in VictoriaMetrics server To avoid recording rules results and alerts state duplication in VictoriaMetrics server
don't forget to configure [deduplication](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication). don't forget to configure [deduplication](https://docs.victoriametrics.com/single-server-victoriametrics/#deduplication).
@ -661,7 +661,7 @@ or reducing resolution) and push results to "cold" cluster.
-remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results -remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheus # vminsert addr to persist recording rules results
``` ```
<img alt="vmalert multi cluster" src="vmalert_multicluster.webp"> ![vmalert multi cluster](vmalert_multicluster.webp)
Please note, [replay](#rules-backfilling) feature may be used for transforming historical data. Please note, [replay](#rules-backfilling) feature may be used for transforming historical data.
@ -675,7 +675,7 @@ For persisting recording or alerting rule results `vmalert` requires `-remoteWri
But this flag supports only one destination. To persist rule results to multiple destinations But this flag supports only one destination. To persist rule results to multiple destinations
we recommend using [vmagent](https://docs.victoriametrics.com/vmagent/) as fan-out proxy: we recommend using [vmagent](https://docs.victoriametrics.com/vmagent/) as fan-out proxy:
<img alt="vmalert multiple remote write destinations" src="vmalert_multiple_rw.webp"> ![vmalert multiple remote write destinations](vmalert_multiple_rw.webp)
In this topology, `vmalert` is configured to persist rule results to `vmagent`. And `vmagent` In this topology, `vmalert` is configured to persist rule results to `vmagent`. And `vmagent`
is configured to fan-out received data to two or more destinations. is configured to fan-out received data to two or more destinations.
@ -848,12 +848,12 @@ Data delay is one of the most common issues with rules execution.
vmalert executes configured rules within certain intervals at specifics timestamps. vmalert executes configured rules within certain intervals at specifics timestamps.
It expects that the data is already present in configured `-datasource.url` at the moment of time when rule is executed: It expects that the data is already present in configured `-datasource.url` at the moment of time when rule is executed:
<img alt="vmalert expected evaluation" src="vmalert_ts_normal.gif"> ![vmalert expected evaluation](vmalert_ts_normal.gif)
Usually, troubles start to appear when data in `-datasource.url` is delayed or absent. In such cases, evaluations Usually, troubles start to appear when data in `-datasource.url` is delayed or absent. In such cases, evaluations
may get empty response from the datasource and produce empty recording rules or reset alerts state: may get empty response from the datasource and produce empty recording rules or reset alerts state:
<img alt="vmalert evaluation when data is delayed" src="vmalert_ts_data_delay.gif"> ![vmalert evaluation when data is delayed](vmalert_ts_data_delay.gif)
Try the following recommendations to reduce the chance of hitting the data delay issue: Try the following recommendations to reduce the chance of hitting the data delay issue:
* Always configure group's `-evaluationInterval` to be bigger or at least equal to * Always configure group's `-evaluationInterval` to be bigger or at least equal to
@ -893,7 +893,7 @@ state updates for each rule starting from [v1.86](https://docs.victoriametrics.c
To check updates, click on `Details` link next to rule's name on `/vmalert/groups` page To check updates, click on `Details` link next to rule's name on `/vmalert/groups` page
and check the `Last updates` section: and check the `Last updates` section:
<img alt="vmalert state" src="vmalert_state.webp"> ![vmalert state](vmalert_state.webp)
Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of
HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were
@ -905,7 +905,7 @@ for more details.
vmalert allows configuring more detailed logging for specific alerting rule starting from [v1.82](https://docs.victoriametrics.com/changelog/#v1820). vmalert allows configuring more detailed logging for specific alerting rule starting from [v1.82](https://docs.victoriametrics.com/changelog/#v1820).
Just set `debug: true` in rule's configuration and vmalert will start printing additional log messages: Just set `debug: true` in rule's configuration and vmalert will start printing additional log messages:
```terminal ```shell-session
2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms) 2022-09-15T13:35:41.155Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:41+02:00: query returned 0 samples (elapsed: 5.896041ms)
2022-09-15T13:35:56.149Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29+by%28instance%29+%3E+0&step=15s&time=1663248945" 2022-09-15T13:35:56.149Z DEBUG datasource request: executing POST request with params "denyPartialResponse=true&query=sum%28vm_tcplistener_conns%7Binstance%3D%22localhost%3A8429%22%7D%29+by%28instance%29+%3E+0&step=15s&time=1663248945"
2022-09-15T13:35:56.178Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: query returned 1 samples (elapsed: 28.368208ms) 2022-09-15T13:35:56.178Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:35:56+02:00: query returned 1 samples (elapsed: 28.368208ms)

2
docs/vmauth.md
View file

@ -9,8 +9,6 @@ title: vmauth
aliases: aliases:
- /vmauth.html - /vmauth.html
--- ---
# vmauth
`vmauth` is an HTTP proxy, which can [authorize](https://docs.victoriametrics.com/vmauth/#authorization), [route](https://docs.victoriametrics.com/vmauth/#routing) `vmauth` is an HTTP proxy, which can [authorize](https://docs.victoriametrics.com/vmauth/#authorization), [route](https://docs.victoriametrics.com/vmauth/#routing)
and [load balance](https://docs.victoriametrics.com/vmauth/#load-balancing) requests across [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) components and [load balance](https://docs.victoriametrics.com/vmauth/#load-balancing) requests across [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) components
or any other HTTP backends. or any other HTTP backends.

2
docs/vmbackup.md
View file

@ -9,8 +9,6 @@ title: vmbackup
aliases: aliases:
- /vmbackup.html - /vmbackup.html
--- ---
# vmbackup
`vmbackup` creates VictoriaMetrics data backups from [instant snapshots](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots). `vmbackup` creates VictoriaMetrics data backups from [instant snapshots](https://docs.victoriametrics.com/single-server-victoriametrics/#how-to-work-with-snapshots).
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup. `vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.

8
docs/vmbackupmanager.md
View file

@ -117,11 +117,11 @@ The result on the GCS bucket
* The root folder * The root folder
<img alt="root folder" src="vmbackupmanager_root_folder.webp"> ![root folder](vmbackupmanager_root_folder.webp)
* The latest folder * The latest folder
<img alt="latest folder" src="vmbackupmanager_latest_folder.webp"> ![latest folder](vmbackupmanager_latest_folder.webp)
`vmbackupmanager` uses [smart backups](https://docs.victoriametrics.com/vmbackup/#smart-backups) technique in order `vmbackupmanager` uses [smart backups](https://docs.victoriametrics.com/vmbackup/#smart-backups) technique in order
to accelerate backups and save both data transfer costs and data copying costs. This includes server-side copy of already existing to accelerate backups and save both data transfer costs and data copying costs. This includes server-side copy of already existing
@ -147,7 +147,7 @@ Backup retention policy is controlled by:
Let’s assume we have a backup manager collecting daily backups for the past 10 days. Let’s assume we have a backup manager collecting daily backups for the past 10 days.
<img alt="retention policy daily before retention cycle" src="vmbackupmanager_rp_daily_1.webp"> ![retention policy daily before retention cycle](vmbackupmanager_rp_daily_1.webp)
We enable backup retention policy for backup manager by using following configuration: We enable backup retention policy for backup manager by using following configuration:
@ -172,7 +172,7 @@ info app/vmbackupmanager/retention.go:106 daily backups to delete [daily/2
The result on the GCS bucket. We see only 3 daily backups: The result on the GCS bucket. We see only 3 daily backups:
<img alt="retention policy daily after retention cycle" src="vmbackupmanager_rp_daily_2.webp"> [retention policy daily after retention cycle](vmbackupmanager_rp_daily_2.webp "retention policy daily after retention cycle")
### Protection backups against deletion by retention policy ### Protection backups against deletion by retention policy

2
docs/vmctl.md
View file

@ -9,8 +9,6 @@ title: vmctl
aliases: aliases:
- /vmctl.html - /vmctl.html
--- ---
# vmctl
VictoriaMetrics command-line tool (vmctl) provides the following list of actions: VictoriaMetrics command-line tool (vmctl) provides the following list of actions:
- migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API - migrate data from [Prometheus](#migrating-data-from-prometheus) to VictoriaMetrics using snapshot API
- migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics - migrate data from [Thanos](#migrating-data-from-thanos) to VictoriaMetrics

8
docs/vmgateway.md
View file

@ -9,13 +9,11 @@ title: vmgateway
aliases: aliases:
- /vmgateway.html - /vmgateway.html
--- ---
# vmgateway
***vmgateway is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/). ***vmgateway is a part of [enterprise package](https://docs.victoriametrics.com/enterprise/).
It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). It is available for download and evaluation at [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/).*** See how to request a free trial license [here](https://victoriametrics.com/products/enterprise/trial/).***
<img alt="vmgateway" src="vmgateway-overview.webp"> ![vmgateway](vmgateway-overview.webp)
`vmgateway` is a proxy for the VictoriaMetrics Time Series Database (TSDB). It provides the following features: `vmgateway` is a proxy for the VictoriaMetrics Time Series Database (TSDB). It provides the following features:
@ -30,7 +28,7 @@ See how to request a free trial license [here](https://victoriametrics.com/produ
## Access Control ## Access Control
<img alt="vmgateway-ac" src="vmgateway-access-control.webp"> ![vmgateway-ac](vmgateway-access-control.webp)
`vmgateway` supports jwt based authentication. With jwt payload can be configured to give access to specific tenants and labels as well as to read/write. `vmgateway` supports jwt based authentication. With jwt payload can be configured to give access to specific tenants and labels as well as to read/write.
@ -95,7 +93,7 @@ curl 'http://localhost:8431/api/v1/series/count' -H 'Authorization: Bearer incor
## Rate Limiter ## Rate Limiter
<img alt="vmgateway-rl" src="vmgateway-rate-limiting.webp"> ![vmgateway-rl](vmgateway-rate-limiting.webp)
Limits incoming requests by given, pre-configured limits. It supports read and write limiting by tenant. Limits incoming requests by given, pre-configured limits. It supports read and write limiting by tenant.

2
docs/vmrestore.md
View file

@ -9,8 +9,6 @@ title: vmrestore
aliases: aliases:
- /vmrestore.html - /vmrestore.html
--- ---
# vmrestore
`vmrestore` restores data from backups created by [vmbackup](https://docs.victoriametrics.com/vmbackup/). `vmrestore` restores data from backups created by [vmbackup](https://docs.victoriametrics.com/vmbackup/).
Restore process can be interrupted at any time. It is automatically resumed from the interruption point Restore process can be interrupted at any time. It is automatically resumed from the interruption point