github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2025-01-10 15:14:09 +00:00
Code Issues Projects Releases Packages Wiki Activity

Automatic update operator docs from VictoriaMetrics/operator@c7125bd (#5102)

Browse source
This commit is contained in:
Github Actions 2023-10-02 20:50:08 +08:00 committed by GitHub
parent 7e744f86cb
commit 62de314510
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 5059 additions and 1913 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

132
docs/operator/CHANGELOG.md
View file

@ -1,16 +1,22 @@
---
sort: 10
weight: 10
title: CHANGELOG
---
# CHANGELOG
## Next release
### Features
- [vmoperator](https://docs.victoriametrics.com/operator/): upgrade vmagent/vmauth's default config-reloader image.
- [vmoperator](./README.md): upgrade vmagent/vmauth's default config-reloader image.
### Fixes
- [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmcluster): remove redundant annotation `operator.victoriametrics/last-applied-spec` from created workloads like vmstorage statefulset.
- [vmoperator](https://docs.victoriametrics.com/operator/): properly resize statefulset's multiple pvc when needed and allowable, before they could be updated with wrong size.
- [vmoperator](https://docs.victoriametrics.com/operator/): fix wrong api group of endpointsices, before vmagent won't able to access endpointsices resources with default rbac rule.
- [vmcluster](./api.html#vmcluster): remove redundant annotation `operator.victoriametrics/last-applied-spec` from created workloads like vmstorage statefulset.
- [vmoperator](./README.md): properly resize statefulset's multiple pvc when needed and allowable, before they could be updated with wrong size.
- [vmoperator](./README.md): fix wrong api group of endpointsices, before vmagent won't able to access endpointsices resources with default rbac rule.
<a name="v0.38.0"></a>
## [v0.38.0](https://github.com/VictoriaMetrics/operator/releases/tag/v0.38.0) - 11 Sep 2023
@ -19,13 +25,13 @@
### Fixes
- [vmuser](https://docs.victoriametrics.com/operator/api.html#vmuser): [Enterprise] fixes ip_filters indent for url_prefix. Previously it wasn't possible to use ip_filters with multiple target refs
- [vmoperator](https://docs.victoriametrics.com/operator/): turn off `EnableStrictSecurity` by default. Before, upgrade operator to v0.36.0+ could fail components with volume attached, see [this issue](https://github.com/VictoriaMetrics/operator/issues/749) for details.
- [vmoperator](https://docs.victoriametrics.com/operator/): bump default version of VictoriaMetrics components to [1.93.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.4).
- [vmuser](./api.md#vmuser): [Enterprise] fixes ip_filters indent for url_prefix. Previously it wasn't possible to use ip_filters with multiple target refs
- [vmoperator](./README.md): turn off `EnableStrictSecurity` by default. Before, upgrade operator to v0.36.0+ could fail components with volume attached, see [this issue](https://github.com/VictoriaMetrics/operator/issues/749) for details.
- [vmoperator](./README.md): bump default version of VictoriaMetrics components to [1.93.4](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.4).
### Features
- [vmoperator](https://docs.victoriametrics.com/operator/) add ability to print default values for all [operator variables](https://docs.victoriametrics.com/operator/vars.html). See [this issue](https://github.com/VictoriaMetrics/operator/issues/675) for details.
- [vmoperator](./README.md) add ability to print default values for all [operator variables](./vars.md). See [this issue](https://github.com/VictoriaMetrics/operator/issues/675) for details.
<a name="v0.37.1"></a>
## [v0.37.1](https://github.com/VictoriaMetrics/operator/releases/tag/v0.37.1) - 02 Sep 2023
@ -41,18 +47,18 @@
### Fixes
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): fix unmarshalling for streaming aggregation `match` field.
- [vmagent](./api.md#vmagent): fix unmarshalling for streaming aggregation `match` field.
### Features
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): support [multiple if conditions](https://docs.victoriametrics.com/vmagent.html#relabeling:~:text=the%20if%20option%20may%20contain%20more%20than%20one%20filter) for relabeling. See [this issue](https://github.com/VictoriaMetrics/operator/issues/730) for details.
- [vmagent](./api.md#vmagent): support [multiple if conditions](https://docs.victoriametrics.com/vmagent.html#relabeling:~:text=the%20if%20option%20may%20contain%20more%20than%20one%20filter) for relabeling. See [this issue](https://github.com/VictoriaMetrics/operator/issues/730) for details.
<a name="v0.36.1"></a>
## [v0.36.1](https://github.com/VictoriaMetrics/operator/releases/tag/v0.36.0) - 25 Aug 2023
### Fixes
- [vmselect](https://docs.victoriametrics.com/operator/api.html#vmcluster): fix cache directory when `cacheDataPath` not specified, before it will use `/tmp` which is protect by default strict securityContext.
- [vmselect](./api.md#vmcluster): fix cache directory when `cacheDataPath` not specified, before it will use `/tmp` which is protect by default strict securityContext.
### Features
@ -61,7 +67,7 @@
### Breaking changes
- **[vmalert](https://docs.victoriametrics.com/operator/api.html#vmalert): Field `OAuth2` was renamed to `oauth2` due to compatibility issue. If you defined `OAuth2` with below fields in vmalert objects using operator before v0.36.0, these fields must be reapplied with new tag `oauth2` after upgrading. See [this issue](https://github.com/VictoriaMetrics/operator/issues/522) and [this PR](https://github.com/VictoriaMetrics/operator/pull/689) for details.**
- **[vmalert](./api.md#vmalert): Field `OAuth2` was renamed to `oauth2` due to compatibility issue. If you defined `OAuth2` with below fields in vmalert objects using operator before v0.36.0, these fields must be reapplied with new tag `oauth2` after upgrading. See [this issue](https://github.com/VictoriaMetrics/operator/issues/522) and [this PR](https://github.com/VictoriaMetrics/operator/pull/689) for details.**
- **Affected fields:**
- **`VMAlert.spec.datasource.OAuth2` -> `VMAlert.spec.datasource.oauth2`,**
- **`VMAlert.spec.notifier.OAuth2` -> `VMAlert.spec.notifier.oauth2`,**
@ -69,7 +75,7 @@
- **`VMAlert.spec.remoteRead.OAuth2` -> `VMAlert.spec.remoteRead.oauth2`,**
- **`VMAlert.spec.remoteWrite.OAuth2` -> `VMAlert.spec.remoteWrite.oauth2`,**
- **[vmalert](https://docs.victoriametrics.com/operator/api.html#vmalert): Field `bearerTokenFilePath` was renamed to `bearerTokenFile` due to compatibility issue. If you defined `bearerTokenFilePath` with below fields in vmalert objects using operator before v0.36.0, these fields must be reapplied with new tag `bearerTokenFile` after upgrading. See [this issue](https://github.com/VictoriaMetrics/operator/issues/522) and [this PR](https://github.com/VictoriaMetrics/operator/pull/688/) for details.**
- **[vmalert](./api.md#vmalert): Field `bearerTokenFilePath` was renamed to `bearerTokenFile` due to compatibility issue. If you defined `bearerTokenFilePath` with below fields in vmalert objects using operator before v0.36.0, these fields must be reapplied with new tag `bearerTokenFile` after upgrading. See [this issue](https://github.com/VictoriaMetrics/operator/issues/522) and [this PR](https://github.com/VictoriaMetrics/operator/pull/688/) for details.**
- **Affected fields:**
- **`VMAlert.spec.datasource.bearerTokenFilePath` --> `VMAlert.spec.datasource.bearerTokenFile`,**
- **`VMAlert.spec.notifier.bearerTokenFilePath` --> `VMAlert.spec.notifier.bearerTokenFile`,**
@ -82,22 +88,22 @@
- operator set resource requests for config-reloader container by default. See [this PR](https://github.com/VictoriaMetrics/operator/pull/695/) for details.
- fix `attachMetadata` value miscovert for scrape objects. See [this issue](https://github.com/VictoriaMetrics/operator/issues/697) and [this PR](https://github.com/VictoriaMetrics/operator/pull/698) for details.
- fix volumeClaimTemplates change check for objects that generate statefulset, like vmstorage, vmselect. Before, the statefulset won't be recreated if additional `claimTemplates` object changed. See [this issue](https://github.com/VictoriaMetrics/operator/issues/507) and [this PR](https://github.com/VictoriaMetrics/operator/pull/719) for details.
- [vmalert](https://docs.victoriametrics.com/operator/api.html#vmalert): fix `tlsCAFile` argument value generation when using secret or configMap. See [this issue](https://github.com/VictoriaMetrics/operator/issues/699) and [this PR](https://github.com/VictoriaMetrics/operator/issues/699) for details.
- [vmalertmanager](https://docs.victoriametrics.com/operator/api.html#vmalertmanager): fix default request memory and apply default resources if not set. See [this issue](https://github.com/VictoriaMetrics/operator/issues/706) and [this PR](https://github.com/VictoriaMetrics/operator/pull/710) for details.
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): fix missing additional VolumeClaimTemplates when using `ClaimTemplates` under StatefulMode.
- [vmalert](./api.md#vmalert): fix `tlsCAFile` argument value generation when using secret or configMap. See [this issue](https://github.com/VictoriaMetrics/operator/issues/699) and [this PR](https://github.com/VictoriaMetrics/operator/issues/699) for details.
- [vmalertmanager](./api.md#vmalertmanager): fix default request memory and apply default resources if not set. See [this issue](https://github.com/VictoriaMetrics/operator/issues/706) and [this PR](https://github.com/VictoriaMetrics/operator/pull/710) for details.
- [vmagent](./api.md#vmagent): fix missing additional VolumeClaimTemplates when using `ClaimTemplates` under StatefulMode.
### Features
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): add [example config](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmagent_stateful_with_sharding.yaml) for vmagent statefulmode.
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent)/[vmsingle](https://docs.victoriametrics.com/operator/api.html#vmsingle): adapt new features in streaming aggregation:
- [vmagent](./api.md#vmagent): add [example config](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmagent_stateful_with_sharding.yaml) for vmagent statefulmode.
- [vmagent](./api.md#vmagent)/[vmsingle](./api.md#vmsingle): adapt new features in streaming aggregation:
- support `streamAggr.dropInput`, see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243) for details;
- support list for `match` parameter, see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635) for details;
- support `staleness_interval`, see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667) for details.
- [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmagent): add [example config](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmcluster_with_additional_claim.yaml) for cluster with custom storage claims.
- [vmrule](https://docs.victoriametrics.com/operator/api.html#vmrule): support `update_entries_limit` field in rules, refer to [alerting rules](https://docs.victoriametrics.com/vmalert.html#alerting-rules). See [this PR](https://github.com/VictoriaMetrics/operator/pull/691) for details.
- [vmrule](https://docs.victoriametrics.com/operator/api.html#vmrule): support `keep_firing_for` field in rules, refer to [alerting rules](https://docs.victoriametrics.com/vmalert.html#alerting-rules). See [this PR](https://github.com/VictoriaMetrics/operator/pull/711) for details.
- [vmoperator parameters](https://docs.victoriametrics.com/operator/vars.html): Add option `VM_ENABLESTRICTSECURITY` and enable strict security context by default. See [this issue](https://github.com/VictoriaMetrics/operator/issues/637), [this](https://github.com/VictoriaMetrics/operator/pull/692/) and [this](https://github.com/VictoriaMetrics/operator/pull/712) PR for details.
- [vmoperator parameters](https://docs.victoriametrics.com/operator/vars.html): change option `VM_PSPAUTOCREATEENABLED` default value from `true` to `false` cause PodSecurityPolicy already got deprecated since [kubernetes v1.25](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#psp-v125). See [this pr](https://github.com/VictoriaMetrics/operator/pull/726) for details.
- [vmcluster](./api.md#vmagent): add [example config](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmcluster_with_additional_claim.yaml) for cluster with custom storage claims.
- [vmrule](./api.md#vmrule): support `update_entries_limit` field in rules, refer to [alerting rules](https://docs.victoriametrics.com/vmalert.html#alerting-rules). See [this PR](https://github.com/VictoriaMetrics/operator/pull/691) for details.
- [vmrule](./api.md#vmrule): support `keep_firing_for` field in rules, refer to [alerting rules](https://docs.victoriametrics.com/vmalert.html#alerting-rules). See [this PR](https://github.com/VictoriaMetrics/operator/pull/711) for details.
- [vmoperator parameters](./vars.md): Add option `VM_ENABLESTRICTSECURITY` and enable strict security context by default. See [this issue](https://github.com/VictoriaMetrics/operator/issues/637), [this](https://github.com/VictoriaMetrics/operator/pull/692/) and [this](https://github.com/VictoriaMetrics/operator/pull/712) PR for details.
- [vmoperator parameters](./vars.md): change option `VM_PSPAUTOCREATEENABLED` default value from `true` to `false` cause PodSecurityPolicy already got deprecated since [kubernetes v1.25](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#psp-v125). See [this pr](https://github.com/VictoriaMetrics/operator/pull/726) for details.
[Changes][v0.36.0]
@ -106,7 +112,7 @@
### Fixes
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): fixes regression with remoteWrite authorization (basicAuth/token). When `UseCustomConfigReloader` option was set, operator incorrectly rendered mounts for `vmagent` container. https://github.com/VictoriaMetrics/operator/commit/f2b8cf701a33f91cef19848c857fd6efb7db59dd
- [vmagent](./api.md#vmagent): fixes regression with remoteWrite authorization (basicAuth/token). When `UseCustomConfigReloader` option was set, operator incorrectly rendered mounts for `vmagent` container. https://github.com/VictoriaMetrics/operator/commit/f2b8cf701a33f91cef19848c857fd6efb7db59dd
[Changes][v0.35.1]
@ -116,19 +122,19 @@
### Fixes
* [vmuser](https://docs.victoriametrics.com/operator/api.html#vmuser): fix vmselect url_map in vmuser. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/655). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmalert](https://docs.victoriametrics.com/operator/api.html#vmalert): correctly set default port for vmauth components discovery. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/658). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmuser](https://docs.victoriametrics.com/operator/api.html#vmuser): remove rate limit on delete. In https://github.com/VictoriaMetrics/operator/pull/672. Thanks [@Haleygo](https://github.com/Haleygo)
* [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmcluster): fix spec change check. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/677). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmuser](./api.md#vmuser): fix vmselect url_map in vmuser. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/655). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmalert](./api.md#vmalert): correctly set default port for vmauth components discovery. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/658). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmuser](./api.md#vmuser): remove rate limit on delete. In https://github.com/VictoriaMetrics/operator/pull/672. Thanks [@Haleygo](https://github.com/Haleygo)
* [vmcluster](./api.md#vmcluster): fix spec change check. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/677). Thanks [@Haleygo](https://github.com/Haleygo)
* Correctly publish multi-arch release at https://github.com/VictoriaMetrics/operator/pull/681. Thanks [@Haleygo](https://github.com/Haleygo)
### Features
* [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): add validation when generate static scrape config. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/677). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): add validation for slack receiver url. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/661). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmauth](https://docs.victoriametrics.com/operator/api.html#vmauth)/[vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): implement configuration initiation for custom config reloader. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/619). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmagent](./api.md#vmagent): add validation when generate static scrape config. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/677). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): add validation for slack receiver url. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/661). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmauth](./api.md#vmauth)/[vmagent](./api.md#vmagent): implement configuration initiation for custom config reloader. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/619). Thanks [@Haleygo](https://github.com/Haleygo)
* add more generators Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/668
* [vmsingle](https://docs.victoriametrics.com/operator/api.html#vmsingle): add status field. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/670). Thanks [@Haleygo](https://github.com/Haleygo)
* [vmsingle](./api.md#vmsingle): add status field. See [this issue for details](https://github.com/VictoriaMetrics/operator/issues/670). Thanks [@Haleygo](https://github.com/Haleygo)
[Changes][v0.35.0]
@ -138,9 +144,9 @@
### Fixes
- [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmcluster): fail fast on misconfigured or missing kubernetes pods. It should prevent rare bug with cascade pod deletion. See this [issue](https://github.com/VictoriaMetrics/operator/issues/643) for details
- [vmauth](https://docs.victoriametrics.com/operator/api.html#vmauth)/[vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): correctly renders initConfig image with global container registry domain. See this [issue](https://github.com/VictoriaMetrics/operator/issues/654) for details.
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): correctly set RBAC permissions for single namespace mode and custom config reloader image. See this [issue](https://github.com/VictoriaMetrics/operator/issues/653) for details.
- [vmcluster](./api.md#vmcluster): fail fast on misconfigured or missing kubernetes pods. It should prevent rare bug with cascade pod deletion. See this [issue](https://github.com/VictoriaMetrics/operator/issues/643) for details
- [vmauth](./api.md#vmauth)/[vmagent](./api.md#vmagent): correctly renders initConfig image with global container registry domain. See this [issue](https://github.com/VictoriaMetrics/operator/issues/654) for details.
- [vmagent](./api.md#vmagent): correctly set RBAC permissions for single namespace mode and custom config reloader image. See this [issue](https://github.com/VictoriaMetrics/operator/issues/653) for details.
[Changes][v0.34.1]
@ -154,18 +160,18 @@
### Fixes
- [vmnodescrape](https://docs.victoriametrics.com/operator/api.html#vmnodescrape): fixed selectors for Exists and NotExists operators with empty label Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/646
- [vmrule](https://docs.victoriametrics.com/operator/api.html#vmrule): Add config for vmrule in validating webhook Thanks in https://github.com/VictoriaMetrics/operator/pull/650
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): skips misconfigured objects with missed secret references: https://github.com/VictoriaMetrics/operator/issues/648
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): correctly renders initContainer for configuration download: https://github.com/VictoriaMetrics/operator/issues/649
- [vmnodescrape](./api.md#vmnodescrape): fixed selectors for Exists and NotExists operators with empty label Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/646
- [vmrule](./api.md#vmrule): Add config for vmrule in validating webhook Thanks in https://github.com/VictoriaMetrics/operator/pull/650
- [vmagent](./api.md#vmagent): skips misconfigured objects with missed secret references: https://github.com/VictoriaMetrics/operator/issues/648
- [vmagent](./api.md#vmagent): correctly renders initContainer for configuration download: https://github.com/VictoriaMetrics/operator/issues/649
### Features
- [vmalertmanager](https://docs.victoriametrics.com/operator/api.html#vmalertmanager): Bump alertmanager to v0.25.0 Thanks [@tamcore](https://github.com/tamcore) in https://github.com/VictoriaMetrics/operator/pull/636
- [vmcluster](https://docs.victoriametrics.com/operator/api.html#vmcluster): added `clusterNativePort` field to VMSelect/VMInsert for multi-level cluster setup ([#634](https://github.com/VictoriaMetrics/operator/issues/634)) Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/639
- [vmrule](https://docs.victoriametrics.com/operator/api.html#vmrule): add notifierHeader field in vmrule spec Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/622
- [vmpodscrape](https://docs.victoriametrics.com/operator/api.html#vmpodscrape): adds FilterRunning option as prometheus does in https://github.com/VictoriaMetrics/operator/pull/640
- [vmauth](https://docs.victoriametrics.com/operator/api.html#vmauth): adds latest features in https://github.com/VictoriaMetrics/operator/pull/642
- [vmalertmanager](./api.md#vmalertmanager): Bump alertmanager to v0.25.0 Thanks [@tamcore](https://github.com/tamcore) in https://github.com/VictoriaMetrics/operator/pull/636
- [vmcluster](./api.md#vmcluster): added `clusterNativePort` field to VMSelect/VMInsert for multi-level cluster setup ([#634](https://github.com/VictoriaMetrics/operator/issues/634)) Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/639
- [vmrule](./api.md#vmrule): add notifierHeader field in vmrule spec Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/622
- [vmpodscrape](./api.md#vmpodscrape): adds FilterRunning option as prometheus does in https://github.com/VictoriaMetrics/operator/pull/640
- [vmauth](./api.md#vmauth): adds latest features in https://github.com/VictoriaMetrics/operator/pull/642
[Changes][v0.34.0]
@ -175,22 +181,22 @@
### Fixes
- [vmalert](https://docs.victoriametrics.com/operator/api.html#vmalert): skip bad rules and improve logging for rules exceed max configmap size https://github.com/VictoriaMetrics/operator/commit/bb754d5c20bb371a197cd6ff5afac1ba86a4d92b
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): fixed error with headers in VMAlertmanagerConfig.Receivers.EmailConfigs.Headers unmarshalling. Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/610
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): fixed keepInput setting for streaming aggregation. Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/618
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): fix webhook config maxAlerts not work. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/625
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): Remove single quotes from remote write headers. Thanks [@axelsccp](https://github.com/axelsccp) in https://github.com/VictoriaMetrics/operator/pull/613
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): fix parse route error and some comments. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/630
- [vmuser](https://docs.victoriametrics.com/operator/api.html#vmuser): properly removes finalizers for objects https://github.com/VictoriaMetrics/operator/commit/8f10113920a353f21fbcc8637076905f2e57bb34
- [vmalert](./api.md#vmalert): skip bad rules and improve logging for rules exceed max configmap size https://github.com/VictoriaMetrics/operator/commit/bb754d5c20bb371a197cd6ff5afac1ba86a4d92b
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): fixed error with headers in VMAlertmanagerConfig.Receivers.EmailConfigs.Headers unmarshalling. Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/610
- [vmagent](./api.md#vmagent): fixed keepInput setting for streaming aggregation. Thanks [@Amper](https://github.com/Amper) in https://github.com/VictoriaMetrics/operator/pull/618
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): fix webhook config maxAlerts not work. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/625
- [vmagent](./api.md#vmagent): Remove single quotes from remote write headers. Thanks [@axelsccp](https://github.com/axelsccp) in https://github.com/VictoriaMetrics/operator/pull/613
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): fix parse route error and some comments. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/630
- [vmuser](./api.md#vmuser): properly removes finalizers for objects https://github.com/VictoriaMetrics/operator/commit/8f10113920a353f21fbcc8637076905f2e57bb34
### Features
- [vmalertmanager](https://docs.victoriametrics.com/operator/api.html#vmalertmanager): add option to disable route continue enforce. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/621
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): support set require_tls to false. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/624
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): add sanity check. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/627
- [vmalertmanager](./api.md#vmalertmanager): add option to disable route continue enforce. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/621
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): support set require_tls to false. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/624
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): add sanity check. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/627
- Makefile: bump Alpine base image to latest v3.17.3. Thanks [@denisgolius](https://github.com/denisgolius) in https://github.com/VictoriaMetrics/operator/pull/628
- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/api.html#vmalertmanagerconfig): support sound field in pushover config. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/631
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent)/[vmauth](https://docs.victoriametrics.com/operator/api.html#vmauth): download initial config with initContainer https://github.com/VictoriaMetrics/operator/commit/612e7c8f40659731e7938ef9556eb088c67eb4b7
- [vmalertmanagerconfig](./api.md#vmalertmanagerconfig): support sound field in pushover config. Thanks [@Haleygo](https://github.com/Haleygo) in https://github.com/VictoriaMetrics/operator/pull/631
- [vmagent](./api.md#vmagent)/[vmauth](./api.md#vmauth): download initial config with initContainer https://github.com/VictoriaMetrics/operator/commit/612e7c8f40659731e7938ef9556eb088c67eb4b7
[Changes][v0.33.0]
@ -201,7 +207,7 @@
### Fixes
- config: fixes typo at default vm apps version https://github.com/VictoriaMetrics/operator/issues/608
- [vmsingle](https://docs.victoriametrics.com/operator/api.html#vmsingle): conditionally adds stream aggregation config https://github.com/VictoriaMetrics/operator/commit/4a0ca54113afcde439ca4c77e22d3ef1c0d36241
- [vmsingle](./api.md#vmsingle): conditionally adds stream aggregation config https://github.com/VictoriaMetrics/operator/commit/4a0ca54113afcde439ca4c77e22d3ef1c0d36241
[Changes][v0.32.1]
@ -215,10 +221,10 @@
### Features
- [vmauth](https://docs.victoriametrics.com/operator/api.html#vmauth): automatically configures `proxy-protocol` client and `reloadAuthKey` for `config-reloader` container. https://github.com/VictoriaMetrics/operator/commit/611819233bf595a4dbd04b07d7be24b7e994379c
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): adds `scrapeTimeout` global configuration for `VMAgent` https://github.com/VictoriaMetrics/operator/commit/d1d5024c6befa0961f8d56c82a0554935a4b1878
- [vmagent](https://docs.victoriametrics.com/operator/api.html#vmagent): adds [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html) for `remoteWrite` targets https://github.com/VictoriaMetrics/operator/commit/b8baa6c2b72bdda64ebfcc9c3d86d846cd9b3c98 Thanks [@Amper](https://github.com/Amper)
- [vmsingle](https://docs.victoriametrics.com/operator/api.html#vmsingle): adds [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html) as global configuration for database https://github.com/VictoriaMetrics/operator/commit/b8baa6c2b72bdda64ebfcc9c3d86d846cd9b3c98 Thanks [@Amper](https://github.com/Amper)
- [vmauth](./api.md#vmauth): automatically configures `proxy-protocol` client and `reloadAuthKey` for `config-reloader` container. https://github.com/VictoriaMetrics/operator/commit/611819233bf595a4dbd04b07d7be24b7e994379c
- [vmagent](./api.md#vmagent): adds `scrapeTimeout` global configuration for `VMAgent` https://github.com/VictoriaMetrics/operator/commit/d1d5024c6befa0961f8d56c82a0554935a4b1878
- [vmagent](./api.md#vmagent): adds [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html) for `remoteWrite` targets https://github.com/VictoriaMetrics/operator/commit/b8baa6c2b72bdda64ebfcc9c3d86d846cd9b3c98 Thanks [@Amper](https://github.com/Amper)
- [vmsingle](./api.md#vmsingle): adds [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html) as global configuration for database https://github.com/VictoriaMetrics/operator/commit/b8baa6c2b72bdda64ebfcc9c3d86d846cd9b3c98 Thanks [@Amper](https://github.com/Amper)
[Changes][v0.32.0]
@ -233,8 +239,8 @@
### Features
- [vmalertmanager](https://docs.victoriametrics.com/operator/api.html#vmalertmanager): Add support of vmalertmanager.spec.templates and autoreload dirs for templates and configmaps thanks [@Amper](https://github.com/Amper) https://github.com/VictoriaMetrics/operator/issues/590 https://github.com/VictoriaMetrics/operator/issues/592
- [vmalertmanager](https://docs.victoriametrics.com/operator/api.html#vmalertmanager): Add support "%SHARD_NUM%" placeholder for vmagent sts/deployment Thanks [@Amper](https://github.com/Amper) https://github.com/VictoriaMetrics/operator/issues/508
- [vmalertmanager](./api.md#vmalertmanager): Add support of vmalertmanager.spec.templates and autoreload dirs for templates and configmaps thanks [@Amper](https://github.com/Amper) https://github.com/VictoriaMetrics/operator/issues/590 https://github.com/VictoriaMetrics/operator/issues/592
- [vmalertmanager](./api.md#vmalertmanager): Add support "%SHARD_NUM%" placeholder for vmagent sts/deployment Thanks [@Amper](https://github.com/Amper) https://github.com/VictoriaMetrics/operator/issues/508
[Changes][v0.31.0]

67
docs/operator/FAQ.md
View file

@ -1,17 +1,14 @@
---
sort: 15
weight: 15
sort: 9
weight: 9
title: FAQ
menu:
docs:
parent: "operator"
weight: 15
identifier: "faq-operator"
aliases:
- /operator/FAQ.html
---
# FAQ
# FAQ (Frequency Asked Questions)
## How do you monitor the operator itself?
You can read about vmoperator monitoring in [this document](./monitoring.md).
## How to change VMStorage PVC storage class
@ -29,3 +26,53 @@ With Operator deployment:
1. Run `kubectl delete statefulset --cascade=orphan {vmstorage-sts}`
1. Update VMCluster spec to use new storage class
1. Apply cluster configuration
## How to override image registry
You can use `VM_CONTAINERREGISTRY` parameter for operator:
- See details about tuning [operator settings here](./setup.md#settings).
- See [available operator settings](./vars.md) here.
## How to set up automatic backups?
You can read about backups:
- for `VMSingle`: [Backup automation](./resources/vmsingle.md#backup-automation)
- for `VMCluster`: [Backup automation](./resources/vmcluster.md#backup-automation)
## How to migrate from Prometheus-operator to VictoriaMetrics operator?
You can read about migration from prometheus operator on [this page](./migration.md).
## How to turn off conversion for prometheus resources
You can read about it on [this page](./migration.md#objects-convesion).
## My VM objects are not deleted/changed when I delete/change Prometheus objects
You can read about it in following sections of "Migration from prometheus-operator" docs:
- [Deletion synchronization](./migration.md#deletion-synchronization)
- [Update synchronization](./migration.md#update-synchronization)
- [Labels synchronization](./migration.md#labels-synchronization)
## What permissions does an operator need to run in a cluster?
You can read about needed permissions for operator in [this document](./security.md#roles).
## How to know the version of VM components in the operator?
See [printDefaults mode](./configuration.md).
In addition, you can use [Release notes](https://github.com/VictoriaMetrics/operator/releases)
or [CHANGELOG](https://github.com/VictoriaMetrics/operator/blob/master/docs/CHANGELOG.md).
- that's where we describe default version of VictoriaMetrics components.
## How to run VictoriaMetrics operator with permissions for one namespace only?
See this document for details: [Configuration -> Namespaced mode](./configuration.md#namespaced-mode).
## What versions of Kubernetes is the operator compatible with?
Operator tested at kubernetes versions from 1.16 to 1.27.

100
docs/operator/README.md
View file

@ -1,21 +1,91 @@
---
sort: 27
sort: 0
weight: 0
title: VictoriaMetrics Operator
disableToc: true
---
# VictoriaMetrics Operator
1. [VictoriaMetrics Operator](VictoriaMetrics-Operator.html)
1. [Additional Scrape Configuration](additional-scrape.html)
1. [API Docs](api.html)
1. [Authorization and exposing components](auth.html)
1. [vmbackupmanager](backups.html)
1. [Design](design.html)
1. [High Availability](high-availability.html)
1. [VMAlert, VMAgent, VMAlertmanager, VMSingle version](managing-versions.html)
1. [Victoria Metrics Operator Quick Start](quick-start.html)
1. [VMAgent relabel](relabeling.html)
1. [CRD Validation](resources-validation.html)
1. [Security](security.html)
1. [Auto Generated vars for package config](vars.html)
Operator serves to make running VictoriaMetrics applications on top of Kubernetes as easy as possible while preserving Kubernetes-native configuration options.
VictoriaMetrics Operator (`vmoperator`) is the classic kubernetes-operator for VictoriaMetrics with many [great features](#features).
It allows you to manage Victoria Metrics components in Kubernetes or OpenShift clusters
in a declarative style according to [GitOps](https://www.redhat.com/en/topics/devops/what-is-gitops)
and [IaC](https://en.wikipedia.org/wiki/Infrastructure_as_code) concepts.
VictoriaMetrics also provides [helm charts](https://github.com/VictoriaMetrics/helm-charts) without operator.
Operator makes the same, simplifies it and provides [advanced features](#features).
Learn more about [key concepts](#key-concepts) of `vmoperator` and follow the **[quick start guide](./quick-start.md)** for a better experience.
## Features of vmoperator
- Deployment and management in a kubernetes clusters of any number of VictoriaMetrics applications (like vmsingle/vmcluster instances and another components like vmauth, vmagent, vmalert, etc...)
- Seamless [migration from prometheus-operator](./migration.md) with auto-conversion of prometheus [custom resources](#custom-resources)
- Simple VictoriaMetrics cluster installation, configuring, upgrading and managing with [crd-objects](./resources/README.md).
- Ability to delegate the configuration (parts of configuration) of applications monitoring to the end-users and managing access to different configurations or configuration sections.
- Integration with VictoriaMetrics [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html) - advanced tools for making backups. Check [Backup automation for VMSingle](./resources/vmsingle.md#backup-automation) or [Backup automation for VMCluster](./resources/vmcluster.md#backup-automation).
- Everything you need for monitoring out of the box in [k8s-stack helm chart](https://victoriametrics.github.io/helm-charts/charts/victoria-metrics-k8s-stack/) with ready-made usecases and solutions.
- Ability to template your own deployment scenarios.
## Key Concepts
### Kubernetes-operators
[Kubernetes-operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) are software extensions
for Kubernetes that make use of [custom resources](#custom-resources) to manage applications and their components.
Operators follow Kubernetes principles, notably the control loop.
It can be said that operators are custom controllers for Kubernetes that allow you to create business logic for custom resources.
Design and implementation of `vmoperator` inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator).
Useful links:
- [Custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
- [Custom resource definitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/)
- [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
- [Operator best practices](https://sdk.operatorframework.io/docs/best-practices/)
### Custom resources
Kubernetes-Operators use [custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
for interaction. Custom resources are a mechanism built into Kubernetes that allows you to create your own extensions for Kubernetes,
working on the same principles as those built into Kubernetes APIs. Custom resources make Kubernetes so modular and extensible.
In addition, thanks to CRD ([Custom Resource Definitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/)),
the mechanism of custom resources allows you to declare an API in the format of the OpenAPI specification and verify that the resources correspond to this API.
### Reconciliation cycle
The main task of the operator is to bring the state of the cluster in line with what is declared by the user in the custom resources.
This process of constant monitoring and adjustment is called the "Reconciliation cycle" - it is the operator's workflow.
The basic workflow of working with the operator can be simplified as the following diagram:
<img src="README_operator-workflow.png" width="1200">
- Operator declares and owns [resources of Victoria Metrics](./resources/README.md).
- Kubernetes validates of the resource according to the specification from CRD (see more in [custom resources](#custom-resources)).
- Operator subscribed to change events (`create`, `update`, `delete`) for related resources.
- When an event occurs, the operator reacts and updates the state of the objects in the cluster.
- For some objects in the cluster the reconciliation cycle is performed at a given interval, even without the occurrence of change events (see `VM_FORCERESYNCINTERVAL`).
### Next steps
- [Quick Start Guide](./quick-start.md)
- [Setup](./setup.md)
- [Security](./security.md)
- [Configuration](./configuration.md)
- [Migration from Prometheus](./migration.md)
- [Monitoring](./monitoring.md)
- [Authorization and exposing components](./auth.md)
- [High Availability](./high-availability.md)
- [Enterprise](./enterprise.md)
- [Custom resources](./resources/README.md)
If you have any questions, check out our [FAQ](./FAQ.md)
and feel free to can ask them:
- [VictoriaMetrics Slack](https://victoriametrics.slack.com/)
- [VictoriaMetrics Telegram](https://t.me/VictoriaMetrics_en)
If you have any suggestions or find a bug, please create an issue
on [GitHub](https://github.com/VictoriaMetrics/operator/issues/new).

12
docs/operator/api.md
View file

@ -1,15 +1,11 @@
---
sort: 16
sort: 12
weight: 12
title: API Docs
weight: 16
menu:
docs:
parent: "operator"
weight: 16
aliases:
- /operator/api.html
---
<!-- this doc autogenerated - don't edit it manually -->
# API Docs
This Document documents the types introduced by the VictoriaMetrics to be consumed by users.

55
docs/operator/auth.md
View file

@ -1,27 +1,23 @@
---
sort: 4
weight: 4
sort: 7
weight: 7
title: Authorization and exposing components
menu:
docs:
parent: "operator"
weight: 4
aliases:
- /operator/auth.html
---
# Authorization and exposing components
## Exposing components
CRD objects doesn't have `ingress` configuration.
Instead, you can use [VMAuth](./resources/vmauth.md) as proxy between ingress-controller and VictoriaMetrics components.
CRD objects doesn't have `ingress` configuration. Instead, you can use `VMAuth` as proxy between ingress-controller and VM app components.
It adds missing authorization and access control features and enforces it.
Access can be given with `VMUser` definition. It supports basic auth and bearer token authentication.
Access can be given with [VMUser](./resources/vmuser.md) definition.
It supports basic auth and bearer token authentication:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
@ -30,12 +26,12 @@ spec:
userNamespaceSelector: {}
userSelector: {}
ingress: {}
EOF
unauthorizedAccessConfig: []
```
Advanced configuration with cert-manager annotations:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
@ -61,14 +57,12 @@ spec:
class_name: nginx
tlsHosts:
- vm-access.example.com
EOF
```
Simple static routing with read-only access to vmagent for username - `user-1` with password `Asafs124142`:
simple static routing with read-only access to vmagent for username - `user-1` with password `Asafs124142`
```yaml
# curl vmauth:8427/metrics -u 'user-1:Asafs124142'
cat << EOF | kubectl apply -f
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
@ -79,14 +73,12 @@ spec:
- static:
url: http://vmagent-base.default.svc:8429
paths: ["/targets/api/v1","/targets","/metrics"]
EOF
```
With bearer token access:
```yaml
# curl vmauth:8427/metrics -H 'Authorization: Bearer Asafs124142'
cat << EOF | kubectl apply -f
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
@ -97,13 +89,12 @@ spec:
- static:
url: http://vmagent-base.default.svc:8429
paths: ["/targets/api/v1","/targets","/metrics"]
EOF
```
It's also possible to use service discovery for objects:
```yaml
# curl vmauth:8427/metrics -H 'Authorization: Bearer Asafs124142'
cat << EOF | kubectl apply -f
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
@ -116,12 +107,11 @@ spec:
name: base
namespace: default
paths: ["/targets/api/v1","/targets","/metrics"]
EOF
```
Cluster components supports auto path generation for single tenant view:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
@ -143,17 +133,15 @@ spec:
url: http://vmselect-test-persistent.default.svc:8481/
paths:
- /internal/resetRollupResultCache
EOF
```
For each `VMUser` operator generates corresponding secret with username/password or bearer token at the same namespace as `VMUser`.
## Basic auth for targets
To authenticate a `VMServiceScrape`s over a metrics endpoint use [`basicAuth`](https://docs.victoriametrics.com/operator/api.html#basicauth)
To authenticate a `VMServiceScrape`s over a metrics endpoint use [`basicAuth`](./api.md#basicauth):
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
@ -173,11 +161,9 @@ spec:
selector:
matchLabels:
app: myapp
EOF
```
```yaml
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
@ -186,5 +172,14 @@ data:
password: dG9vcg== # toor
user: YWRtaW4= # admin
type: Opaque
EOF
```
## Unauthorized access
You can expose some routes without authorization with `unauthorizedAccessConfig`.
Check more details in [VMAuth docs -> Unauthorized access](./resources/vmauth.md#unauthorized-access).
More details about features of `VMAuth` and `VMUser` you can read in:
- [VMAuth docs](./resources/vmauth.md),
- [VMUser docs](./resources/vmuser.md).

260
docs/operator/configuration.md Normal file
View file

@ -0,0 +1,260 @@
---
sort: 4
weight: 4
title: Configuration
---
# Configuration
Operator configured by env variables, list of it can be found
on [Variables](./vars.md) page.
It defines default configuration options, like images for components, timeouts, features.
In addition, the operator has a special startup mode for outputting all variables, their types and default values.
For instance, with this mode you can know versions of VM components, which are used by default:
```console
./operator --printDefaults
# This application is configured via the environment. The following environment variables can be used:
#
# KEY TYPE DEFAULT REQUIRED DESCRIPTION
# VM_USECUSTOMCONFIGRELOADER True or False false
# VM_CUSTOMCONFIGRELOADERIMAGE String victoriametrics/operator:config-reloader-v0.32.0
# VM_VMALERTDEFAULT_IMAGE String victoriametrics/vmalert
# VM_VMALERTDEFAULT_VERSION String v1.93.3
# VM_VMALERTDEFAULT_USEDEFAULTRESOURCES True or False true
# VM_VMALERTDEFAULT_RESOURCE_LIMIT_MEM String 500Mi
# VM_VMALERTDEFAULT_RESOURCE_LIMIT_CPU String 200m
# ...
```
You can choose output format for variables with `--printFormat` flag, possible values: `json`, `yaml`, `list` and `table` (default):
```console
.operator --printDefaults --printFormat=json
# {
# 'VM_USECUSTOMCONFIGRELOADER': 'false',
# 'VM_CUSTOMCONFIGRELOADERIMAGE': 'victoriametrics/operator:config-reloader-v0.32.0',
# 'VM_VMALERTDEFAULT_IMAGE': 'victoriametrics/vmalert',
# 'VM_VMALERTDEFAULT_VERSION': 'v1.93.3',
# ...
# 'VM_FORCERESYNCINTERVAL': '60s',
# 'VM_ENABLESTRICTSECURITY': 'true'
# }
```
## Conversion of prometheus-operator objects
You can read detailed instructions about configuring prometheus-objects conversion in [this document](./migration.md).
## Helm-charts
In [helm-charts](https://github.com/VictoriaMetrics/helm-charts) some important configuration parameters are implemented as separate flags in `values.yaml`:
### victoria-metrics-k8s-stack
For possible values refer to [parameters](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack#parameters).
Also, checkout [here possible ENV variables](./vars.md) to configure operator behaviour.
ENV variables can be set in the `victoria-metrics-operator.env` section.
```yaml
# values.yaml
victoria-metrics-operator:
image:
# -- Image repository
repository: victoriametrics/operator
# -- Image tag
tag: v0.35.0
# -- Image pull policy
pullPolicy: IfNotPresent
# -- Tells helm to remove CRD after chart remove
cleanupCRD: true
cleanupImage:
repository: gcr.io/google_containers/hyperkube
tag: v1.18.0
pullPolicy: IfNotPresent
operator:
# -- By default, operator converts prometheus-operator objects.
disable_prometheus_converter: false
# -- Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD
prometheus_converter_add_argocd_ignore_annotations: false
# -- Enables ownership reference for converted prometheus-operator objects,
# it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
enable_converter_ownership: false
# -- By default, operator creates psp for its objects.
psp_auto_creation_enabled: true
# -- Enables custom config-reloader, bundled with operator.
# It should reduce vmagent and vmauth config sync-time and make it predictable.
useCustomConfigReloader: false
# -- extra settings for the operator deployment. full list Ref: [https://github.com/VictoriaMetrics/operator/blob/master/vars.md](https://github.com/VictoriaMetrics/operator/blob/master/vars.md)
env:
# -- default version for vmsingle
- name: VM_VMSINGLEDEFAULT_VERSION
value: v1.43.0
# -- container registry name prefix, e.g. docker.io
- name: VM_CONTAINERREGISTRY
value: ""
# -- image for custom reloader (see the useCustomConfigReloader parameter)
- name: VM_CUSTOMCONFIGRELOADERIMAGE
value: victoriametrics/operator:config-reloader-v0.32.0
# By default, the operator will watch all the namespaces
# If you want to override this behavior, specify the namespace it needs to watch separated by a comma.
# Ex: my_namespace1,my_namespace2
watchNamespace: ""
# Count of operator instances (can be increased for HA mode)
replicaCount: 1
# -- VM operator log level
# -- possible values: info and error.
logLevel: "info"
# -- Resource object
resources:
{}
# limits:
# cpu: 120m
# memory: 320Mi
# requests:
# cpu: 80m
# memory: 120Mi
```
### victoria-metrics-operator
For possible values refer to [parameters](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator#parameters).
Also, checkout [here possible ENV variables](./vars.md) to configure operator behaviour.
ENV variables can be set in the `env` section.
```yaml
# values.yaml
image:
# -- Image repository
repository: victoriametrics/operator
# -- Image tag
tag: v0.35.0
# -- Image pull policy
pullPolicy: IfNotPresent
operator:
# -- By default, operator converts prometheus-operator objects.
disable_prometheus_converter: false
# -- Compare-options and sync-options for prometheus objects converted by operator for properly use with ArgoCD
prometheus_converter_add_argocd_ignore_annotations: false
# -- Enables ownership reference for converted prometheus-operator objects,
# it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
enable_converter_ownership: false
# -- By default, operator creates psp for its objects.
psp_auto_creation_enabled: true
# -- Enables custom config-reloader, bundled with operator.
# It should reduce vmagent and vmauth config sync-time and make it predictable.
useCustomConfigReloader: false
# -- extra settings for the operator deployment. full list Ref: [https://github.com/VictoriaMetrics/operator/blob/master/vars.md](https://github.com/VictoriaMetrics/operator/blob/master/vars.md)
env:
# -- default version for vmsingle
- name: VM_VMSINGLEDEFAULT_VERSION
value: v1.43.0
# -- container registry name prefix, e.g. docker.io
- name: VM_CONTAINERREGISTRY
value: ""
# -- image for custom reloader (see the useCustomConfigReloader parameter)
- name: VM_CUSTOMCONFIGRELOADERIMAGE
value: victoriametrics/operator:config-reloader-v0.32.0
# By default, the operator will watch all the namespaces
# If you want to override this behavior, specify the namespace it needs to watch separated by a comma.
# Ex: my_namespace1,my_namespace2
watchNamespace: ""
# Count of operator instances (can be increased for HA mode)
replicaCount: 1
# -- VM operator log level
# -- possible values: info and error.
logLevel: "info"
# -- Resource object
resources:
{}
# limits:
# cpu: 120m
# memory: 320Mi
# requests:
# cpu: 80m
# memory: 120Mi
```
## Namespaced mode
By default, the operator will watch all namespaces, but it can be configured to watch only specific namespace.
If you want to override this behavior, specify the namespace:
- in the `WATCH_NAMESPACE` environment variable.
- in the `watchNamespace` field in the `values.yaml` file of helm-charts.
The operator supports only single namespace for watching.
You can find example of RBAC manifests for single-namespace mode in
[this file](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/operator_rbac_for_single_namespace.yaml).
## Monitoring of cluster components
By default, operator creates [VMServiceScrape](./resources/vmservicescrape.md)
object for each component that it manages.
You can disable this behaviour with `VM_DASABLESELFSERVICASCRAPECREATION` environment variable:
```shell
VM_DASABLESELFSERVICASCRAPECREATION=false
```
Also, you can override default configuration for self-scraping with `ServiceScrapeSpec` field in each deployable resource
(`vmcluster/select`, `vmcluster/insert`, `vmcluster/storage`, `vmagent`, `vmalert`, `vmalertmanager`, `vmauth`, `vmsingle`):
## CRD Validation
Operator supports validation admission webhook [docs](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
It checks resources configuration and returns errors to caller before resource will be created at kubernetes api.
This should reduce errors and simplify debugging.
Validation hooks at operator side must be enabled with flags:
```console
./operator
--webhook.enable
# optional configuration for certDir and tls names.
--webhook.certDir=/tmp/k8s-webhook-server/serving-certs/
--webhook.keyName=tls.key
--webhook.certName=tls.crt
```
You have to mount correct certificates at give directory.
It can be simplified with cert-manager and kustomize command:
```console
kustomize build config/deployments/webhook/
```
### Requirements
- Valid certificate with key must be provided to operator
- Valid CABundle must be added to the `ValidatingWebhookConfiguration`
### Useful links
- [k8s admission webhooks](https://banzaicloud.com/blog/k8s-admission-webhooks/)
- [olm webhooks](https://docs.openshift.com/container-platform/4.5/operators/user/olm-webhooks.html)

36
docs/operator/enterprise.md Normal file
View file

@ -0,0 +1,36 @@
---
sort: 13
weight: 13
title: Enterprise features
---
# Using operator with enterprise features
Operator doesn't have enterprise version for itself, but it supports
[enterprise features for VictoriaMetrics components](https://docs.victoriametrics.com/enterprise.html):
- [VMAgent Enterprise features](./resources/vmagent.md#enterprise-features):
- [Reading metrics from kafka](./resources/vmagent.md#reading-metrics-from-kafka)
- [Writing metrics to kafka](./resources/vmagent.md#writing-metrics-to-kafka)
- [VMAlert Enterprise features](./resources/vmalert.md#enterprise-features):
- [Reading rules from object storage](./resources/vmalert.md#reading-rules-from-object-storage)
- [Multitenancy](./resources/vmalert.md#multitenancy)
- [VMAuth Enterprise features](./resources/vmauth.md#enterprise-features)
- [IP Filters](./resources/vmauth.md#ip-filters)
- [VMCluster Enterprise features](./resources/vmcluster.md#enterprise-features)
- [Downsampling](./resources/vmcluster.md#downsampling)
- [Multiple retentions / Retention filters](./resources/vmcluster.md#retention-filters)
- [Advanced per-tenant statistic](./resources/vmcluster.md#advanced-per-tenant-statistic)
- [mTLS protection](./resources/vmcluster.md#mtls-protection)
- [Backup atomation](./resources/vmcluster.md#backup-atomation)
- [VMRule Enterprise features](./resources/vmrule.md#enterprise-features)
- [Multitenancy](./resources/vmrule.md#multitenancy)
- [VMSingle Enterprise features](./resources/vmsingle.md#enterprise-features)
- [Downsampling](./resources/vmsingle.md#downsampling)
- [Retention filters](./resources/vmsingle.md#retention-filters)
- [Backup atomation](./resources/vmsingle.md#backup-atomation)
- [VMUser Enterprise features](./resources/vmuser.md#enterprise-features)
- [IP Filters](./resources/vmuser.md#ip-filters)
More information about enterprise features you can read
on [VictoriaMetrics Enterprise page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).

392
docs/operator/high-availability.md
View file

@ -1,381 +1,43 @@
---
sort: 7
weight: 7
sort: 8
weight: 8
title: High Availability
menu:
docs:
parent: "operator"
weight: 7
aliases:
- /operator/high-availability.html
---
# High Availability
High availability is not only important for customer-facing software but if the monitoring infrastructure is not highly available, then there is a risk that operations people are not notified of alerts. Therefore, high availability must be just as thought through for the monitoring stack, as for anything else.
High availability is not only important for customer-facing software but if the monitoring infrastructure is not highly available, then there is a risk that operations people are not notified of alerts.
Therefore, high availability must be just as thought through for the monitoring stack, as for anything else.
## VMAgent
## Components
To run VMAgent in a highly available manner you have to configure deduplication at Victoria Metrics first [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/Single-server-VictoriaMetrics.md#deduplication)
VictoriaMetrics operator support high availability for each component of the monitoring stack:
Then increase replicas for VMAgent.
- [VMAgent](./resources/vmagent.md#high-availability)
- [VMAlert](./resources/vmalert.md#high-availability)
- [VMAlertmanager](./resources/vmalertmanager.md#high-availability)
- [VMAuth](./resources/vmauth.md#high-availability)
- [VMCluster](./resources/vmcluster.md#high-availability)
create `VMSingle` with dedup flag:
More details you can find in the section **[High Availability for resources](./resources/README.md#high-availability)**.
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
extraArgs:
dedup.minScrapeInterval: 60s
EOF
```
## Operator
create `VMAgent` with 2 replicas:
VictoriaMetrics operator can be safely scaled horizontally, but only one replica of the operator can
process [the reconciliation](./README.md#reconciliation-cycle) at a time -
it uses a leader election mechanism to ensure that only one replica is active at a time.
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeNamespaceSelector: {}
podScrapeNamespaceSelector: {}
podScrapeSelector: {}
serviceScrapeSelector: {}
scrapeInterval: 60s
vmAgentExternalLabelName: vmagent-ha
replicaCount: 2
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
If one of replicas of the operator will be failed, then another replica will be elected as a leader and will continue to work -
operator replication affects how quickly this happens.
```
[CRD validation](./configuration.md#crd-validation) workload is fully
distributed among the available operator replicas.
### Sharding
In addition, you can safely use for operator such features
as [assigning and distributing to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)
(like [node selector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector),
[affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity),
[topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints),
[taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), etc...)
Sharding for `VMAgent` distributes scraping between multiple deployments of `VMAgent`.
more info https://victoriametrics.github.io/vmagent.html#scraping-big-number-of-targets
Example usage:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeNamespaceSelector: {}
podScrapeNamespaceSelector: {}
podScrapeSelector: {}
serviceScrapeSelector: {}
scrapeInterval: 60s
vmAgentExternalLabelName: vmagent-ha
shardCount: 5
replicaCount: 2
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
shard-num: '%SHARD_NUM%'
topologyKey: kubernetes.io/hostname
EOF
```
This configuration produces 5 deployments with 2 replicas at each. Each deployment has its own shard num
and scrapes only 1/5 of all targets.
You can use special placeholder `%SHARD_NUM%` in any field of `VMAgent` specification.
and operator will replace it with current shard num of vmagent when creating deployment or statefullset for vmagent.
In the example above, the `%SHARD_NUM%` placeholder is used in the `podAntiAffinity` section,
which recommend to scheduler that pods with the same shard num (label `shard-num` in the pod template)
are not deployed on the same node. You can use another `topologyKey` for availability zone or region instead of nodes.
### StatefulMode
In `StatefulMode` `VMAgent` doesn't lose state of the PersistentQueue (file-based buffer size for unsent data) on pod restarts.
Operator creates `StatefulSet` and, with provided `PersistentVolumeClaimTemplate` at `StatefulStorage` configuration param, metrics queue is stored on disk.
Example of configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-persistent
spec:
selectAllByDefault: true
vmAgentExternalLabelName: vmagent-ha
replicaCount: 2
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
statefulMode: true
statefulStorage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
```
## VMAlert
It can be launched with multiple replicas without an additional configuration, alertmanager is responsible for alert deduplication.
Note, if you want to use `VMAlert` with high-available `VMAlertmanager`, which has more than 1 replica. You have to specify all pod fqdns
at `VMAlert.spec.notifiers.[url]`. Or you can use service discovery for notifier, examples:
alertmanager:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: vmalertmanager-example-alertmanager
labels:
app: vm-operator
type: Opaque
stringData:
alertmanager.yaml: |
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example
namespace: default
labels:
usage: dedicated
spec:
replicaCount: 2
configSecret: vmalertmanager-example-alertmanager
configSelector: {}
configNamespaceSelector: {}
```
vmalert with fqdns:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-ha
namespace: default
spec:
datasource:
url: http://vmsingle-example.default.svc:8429
notifiers:
- url: http://vmalertmanager-example-0.vmalertmanager-example.default.svc:9093
- url: http://vmalertmanager-example-1.vmalertmanager-example.default.svc:9093
```
vmalert with service discovery:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-ha
namespace: default
spec:
datasource:
url: http://vmsingle-example.default.svc:8429
notifiers:
- selector:
namespaceSelector:
matchNames:
- default
labelSelector:
matchLabels:
usage: dedicated
```
## VMSingle
It doesn't support high availability by default, for such purpose use VMCluster or duplicate the setup.
## VMCluster
The cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling.
For using the cluster version you have to create the corresponding CRD object:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster-persistent
spec:
retentionPeriod: "4"
replicationFactor: 2
vmstorage:
replicaCount: 2
storageDataPath: "/vm-data"
podMetadata:
labels:
owner: infra
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmstorage"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "2"
memory: 2048Mi
vmselect:
replicaCount: 2
cacheMountPath: "/select-cache"
podMetadata:
labels:
owner: infra
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmselect"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 2Gi
resources:
limits:
cpu: "1"
memory: "500Mi"
vminsert:
replicaCount: 2
podMetadata:
labels:
owner: infra
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vminsert"
topologyKey: "kubernetes.io/hostname"
resources:
limits:
cpu: "1"
memory: "500Mi"
EOF
```
Then wait for the cluster becomes ready
```console
kubectl get vmclusters -w
NAME INSERT COUNT STORAGE COUNT SELECT COUNT AGE STATUS
example-vmcluster-persistent 2 2 2 2s expanding
example-vmcluster-persistent 2 2 2 30s operational
```
Get links for connection by executing the command:
```console
kubectl get svc -l app.kubernetes.io/instance=example-vmcluster-persistent
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vminsert-example-vmcluster-persistent ClusterIP 10.96.34.94 <none> 8480/TCP 69s
vmselect-example-vmcluster-persistent ClusterIP None <none> 8481/TCP 79s
vmstorage-example-vmcluster-persistent ClusterIP None <none> 8482/TCP,8400/TCP,8401/TCP 85s
```
Now you can connect vmagent to vminsert and vmalert to vmselect
>NOTE do not forget to create rbac for vmagent
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeNamespaceSelector: {}
serviceScrapeSelector: {}
podScrapeNamespaceSelector: {}
podScrapeSelector: {}
# Add fields here
replicaCount: 1
remoteWrite:
- url: "http://vminsert-example-vmcluster-persistent.default.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"
EOF
```
Config for vmalert
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
# Add fields here
replicas: 1
datasource:
url: "http://vmselect-example-vmcluster-persistent.default.svc.cluster.local:8481/select/0/prometheus"
notifier:
url: "http://alertmanager-operated.default.svc:9093"
evaluationInterval: "10s"
ruleSelector: {}
EOF
```
## Alertmanager
The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fire alerts against *all* instances of an Alertmanager cluster.
The Alertmanager, starting with the `v0.5.0` release, ships with a high availability mode. It implements a gossip protocol to synchronize instances of an Alertmanager cluster regarding notifications that have been sent out, to prevent duplicate notifications. It is an AP (available and partition tolerant) system. Being an AP system means that notifications are guaranteed to be sent at least once.
The Victoria Metrics Operator ensures that Alertmanager clusters are properly configured to run highly available on Kubernetes.
In addition, don't forget about [monitoring for the operator](./monitoring.md).

203
docs/operator/migration.md Normal file
View file

@ -0,0 +1,203 @@
---
sort: 5
weight: 5
title: Migration from Prometheus
---
# Migration from prometheus-operator
Design and implementation inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator).
It's great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.
So you can use familiar CRD objects: `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe` and `AlertmanagerConfig`.
Or you can use VictoriaMetrics CRDs:
- `VMServiceScrape` (instead of `ServiceMonitor`) - defines scraping metrics configuration from pods backed by services. [See details](./resources/vmservicescrape.md).
- `VMPodScrape` (instead of `PodMonitor`) - defines scraping metrics configuration from pods. [See details](./resources/vmpodscrape.md).
- `VMRule` (instead of `PrometheusRule`) - defines alerting or recording rules. [See details](./resources/vmrule.md).
- `VMProbe` (instead of `Probe`) - defines a probing configuration for targets with blackbox exporter. [See details](./resources/vmprobe.md).
- `VMAlertmanagerConfig` (instead of `AlertmanagerConfig`) - defines a configuration for AlertManager. [See details](./resources/vmalertmanagerconfig.md).
Note that Prometheus CRDs are not supplied with the VictoriaMetrics operator,
so you need to [install them separately](https://github.com/prometheus-operator/prometheus-operator/releases).
VictoriaMetrics operator supports conversion from Prometheus CRD of
version `monitoring.coreos.com/v1` for kinds `ServiceMonitor`, `PodMonitor`, `PrometheusRule`, `Probe`
and version `monitoring.coreos.com/v1alpha1` for kind `AlertmanagerConfig`.
The default behavior of the operator is as follows:
- It **converts** all existing Prometheus `ServiceMonitor`, `PodMonitor`, `PrometheusRule` and `Probe` objects into corresponding VictoriaMetrics Operator objects.
- It **syncs** updates (including labels) from Prometheus `ServiceMonitor`, `PodMonitor`, `PrometheusRule` and `Probe` objects to corresponding VictoriaMetrics Operator objects.
- It **DOES NOT delete** converted objects after original ones are deleted.
With this configuration removing prometheus-operator API objects wouldn't delete any converted objects. So you can safely migrate or run two operators at the same time.
You can change default behavior with operator configuration - [see details below](#objects-conversion).
## Objects conversion
By default, the vmoperator converts all existing [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator)
API objects into corresponding VictoriaMetrics Operator objects ([see above](#migration-from-prometheus-operator)),
i.e. creates resources of VictoriaMetrics similar to Prometheus resources in the same namespace.
You can control this behaviour by setting env variable for operator:
```console
# disable convertion for each object
VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR=false
VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROBE=false
```
For [victoria-metrics-operator helm-chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md) you can use following way:
```yaml
# values.yaml
# ...
operator:
# -- By default, operator converts prometheus-operator objects.
disable_prometheus_converter: true
# ...
```
Otherwise, VictoriaMetrics Operator would try to discover prometheus-operator API and convert it.
<img src="migration_prometheus-conversion.png" width="1200">
For more information about the operator's workflow, see [this doc](./README.md).
## Deletion synchronization
By default, the operator doesn't make converted objects disappear after original ones are deleted. To change this behaviour
configure adding `OwnerReferences` to converted objects with following [operator parameter](./setup.md#settings):
```console
VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true
```
For [victoria-metrics-operator helm-chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md) you can use following way:
```yaml
# values.yaml
# ...
operator:
# -- Enables ownership reference for converted prometheus-operator objects,
# it will remove corresponding victoria-metrics objects in case of deletion prometheus one.
enable_converter_ownership: true
# ...
```
Converted objects will be linked to the original ones and will be deleted by kubernetes after the original ones are deleted.
## Update synchronization
Conversion of api objects can be controlled by annotations, added to `VMObject`s.
Annotation `operator.victoriametrics.com/ignore-prometheus-updates` controls updates from Prometheus api objects.
By default, it set to `disabled`. You define it to `enabled` state and all updates from Prometheus api objects will be ignored.
Example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/ignore-prometheus-updates: enabled
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
Annotation `operator.victoriametrics.com/ignore-prometheus-updates` can be set on one of the resources:
- [VMServiceScrape](./resources/vmservicescrape.md)
- [VMPodScrape](./resources/vmpodscrape.md)
- [VMRule](./resources/vmrule.md)
- [VMProbe](./resources/vmprobe.md)
- [VMAlertmanagerConfig](./resources/vmalertmanagerconfig.md)
And annotation doesn't make sense for [VMStaticScrape](./resources/vmstaticscrape.md)
and [VMNodeScrape](./resources/vmnodescrape.md) because these objects are not created as a result of conversion.
## Labels and annotations synchronization
Conversion of api objects can be controlled by annotations, added to `VMObject`s.
Annotation `operator.victoriametrics.com/merge-meta-strategy` controls syncing of metadata labels and annotations
between `VMObject`s and `Prometheus` api objects during updates to `Prometheus` objects.
By default, it has `prefer-prometheus`. And annotations and labels will be used from `Prometheus` objects, manually set values will be dropped.
You can set it to `prefer-victoriametrics`. In this case all labels and annotations applied to `Prometheus` object will be ignored and `VMObject` will use own values.
Two additional strategies annotations -`merge-victoriametrics-priority` and `merge-prometheus-priority` merges labelSets into one combined labelSet, with priority.
Example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
Annotation `operator.victoriametrics.com/merge-meta-strategy` can be set on one of the resources:
- [VMServiceScrape](./resources/vmservicescrape.md)
- [VMPodScrape](./resources/vmpodscrape.md)
- [VMRule](./resources/vmrule.md)
- [VMProbe](./resources/vmprobe.md)
- [VMAlertmanagerConfig](./resources/vmalertmanagerconfig.md)
And annotation doesn't make sense for [VMStaticScrape](./resources/vmstaticscrape.md)
and [VMNodeScrape](./resources/vmnodescrape.md) because these objects are not created as a result of conversion.
You can filter labels for syncing
with [operator parameter](./setup.md#settings) `VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES`:
```console
# it excludes all labels that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES=helm.sh,argoproj.io
```
In the same way, annotations with specified prefixes can be excluded from synchronization
with [operator parameter](./setup.md#settings) `VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES`:
```console
# it excludes all annotations that start with "helm.sh" or "argoproj.io" from synchronization
VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES=helm.sh,argoproj.io
```
## Using converter with ArgoCD
If you use ArgoCD, you can allow ignoring objects at ArgoCD converted from Prometheus CRD
with [operator parameter](./setup.md#settings) `VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS`.
It helps to properly use converter with ArgoCD and should help prevent out-of-sync issues with argo-cd based deployments:
```console
# adds compare-options and sync-options for prometheus objects converted by operator
VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS=true
```
## Data migration
You can use [vmctl](https://docs.victoriametrics.com/vmctl.html) for migrating your data from Prometheus to VictoriaMetrics.
See [this doc](https://docs.victoriametrics.com/vmctl.html#migrating-data-from-prometheus) for more details.

72
docs/operator/monitoring.md Normal file
View file

@ -0,0 +1,72 @@
---
sort: 6
weight: 6
title: Monitoring
---
# Monitoring of VictoriaMetrics Operator
VictoriaMetrics operator exports internal metrics in Prometheus exposition format at `/metrics` page.
These metrics can be scraped via [vmagent](./resources/vmagent.md) or Prometheus.
## Dashboard
Official Grafana dashboard available for [vmoperator](https://grafana.com/grafana/dashboards/17869-victoriametrics-operator/).
<img src="monitoring_operator-dashboard.jpeg" width=1200>
Graphs on the dashboards contain useful hints - hover the `i` icon in the top left corner of each graph to read it.
<!-- TODO: alerts for operator -->
## Configuration
### Helm-chart victoria-metrics-k8s-stack
In [victoria-metrics-k8s-stack](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/README.md) helm-chart operator self-scrapes metrics by default.
This helm-chart also includes [official grafana dashboard for operator](#dashboard).
### Helm-chart victoria-metrics-operator
With [victoria-metrics-operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator/README.md) you can use following parameter in `values.yaml`:
```yaml
# values.yaml
#...
# -- configures monitoring with serviceScrape. VMServiceScrape must be pre-installed
serviceMonitor:
enabled: true
```
This parameter makes helm-chart to create a scrape-object for installed operator instance.
You will also need to deploy a (vmsingle)[./resources/vmsingle.md] where the metrics will be collected.
### Pure operator installation
With pure operator installation you can use config with separate vmsingle and scrape object for operator like that:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: vmoperator
namespace: monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/instance: vm-operator
app.kubernetes.io/name: victoria-metrics-operator
endpoints:
- port: http
namespaceSelector:
matchNames:
- monitoring
```
See more info about object [VMServiceScrape](./resources/vmservicescrape.md).
You will also need a [vmsingle](https://docs.victoriametrics.com/vmoperatos/resources/vmsingle.html) where the metrics will be collected.

1891
docs/operator/quick-start.md
View file

File diff suppressed because it is too large Load diff

220
docs/operator/resources/README.md Normal file
View file

@ -0,0 +1,220 @@
---
sort: 14
weight: 14
title: Custom resources
---
# Custom resource kinds
This documentation section describes the design and interaction between the custom resource definitions (CRD) that the Victoria
Metrics Operator introduces.
[Operator](../README.md) introduces the following custom resources:
- [VMAgent](./vmagent.md)
- [VMAlert](./vmalert.md)
- [VMAlertManager](./vmalertmanager.md)
- [VMAlertManagerConfig](./vmalertmanagerconfig.md)
- [VMAuth](./vmauth.md)
- [VMCluster](./vmcluster.md)
- [VMNodeScrape](./vmnodescrape.md)
- [VMPodScrape](./vmpodscrape.md)
- [VMProbe](./vmprobe.md)
- [VMRule](./vmrule.md)
- [VMServiceScrape](./vmservicescrape.md)
- [VMStaticScrape](./vmstaticscrape.md)
- [VMSingle](./vmsingle.md)
- [VMUser](./vmuser.md)
Here is the scheme of relations between the custom resources:
<img src="README_cr-relations.png" width="1200">
## Specification
You can find the specification for the custom resources on **[API Docs](../api.md)**.
### Extra arguments
If you can't find necessary field in the specification of custom resource,
you can use `extraArgs` field for passing additional arguments to the application.
Field `extraArgs` is supported for the following custom resources:
- [VMAgent spec](../api.md#vmagentspec)
- [VMAlert spec](../api.md#vmalertspec)
- [VMAlertManager spec](../api.md#vmalertmanagerspec)
- [VMAuth spec](../api.md#vmauthspec)
- [VMCluster/vmselect spec](../api.md#vmselect)
- [VMCluster/vminsert spec](../api.md#vminsert)
- [VMCluster/vmstorage spec](../api.md#vmstorage)
- [VMSingle spec](../api.md#vmsinglespec)
Supported flags for each application can be found the in the corresponding documentation:
- [VMAgent flags](https://docs.victoriametrics.com/vmagent.html#advanced-usage)
- [VMAlert](https://docs.victoriametrics.com/vmalert.html#configuration)
- [VMAuth](https://docs.victoriametrics.com/vmauth.html#advanced-usage)
- [VMCluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#list-of-command-line-flags)
- [VMSingle](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#list-of-command-line-flags)
Usage example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: vmsingle-example-exrtaargs
spec:
retentionPeriod: "1"
extraArgs:
dedup.minScrapeInterval: 60s
# ...
```
### Extra environment variables
Flag can be replaced with environment variable, it's useful for retrieving value from secret.
You can use `extraEnvs` field for passing additional arguments to the application.
Usage example:
```yaml
kind: VMSingle
metadata:
name: vmsingle-example--exrtaenvs
spec:
retentionPeriod: "1"
extraEnvs:
- name: DEDUP_MINSCRAPEINTERVAL
valueFrom:
secretKeyRef:
name: vm-secret
key: dedup
```
This feature really useful for using with
[`-envflag.enable` command-line argument](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#environment-variables).
## Examples
Page for every custom resource contains examples section:
- [VMAgent examples](./vmagent.md#examples)
- [VMAlert examples](./vmalert.md#examples)
- [VMAlertmanager examples](./vmalertmanager.md#examples)
- [VMAlertmanagerConfig examples](./vmalertmanagerconfig.md#examples)
- [VMAuth examples](./vmauth.md#examples)
- [VMCluster examples](./vmcluster.md#examples)
- [VMNodeScrape examples](./vmnodescrape.md#examples)
- [VMPodScrape examples](./vmpodscrape.md#examples)
- [VMProbe examples](./vmprobe.md#examples)
- [VMRule examples](./vmrule.md#examples)
- [VMServiceScrape examples](./vmservicescrape.md#examples)
- [VMStaticScrape examples](./vmstaticscrape.md#examples)
- [VMSingle examples](./vmsingle.md#examples)
- [VMUser examples](./vmuser.md#examples)
In addition, you can find examples of the custom resources for VIctoriMetrics operator in
the **[examples directory](https://github.com/VictoriaMetrics/operator/tree/master/config/examples) of operator repository**.
## Managing versions of VM
Every custom resource with deployable application has a fields for specifying version (docker image) of component:
- [Managing versions for VMAgent](./vmagent.md#version-management)
- [Managing versions for VMAlert](./vmalert.md#version-management)
- [Managing versions for VMAlertmanager](./vmalertmanager.md#version-management)
- [Managing versions for VMAuth](./vmauth.md#version-management)
- [Managing versions for VMCluster](./vmcluster.md#version-management)
- [Managing versions for VMSingle](./vmsingle.md#version-management)
## High availability
VictoriaMetrics operator support high availability for each component of the monitoring stack:
- [VMAgent](./vmagent.md#high-availability)
- [VMAlert](./vmalert.md#high-availability)
- [VMAlertmanager](./vmalertmanager.md#high-availability)
- [VMAuth](./vmauth.md#high-availability)
- [VMCluster](./vmcluster.md#high-availability)
In addition, these CRD support common features, that can be used to increase high availability - resources above have the following fields:
- `affinity` - to schedule pods on different nodes ([affinity and anti-affinity in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)),
- `tolerations` - to schedule pods on nodes with taints ([taints and tolerations in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)),
- `nodeSelector` - to schedule pods on nodes with specific labels ([node selector in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)),
- `topologySpreadConstraints` - to schedule pods on different nodes in the same topology ([topology spread constraints in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints)).
See details about these fields in the [Specification](#specification).
## Enterprise features
Operator supports following [Enterprise features for VictoriaMetrics components](https://docs.victoriametrics.com/enterprise.html):
- [VMAgent Enterprise features](./vmagent.md#enterprise-features):
- [Reading metrics from kafka](./vmagent.md#reading-metrics-from-kafka)
- [Writing metrics to kafka](./vmagent.md#writing-metrics-to-kafka)
- [VMAlert Enterprise features](./vmalert.md#enterprise-features):
- [Reading rules from object storage](./vmalert.md#reading-rules-from-object-storage)
- [Multitenancy](./vmalert.md#multitenancy)
- [VMAuth Enterprise features](./vmauth.md#enterprise-features)
- [IP Filters](./vmauth.md#ip-filters)
- [VMCluster Enterprise features](./vmcluster.md#enterprise-features)
- [Downsampling](./vmcluster.md#downsampling)
- [Multiple retentions / Retention filters](./vmcluster.md#retention-filters)
- [Advanced per-tenant statistic](./vmcluster.md#advanced-per-tenant-statistic)
- [mTLS protection](./vmcluster.md#mtls-protection)
- [Backup atomation](./vmcluster.md#backup-atomation)
- [VMRule Enterprise features](./vmrule.md#enterprise-features)
- [Multitenancy](./vmrule.md#multitenancy)
- [VMSingle Enterprise features](./vmsingle.md#enterprise-features)
- [Downsampling](./vmsingle.md#downsampling)
- [Retention filters](./vmsingle.md#retention-filters)
- [Backup atomation](./vmsingle.md#backup-atomation)
- [VMUser Enterprise features](./vmuser.md#enterprise-features)
- [IP Filters](./vmuser.md#ip-filters)
More information about enterprise features you can read
on [VictoriaMetrics Enterprise page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
## Configuration synchronization
### Basic concepts
VictoriaMetrics applications, like many other applications with configuration file deployed at Kubernetes, uses `ConfigMaps` and `Secrets` for configuration files.
Usually, it's out of application scope to watch for configuration on-disk changes.
Applications reload their configuration by a signal from a user or some other tool, that knows how to watch for updates.
At Kubernetes, the most popular design for this case is a sidecar container, that watches for configuration file changes and sends an HTTP request to the application.
`Configmap` or `Secret` that mounted at `Pod` holds a copy of its content.
Kubernetes component `kubelet` is responsible for content synchronization between an object at Kubernetes API and a file served on disk.
It's not efficient to sync its content immediately, and `kubelet` eventually synchronizes it. There is a configuration option, that controls this period.
That's why, applications managed by operator don't receive changes immediately. It usually takes 1-2 min, before content will be updated.
It may trigger errors when an application was deleted, but [`VMAgent`](./README.mdvmagent) still tries to scrape it.
### Possible mitigations
The naive solution for this case decrease the synchronization period. But it configures globally and may be hard for operator users.
That's why operator uses a few hacks.
For `ConfigMap` updates, operator changes annotation with a time of `Configmap` content update. It triggers `ConfigMap`'s content synchronization by kubelet immediately.
It's the case for `VMAlert`, it uses `ConfigMap` as a configuration source.
For `Secret` it doesn't work. And operator offers its implementation for side-car container. It can be configured with env variable for operator:
```
- name: VM_USECUSTOMCONFIGRELOADER
value: "true"
```
If it's defined, operator uses own [config-reloader](https://github.com/VictoriaMetrics/operator/tree/master/internal/config-reloader)
instead of [prometheus-config-reload](https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader).
It watches corresponding `Secret` for changes with Kubernetes API watch call and writes content into emptyDir.
This emptyDir shared with the application.
In case of content changes, `config-reloader` sends HTTP requests to the application.
It greatly reduces the time for configuration synchronization.

720
docs/operator/resources/vmagent.md Normal file
View file

@ -0,0 +1,720 @@
# VMAgent
`VMAgent` represents agent, which helps you collect metrics from various sources and stores them in VictoriaMetrics.
The `VMAgent` CRD declaratively defines a desired [VMAgent](https://docs.victoriametrics.com/vmagent)
setup to run in a Kubernetes cluster.
It requires access to Kubernetes API and you can create RBAC for it first, it can be found
at [`examples/vmagent_rbac.yaml`](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmagent_rbac.yaml)
Or you can use default rbac account, that will be created for `VMAgent` by operator automatically.
For each `VMAgent` resource Operator deploys a properly configured `Deployment` in the same namespace.
The VMAgent `Pod`s are configured to mount a `Secret` prefixed with `<VMAgent-name>` containing the configuration
for VMAgent.
For each `VMAgent` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
name `<VMAgent-name>`.
The CRD specifies which `VMServiceScrape` should be covered by the deployed VMAgent instances based on label selection.
The Operator then generates a configuration based on the included `VMServiceScrape`s and updates the `Secret` which
contains the configuration. It continuously does so for all changes that are made to the `VMServiceScrape`s or the
`VMAgent` resource itself.
If no selection of `VMServiceScrape`s is provided - Operator leaves management of the `Secret` to the user,
so user can set custom configuration while still benefiting from the Operator's capabilities of managing VMAgent setups.
## Specification
You can see the full actual specification of the `VMAgent` resource in the **[API docs -> VMAgent](../api.md#vmagent)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## Scraping
`VMAgent` supports scraping targets with:
- [VMServiceScrape](./vmservicescrape.md),
- [VMPodScrape](./vmpodscrape.md),
- [VMNodeScrape](./vmnodescrape.md),
- [VMStaticScrape](./vmstaticscrape.md),
- [VMProbe](./vmprobe.md).
These objects tell VMAgent from which targets and how to collect metrics and
generate part of [VMAgent](./vmagent.md) scrape configuration.
For filtering scrape objects `VMAgent` uses selectors.
Selectors are defined with suffixes - `NamespaceSelector` and `Selector` for each type of scrape objects in spec of `VMAgent`:
- `serviceScrapeNamespaceSelector` and `serviceScrapeSelector` for selecting [VMServiceScrape](./vmservicescrape.md) objects,
- `podScrapeNamespaceSelector` and `podScrapeSelector` for selecting [VMPodScrape](./vmpodscrape.md) objects,
- `probeNamespaceSelector` and `probeSelector` for selecting [VMProbe](./vmprobe.md) objects,
- `staticScrapeNamespaceSelector` and `staticScrapeSelector` for selecting [VMStaticScrape](./vmstaticscrape.md) objects,
- `nodeScrapeNamespaceSelector` and `nodeScrapeSelector` for selecting [VMNodeScrape](./vmnodescrape.md) objects.
It allows configuring objects access control across namespaces and different environments.
Specification of selectors you can see in [this doc](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta).
In addition to the above selectors, the filtering of objects in a cluster is affected by the field `selectAllByDefault` of `VMAgent` spec and environment variable `WATCH_NAMESPACE` for operator.
Following rules are applied:
- If `*NamespaceSelector` and `*Selector` both undefined, then by default select nothing. With option set - `spec.selectAllByDefault: true`, select all objects of given type.
- If `*NamespaceSelector` defined, `*Selector` undefined, then all objects are matching at namespaces for given `*NamespaceSelector`.
- If `*NamespaceSelector` undefined, `*Selector` defined, then all objects at `VMAgent`'s namespaces are matching for given `*Selector`.
- If `*NamespaceSelector` and `*Selector` both defined, then only objects at namespaces matched `*NamespaceSelector` for given `*Selector` are matching.
Here's a more visual and more detailed view:
| `*NamespaceSelector` | `*Selector` | `selectAllByDefault` | `WATCH_NAMESPACE` | Selected objects |
|----------------------|-------------|----------------------|-------------------|-------------------------------------------------------------------------------------------------------|
| undefined | undefined | false | undefined | nothing |
| undefined | undefined | **true** | undefined | all objects of given type (`*`) in the cluster |
| **defined** | undefined | any | undefined | all objects of given type (`*`) at namespaces for given `*NamespaceSelector` |
| undefined | **defined** | any | undefined | all objects of given type (`*`) only at `VMAgent`'s namespace are matching for given `Selector |
| **defined** | **defined** | any | undefined | all objects of given type (`*`) only at namespaces matched `*NamespaceSelector` for given `*Selector` |
| any | undefined | any | **defined** | all objects of given type (`*`) only at `VMAgent`'s namespace |
| any | **defined** | any | **defined** | all objects of given type (`*`) only at `VMAgent`'s namespace for given `*Selector` |
More details about `WATCH_NAMESPACE` variable you can read in [this doc](../configuration.md#namespaced-mode).
Here are some examples of `VMAgent` configuration with selectors:
```yaml
# select all scrape objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-select-all
spec:
# ...
selectAllByDefault: true
---
# select all scrape objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-select-ns
spec:
# ...
serviceScrapeNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
podScrapeNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
nodeScrapeNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
staticScrapeNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
probeNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
```
## High availability
<!-- TODO: health checks -->
### Replication and deduplication
To run VMAgent in a highly available manner at first you have to configure deduplication in Victoria Metrics
according [this doc for VMSingle](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication)
or [this doc for VMCluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#deduplication).
You can do it with `extraArgs` on [`VMSingle`](./vmsingle.md):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: vmsingle-example
spec:
# ...
extraArgs:
dedup.minScrapeInterval: 30s
# ...
```
For [`VMCluster`](./vmcluster.md) you can do it with `vmstorage.extraArgs` and `vmselect.extraArgs`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-example
spec:
# ...
vmselect:
extraArgs:
dedup.minScrapeInterval: 30s
# ...
vmstorage:
extraArgs:
dedup.minScrapeInterval: 30s
# ...
```
Deduplication is automatically enabled with `replicationFactor > 1` on `VMCLuster`.
After enabling deduplication you can increase replicas for VMAgent.
For instance, let's create `VMAgent` with 2 replicas:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ha-example
spec:
# ...
selectAllByDefault: true
vmAgentExternalLabelName: vmagent_ha
remoteWrite:
- url: "http://vmsingle-example.default.svc:8429/api/v1/write"
# Replication:
scrapeInterval: 30s
replicaCount: 2
# ...
```
Now, even if something happens to one of the vmagent, you'll still have the data.
### StatefulMode
VMAgent supports [persistent buffering](https://docs.victoriametrics.com/vmagent.html#replication-and-high-availability)
for sending data to remote storage. By default, operator set `-remoteWrite.tmpDataPath` for `VMAgent` to `/tmp` (that use k8s ephemeral storage)
and `VMAgent` loses state of the PersistentQueue on pod restarts.
In `StatefulMode` `VMAgent` doesn't lose state of the PersistentQueue (file-based buffer size for unsent data) on pod restarts.
Operator creates `StatefulSet` and, with provided `PersistentVolumeClaimTemplate` at `StatefulStorage` configuration param, metrics queue is stored on disk.
Example of configuration for `StatefulMode`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ha-example
spec:
# ...
selectAllByDefault: true
vmAgentExternalLabelName: vmagent_ha
remoteWrite:
- url: "http://vmsingle-example.default.svc:8429/api/v1/write"
# Replication:
scrapeInterval: 30s
replicaCount: 2
# StatefulMode:
statefulMode: true
statefulStorage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
# ...
```
### Sharding
Operator supports sharding with [cluster mode of vmagent](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets)
for **scraping big number of targets**.
Sharding for `VMAgent` distributes scraping between multiple deployments of `VMAgent`.
Example usage (it is a complete example of `VMAgent` with high availability features):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ha-example
spec:
# ...
selectAllByDefault: true
vmAgentExternalLabelName: vmagent_ha
remoteWrite:
- url: "http://vmsingle-example.default.svc:8429/api/v1/write"
# Replication:
scrapeInterval: 30s
replicaCount: 2
# StatefulMode:
statefulMode: true
statefulStorage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
# Sharding
shardCount: 5
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
shard-num: '%SHARD_NUM%'
topologyKey: kubernetes.io/hostname
# ...
```
This configuration produces `5` deployments with `2` replicas at each.
Each deployment has its own shard num and scrapes only `1/5` of all targets.
Also, you can use special placeholder `%SHARD_NUM%` in fields of `VMAgent` specification
and operator will replace it with current shard num of vmagent when creating deployment or statefullset for vmagent.
In the example above, the `%SHARD_NUM%` placeholder is used in the `podAntiAffinity` section,
which recommend to scheduler that pods with the same shard num (label `shard-num` in the pod template)
are not deployed on the same node. You can use another `topologyKey` for availability zone or region instead of nodes.
**Note** that at the moment operator doesn't use `-promscrape.cluster.replicationFactor` parameter of `VMAgent` and
creates `replicaCount` of replicas for each shard (which leads greater resource consumption).
This will be fixed in the future, more details can be seen in [this issue](https://github.com/VictoriaMetrics/operator/issues/604).
Also see [this example](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmagent_stateful_with_sharding.yaml).
## Additional scrape configuration
AdditionalScrapeConfigs is an additional way to add scrape targets in `VMAgent` CRD.
There are two options for adding targets into `VMAgent`:
- [inline configuration into CRD](#inline-additional-scrape-configuration-in-vmagent-crd),
- [defining it as a Kubernetes Secret](#define-additional-scrape-configuration-as-a-kubernetes-secret).
No validation happens during the creation of configuration. However, you must validate job specs, and it must follow job spec configuration.
Please check [scrape_configs documentation](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) as references.
### Inline Additional Scrape Configuration in VMAgent CRD
You need to add scrape configuration directly to the `vmagent spec.inlineScrapeConfig`. It is raw text in YAML format.
See example below
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
# ...
selectAllByDefault: true
inlineScrapeConfig: |
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
remoteWrite:
- url: "http://vmsingle-example.default.svc:8429/api/v1/write"
# ...
```
**Note**: Do not use passwords and tokens with inlineScrapeConfig use Secret instead.
## Define Additional Scrape Configuration as a Kubernetes Secret
You need to define Kubernetes Secret with a key.
The key is `prometheus-additional.yaml` in the example below:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
stringData:
prometheus-additional.yaml: |
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
```
After that, you need to specify the secret's name and key in VMAgent CRD in `additionalScrapeConfigs` section:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
# ...
selectAllByDefault: true
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yaml
remoteWrite:
- url: "http://vmsingle-example.default.svc:8429/api/v1/write"
# ...
```
**Note**: You can specify only one Secret in the VMAgent CRD configuration so use it for all additional scrape configurations.
## Relabeling
`VMAgent` supports global relabeling for all metrics and per remoteWrite target relabel config.
Note in some cases, you don't need relabeling, `key=value` label pairs can be added to the all scrapped metrics with `spec.externalLabels` for `VMAgent`:
```yaml
# simple label add config
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
externalLabels:
clusterid: some_cluster
```
`VMAgent` CR supports relabeling with [custom configMap](#relabeling-config-in-configmap)
or [inline defined at CRD](#inline-relabeling-config).
### Relabeling config in Configmap
Quick tour how to create `ConfigMap` with relabeling configuration:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: vmagent-relabel
data:
global-relabel.yaml: |
- target_label: bar
- source_labels: [aa]
separator: "foobar"
regex: "foo.+bar"
target_label: aaa
replacement: "xxx"
- action: keep
source_labels: [aaa]
- action: drop
source_labels: [aaa]
target-1-relabel.yaml: |
- action: keep_if_equal
source_labels: [foo, bar]
- action: drop_if_equal
source_labels: [foo, bar]
```
Second, add `relabelConfig` to `VMagent` spec for global relabeling with name of `Configmap` - `vmagent-relabel` and key `global-relabel.yaml`.
For relabeling per remoteWrite target, add `urlRelabelConfig` name of `Configmap` - `vmagent-relabel`
and key `target-1-relabel.yaml` to one of remoteWrite target for relabeling only for those target:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
# ...
selectAllByDefault: true
relabelConfig:
name: "vmagent-relabel"
key: "global-relabel.yaml"
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
urlRelabelConfig:
name: "vmagent-relabel"
key: "target-1-relabel.yaml"
```
### Inline relabeling config
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-example
spec:
# ...
selectAllByDefault: true
inlineRelabelConfig:
- target_label: bar
- source_labels: [aa]
separator: "foobar"
regex: "foo.+bar"
target_label: aaa
replacement: "xxx"
- action: keep
source_labels: [aaa]
- action: drop
source_labels: [aaa]
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
inlineUrlRelabelConfig:
- action: keep_if_equal
source_labels: [foo, bar]
- action: drop_if_equal
source_labels: [foo, bar]
```
### Combined example
It's also possible to use both features in combination.
First will be added relabeling configs from `inlineRelabelConfig`, then `relabelConfig` from configmap.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: vmagent-relabel
data:
global-relabel.yaml: |
- target_label: bar
- source_labels: [aa]
separator: "foobar"
regex: "foo.+bar"
target_label: aaa
replacement: "xxx"
- action: keep
source_labels: [aaa]
- action: drop
source_labels: [aaa]
target-1-relabel.yaml: |
- action: keep_if_equal
source_labels: [foo, bar]
- action: drop_if_equal
source_labels: [foo, bar]
```
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
# ...
selectAllByDefault: true
inlineRelabelConfig:
- target_label: bar1
- source_labels: [aa]
relabelConfig:
name: "vmagent-relabel"
key: "global-relabel.yaml"
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
urlRelabelConfig:
name: "vmagent-relabel"
key: "target-1-relabel.yaml"
inlineUrlRelabelConfig:
- action: keep_if_equal
source_labels: [foo1, bar2]
```
Resulted configmap, mounted to `VMAgent` pod:
```yaml
apiVersion: v1
data:
global_relabeling.yaml: |
- target_label: bar1
- source_labels:
- aa
- target_label: bar
- source_labels: [aa]
separator: "foobar"
regex: "foo.+bar"
target_label: aaa
replacement: "xxx"
- action: keep
source_labels: [aaa]
- action: drop
source_labels: [aaa]
url_rebaling-1.yaml: |
- source_labels:
- foo1
- bar2
action: keep_if_equal
- action: keep_if_equal
source_labels: [foo, bar]
- action: drop_if_equal
source_labels: [foo, bar]
kind: ConfigMap
metadata:
finalizers:
- apps.victoriametrics.com/finalizer
labels:
app.kubernetes.io/component: monitoring
app.kubernetes.io/instance: example-vmagent
app.kubernetes.io/name: vmagent
managed-by: vm-operator
name: relabelings-assets-vmagent-example-vmagent
namespace: default
ownerReferences:
- apiVersion: operator.victoriametrics.com/v1beta1
blockOwnerDeletion: true
controller: true
kind: VMAgent
name: example-vmagent
uid: 7e9fb838-65da-4443-a43b-c00cd6c4db5b
```
### Additional information
`VMAgent` also has some extra options for relabeling actions, you can check it [docs](https://docs.victoriametrics.com/vmagent#relabeling).
## Version management
To set `VMAgent` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
# ...
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Enterprise features
VMAgent supports feature [Kafka integration](https://docs.victoriametrics.com/vmagent.html#kafka-integration)
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
For using Enterprise version of [vmagent](https://docs.victoriametrics.com/vmagent.html)
you need to change version of `vmagent` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMAgent`:
After that you can pass [Kafka integration](https://docs.victoriametrics.com/vmagent.html#kafka-integration)
flags to `VMAgent` with [extraArgs](./README.md#extra-arguments).
### Reading metrics from Kafka
Here are complete example for [Reading metrics from Kafka](https://docs.victoriametrics.com/vmagent.html#reading-metrics-from-kafka):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmagent
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmagent enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: reading metrics from kafka
# more details about kafka integration you can read on https://docs.victoriametrics.com/vmagent.html#kafka-integration
# more details about these and other flags you can read on https://docs.victoriametrics.com/vmagent.html#command-line-flags-for-kafka-consumer
kafka.consumer.topic.brokers: localhost:9092
kafka.consumer.topic.format: influx
kafka.consumer.topic: metrics-by-telegraf
kafka.consumer.topic.groupID: some-id
# ...other fields...
```
### Writing metrics to Kafka
Here are complete example for [Writing metrics to Kafka](https://docs.victoriametrics.com/vmagent.html#writing-metrics-to-kafka):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: vmagent-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmagent
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmagent enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: writing metrics to Kafka
# more details about kafka integration you can read on https://docs.victoriametrics.com/vmagent.html#kafka-integration
remoteWrite:
# sasl with username and password
- url: kafka://broker-1:9092/?topic=prom-rw-1&security.protocol=SASL_SSL&sasl.mechanisms=PLAIN
# it requires to create kubernetes secret `kafka-basic-auth` with keys `username` and `password` in the same namespace
basicAuth:
username:
name: kafka-basic-auth
key: username
password:
name: kafka-basic-auth
key: password
# sasl with username and password from secret and tls
- url: kafka://localhost:9092/?topic=prom-rw-2&security.protocol=SSL
# it requires to create kubernetes secret `kafka-tls` with keys `ca.pem`, `cert.pem` and `key.pem` in the same namespace
tlsConfig:
ca:
secret:
name: kafka-tls
key: ca.pem
cert:
secret:
name: kafka-tls
key: cert.pem
keySecret:
name: kafka-tls
key: key.pem
# ...other fields...
```
## Examples
```yaml
kind: VMAgent
metadata:
name: vmagent-example
spec:
selectAllByDefault: true
replicaCount: 1
scrapeInterval: 30s
scrapeTimeout: 10s
vmAgentExternalLabelName: example
externalLabels:
cluster: my-cluster
remoteWrite:
- url: "http://vmsingle-example.default.svc:8428/api/v1/write"
inlineRelabelConfig:
- action: labeldrop
regex: "temp.*"
```

362
docs/operator/resources/vmalert.md Normal file
View file

@ -0,0 +1,362 @@
# VMAlert
`VMAlert` - executes a list of given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) rules against configured address.
The `VMAlert` CRD declaratively defines a desired [VMAlert](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert)
setup to run in a Kubernetes cluster.
It has few required config options - `datasource` and `notifier` are required, for other config parameters
check [doc](../api.md#vmalert).
For each `VMAlert` resource, the Operator deploys a properly configured `Deployment` in the same namespace.
The VMAlert `Pod`s are configured to mount a list of `Configmaps` prefixed with `<VMAlert-name>-number` containing
the configuration for alerting rules.
For each `VMAlert` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
name `<VMAlert-name>`.
## Specification
You can see the full actual specification of the `VMAlert` resource in the **[API docs -> VMAlert](../api.md#vmalert)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## Rules
The CRD specifies which `VMRule`s should be covered by the deployed `VMAlert` instances based on label selection.
The Operator then generates a configuration based on the included `VMRule`s and updates the `Configmaps` containing
the configuration. It continuously does so for all changes that are made to `VMRule`s or to the `VMAlert` resource itself.
Alerting rules are filtered by selectors `ruleNamespaceSelector` and `ruleSelector` in `VMAlert` CRD definition.
For selecting rules from all namespaces you must specify it to empty value:
```yaml
spec:
ruleNamespaceSelector: {}
```
[VMRUle](./vmrule.md) objects generate part of `VMAlert` configuration.
For filtering rules `VMAlert` uses selectors `ruleNamespaceSelector` and `ruleSelector`.
It allows configuring rules access control across namespaces and different environments.
Specification of selectors you can see in [this doc](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta).
In addition to the above selectors, the filtering of objects in a cluster is affected by the field `selectAllByDefault` of `VMAlert` spec and environment variable `WATCH_NAMESPACE` for operator.
Following rules are applied:
- If `ruleNamespaceSelector` and `ruleSelector` both undefined, then by default select nothing. With option set - `spec.selectAllByDefault: true`, select all vmrules.
- If `ruleNamespaceSelector` defined, `ruleSelector` undefined, then all vmrules are matching at namespaces for given `ruleNamespaceSelector`.
- If `ruleNamespaceSelector` undefined, `ruleSelector` defined, then all vmrules at `VMAgent`'s namespaces are matching for given `ruleSelector`.
- If `ruleNamespaceSelector` and `ruleSelector` both defined, then only vmrules at namespaces matched `ruleNamespaceSelector` for given `ruleSelector` are matching.
Here's a more visual and more detailed view:
| `ruleNamespaceSelector` | `ruleSelector` | `selectAllByDefault` | `WATCH_NAMESPACE` | Selected rules |
|-------------------------|----------------|----------------------|-------------------|------------------------------------------------------------------------------------------------------|
| undefined | undefined | false | undefined | nothing |
| undefined | undefined | **true** | undefined | all vmrules in the cluster |
| **defined** | undefined | any | undefined | all vmrules are matching at namespaces for given `ruleNamespaceSelector` |
| undefined | **defined** | any | undefined | all vmrules only at `VMAlert`'s namespace are matching for given `ruleSelector` |
| **defined** | **defined** | any | undefined | all vmrules only at namespaces matched `ruleNamespaceSelector` for given `ruleSelector` are matching |
| any | undefined | any | **defined** | all vmrules only at `VMAlert`'s namespace |
| any | **defined** | any | **defined** | all vmrules only at `VMAlert`'s namespace for given `ruleSelector` are matching |
More details about `WATCH_NAMESPACE` variable you can read in [this doc](../configuration.md#namespaced-mode).
Here are some examples of `VMAlert` configuration with selectors:
```yaml
# select all rule objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: vmalert-select-all
spec:
# ...
selectAllByDefault: true
---
# select all rule objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: vmalert-select-ns
spec:
# ...
ruleNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
```
## High availability
`VMAlert` can be launched with multiple replicas without an additional configuration as far [alertmanager](./vmalertmanager.md) is responsible for alert deduplication.
Note, if you want to use `VMAlert` with high-available [`VMAlertmanager`](./vmalertmanager.md), which has more than 1 replica.
You have to specify all pod fqdns at `VMAlert.spec.notifiers.[url]`. Or you can use service discovery for notifier, examples:
- alertmanager:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: vmalertmanager-example-alertmanager
labels:
app: vm-operator
type: Opaque
stringData:
alertmanager.yaml: |
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
# ...
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example
namespace: default
labels:
usage: dedicated
spec:
replicaCount: 2
configSecret: vmalertmanager-example-alertmanager
configSelector: {}
configNamespaceSelector: {}
# ...
```
- vmalert with fqdns:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-ha
namespace: default
spec:
replicaCount: 2
datasource:
url: http://vmsingle-example.default.svc:8429
notifiers:
- url: http://vmalertmanager-example-0.vmalertmanager-example.default.svc:9093
- url: http://vmalertmanager-example-1.vmalertmanager-example.default.svc:9093
evaluationInterval: "10s"
ruleSelector: {}
# ...
```
- vmalert with service discovery:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-ha
namespace: default
spec:
replicaCount: 2
datasource:
url: http://vmsingle-example.default.svc:8429
notifiers:
- selector:
namespaceSelector:
matchNames:
- default
labelSelector:
matchLabels:
usage: dedicated
evaluationInterval: "10s"
ruleSelector: {}
# ...
```
In addition, you need to specify `remoteWrite` and `remoteRead` urls for restoring alert states after restarts:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-ha
namespace: default
spec:
replicaCount: 2
evaluationInterval: "10s"
selectAllByDefault: true
datasource:
url: http://vmselect-demo.vm.svc:8481/select/0/prometheus
notifiers:
- url: http://vmalertmanager-example-0.vmalertmanager-example.default.svc:9093
- url: http://vmalertmanager-example-1.vmalertmanager-example.default.svc:9093
remoteWrite:
url: http://vminsert-demo.vm.svc:8480/insert/0/prometheus
remoteRead:
url: http://vmselect-demo.vm.svc:8481/select/0/prometheus
```
More details about `remoteWrite` and `remoteRead` you can read in [vmalert docs](https://docs.victoriametrics.com/vmalert.html#alerts-state-on-restarts).
## Version management
To set `VMAlert` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
# ...
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Enterprise features
VMAlert supports features [Reading rules from object storage](https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage)
and [Multitenancy](https://docs.victoriametrics.com/vmalert.html#multitenancy)
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
For using Enterprise version of [vmalert](https://docs.victoriametrics.com/vmalert.html)
you need to change version of `VMAlert` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMAlert`:
### Reading rules from object storage
After that you can pass `-rule` command-line argument with `s3://` or `gs://`
to `VMAlert` with [extraArgs](./README.md#extra-arguments).
More details about reading rules from object storage you can read in [vmalert docs](https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage).
Here are complete example for [Reading rules from object storage](https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: vmalert-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmalert
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmalert enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Reading rules from object storage
# more details about reading rules from object storage you can read on https://docs.victoriametrics.com/vmalert.html#reading-rules-from-object-storage
rule: s3://bucket/dir/alert.rules
# ...other fields...
```
### Multitenancy
After enabling enterprise version you can use [Multitenancy](https://docs.victoriametrics.com/vmalert.html#multitenancy)
feature in `VMAlert`.
For that you need to set `clusterMode` commad-line flag
with [extraArgs](./README.md#extra-arguments)
and specify `tenant` field for groups
in [VMRule](./vmrule.md#enterprise-features):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: vmalert-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmalert
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmalert enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Multitenancy
# more details about multitenancy you can read on https://docs.victoriametrics.com/vmalert.html#multitenancy
clusterMode: true
# ...other fields...
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: vmrule-ent-example
spec:
groups:
- name: vmalert-1
rules:
# using enterprise features: Multitenancy
# more details about multitenancy you can read on https://docs.victoriametrics.com/vmalert.html#multitenancy
- tenant: 1
alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
job: "{{ $labels.job }}"
annotations:
value: "{{ $value }}"
description: 'error reloading vmalert config, reload count for 5 min {{ $value }}'
```
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
replicaCount: 1
datasource:
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
notifier:
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
evaluationInterval: "30s"
selectAllByDefault: true
```

270
docs/operator/resources/vmalertmanager.md Normal file
View file

@ -0,0 +1,270 @@
# VMAlertmanager
`VMAlertmanager` - represents [alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) configuration.
The `VMAlertmanager` CRD declaratively defines a desired Alertmanager setup to run in a Kubernetes cluster.
It provides options to configure replication and persistent storage.
For each `Alertmanager` resource, the Operator deploys a properly configured `StatefulSet` in the same namespace.
The Alertmanager pods are configured to include a `Secret` called `<alertmanager-name>` which holds the used
configuration file in the key `alertmanager.yaml`.
When there are two or more configured replicas the Operator runs the Alertmanager instances in high availability mode.
## Specification
You can see the full actual specification of the `VMAlertmanager` resource in the **[API docs -> VMAlert](../api.md#vmalertmanager)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## Configuration
The operator generates a configuration file for `VMAlertmanager` based on user input at the definition of `CRD`.
Generated config stored at `Secret` created by the operator, it has the following name template `vmalertmanager-CRD_NAME-config`.
This configuration file is mounted at `VMAlertmanager` `Pod`. A special side-car container tracks its changes and sends config-reload signals to `alertmanager` container.
### Using secret
Basically, you can use the global configuration defined at manually created `Secret`. This `Secret` must be created before `VMAlertmanager`.
Name of the `Secret` must be defined at `VMAlertmanager` `spec.configSecret` option:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: vmalertmanager-example-alertmanager
labels:
app: vm-operator
type: Opaque
stringData:
alertmanager.yaml: |
global:
resolve_timeout: 5m
route:
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
replicaCount: 2
configSecret: vmalertmanager-example-alertmanager
```
### Using inline raw config
Also, if there is no secret data at configuration, or you just want to redefine some global variables for `alertmanager`.
You can define configuration at `spec.configRawYaml` section of `VMAlertmanager` configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
replicaCount: 2
configRawYaml: |
global:
resolve_timeout: 5m
route:
receiver: 'default'
group_interval: 5m
repeat_interval: 12h
receivers:
- name: 'default'
```
If both `configSecret` and `configRawYaml` are defined, only configuration from `configRawYaml` will be used. Values from `configRawYaml` will be ignored.
### Using VMAlertmanagerConfig
See details at [VMAlertmanagerConfig](./vmalertmanagerconfig.md).
The CRD specifies which `VMAlertmanagerConfig`s should be covered by the deployed `VMAlertmanager` instances based on label selection.
The Operator then generates a configuration based on the included `VMAlertmanagerConfig`s and updates the `Configmaps` containing
the configuration. It continuously does so for all changes that are made to `VMAlertmanagerConfig`s or to the `VMAlertmanager` resource itself.
Configs are filtered by selectors `configNamespaceSelector` and `configSelector` in `VMAlertmanager` CRD definition.
For selecting rules from all namespaces you must specify it to empty value:
```yaml
spec:
configNamespaceSelector: {}
```
[VMAlertmanagerConfig](./vmalertmanagerconfig.md) objects are
generates part of [VMAlertmanager](./vmalertmanager.md) configuration.
For filtering rules `VMAlertmanager` uses selectors `configNamespaceSelector` and `configSelector`.
It allows configuring rules access control across namespaces and different environments.
Specification of selectors you can see in [this doc](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta).
In addition to the above selectors, the filtering of objects in a cluster is affected by the field `selectAllByDefault`
of `VMAlertmanager` spec and environment variable `WATCH_NAMESPACE` for operator.
Following rules are applied:
- If `configNamespaceSelector` and `configSelector` both undefined, then by default select nothing. With option set - `spec.selectAllByDefault: true`, select all vmalertmanagerconfigs.
- If `configNamespaceSelector` defined, `configSelector` undefined, then all vmalertmaangerconfigs are matching at namespaces for given `configNamespaceSelector`.
- If `configNamespaceSelector` undefined, `configSelector` defined, then all vmalertmaangerconfigs at `VMAgent`'s namespaces are matching for given `configSelector`.
- If `configNamespaceSelector` and `configSelector` both defined, then only vmalertmaangerconfigs at namespaces matched `configNamespaceSelector` for given `configSelector` are matching.
Here's a more visual and more detailed view:
| `configNamespaceSelector` | `configSelector` | `selectAllByDefault` | `WATCH_NAMESPACE` | Selected rules |
|---------------------------|------------------|----------------------|-------------------|------------------------------------------------------------------------------------------------------------------------|
| undefined | undefined | false | undefined | nothing |
| undefined | undefined | **true** | undefined | all vmalertmaangerconfigs in the cluster |
| **defined** | undefined | any | undefined | all vmalertmaangerconfigs are matching at namespaces for given `configNamespaceSelector` |
| undefined | **defined** | any | undefined | all vmalertmaangerconfigs only at `VMAlertmanager`'s namespace are matching for given `ruleSelector` |
| **defined** | **defined** | any | undefined | all vmalertmaangerconfigs only at namespaces matched `configNamespaceSelector` for given `configSelector` are matching |
| any | undefined | any | **defined** | all vmalertmaangerconfigs only at `VMAlertmanager`'s namespace |
| any | **defined** | any | **defined** | all vmalertmaangerconfigs only at `VMAlertmanager`'s namespace for given `configSelector` are matching |
More details about `WATCH_NAMESPACE` variable you can read in [this doc](../configuration.md#namespaced-mode).
Here are some examples of `VMAlertmanager` configuration with selectors:
```yaml
# select all config objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: vmalertmanager-select-all
spec:
# ...
selectAllByDefault: true
---
# select all config objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: vmalertmanager-select-ns
spec:
# ...
configNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
```
### Extra configuration files
`VMAlertmanager` specification has the following fields, that can be used to configure without editing raw configuration file:
- `spec.templates` - list of keys in `ConfigMaps`, that contains template files for `alertmanager`, e.g.:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
replicaCount: 2
templates:
- Name: alertmanager-templates
Key: my-template-1.tmpl
- Name: alertmanager-templates
Key: my-template-2.tmpl
---
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-templates
data:
my-template-1.tmpl: |
{{ define "hello" -}}
hello, Victoria!
{{- end }}
my-template-2.tmpl: """
```
These templates will be automatically added to `VMAlertmanager` configuration and will be automatically reloaded on changes in source `ConfigMap`.
- `spec.configMaps` - list of `ConfigMap` names (in the same namespace) that will be mounted at `VMAlertmanager`
workload and will be automatically reloaded on changes in source `ConfigMap`. Mount path is `/etc/vm/configs/<configmap-name>`.
### Behavior without provided config
If no configuration is provided, operator configures stub configuration with blackhole route.
## High Availability
The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fire alerts against *all* instances of an Alertmanager cluster.
The Alertmanager, starting with the `v0.5.0` release, ships with a high availability mode.
It implements a gossip protocol to synchronize instances of an Alertmanager cluster
regarding notifications that have been sent out, to prevent duplicate notifications.
It is an AP (available and partition tolerant) system. Being an AP system means that notifications are guaranteed to be sent at least once.
The Victoria Metrics Operator ensures that Alertmanager clusters are properly configured to run highly available on Kubernetes.
## Version management
To set `VMAlertmanager` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-vmalertmanager
spec:
image:
repository: prom/alertmanager
tag: v0.25.0
pullPolicy: Always
# ...
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-vmalertmanager
spec:
image:
repository: prom/alertmanager
tag: v0.25.0
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: vmalertmanager-example
spec:
replicaCount: 1
configRawYaml: |
global:
resolve_timeout: 5m
route:
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://localhost:30502/'
```

101
docs/operator/resources/vmalertmanagerconfig.md Normal file
View file

@ -0,0 +1,101 @@
# VMAlertmanagerConfig
The `VMAlertmanagerConfig` provides way to configure [VMAlertmanager](./vmalertmanager.md)
configuration with CRD. It allows to define different configuration parts, which will be merged by operator into config.
It behaves like other config parts - `VMServiceScrape` and etc.
Read [Usage](#usage) and [Special case](#special-case) before using.
## Specification
You can see the full actual specification of the `VMAlertmanagerConfig` resource in
the **[API docs -> VMAlertmanagerConfig](../api.md#vmalertmanagerconfig)**.
Also, you can check out the [examples](#examples) section.
## Usage
`VMAlertmanagerConfig` allows delegating notification configuration to the kubernetes cluster users.
The application owner may configure notifications by defining it at `VMAlertmanagerConfig`.
With the combination of `VMRule` and `VMServiceScrape` it allows delegating configuration observability to application owners, and uses popular `GitOps` practice.
Operator combines `VMAlertmanagerConfig`s into a single configuration file for `VMAlertmanager`.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
name: example-email-web
namespace: production
spec:
route:
receiver: email
group_interval: 1m
routes:
- receiver: email
matchers:
- {severity =~ "warning|critical", app_name = "blog"}
receivers:
- name: email
email_configs:
- to: some-email@example.com
from: alerting@example.com
smarthost: example.com:25
text: ALARM
```
#### Special Case
VMAlertmanagerConfig has enforced namespace matcher.
Alerts must have a proper namespace label, with the same value as name of namespace for VMAlertmanagerConfig.
It can be disabled, by setting the following value to the VMAlertmanager: `spec.disableNamespaceMatcher: true`.
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
name: example
namespace: default
spec:
inhibit_rules:
- equals: []
target_matchers: []
source_matchers: []
route:
routes:
- receiver: webhook
continue: true
receiver: email
group_by: []
continue: false
matchers:
- job = "alertmanager"
group_wait: 30s
group_interval: 45s
repeat_interval: 1h
mute_time_intervals:
- name: base
time_intervals:
- times:
- start_time: ""
end_time: ""
weekdays: []
days_of_month: []
months: []
years: []
receivers:
email_configs: []
webhook_configs:
- url: http://some-other-wh
pagerduty_configs: []
pushover_configs: []
slack_configs: []
opsgenie_configs: []
victorops_configs: []
wechat_configs: []
```

237
docs/operator/resources/vmauth.md Normal file
View file

@ -0,0 +1,237 @@
# VMAuth
The `VMAuth` CRD provides mechanism for exposing application with authorization to outside world or to other applications inside kubernetes cluster.
For first case, user can configure `ingress` setting at `VMAuth` CRD. For second one, operator will create secret with `username` and `password` at `VMUser` CRD name.
So it will be possible to access these credentials from any application by targeting corresponding kubernetes secret.
## Specification
You can see the full actual specification of the `VMAuth` resource in
the **[API docs -> VMAuth](../api.md#vmauth)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## Users
The CRD specifies which `VMUser`s should be covered by the deployed `VMAuth` instances based on label selection.
The Operator then generates a configuration based on the included `VMUser`s and updates the `Configmaps` containing
the configuration. It continuously does so for all changes that are made to `VMUser`s or to the `VMAuth` resource itself.
[VMUser](./vmrule.md) objects generate part of `VMAuth` configuration.
For filtering users `VMAuth` uses selectors `userNamespaceSelector` and `userSelector`.
It allows configuring rules access control across namespaces and different environments.
Specification of selectors you can see in [this doc](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#labelselector-v1-meta).
In addition to the above selectors, the filtering of objects in a cluster is affected by the field `selectAllByDefault` of `VMAuth` spec and environment variable `WATCH_NAMESPACE` for operator.
Following rules are applied:
- If `userNamespaceSelector` and `userSelector` both undefined, then by default select nothing. With option set - `spec.selectAllByDefault: true`, select all vmusers.
- If `userNamespaceSelector` defined, `userSelector` undefined, then all vmusers are matching at namespaces for given `userNamespaceSelector`.
- If `userNamespaceSelector` undefined, `userSelector` defined, then all vmusers at `VMAgent`'s namespaces are matching for given `userSelector`.
- If `userNamespaceSelector` and `userSelector` both defined, then only vmusers at namespaces matched `userNamespaceSelector` for given `userSelector` are matching.
Here's a more visual and more detailed view:
| `userNamespaceSelector` | `userSelector` | `selectAllByDefault` | `WATCH_NAMESPACE` | Selected rules |
|-------------------------|----------------|----------------------|-------------------|------------------------------------------------------------------------------------------------------|
| undefined | undefined | false | undefined | nothing |
| undefined | undefined | **true** | undefined | all vmusers in the cluster |
| **defined** | undefined | any | undefined | all vmusers are matching at namespaces for given `userNamespaceSelector` |
| undefined | **defined** | any | undefined | all vmusers only at `VMAuth`'s namespace are matching for given `userSelector` |
| **defined** | **defined** | any | undefined | all vmusers only at namespaces matched `userNamespaceSelector` for given `userSelector` are matching |
| any | undefined | any | **defined** | all vmusers only at `VMAuth`'s namespace |
| any | **defined** | any | **defined** | all vmusers only at `VMAuth`'s namespace for given `userSelector` are matching |
More details about `WATCH_NAMESPACE` variable you can read in [this doc](../configuration.md#namespaced-mode).
Here are some examples of `VMAuth` configuration with selectors:
```yaml
# select all user objects in the cluster
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: vmauth-select-all
spec:
# ...
selectAllByDefault: true
---
# select all user objects in specific namespace (my-namespace)
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: vmauth-select-ns
spec:
# ...
userNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
```
## Unauthorized access
You can configure `VMAuth` to allow unauthorized access for specified routes with `unauthorizedAccessConfig` field.
For instance:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: vmauth-unauthorized-example
spec:
unauthorizedAccessConfig:
- paths: ["/metrics"]
urls:
- http://vmsingle-example.default.svc:8428
```
In this example every user can access `/metrics` route and get vmsingle metrics without authorization.
In addition, `unauthorizedAccessConfig` in [Enterprise version](#enterprise-features) supports [IP Filters](#ip-filters)
with `ip_filters` field.
## High availability
The `VMAuth` resource is stateless, so it can be scaled horizontally by increasing the number of replicas:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: vmauth-example
spec:
replicas: 3
# ...
```
## Version management
To set `VMAuth` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: example-vmauth
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
# ...
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: example-vmauth
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Enterprise features
Custom resource `VMAuth` supports feature [IP filters](https://docs.victoriametrics.com/vmauth.html#ip-filters)
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
For using Enterprise version of [vmauth](https://docs.victoriametrics.com/vmauth.html)
you need to change version of `vmauth` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMAuth`:
### IP Filters
After that you can use [IP filters for `VMUser`](./vmuser.md#enterprise-features)
and field `ip_filters` for `VMAuth`.
Here are complete example with described above:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: vmauth-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmauth
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmauth enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: ip filters for vmauth
# more details about ip filters you can read in https://docs.victoriametrics.com/vmauth.html#ip-filters
ip_filters:
allow_list:
- 10.0.0.0/24
- 1.2.3.4
deny_list:
- 5.6.7.8
# allow read vmsingle metrics without authorization for users from internal network
unauthorizedAccessConfig:
- paths: ["/metrics"]
urls: ["http://vmsingle-example.default.svc:8428"]
ip_filters:
allow_list:
- 192.168.0.0/16
- 10.0.0.0/8
# ...other fields...
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
name: vmuser-ent-example
spec:
username: simple-user
password: simple-password
# using enterprise features: ip filters for vmuser
# more details about ip filters you can read in https://docs.victoriametrics.com/vmuser.html#enterprise-features
ip_filters:
allow_list:
- 10.0.0.0/24
- 1.2.3.4
deny_list:
- 5.6.7.8
```
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: example
namespace: default
spec:
selectAllByDefault: true
ingress:
class_name: nginx # <-- change this to your ingress-controller
host: vm-demo.k8s.orb.local # <-- change this to your domain
```

612
docs/operator/resources/vmcluster.md Normal file
View file

@ -0,0 +1,612 @@
# VMCluster
`VMCluster` represents a high-available and fault-tolerant version of VictoriaMetrics database.
The `VMCluster` CRD defines a [cluster version VM](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
For each `VMCluster` resource, the Operator creates:
- `VMStorage` as `StatefulSet`,
- `VMSelect` as `StatefulSet`
- and `VMInsert` as deployment.
For `VMStorage` and `VMSelect` headless services are created. `VMInsert` is created as service with clusterIP.
There is a strict order for these objects creation and reconciliation:
1. `VMStorage` is synced - the Operator waits until all its pods are ready;
1. Then it syncs `VMSelect` with the same manner;
1. `VMInsert` is the last object to sync.
All [statefulsets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) are created
with [OnDelete](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#on-delete) update type.
It allows to manually manage the rolling update process for Operator by deleting pods one by one and waiting for the ready status.
Rolling update process may be configured by the operator env variables.
The most important is `VM_PODWAITREADYTIMEOUT=80s` - it controls how long to wait for pod's ready status.
## Specification
You can see the full actual specification of the `VMCluster` resource in the **[API docs -> VMCluster](../api.md#vmcluster)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## High availability
The cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling.
First, we recommend familiarizing yourself with the high availability tools provided by "VictoriaMetrics Cluster" itself:
- [High availability](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#high-availability),
- [Cluster availability](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-availability),
- [Replication and data safety](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#replication-and-data-safety).
`VMCluster` supports all listed in the above-mentioned articles parameters and features:
- `replicationFactor` - the number of replicas for each metric.
- for every component of cluster (`vmstorage` / `vmselect` / `vminsert`):
- `replicaCount` - the number of replicas for components of cluster.
- `affinity` - the affinity (the pod's scheduling constraints) for components pods. See more details in [kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
- `topologySpreadConstraints` - controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. See more details in [kubernetes docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/).
In addition, operator:
- uses k8s services or vmauth for load balancing between `vminsert` and `vmselect` components,
- uses health checks for to determine the readiness of components for work after restart,
- allows to horizontally scale all cluster components just by changing `replicaCount` field.
Here is an example of a `VMCluster` resource with HA features:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster-persistent
spec:
replicationFactor: 2
vmstorage:
replicaCount: 10
storageDataPath: "/vm-data"
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmstorage"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "2"
memory: 2048Mi
vmselect:
replicaCount: 3
cacheMountPath: "/select-cache"
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vmselect"
topologyKey: "kubernetes.io/hostname"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 2Gi
resources:
limits:
cpu: "1"
memory: "500Mi"
vminsert:
replicaCount: 4
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app.kubernetes.io/name"
operator: In
values:
- "vminsert"
topologyKey: "kubernetes.io/hostname"
resources:
limits:
cpu: "1"
memory: "500Mi"
```
## Version management
For `VMCluster` you can specify tag name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and repository setting per cluster object:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster
spec:
vmstorage:
replicaCount: 2
image:
repository: victoriametrics/vmstorage
tag: v1.93.4-cluster
pullPolicy: Always
vmselect:
replicaCount: 2
image:
repository: victoriametrics/vmselect
tag: v1.93.4-cluster
pullPolicy: Always
vminsert:
replicaCount: 2
image:
repository: victoriametrics/vminsert
tag: v1.93.4-cluster
pullPolicy: Always
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo,
but `imagePullSecrets` is global setting for all `VMCluster` specification:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster
spec:
vmstorage:
replicaCount: 2
image:
repository: victoriametrics/vmstorage
tag: v1.93.4-cluster
pullPolicy: Always
vmselect:
replicaCount: 2
image:
repository: victoriametrics/vmselect
tag: v1.93.4-cluster
pullPolicy: Always
vminsert:
replicaCount: 2
image:
repository: victoriametrics/vminsert
tag: v1.93.4-cluster
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Enterprise features
VMCluster supports following features
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise):
- [Downsampling](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling)
- [Multiple retentions / Retention filters](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters)
- [Advanced per-tenant statistic](https://docs.victoriametrics.com/PerTenantStatistic.html)
- [mTLS for cluster components](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection)
- [Backup automation](https://docs.victoriametrics.com/vmbackupmanager.html)
VMCluster doesn't support yet feature
[Automatic discovery for vmstorage nodes](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#automatic-vmstorage-discovery).
For using Enterprise version of [vmcluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html)
you need to change version of `VMCluster` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMCluster`.
### Downsampling
After that you can pass [Downsampling](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling)
flag to `VMCluster/vmselect` and `VMCluster/vmstorage` with [extraArgs](./README.md#extra-arguments) too.
Here are complete example for [Downsampling](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-ent-example
spec:
vmselect:
# enabling enterprise features for vmselect
image:
# enterprise version of vmselect
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmselect enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Downsampling
# more details about downsampling you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling
downsampling.period: 30d:5m,180d:1h,1y:6h,2y:1d
vmstorage:
# enabling enterprise features for vmstorage
image:
# enterprise version of vmstorage
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmstorage enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Downsampling
# more details about downsampling you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#downsampling
downsampling.period: 30d:5m,180d:1h,1y:6h,2y:1d
# ...other fields...
```
### Retention filters
You can pass [Retention filters](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters)
flag to `VMCluster/vmstorage` with [extraArgs](./README.md#extra-arguments).
Here are complete example for [Retention filters](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-ent-example
spec:
vmstorage:
# enabling enterprise features for vmstorage
image:
# enterprise version of vmstorage
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmstorage enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Retention filters
# more details about retention filters you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#retention-filters
retentionFilter: '{vm_account_id="5",env="dev"}:5d,{vm_account_id="5",env="prod"}:5y'
# ...other fields...
```
### Advanced per-tenant statistic
For using [Advanced per-tenant statistic](https://docs.victoriametrics.com/PerTenantStatistic.html)
you only need to [enable Enterprise version of vmcluster components](#enterprise-features)
and operator will automatically create
[Scrape objects](./vmagent.md#scraping) for cluster components.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-ent-example
spec:
vmselect:
# enabling enterprise features for vmselect
image:
# enterprise version of vmselect
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmselect enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
vminsert:
# enabling enterprise features for vminsert
image:
# enterprise version of vminsert
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vminsert enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
vmstorage:
# enabling enterprise features for vmstorage
image:
# enterprise version of vmstorage
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmstorage enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# ...other fields...
```
After that [VMAgent](./vmagent.md) will automatically
scrape [Advanced per-tenant statistic](https://docs.victoriametrics.com/PerTenantStatistic.html) for cluster components.
### mTLS protection
You can pass [mTLS protection](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection)
flags to `VMCluster/vmstorage`, `VMCluster/vmselect` and `VMCluster/vminsert` with [extraArgs](./README.md#extra-arguments) and mount secret files
with `extraVolumes` and `extraVolumeMounts` fields.
Here are complete example for [mTLS protection](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-ent-example
spec:
vmselect:
# enabling enterprise features for vmselect
image:
# enterprise version of vmselect
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vmselect enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: mTLS protection
# more details about mTLS protection you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
cluster.tls: true
cluster.tlsCAFile: /etc/mtls/ca.crt
cluster.tlsCertFile: /etc/mtls/vmselect.crt
cluster.tlsKeyFile: /etc/mtls/vmselect.key
extraVolumes:
- name: mtls
secret:
secretName: mtls
extraVolumeMounts:
- name: mtls
mountPath: /etc/mtls
vminsert:
# enabling enterprise features for vminsert
image:
# enterprise version of vminsert
tag: v1.93.5-enterprise-cluster
extraArgs:
# should be true and means that you have the legal right to run a vminsert enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: mTLS protection
# more details about mTLS protection you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
cluster.tls: true
cluster.tlsCAFile: /etc/mtls/ca.crt
cluster.tlsCertFile: /etc/mtls/vminsert.crt
cluster.tlsKeyFile: /etc/mtls/vminsert.key
extraVolumes:
- name: mtls
secret:
secretName: mtls
extraVolumeMounts:
- name: mtls
mountPath: /etc/mtls
vmstorage:
# enabling enterprise features for vmstorage
image:
# enterprise version of vmstorage
tag: v1.93.5-enterprise-cluster
env:
- name: POD
valueFrom:
fieldRef:
fieldPath: metadata.name
extraArgs:
# should be true and means that you have the legal right to run a vmstorage enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: mTLS protection
# more details about mTLS protection you can read on https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#mtls-protection
cluster.tls: true
cluster.tlsCAFile: /etc/mtls/ca.crt
cluster.tlsCertFile: /etc/mtls/$(POD).crt
cluster.tlsKeyFile: /etc/mtls/$(POD).key
extraVolumes:
- name: mtls
secret:
secretName: mtls
extraVolumeMounts:
- name: mtls
mountPath: /etc/mtls
# ...other fields...
---
apiVersion: v1
kind: Secret
metadata:
name: mtls
namespace: default
stringData:
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
mtls-vmstorage-0.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
mtls-vmstorage-0.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
mtls-vmstorage-1.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
mtls-vmstorage-1.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
vminsert.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
vminsert.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
vmselect.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
vmselect.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
```
Example commands for generating certificates you can read
on [this page](https://gist.github.com/f41gh7/76ed8e5fb1ebb9737fe746bae9175ee6#generate-self-signed-ca-with-key).
### Backup automation
You can check [vmbackupmanager documentation](https://docs.victoriametrics.com/vmbackupmanager.html) for backup automation.
It contains a description of the service and its features. This section covers vmbackumanager integration in vmoperator.
`VMCluster` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups.
It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage).
Here is a complete example for backup configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-ent-example
spec:
vmstorage:
vmBackup:
# should be true and means that you have the legal right to run a vmstorage enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
acceptEULA: true
# using enterprise features: Backup automation
# more details about backup automation you can read on https://docs.victoriametrics.com/vmbackupmanager.html
destination: "s3://your_bucket/folder"
credentialsSecret:
name: remote-storage-keys
key: credentials
# ...other fields...
---
apiVersion: v1
kind: Secret
metadata:
name: remote-storage-keys
type: Opaque
stringData:
credentials: |-
[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
```
**NOTE**: for cluster version operator adds suffix for destination: `"s3://your_bucket/folder"`, it becomes `"s3://your_bucket/folder/$(POD_NAME)"`.
It's needed to make consistent backups for each storage node.
You can read more about backup configuration options and mechanics [here](https://docs.victoriametrics.com/vmbackupmanager.html)
Possible configuration options for backup crd can be found at [link](../api.md#vmbackup)
**Using VMBackupmanager for restoring backups** in Kubernetes environment is described [here](https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-in-kubernetes).
Also see VMCLuster example spec [here](https://github.com/VictoriaMetrics/operator/blob/master/config/examples/vmcluster_with_backuper.yaml).
## Examples
### Minimal example without persistence
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: vmcluster-example-minimal
spec:
# ...
retentionPeriod: "1"
vmstorage:
replicaCount: 2
vmselect:
replicaCount: 2
vminsert:
replicaCount: 2
```
### With persistence
```yaml
kind: VMCluster
metadata:
name: vmcluster-example-persistent
spec:
# ...
retentionPeriod: "4"
replicationFactor: 2
vmstorage:
replicaCount: 2
storageDataPath: "/vm-data"
storage:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "0.5"
memory: 500Mi
vmselect:
replicaCount: 2
cacheMountPath: "/select-cache"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 2Gi
resources:
limits:
cpu: "0.3"
memory: "300Mi"
vminsert:
replicaCount: 2
```

46
docs/operator/resources/vmnodescrape.md Normal file
View file

@ -0,0 +1,46 @@
# VMNodeScrape
The `VMNodeScrape` CRD provides discovery mechanism for scraping metrics kubernetes nodes,
it is useful for node exporters monitoring.
`VMNodeScrape` object generates part of [VMAgent](./vmagent.md) configuration.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc.).
By specifying configuration at CRD, operator generates config
for [VMAgent](./vmagent.md) and syncs it. It's useful for cadvisor scraping,
node-exporter or other node-based exporters. `VMAgent` `nodeScrapeSelector` must match `VMNodeScrape` labels.
More information about selectors you can find in [this doc](./vmagent.md#scraping).
## Specification
You can see the full actual specification of the `VMNodeScrape` resource in
the **[API docs -> VMNodeScrape](../api.md#vmnodescrape)**.
Also, you can check out the [examples](#examples) section.
## Examples
### Cadvisor scraping
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
name: cadvisor-metrics
spec:
scheme: "https"
tlsConfig:
insecureSkipVerify: true
caFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token"
relabelConfigs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- targetLabel: __address__
replacement: kubernetes.default.svc:443
- sourceLabels: [__meta_kubernetes_node_name]
regex: (.+)
targetLabel: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
```

64
docs/operator/resources/vmpodscrape.md Normal file
View file

@ -0,0 +1,64 @@
# VMPodScrape
The `VMPodScrape` CRD allows to declaratively define how a dynamic set of pods should be monitored.
Use label selections to match pods for scraping. This allows an organization to introduce conventions
for how metrics should be exposed. Following these conventions new services will be discovered automatically without
need to reconfigure.
`VMPodScrape` object generates part of [VMAgent](./vmagent.md) configuration with
[kubernetes service discovery](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) role `pod` having specific labels and ports.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc.).
A `Pod` is a collection of one or more containers which can expose Prometheus metrics on a number of ports.
The `VMPodScrape` object discovers pods and generates the relevant scraping configuration.
The `PodMetricsEndpoints` section of the `VMPodScrapeSpec` is used to configure which ports of a pod are going to be
scraped for metrics and with which parameters.
Both `VMPodScrapes` and discovered targets may belong to any namespace. It is important for cross-namespace monitoring
use cases, e.g. for meta-monitoring. Using the `namespaceSelector` of the `VMPodScrapeSpec` one can restrict the
namespaces from which `Pods` are discovered from. To discover targets in all namespaces the `namespaceSelector` has to
be empty:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: example-pod-scrape
spec:
namespaceSelector:
any: true
```
More information about selectors you can find in [this doc](./vmagent.md#scraping).
## Specification
You can see the full actual specification of the `VMPodScrape` resource in
the **[API docs -> VMPodScrape](../api.md#vmpodscrape)**.
Also, you can check out the [examples](#examples) section.
## Migration from Prometheus
The `VMPodScrape` CRD from VictoriaMetrics Operator is a drop-in replacement
for the Prometheus `PodMonitor` from prometheus-operator.
More details about migration from prometheus-operator you can read in [this doc](../migration.md).
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: example-pod-scrape
spec:
podMetricsEndpoints:
- port: web
scheme: http
selector:
matchLabels:
owner: dev
```

226
docs/operator/resources/vmprobe.md Normal file
View file

@ -0,0 +1,226 @@
# VMProbe
The `VMProbe` CRD provides probing target ability with some external prober.
The most common prober is [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
By specifying configuration at CRD, operator generates config for [VMAgent](./vmagent.md)
and syncs it. It's possible to use static targets or use standard k8s discovery mechanism with `Ingress`.
`VMProbe` object generates part of [VMAgent](./vmagent.md) configuration;
It has various options for scraping configuration of target (with basic auth, tls access, by specific port name etc.).
You have to configure blackbox exporter before you can use this feature.
The second requirement is [VMAgent](./vmagent.md) selectors,
it must match your `VMProbe` by label or namespace selector. `VMAgent` `probeSelector` must match `VMProbe` labels.
See more details about selectors [here](./vmagent.md#scraping).
## Specification
You can see the full actual specification of the `VMProbe` resource in
the **[API docs -> VMProbe](../api.md#vmprobe)**.
Also, you can check out the [examples](#examples) section.
## Migration from Prometheus
The `VMProbe` CRD from VictoriaMetrics Operator is a drop-in replacement
for the Prometheus `Probe` from prometheus-operator.
More details about migration from prometheus-operator you can read in [this doc](../migration.md).
## Examples
### Static targets
It will probe `VMAgent` with url - `vmagent-example-vmagent.default.svc:9115/heath` with blackbox url:
`prometheus-blackbox-exporter.default.svc:9115` and module `http_2xx`
(it was specified at [blackbox configmap](#blackbox-exporter)).
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
name: vmprobe-static-example
spec:
jobName: static-probe
vmProberSpec:
# by default scheme http, and path is /probe
url: prometheus-blackbox-exporter.default.svc:9115
module: http_2xx
targets:
staticConfig:
targets:
- vmagent-example-vmagent.default.svc:8429/health
interval: 2s
```
After adding target to `VMAgent` configuration it starts probing itself throw blackbox exporter.
### Ingress targets
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
name: vmprobe-ingress-example
spec:
vmProberSpec:
# by default scheme http, and path is /probe
url: prometheus-blackbox-exporter.default.svc:9115
module: http_2xx
targets:
ingress:
selector:
matchLabels:
app: victoria-metrics-single
interval: 10s
```
This configuration will add 2 additional targets for probing: `vmsingle2.example.com` and `vmsingle.example.com`.
But probes will be unsuccessful, because there is no such hosts.
### Related resources
Following resources will be used for the examples below:
#### Blackbox exporter
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
data:
blackbox.yaml: |
modules:
http_2xx:
http:
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/1.1
- HTTP/2.0
valid_status_codes: []
prober: http
timeout: 5s
---
kind: Service
apiVersion: v1
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
protocol: TCP
selector:
app: prometheus-blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-blackbox-exporter
template:
metadata:
labels:
app: prometheus-blackbox-exporter
spec:
containers:
- name: blackbox-exporter
image: "prom/blackbox-exporter:v0.17.0"
args:
- "--config.file=/config/blackbox.yaml"
resources:
{}
ports:
- containerPort: 9115
name: http
livenessProbe:
httpGet:
path: /health
port: http
readinessProbe:
httpGet:
path: /health
port: http
volumeMounts:
- mountPath: /config
name: config
volumes:
- name: config
configMap:
name: prometheus-blackbox-exporter
```
### VMSingle
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
removePvcAfterDelete: true
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
labels:
app: victoria-metrics-single
name: victoria-metrics-single
spec:
rules:
- host: vmsingle.example.com
http:
paths:
- backend:
serviceName: vmsingle-example-vmsingle-persisted
servicePort: 8428
path: /
- host: vmsingle2.example.com
http:
paths:
- backend:
serviceName: vmsingle-example-vmsingle-persisted
servicePort: 8428
path: /
```
### VMAgent
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
selectAllByDefault: true
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
```

99
docs/operator/resources/vmrule.md Normal file
View file

@ -0,0 +1,99 @@
# VMRule
`VMRule` represents [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) rules
for [VMAlert](./vmalert.md) instances.
The `VMRule` CRD declaratively defines a desired Prometheus rule to be consumed by one or more VMAlert instances.
`VMRule` object generates [VMAlert](./vmalert.md)
with ruleset defined at `VMRule` spec.
Alerts and recording rules can be saved and applied as YAML files, and dynamically loaded without requiring any restart.
See more details about rule configuration in [VMAlert docs](https://docs.victoriametrics.com/vmalert.html#quickstart).
## Specification
You can see the full actual specification of the `VMRule` resource in
the **[API docs -> VMRule](../api.md#vmrule)**.
Also, you can check out the [examples](#examples) section.
## Enterprise features
Custom resource `VMRule` supports feature [Multitenancy](https://docs.victoriametrics.com/vmalert.html#multitenancy)
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
### Multitenancy
For using [Multitenancy](https://docs.victoriametrics.com/vmalert.html#multitenancy) in `VMRule`
you need to **[enable VMAlert Enterprise](./vmalert.md#enterprise-features)**.
After that you can add `tenant` field for groups in `VMRule`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: vmrule-ent-example
spec:
groups:
- name: vmalert-1
rules:
# using enterprise features: Multitenancy
# more details about multitenancy you can read on https://docs.victoriametrics.com/vmalert.html#multitenancy
- tenant: 1
alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
job: "{{ $labels.job }}"
annotations:
value: "{{ $value }}"
description: 'error reloading vmalert config, reload count for 5 min {{ $value }}'
```
## Examples
### Alerting rule
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: vmrule-alerting-example
spec:
groups:
- name: vmalert
rules:
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
job: "{{ $labels.job }}"
annotations:
value: "{{ $value }}"
description: 'error reloading vmalert config, reload count for 5 min {{ $value }}'
```
### Recording rule
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: vmrule-recording-example
spec:
groups:
- name: vmalert
interval: 1m
rules:
- alert: vmalert config reload error
expr: |-
sum by (cluster, namespace, job) (
rate(vm_http_request_errors_total[5m])
)
```

77
docs/operator/resources/vmservicescrape.md Normal file
View file

@ -0,0 +1,77 @@
# VMServiceScrape
The `VMServiceScrape` CRD allows to define a dynamic set of services for monitoring. Services
and scraping configurations can be matched via label selections. This allows an organization to introduce conventions
for how metrics should be exposed. Following these conventions new services will be discovered automatically without
need to reconfigure.
`VMServiceScrape` object generates part of [VMAgent](./vmagent.md) configuration with
[kubernetes service discovery](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) targets by corresponding `Service`.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc.).
Monitoring configuration based on `discoveryRole` setting. By default, `endpoints` is used to get objects from kubernetes api.
It's also possible to use `discoveryRole: service` or `discoveryRole: endpointslices`.
`Endpoints` objects are essentially lists of IP addresses.
Typically, `Endpoints` objects are populated by `Service` object. `Service` object discovers `Pod`s by a label
selector and adds those to the `Endpoints` object.
A `Service` may expose one or more service ports backed by a list of one or multiple endpoints pointing to
specific `Pod`s. The same reflected in the respective `Endpoints` object as well.
The `VMServiceScrape` object discovers `Endpoints` objects and configures [VMAgent](./vmagent.md) to monitor `Pod`s.
The `Endpoints` section of the `VMServiceScrapeSpec` is used to configure which `Endpoints` ports should be scraped.
For advanced use cases, one may want to monitor ports of backing `Pod`s, which are not a part of the service endpoints.
Therefore, when specifying an endpoint in the `endpoints` section, they are strictly used.
**Note:** `endpoints` (lowercase) is the field in the `VMServiceScrape` CRD, while `Endpoints` (capitalized) is the Kubernetes object kind.
Both `VMServiceScrape` and discovered targets may belong to any namespace. It is important for cross-namespace monitoring
use cases, e.g. for meta-monitoring. Using the `serviceScrapeSelector` of the `VMAgentSpec`
one can restrict the namespaces from which `VMServiceScrape`s are selected from by the respective [VMAgent](./vmagent.md) server.
Using the `namespaceSelector` of the `VMServiceScrape` one can restrict the namespaces from which `Endpoints` can be
discovered from. To discover targets in all namespaces the `namespaceSelector` has to be empty:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: example-service-scrape
spec:
namespaceSelector: {}
# ...
```
More information about selectors you can find in [this doc](./vmagent.md#scraping).
## Specification
You can see the full actual specification of the `VMServiceScrape` resource in
the **[API docs -> VMServiceScrape](../api.md#vmservicescrape)**.
Also, you can check out the [examples](#examples) section.
## Migration from Prometheus
The `VMServiceScrape` CRD from VictoriaMetrics Operator is a drop-in replacement
for the Prometheus `ServiceMonitor` from prometheus-operator.
More details about migration from prometheus-operator you can read in [this doc](../migration.md).
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: example-app
labels:
team: frontend
spec:
selector:
matchLabels:
app: example-app
endpoints:
- port: web
```

282
docs/operator/resources/vmsingle.md Normal file
View file

@ -0,0 +1,282 @@
# VMSingle
`VMSingle` represents database for storing metrics.
The `VMSingle` CRD declaratively defines a [single-node VM](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
installation to run in a Kubernetes cluster.
For each `VMSingle` resource, the Operator deploys a properly configured `Deployment` in the same namespace.
The VMSingle `Pod`s are configured to mount an empty dir or `PersistentVolumeClaimSpec` for storing data.
Deployment update strategy set to [recreate](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#recreate-deployment).
No more than one replica allowed.
For each `VMSingle` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with name from `VMSingle.metadata.name`.
## Specification
You can see the full actual specification of the `VMSingle` resource in the **[API docs -> VMSingle](../api.md#vmsingle)**.
If you can't find necessary field in the specification of the custom resource,
see [Extra arguments section](./README.md#extra-arguments).
Also, you can check out the [examples](#examples) section.
## High availability
`VMSingle` doesn't support high availability by default, for such purpose
use [`VMCluster`](./vmcluster.md) instead or duplicate the setup.
## Version management
To set `VMSingle` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
# ...
```
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle
spec:
image:
repository: victoriametrics/victoria-metrics
tag: v1.93.4
pullPolicy: Always
imagePullSecrets:
- name: my-repo-secret
# ...
```
## Enterprise features
VMSingle supports features from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise):
- [Downsampling](https://docs.victoriametrics.com/#downsampling)
- [Multiple retentions / Retention filters](https://docs.victoriametrics.com/#retention-filters)
- [Backup automation](https://docs.victoriametrics.com/vmbackupmanager.html)
For using Enterprise version of [vmsingle](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
you need to change version of `VMSingle` to version with `-enterprise` suffix using [Version management](#version-management).
All the enterprise apps require `-eula` command-line flag to be passed to them.
This flag acknowledges that your usage fits one of the cases listed on [this page](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
So you can use [extraArgs](./README.md#extra-arguments) for passing this flag to `VMSingle`.
### Downsampling
After that you can pass [Downsampling](https://docs.victoriametrics.com/#downsampling)
flag to `VMSingle` with [extraArgs](./README.md#extra-arguments) too.
Here are complete example for [Downsampling](https://docs.victoriametrics.com/#downsampling):
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: vmsingle-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmsingle
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmsingle enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Downsampling
# more details about downsampling you can read on https://docs.victoriametrics.com/#downsampling
downsampling.period: 30d:5m,180d:1h,1y:6h,2y:1d
# ...other fields...
```
### Retention filters
The same method is used to enable retention filters - here are complete example for [Retention filters](https://docs.victoriametrics.com/#retention-filters).
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: vmsingle-ent-example
spec:
# enabling enterprise features
image:
# enterprise version of vmsingle
tag: v1.93.5-enterprise
extraArgs:
# should be true and means that you have the legal right to run a vmsingle enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
eula: true
# using enterprise features: Retention filters
# more details about retention filters you can read on https://docs.victoriametrics.com/#retention-filters
retentionFilter: '{team="juniors"}:3d,{env=~"dev|staging"}:30d'
# ...other fields...
```
### Backup automation
You can check [vmbackupmanager documentation](https://docs.victoriametrics.com/vmbackupmanager.html) for backup automation.
It contains a description of the service and its features. This section covers vmbackumanager integration in vmoperator.
`VMSingle` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups.
It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage).
Here is a complete example for backup configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle
spec:
vmBackup:
# should be true and means that you have the legal right to run a vmsingle enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
acceptEULA: true
# using enterprise features: Backup automation
# more details about backup automation you can read on https://docs.victoriametrics.com/vmbackupmanager.html
destination: "s3://your_bucket/folder"
credentialsSecret:
name: remote-storage-keys
key: credentials
# ...other fields...
---
apiVersion: v1
kind: Secret
metadata:
name: remote-storage-keys
type: Opaque
stringData:
credentials: |-
[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
```
You can read more about backup configuration options and mechanics [here](https://docs.victoriametrics.com/vmbackupmanager.html)
Possible configuration options for backup crd can be found at [link](../api.md#vmbackup)
#### Restoring backups
There are several ways to restore with [vmrestore](https://docs.victoriametrics.com/vmrestore.html) or [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html).
##### Manually mounting disk
You have to stop `VMSingle` by scaling it replicas to zero and manually restore data to the database directory.
Steps:
1. Edit `VMSingle` CRD, set `replicaCount: 0`
1. Wait until database stops
1. SSH to some server, where you can mount `VMSingle` disk and mount it manually
1. Restore files with `vmrestore`
1. Umount disk
1. Edit `VMSingle` CRD, set `replicaCount: 1`
1. Wait database start
##### Using VMRestore init container
1. Add init container with `vmrestore` command to `VMSingle` CRD, example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle
spec:
vmBackup:
# should be true and means that you have the legal right to run a vmsingle enterprise
# that can either be a signed contract or an email with confirmation to run the service in a trial period
# https://victoriametrics.com/legal/esa/
acceptEULA: true
# using enterprise features: Backup automation
# more details about backup automation you can read on https://docs.victoriametrics.com/vmbackupmanager.html
destination: "s3://your_bucket/folder"
credentialsSecret:
name: remote-storage-keys
key: credentials
extraArgs:
runOnStart: "true"
initContainers:
- name: vmrestore
image: victoriametrics/vmrestore:latest
volumeMounts:
- mountPath: /victoria-metrics-data
name: data
- mountPath: /etc/vm/creds
name: secret-remote-storage-keys
readOnly: true
args:
- -storageDataPath=/victoria-metrics-data
- -src=s3://your_bucket/folder/latest
- -credsFilePath=/etc/vm/creds/credentials
# ...other fields...
```
1. Apply it, and db will be restored from S3
1. Remove `initContainers` and apply CRD.
Note that using `VMRestore` will require adjusting `src` for each pod because restore will be handled per-pod.
##### Using VMBackupmanager init container
Using VMBackupmanager restore in Kubernetes environment is described [here](https://docs.victoriametrics.com/vmbackupmanager.html#how-to-restore-in-kubernetes).
Advantages of using `VMBackupmanager` include:
- Automatic adjustment of `src` for each pod when backup is requested
- Graceful handling of case when no restore is required - `VMBackupmanager` will exit with successful status code and won't prevent pod from starting
## Examples
```yaml
kind: VMSingle
metadata:
name: vmsingle-example
spec:
retentionPeriod: "12"
removePvcAfterDelete: true
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
extraArgs:
dedup.minScrapeInterval: 60s
resources:
requests:
memory: 500Mi
cpu: 500m
limits:
memory: 10Gi
cpu: 5
```

37
docs/operator/resources/vmstaticscrape.md Normal file
View file

@ -0,0 +1,37 @@
# VMStaticScrape
The `VMStaticScrape` CRD provides mechanism for scraping metrics from static targets, configured by CRD targets.
`VMStaticScrape` object generates part of [VMAgent](./vmagent.md)
configuration with [static "service discovery"](https://docs.victoriametrics.com/sd_configs.html#static_configs).
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc.).
By specifying configuration at CRD, operator generates config
for [VMAgent](./vmagent.md) and syncs it.
It's useful for external targets management, when service-discovery is not available.
`VMAgent` `staticScrapeSelector` must match `VMStaticScrape` labels.
More information about selectors you can find in [this doc](./vmagent.md#scraping).
## Specification
You can see the full actual specification of the `VMStaticScrape` resource in
the **[API docs -> VMStaticScrape](../api.md#vmstaticscrape)**.
Also, you can check out the [examples](#examples) section.
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMStaticScrape
metadata:
name: vmstaticscrape-sample
spec:
jobName: static
targetEndpoints:
- targets: ["192.168.0.1:9100", "196.168.0.50:9100"]
labels:
env: dev
project: operator
```

135
docs/operator/resources/vmuser.md Normal file
View file

@ -0,0 +1,135 @@
# VMUser
The `VMUser` CRD describes user configuration, its authentication methods `basic auth` or `Authorization` header.
User access permissions, with possible routing information.
User can define routing target with `static` config, by entering target `url`, or with `CRDRef`, in this case,
operator queries kubernetes API, retrieves information about CRD and builds proper url.
## Specification
You can see the full actual specification of the `VMUser` resource in
the **[API docs -> VMUser](../api.md#vmuser)**.
Also, you can check out the [examples](#examples) section.
## Authentication methods
There are two authentication mechanisms: ["Bearer token"](#bearer-token) and ["Basic auth"](#basic-auth) with `username` and `password`.
Only one of them can be used with `VMUser` at one time.
Operator creates `Secret` for every `VMUser` with name - `vmuser-{VMUser.metadata.name}`.
It places `username` + `password` or `bearerToken` into `data` section.
### Bearer token
Bearer token is a way to authenticate user with `Authorization` header.
User defines `token` field in `auth` section.
Also, you can check out the [examples](#examples) section.
### Basic auth
Basic auth is the simplest way to authenticate user. User defines `username` and `password` fields in `auth` section.
If `username` is empty, `metadata.name` from `VMUser` used as `username`.
You can automatically generate `password` if:
- Set `generatePassword: true` field
- Don't fill `password` field
Operator generates random password for this `VMUser`,
this password will be added to the `Secret` for this `VMUser` at `data.password` field.
Also, you can check out the [examples](#examples) section.
## Routing
You can define routes for user in `targetRefs` section.
For every entry in `targetRefs` you can define routing target with `static` config, by entering target `url`,
or with `crd`, in this case, operator queries kubernetes API, retrieves information about CRD and builds proper url.
Here are details about other fields in `targetRefs`:
- `paths` is the same as `src_paths` from [auth config](https://docs.victoriametrics.com/vmauth.html#auth-config)
- `headers` is the same as `headers` from [auth config](https://docs.victoriametrics.com/vmauth.html#auth-config)
- `targetPathSuffix` is the suffix for `url_prefix` (target URL) from [auth config](https://docs.victoriametrics.com/vmauth.html#auth-config)
### Static
The `static` field is the same as `url_prefix` (target URL) from [auth config](https://docs.victoriametrics.com/vmauth.html#auth-config),
it allows you to set a specific static URL.
### CRDRef
The `crd` field is a more convenient form for specifying the components handled by the operator as auth targets.
User can define routing target with `crd` config, by entering `kind`, `name` and `namespace` of CRD.
Operator supports following kinds in `kind` field:
- `VMAgent` for [VMAgent](./vmagent.md)
- `VMAlert` for [VMAlert](./vmalert.md)
- `VMAlertmanager` for [VMAlertmanager](./vmalertmanager.md)
- `VMSingle` for [VMSingle](./vmsingle.md)
- `VMCluster/vmselect`, `VMCluster/vminsert` and `VMCluster/vmstorage` for [VMCluster](./vmcluster.md)
Also, you can check out the [examples](#examples) section.
Additional fields like `path` and `scheme` can be added to `CRDRef` config.
## Enterprise features
Custom resource `VMUser` supports feature [IP filters](https://docs.victoriametrics.com/vmauth.html#ip-filters)
from [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise.html#victoriametrics-enterprise).
### IP Filters
For using [IP filters](https://docs.victoriametrics.com/vmauth.html#ip-filters)
you need to **[enable VMAuth Enterprise](./vmauth.md#enterprise-features)**.
After that you can add `ip_filters` field to `VMUser`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
name: vmuser-ent-example
spec:
username: simple-user
password: simple-password
# using enterprise features: ip filters for vmuser
# more details about ip filters you can read in https://docs.victoriametrics.com/vmuser.html#enterprise-features
ip_filters:
allow_list:
- 10.0.0.0/24
- 1.2.3.4
deny_list:
- 5.6.7.8
```
## Examples
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
name: example
spec:
username: simple-user
password: simple-password
targetRefs:
- crd:
kind: VMSingle
name: example
namespace: default
paths: ["/.*"]
- static:
url: http://vmalert-example.default.svc:8080
paths: ["/api/v1/groups","/api/v1/alerts"]
```
More examples see on [Authorization and exposing components](../auth.md) page
and in [Quickstart guide](../quick-start.md#vmuser).

79
docs/operator/security.md
View file

@ -1,23 +1,43 @@
---
sort: 12
weight: 12
sort: 3
weight: 3
title: Security
menu:
docs:
parent: "operator"
weight: 12
aliases:
- /operator/security.html
---
# Security
VictoriaMetrics operator provides several security features, such as [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), [PodSecurityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).
## Access control
### Roles
## PodSecurityPolicy.
To run in a cluster the operator needs certain permissions, you can see them in [this directory](https://github.com/VictoriaMetrics/operator/tree/master/config/rbac):
By default, operator creates serviceAccount for each cluster resource and binds default `PodSecurityPolicy` to it.
- [`role.yaml` file](https://github.com/VictoriaMetrics/operator/blob/master/config/rbac/role.yaml) - basic set of cluster roles for launching an operator.
- [`leader_election_role.yaml` file](https://github.com/VictoriaMetrics/operator/blob/master/config/rbac/leader_election_role.yaml) - set of roles with permissions to do leader election (is necessary to run the operator in several replicas for high availability).
Also, you can use single-namespace mode with minimal permissions, see [this section](./configuration.md#namespaced-mode) for details.
Also in [the same directory](https://github.com/VictoriaMetrics/operator/tree/master/config/rbac) are files with a set of separate permissions to view or edit [operator resources](./resources/README.md) to organize fine-grained access:
- file `<RESOURCE_NAME>_viewer_role.yaml` - permissions for viewing (`get`, `list` and `watch`) some resource of vmoperator.
- file `<RESOURCE_NAME>_editor_role.yaml` - permissions for editing (`create`, `delete`, `patch`, `update` and `deletecollection`) some resource of vmoperator (also includes viewing permissions).
For instance, [`vmalert_editor_role.yaml` file](https://github.com/VictoriaMetrics/operator/blob/master/config/rbac/vmalert_editor_role.yaml) contain permission
for editing [`vmagent` custom resources](./resources/vmagent.md).
<!-- TODO: service accounts / role bindings? -->
<!-- TODO: resource/roles relations -->
## Security policies
VictoriaMetrics operator provides several security features, such as [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/),
[PodSecurityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).
### PodSecurityPolicy
> PodSecurityPolicy was [deprecated](https://kubernetes.io/docs/concepts/security/pod-security-policy/) in Kubernetes v1.21, and removed from Kubernetes in v1.25.
If your Kubernetes version is under v1.25 and want to use PodSecurityPolicy, you can set env `VM_PSPAUTOCREATEENABLED: "true"` in operator, it will create serviceAccount for each cluster resource and binds default `PodSecurityPolicy` to it.
Default psp:
```yaml
@ -48,18 +68,38 @@ spec:
- nfs
```
This behaviour may be disabled with env variable passed to operator:
```yaml
- name: VM_PSPAUTOCREATEENABLED
value: "false"
```
User may also override default pod security policy with setting: `spec.podSecurityPolicyName: "psp-name"`.
## PodSecurityContext
`PodSecurityContext` can be configured with spec setting. It may be useful for mounted volumes, with `VMSingle` for example:
VictoriaMetrics operator will add default Security Context to managed pods and containers if env `EnableStrictSecurity: "true"` is set.
The following SecurityContext will be applied:
### Pod SecurityContext
1. **RunAsNonRoot: true**
1. **RunAsUser/RunAsGroup/FSGroup: 65534**
'65534' refers to 'nobody' in all the used default images like alpine, busybox.
If you're using customize image, please make sure '65534' is a valid uid in there or specify SecurityContext.
1. **FSGroupChangePolicy: &onRootMismatch**
If KubeVersion>=1.20, use `FSGroupChangePolicy="onRootMismatch"` to skip the recursive permission change
when the root of the volume already has the correct permissions
1. **SeccompProfile: {type: RuntimeDefault}**
Use `RuntimeDefault` seccomp profile by default, which is defined by the container runtime,
instead of using the Unconfined (seccomp disabled) mode.
### Container SecurityContext
1. **AllowPrivilegeEscalation: false**
1. **ReadOnlyRootFilesystem: true**
1. **Capabilities: {drop: [all]}**
Also `SecurityContext` can be configured with spec setting. It may be useful for mounted volumes, with `VMSingle` for example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
@ -89,5 +129,4 @@ spec:
limits:
cpu: "1"
memory: "1512Mi"
```

117
docs/operator/setup.md Normal file
View file

@ -0,0 +1,117 @@
---
sort: 2
weight: 2
title: Setup
---
# VictoriaMetrics Operator Setup
## Installing by helm-charts
You can use one of the following official helm-charts with `vmoperator`:
- [victoria-metrics-operator helm-chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md)
- [victoria-metrics-k8s-stack helm chart](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/README.md)
(includes the `victoria-metrics-operator` helm-chart and other components for full-fledged k8s monitoring, is an alternative for [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)).
For installing VictoriaMetrics operator with helm-chart follow the instructions from README of the corresponding helm-chart
([this](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-operator/README.md)
or [this](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/README.md)).
in addition, you can use [quickstart guide](./quick-start.md) for
installing VictoriaMetrics operator with helm-chart.
## Installing by Manifest
Obtain release from releases page:
[https://github.com/VictoriaMetrics/operator/releases](https://github.com/VictoriaMetrics/operator/releases)
We suggest use the latest release.
```console
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
wget https://github.com/VictoriaMetrics/operator/releases/download/$VM_VERSION/bundle_crd.zip
unzip bundle_crd.zip
```
Operator use `monitoring-system` namespace, but you can install it to specific namespace with command:
```console
sed -i "s/namespace: monitoring-system/namespace: YOUR_NAMESPACE/g" release/operator/*
```
First of all, you have to create [custom resource definitions](https://github.com/VictoriaMetrics/operator):
```console
kubectl apply -f release/crds
```
Then you need RBAC for operator, relevant configuration for the release can be found at `release/operator/rbac.yaml`.
Change configuration for operator at `release/operator/manager.yaml`, possible settings: [operator-settings](/operator/vars.html)
and apply it:
```console
kubectl apply -f release/operator/
```
Check the status of operator
```console
kubectl get pods -n monitoring-system
#NAME READY STATUS RESTARTS AGE
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
```
## Installing by Kustomize
You can install operator using [Kustomize](https://kustomize.io/) by pointing to the remote kustomization file.
```console
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
cat << EOF > kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/VictoriaMetrics/operator/config/default?ref=${VM_VERSION}
images:
- name: victoriametrics/operator
newTag: ${VM_VERSION}
EOF
```
You can change [operator configuration](#configuring), or use your custom namespace see [kustomize-example](https://github.com/YuriKravetc/yurikravetc.github.io/tree/main/Operator/kustomize-example).
Build template
```console
kustomize build . -o monitoring.yaml
```
Apply manifests
```console
kubectl apply -f monitoring.yaml
```
Check the status of operator
```console
kubectl get pods -n monitoring-system
#NAME READY STATUS RESTARTS AGE
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
```
## Installing to ARM
There is no need in an additional configuration for ARM. Operator and VictoriaMetrics have full support for it.
## Configuring
You can read detailed instructions about operator configuring in [this document](./configuration.md).

14
docs/operator/vars.md
View file

@ -1,16 +1,12 @@
---
sort: 14
weight: 14
sort: 11
weight: 11
title: Variables
menu:
docs:
parent: "operator"
weight: 14
aliases:
- /operator/vars.html
---
<!-- this doc autogenerated - don't edit it manually -->
# Auto Generated vars for package config
updated at Wed Sep 27 00:09:29 UTC 2023
updated at Mon Oct 2 12:46:32 UTC 2023
| varible name | variable default value | variable required | variable description |