2023-10-02 12:50:08 +00:00
This documentation section describes the design and interaction between the custom resource definitions (CRD) that the Victoria
Metrics Operator introduces.
2024-08-16 14:32:25 +00:00
[Operator ](https://docs.victoriametrics.com/operator ) introduces the following custom resources:
- [VMAgent ](https://docs.victoriametrics.com/operator/resources/vmagent )
- [VMAlert ](https://docs.victoriametrics.com/operator/resources/vmalert )
- [VMAlertManager ](https://docs.victoriametrics.com/operator/resources/vmalertmanager )
- [VMAlertManagerConfig ](https://docs.victoriametrics.com/operator/resources/vmalertmanagerconfig )
- [VMAuth ](https://docs.victoriametrics.com/operator/resources/vmauth )
- [VMCluster ](https://docs.victoriametrics.com/operator/resources/vmcluster )
- [VMNodeScrape ](https://docs.victoriametrics.com/operator/resources/vmnodescrape )
- [VMPodScrape ](https://docs.victoriametrics.com/operator/resources/vmpodscrape )
- [VMProbe ](https://docs.victoriametrics.com/operator/resources/vmprobe )
- [VMRule ](https://docs.victoriametrics.com/operator/resources/vmrule )
- [VMServiceScrape ](https://docs.victoriametrics.com/operator/resources/vmservicescrape )
- [VMStaticScrape ](https://docs.victoriametrics.com/operator/resources/vmstaticscrape )
- [VMSingle ](https://docs.victoriametrics.com/operator/resources/vmsingle )
- [VMUser ](https://docs.victoriametrics.com/operator/resources/vmuser )
- [VMScrapeConfig ](https://docs.victoriametrics.com/operator/resources/vmscrapeconfig )
2023-10-02 12:50:08 +00:00
Here is the scheme of relations between the custom resources:
2024-08-16 14:32:25 +00:00
2023-10-02 12:50:08 +00:00
## Specification
2024-08-16 14:32:25 +00:00
You can find the specification for the custom resources on ** [API Docs ](https://docs.victoriametrics.com/operator/api )**.
2023-10-02 12:50:08 +00:00
### Extra arguments
If you can't find necessary field in the specification of custom resource,
you can use `extraArgs` field for passing additional arguments to the application.
Field `extraArgs` is supported for the following custom resources:
2024-08-16 14:32:25 +00:00
- [VMAgent spec ](https://docs.victoriametrics.com/operator/api#vmagentspec )
- [VMAlert spec ](https://docs.victoriametrics.com/operator/api#vmalertspec )
- [VMAlertManager spec ](https://docs.victoriametrics.com/operator/api#vmalertmanagerspec )
- [VMAuth spec ](https://docs.victoriametrics.com/operator/api#vmauthspec )
- [VMCluster/vmselect spec ](https://docs.victoriametrics.com/operator/api#vmselect )
- [VMCluster/vminsert spec ](https://docs.victoriametrics.com/operator/api#vminsert )
- [VMCluster/vmstorage spec ](https://docs.victoriametrics.com/operator/api#vmstorage )
- [VMSingle spec ](https://docs.victoriametrics.com/operator/api#vmsinglespec )
2023-10-02 12:50:08 +00:00
Supported flags for each application can be found the in the corresponding documentation:
2024-08-16 14:32:25 +00:00
- [VMAgent ](https://docs.victoriametrics.com/operator/resources/vmagent#advanced-usage )
- [VMAlert ](https://docs.victoriametrics.com/operator/resources/vmalert#configuration )
- [VMAuth ](https://docs.victoriametrics.com/operator/resources/vmauth#advanced-usage )
- [VMCluster ](https://docs.victoriametrics.com/Cluster-VictoriaMetrics#list-of-command-line-flags )
- [VMSingle ](https://docs.victoriametrics.com#list-of-command-line-flags )
2023-10-02 12:50:08 +00:00
Usage example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: vmsingle-example-exrtaargs
spec:
retentionPeriod: "1"
extraArgs:
dedup.minScrapeInterval: 60s
# ...
```
### Extra environment variables
Flag can be replaced with environment variable, it's useful for retrieving value from secret.
You can use `extraEnvs` field for passing additional arguments to the application.
Usage example:
```yaml
kind: VMSingle
metadata:
name: vmsingle-example--exrtaenvs
spec:
retentionPeriod: "1"
extraEnvs:
- name: DEDUP_MINSCRAPEINTERVAL
valueFrom:
secretKeyRef:
name: vm-secret
key: dedup
```
This feature really useful for using with
2024-08-16 14:32:25 +00:00
[`-envflag.enable` command-line argument ](https://docs.victoriametrics.com/#environment-variables ).
2023-10-02 12:50:08 +00:00
## Examples
Page for every custom resource contains examples section:
2024-08-16 14:32:25 +00:00
- [VMAgent examples ](https://docs.victoriametrics.com/operator/resources/vmagent#examples )
- [VMAlert examples ](https://docs.victoriametrics.com/operator/resources/vmalert#examples )
- [VMAlertmanager examples ](https://docs.victoriametrics.com/operator/resources/vmalertmanager#examples )
- [VMAlertmanagerConfig examples ](https://docs.victoriametrics.com/operator/resources/vmalertmanagerconfig#examples )
- [VMAuth examples ](https://docs.victoriametrics.com/operator/resources/vmauth#examples )
- [VMCluster examples ](https://docs.victoriametrics.com/operator/resources/vmcluster#examples )
- [VMNodeScrape examples ](https://docs.victoriametrics.com/operator/resources/vmnodescrape#examples )
- [VMPodScrape examples ](https://docs.victoriametrics.com/operator/resources/vmpodscrape#examples )
- [VMProbe examples ](https://docs.victoriametrics.com/operator/resources/vmprobe#examples )
- [VMRule examples ](https://docs.victoriametrics.com/operator/resources/vmrule#examples )
- [VMServiceScrape examples ](https://docs.victoriametrics.com/operator/resources/vmservicescrape#examples )
- [VMStaticScrape examples ](https://docs.victoriametrics.com/operator/resources/vmstaticscrape#examples )
- [VMSingle examples ](https://docs.victoriametrics.com/operator/resources/vmsingle#examples )
- [VMUser examples ](https://docs.victoriametrics.com/operator/resources/vmuser#examples )
- [VMScrapeConfig examples ](https://docs.victoriametrics.com/operator/resources/vmscrapeconfig#examples )
2023-10-02 12:50:08 +00:00
In addition, you can find examples of the custom resources for VIctoriMetrics operator in
the ** [examples directory ](https://github.com/VictoriaMetrics/operator/tree/master/config/examples ) of operator repository**.
## Managing versions of VM
Every custom resource with deployable application has a fields for specifying version (docker image) of component:
2024-08-16 14:32:25 +00:00
- [Managing versions for VMAgent ](https://docs.victoriametrics.com/operator/resources/vmagent#version-management )
- [Managing versions for VMAlert ](https://docs.victoriametrics.com/operator/resources/vmalert#version-management )
- [Managing versions for VMAlertmanager ](https://docs.victoriametrics.com/operator/resources/vmalertmanager#version-management )
- [Managing versions for VMAuth ](https://docs.victoriametrics.com/operator/resources/vmauth#version-management )
- [Managing versions for VMCluster ](https://docs.victoriametrics.com/operator/resources/vmcluster#version-management )
- [Managing versions for VMSingle ](https://docs.victoriametrics.com/operator/resources/vmsingle#version-management )
2023-10-02 12:50:08 +00:00
2024-02-21 11:52:32 +00:00
## Managing resources
Every custom resource with deployable application has a fields and operator parameters for specifying resources for the component:
2024-08-16 14:32:25 +00:00
- [Managing resources for VMAgent ](https://docs.victoriametrics.com/operator/resources/vmagent#resource-management )
- [Managing resources for VMAlert ](https://docs.victoriametrics.com/operator/resources/vmalert#resource-management )
- [Managing resources for VMAlertmanager ](https://docs.victoriametrics.com/operator/resources/vmalertmanager#resource-management )
- [Managing resources for VMAuth ](https://docs.victoriametrics.com/operator/resources/vmauth#resource-management )
- [Managing resources for VMCluster ](https://docs.victoriametrics.com/operator/resources/vmcluster#resource-management )
- [Managing resources for VMSingle ](https://docs.victoriametrics.com/operator/resources/vmsingle#resource-management )
2024-02-21 11:52:32 +00:00
2023-10-02 12:50:08 +00:00
## High availability
VictoriaMetrics operator support high availability for each component of the monitoring stack:
2024-08-16 14:32:25 +00:00
- [VMAgent ](https://docs.victoriametrics.com/operator/resources/vmagent#high-availability )
- [VMAlert ](https://docs.victoriametrics.com/operator/resources/vmalert#high-availability )
- [VMAlertmanager ](https://docs.victoriametrics.com/operator/resources/vmalertmanager#high-availability )
- [VMAuth ](https://docs.victoriametrics.com/operator/resources/vmauth#high-availability )
- [VMCluster ](https://docs.victoriametrics.com/operator/resources/vmcluster#high-availability )
2023-10-02 12:50:08 +00:00
In addition, these CRD support common features, that can be used to increase high availability - resources above have the following fields:
- `affinity` - to schedule pods on different nodes ([affinity and anti-affinity in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)),
- `tolerations` - to schedule pods on nodes with taints ([taints and tolerations in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)),
- `nodeSelector` - to schedule pods on nodes with specific labels ([node selector in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)),
- `topologySpreadConstraints` - to schedule pods on different nodes in the same topology ([topology spread constraints in kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints)).
See details about these fields in the [Specification ](#specification ).
## Enterprise features
2024-08-16 14:32:25 +00:00
Operator supports following [Enterprise features for VictoriaMetrics components ](https://docs.victoriametrics.com/enterprise ):
- [VMAgent Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmagent#enterprise-features ):
- [Reading metrics from kafka ](https://docs.victoriametrics.com/operator/resources/vmagent#reading-metrics-from-kafka )
- [Writing metrics to kafka ](https://docs.victoriametrics.com/operator/resources/vmagent#writing-metrics-to-kafka )
- [VMAlert Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmalert#enterprise-features ):
- [Reading rules from object storage ](https://docs.victoriametrics.com/operator/resources/vmalert#reading-rules-from-object-storage )
- [Multitenancy ](https://docs.victoriametrics.com/operator/resources/vmalert#multitenancy )
- [VMAuth Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmauth#enterprise-features )
- [IP Filters ](https://docs.victoriametrics.com/operator/resources/vmauth#ip-filters )
- [VMCluster Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmcluster#enterprise-features )
- [Downsampling ](https://docs.victoriametrics.com/operator/resources/vmcluster#downsampling )
- [Multiple retentions / Retention filters ](https://docs.victoriametrics.com/operator/resources/vmcluster#retention-filters )
- [Advanced per-tenant statistic ](https://docs.victoriametrics.com/operator/resources/vmcluster#advanced-per-tenant-statistic )
- [mTLS protection ](https://docs.victoriametrics.com/operator/resources/vmcluster#mtls-protection )
- [Backup automation ](https://docs.victoriametrics.com/operator/resources/vmcluster#backup-automation )
- [VMRule Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmrule#enterprise-features )
- [Multitenancy ](https://docs.victoriametrics.com/operator/resources/vmrule#multitenancy )
- [VMSingle Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmsingle#enterprise-features )
- [Downsampling ](https://docs.victoriametrics.com/operator/resources/vmsingle#downsampling )
- [Retention filters ](https://docs.victoriametrics.com/operator/resources/vmsingle#retention-filters )
- [Backup automation ](https://docs.victoriametrics.com/operator/resources/vmsingle#backup-automation )
- [VMUser Enterprise features ](https://docs.victoriametrics.com/operator/resources/vmuser#enterprise-features )
- [IP Filters ](https://docs.victoriametrics.com/operator/resources/vmuser#ip-filters )
2023-10-02 12:50:08 +00:00
More information about enterprise features you can read
2024-08-16 14:32:25 +00:00
on [VictoriaMetrics Enterprise page ](https://docs.victoriametrics.com/enterprise#victoriametrics-enterprise ).
2023-10-02 12:50:08 +00:00
## Configuration synchronization
### Basic concepts
VictoriaMetrics applications, like many other applications with configuration file deployed at Kubernetes, uses `ConfigMaps` and `Secrets` for configuration files.
Usually, it's out of application scope to watch for configuration on-disk changes.
Applications reload their configuration by a signal from a user or some other tool, that knows how to watch for updates.
At Kubernetes, the most popular design for this case is a sidecar container, that watches for configuration file changes and sends an HTTP request to the application.
`Configmap` or `Secret` that mounted at `Pod` holds a copy of its content.
Kubernetes component `kubelet` is responsible for content synchronization between an object at Kubernetes API and a file served on disk.
It's not efficient to sync its content immediately, and `kubelet` eventually synchronizes it. There is a configuration option, that controls this period.
That's why, applications managed by operator don't receive changes immediately. It usually takes 1-2 min, before content will be updated.
2024-08-16 14:32:25 +00:00
It may trigger errors when an application was deleted, but [`VMAgent` ](https://docs.victoriametrics.com/operator/resources/vmagent ) still tries to scrape it.
2023-10-02 12:50:08 +00:00
### Possible mitigations
The naive solution for this case decrease the synchronization period. But it configures globally and may be hard for operator users.
That's why operator uses a few hacks.
For `ConfigMap` updates, operator changes annotation with a time of `Configmap` content update. It triggers `ConfigMap` 's content synchronization by kubelet immediately.
It's the case for `VMAlert` , it uses `ConfigMap` as a configuration source.
For `Secret` it doesn't work. And operator offers its implementation for side-car container. It can be configured with env variable for operator:
```
- name: VM_USECUSTOMCONFIGRELOADER
value: "true"
```
2024-08-16 14:32:25 +00:00
If it's defined, operator uses own [config-reloader ](https://github.com/VictoriaMetrics/operator/tree/master/cmd/config-reloader )
2023-10-02 12:50:08 +00:00
instead of [prometheus-config-reload ](https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader ).
It watches corresponding `Secret` for changes with Kubernetes API watch call and writes content into emptyDir.
This emptyDir shared with the application.
In case of content changes, `config-reloader` sends HTTP requests to the application.
It greatly reduces the time for configuration synchronization.
