github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-11 14:53:49 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

Browse source
This commit is contained in:
Aliaksandr Valialkin 2022-05-23 11:00:31 +03:00
commit 3dec16702a
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
11 changed files with 387 additions and 98 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -1071,7 +1071,7 @@ By default VictoriaMetrics is tuned for an optimal resource usage under typical
- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage.
- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage.
See also [capacity planning docs](#capacity-planning).
See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs](#capacity-planning).
## High availability

19
app/vmalert/README.md
View file

@ -1013,6 +1013,25 @@ It is recommended using
* `vmalert` is located in `vmutils-*` archives there.
### Docker image
You can build `vmalert` docker image from source and push it to your own docker repository.
Run the following commands from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics):
```bash
make package-vmalert
docker tag victoria-metrics/vmalert:version my-repo:my-version-name
docker push my-repo:my-version-name
```
To run the built image in `victoria-metrics-k8s-stack` or `VMAlert` CR object apply the following config change:
```yaml
kind: VMAlert
spec:
image:
repository: my-repo
tag: my-version-name
```
### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.17.

2
app/vmui/Dockerfile-web
View file

@ -1,4 +1,4 @@
FROM golang:1.18.1 as build-web-stage
FROM golang:1.18.2 as build-web-stage
COPY build /build
WORKDIR /build

2
deployment/docker/Makefile
View file

@ -4,7 +4,7 @@ DOCKER_NAMESPACE := victoriametrics
ROOT_IMAGE ?= alpine:3.15.4
CERTS_IMAGE := alpine:3.15.4
GO_BUILDER_IMAGE := golang:1.18.1-alpine
GO_BUILDER_IMAGE := golang:1.18.2-alpine
BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)-1
BASE_IMAGE := local/base:1.1.3-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

2
docs/README.md
View file

@ -1071,7 +1071,7 @@ By default VictoriaMetrics is tuned for an optimal resource usage under typical
- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage.
- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage.
See also [capacity planning docs](#capacity-planning).
See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs](#capacity-planning).
## High availability

2
docs/Single-server-VictoriaMetrics.md
View file

@ -1075,7 +1075,7 @@ By default VictoriaMetrics is tuned for an optimal resource usage under typical
- `-search.maxTagKeys` limits the number of items, which may be returned from [/api/v1/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names). This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagKeys` to quite low value in order to limit CPU and memory usage.
- `-search.maxTagValues` limits the number of items, which may be returned from [/api/v1/label/.../values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values). This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memory when the database contains big number of unique time series because of [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). In this case it might be useful to set the `-search.maxTagValues` to quite low value in order to limit CPU and memory usage.
See also [capacity planning docs](#capacity-planning).
See also [cardinality limiter](#cardinality-limiter) and [capacity planning docs](#capacity-planning).
## High availability

108
docs/operator/api.MD
View file

@ -12,7 +12,9 @@ This Document documents the types introduced by the VictoriaMetrics to be consum
* [VMAlertmanagerStatus](#vmalertmanagerstatus)
* [EmailConfig](#emailconfig)
* [HTTPConfig](#httpconfig)
* [ImageConfig](#imageconfig)
* [InhibitRule](#inhibitrule)
* [LinkConfig](#linkconfig)
* [MuteTimeInterval](#mutetimeinterval)
* [OpsGenieConfig](#opsgenieconfig)
* [OpsGenieConfigResponder](#opsgenieconfigresponder)
@ -24,6 +26,7 @@ This Document documents the types introduced by the VictoriaMetrics to be consum
* [SlackConfig](#slackconfig)
* [SlackConfirmationField](#slackconfirmationfield)
* [SlackField](#slackfield)
* [TelegramConfig](#telegramconfig)
* [TimeInterval](#timeinterval)
* [TimeRange](#timerange)
* [VMAlertmanagerConfig](#vmalertmanagerconfig)
@ -179,11 +182,13 @@ VMAlertmanagerSpec is a specification of the desired behavior of the VMAlertmana
| priorityClassName | PriorityClassName class assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| additionalPeers | AdditionalPeers allows injecting a set of additional Alertmanagers to peer with to form a highly available cluster. | []string | false |
| clusterAdvertiseAddress | ClusterAdvertiseAddress is the explicit address to advertise in cluster. Needs to be provided for non RFC1918 [1] (public) addresses. [1] RFC1918: https://tools.ietf.org/html/rfc1918 | string | false |
| portName | PortName used for the pods and governing service. This defaults to web | string | false |
| serviceSpec | ServiceSpec that will be added to vmalertmanager service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| selectAllByDefault | SelectAllByDefault changes default behavior for empty CRD selectors, such ConfigSelector. with selectAllScrapes: true and undefined ConfigSelector and ConfigNamespaceSelector Operator selects all exist alertManagerConfigs with selectAllScrapes: false - selects nothing | bool | false |
| configSelector | ConfigSelector defines selector for VMAlertmanagerConfig, result config will be merged with with Raw or Secret config. Works in combination with NamespaceSelector. NamespaceSelector nil - only objects at VMAlertmanager namespace. Selector nil - only objects at NamespaceSelector namespaces. If both nil - behaviour controlled by selectAllByDefault | *[metav1.LabelSelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#labelselector-v1-meta) | false |
@ -192,6 +197,7 @@ VMAlertmanagerSpec is a specification of the desired behavior of the VMAlertmana
| extraEnvs | ExtraEnvs that will be added to VMAuth pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| disableNamespaceMatcher | DisableNamespaceMatcher disables namespace label matcher for VMAlertmanagerConfig It may be useful if alert doesn't have namespace label for some reason | bool | false |
| rollingUpdateStrategy | RollingUpdateStrategy defines strategy for application updates Default is OnDelete, in this case operator handles update process Can be changed for RollingUpdate | [appsv1.StatefulSetUpdateStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#statefulsetupdatestrategy-v1-apps) | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
[Back to TOC](#table-of-contents)
@ -246,6 +252,18 @@ HTTPConfig defines a client HTTP configuration. See https://prometheus.io/docs/a
[Back to TOC](#table-of-contents)
## ImageConfig
ImageConfig is used to attach images to the incident. See https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgx-send-an-alert-event#the-images-property for more information.
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| href | | string | false |
| source | | string | true |
| alt | | string | false |
[Back to TOC](#table-of-contents)
## InhibitRule
InhibitRule defines an inhibition rule that allows to mute alerts when other alerts are already firing. Note, it doesn't support deprecated alertmanager config options. See https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule
@ -258,6 +276,17 @@ InhibitRule defines an inhibition rule that allows to mute alerts when other ale
[Back to TOC](#table-of-contents)
## LinkConfig
LinkConfig is used to attach text links to the incident. See https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgx-send-an-alert-event#the-links-property for more information.
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| href | | string | true |
| text | | string | false |
[Back to TOC](#table-of-contents)
## MuteTimeInterval
MuteTimeInterval for alerts
@ -286,7 +315,7 @@ OpsGenieConfig configures notifications via OpsGenie. See https://prometheus.io/
| priority | Priority level of alert. Possible values are P1, P2, P3, P4, and P5. | string | false |
| details | A set of arbitrary key/value pairs that provide further detail about the incident. | map[string]string | false |
| responders | List of responders responsible for notifications. | [][OpsGenieConfigResponder](#opsgenieconfigresponder) | false |
| httpConfig | HTTP client configuration. | *[HTTPConfig](#httpconfig) | false |
| http_config | HTTP client configuration. | *[HTTPConfig](#httpconfig) | false |
[Back to TOC](#table-of-contents)
@ -315,6 +344,8 @@ PagerDutyConfig configures notifications via PagerDuty. See https://prometheus.i
| url | The URL to send requests to. | string | false |
| client | Client identification. | string | false |
| client_url | Backlink to the sender of notification. | string | false |
| images | Images to attach to the incident. | [][ImageConfig](#imageconfig) | false |
| links | Links to attach to the incident. | [][LinkConfig](#linkconfig) | false |
| description | Description of the incident. | string | false |
| severity | Severity of the incident. | string | false |
| class | The class/type of the event. | string | false |
@ -362,6 +393,7 @@ Receiver defines one or more notification integrations.
| webhook_configs | WebhookConfigs defines webhook notification configurations. | [][WebhookConfig](#webhookconfig) | false |
| victorops_configs | VictorOpsConfigs defines victor ops notification configurations. | [][VictorOpsConfig](#victoropsconfig) | false |
| wechat_configs | WeChatConfigs defines wechat notification configurations. | [][WeChatConfig](#wechatconfig) | false |
| telegram_configs | | [][TelegramConfig](#telegramconfig) | false |
[Back to TOC](#table-of-contents)
@ -455,6 +487,23 @@ See https://api.slack.com/docs/message-attachments#fields for more information.
[Back to TOC](#table-of-contents)
## TelegramConfig
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| send_resolved | SendResolved controls notify about resolved alerts. | *bool | false |
| api_url | APIUrl the Telegram API URL i.e. https://api.telegram.org. | string | false |
| bot_token | BotToken token for the bot https://core.telegram.org/bots/api | *[v1.SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#secretkeyselector-v1-core) | true |
| chat_id | ChatID is ID of the chat where to send the messages. | int | true |
| message | Message is templated message | string | false |
| disable_notifications | DisableNotifications | *bool | false |
| parse_mode | ParseMode for telegram message, supported values are MarkdownV2, Markdown, Markdown and empty string for plain text. | string | false |
| http_config | HTTP client configuration. | *[HTTPConfig](#httpconfig) | false |
[Back to TOC](#table-of-contents)
## TimeInterval
TimeInterval defines intervals of time
@ -540,8 +589,8 @@ VictorOpsConfig configures notifications via VictorOps. See https://prometheus.i
| entity_display_name | Contains summary of the alerted problem. | string | false |
| state_message | Contains long explanation of the alerted problem. | string | false |
| monitoring_tool | The monitoring tool the state message is from. | string | false |
| custom_fields | Additional custom fields for notification. | map[string]string | false |
| http_config | The HTTP client's configuration. | *[HTTPConfig](#httpconfig) | false |
| custom_fields | Adds optional custom fields https://github.com/prometheus/alertmanager/blob/v0.24.0/config/notifiers.go#L537 | map[string]string | false |
[Back to TOC](#table-of-contents)
@ -639,7 +688,7 @@ VMAgentRemoteWriteSpec defines the remote storage configuration for VmAgent
VMAgentSpec defines the desired state of VMAgent
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| ----- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------- |
| podMetadata | PodMetadata configures Labels and Annotations which are propagated to the vmagent pods. | *[EmbeddedObjectMetadata](#embeddedobjectmetadata) | false |
| image | Image - docker image settings for VMAgent if no specified operator uses default config version | [Image](#image) | false |
| imagePullSecrets | ImagePullSecrets An optional list of references to secrets in the same namespace to use for pulling images from registries see http://kubernetes.io/docs/user-guide/images#specifying-imagepullsecrets-on-a-pod | [][v1.LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#localobjectreference-v1-core) | false |
@ -687,7 +736,6 @@ VMAgentSpec defines the desired state of VMAgent
| nodeScrapeNamespaceSelector | NodeScrapeNamespaceSelector defines Namespaces to be selected for VMNodeScrape discovery. Works in combination with Selector. NamespaceSelector nil - only objects at VMAgent namespace. Selector nil - only objects at NamespaceSelector namespaces. If both nil - behaviour controlled by selectAllByDefault | *[metav1.LabelSelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#labelselector-v1-meta) | false |
| staticScrapeSelector | StaticScrapeSelector defines PodScrapes to be selected for target discovery. Works in combination with NamespaceSelector. If both nil - match everything. NamespaceSelector nil - only objects at VMAgent namespace. Selector nil - only objects at NamespaceSelector namespaces. | *[metav1.LabelSelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#labelselector-v1-meta) | false |
| staticScrapeNamespaceSelector | StaticScrapeNamespaceSelector defines Namespaces to be selected for VMStaticScrape discovery. Works in combination with NamespaceSelector. NamespaceSelector nil - only objects at VMAgent namespace. Selector nil - only objects at NamespaceSelector namespaces. If both nil - behaviour controlled by selectAllByDefault | *[metav1.LabelSelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#labelselector-v1-meta) | false |
| inlineScrapeConfig | InlineScrapeConfig allows for an array of scrape jobs to be passed as a YAML formatted string (it is recommended to use the literal block style indicator "\|"). As scrape configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of VMAgent. It is advised to review VMAgent release notes to ensure that no incompatible scrape configs are going to break VMAgent after the upgrade. | string | false |
| additionalScrapeConfigs | AdditionalScrapeConfigs As scrape configs are appended, the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of VMAgent. It is advised to review VMAgent release notes to ensure that no incompatible scrape configs are going to break VMAgent after the upgrade. | *[v1.SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#secretkeyselector-v1-core) | false |
| arbitraryFSAccessThroughSMs | ArbitraryFSAccessThroughSMs configures whether configuration based on a service scrape can access arbitrary files on the file system of the VMAgent container e.g. bearer token files. | [ArbitraryFSAccessThroughSMsConfig](#arbitraryfsaccessthroughsmsconfig) | false |
| insertPorts | InsertPorts - additional listen ports for data ingestion. | *[InsertPorts](#insertports) | false |
@ -695,6 +743,7 @@ VMAgentSpec defines the desired state of VMAgent
| extraArgs | ExtraArgs that will be passed to VMAgent pod for example remoteWrite.tmpDataPath: /tmp it would be converted to flag --remoteWrite.tmpDataPath=/tmp | map[string]string | false |
| extraEnvs | ExtraEnvs that will be added to VMAgent pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| serviceSpec | ServiceSpec that will be added to vmagent service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| shardCount | ShardCount - numbers of shards of VMAgent in this case operator will use 1 deployment/sts per shard with replicas count according to spec.replicas https://victoriametrics.github.io/vmagent.html#scraping-big-number-of-targets | *int | false |
| updateStrategy | UpdateStrategy - overrides default update strategy. works only for deployments, statefulset always use OnDelete. | *[appsv1.DeploymentStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#deploymentstrategy-v1-apps) | false |
| rollingUpdate | RollingUpdate - overrides deployment update params. | *[appsv1.RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#rollingupdatedeployment-v1-apps) | false |
@ -705,6 +754,14 @@ VMAgentSpec defines the desired state of VMAgent
| nodeScrapeRelabelTemplate | NodeScrapeRelabelTemplate defines relabel config, that will be added to each VMNodeScrape. it's useful for adding specific labels to all targets | []*[RelabelConfig](#relabelconfig) | false |
| staticScrapeRelabelTemplate | StaticScrapeRelabelTemplate defines relabel config, that will be added to each VMStaticScrape. it's useful for adding specific labels to all targets | []*[RelabelConfig](#relabelconfig) | false |
| probeScrapeRelabelTemplate | ProbeScrapeRelabelTemplate defines relabel config, that will be added to each VMProbeScrape. it's useful for adding specific labels to all targets | []*[RelabelConfig](#relabelconfig) | false |
| minScrapeInterval | MinScrapeInterval allows limiting minimal scrape interval for VMServiceScrape, VMPodScrape and other scrapes If interval is lower than defined limit, `minScrapeInterval` will be used. | *string | false |
| maxScrapeInterval | MaxScrapeInterval allows limiting maximum scrape interval for VMServiceScrape, VMPodScrape and other scrapes If interval is higher than defined limit, `maxScrapeInterval` will be used. | *string | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| statefulMode | StatefulMode enables StatefulSet for `VMAgent` instead of Deployment it allows using persistent storage for vmagent's persistentQueue | bool | false |
| statefulStorage | StatefulStorage configures storage for StatefulSet | *[StorageSpec](#storagespec) | false |
| statefulRollingUpdateStrategy | StatefulRollingUpdateStrategy allows configuration for strategyType set it to RollingUpdate for disabling operator statefulSet rollingUpdate | [appsv1.StatefulSetUpdateStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#statefulsetupdatestrategy-v1-apps) | false |
| inlineScrapeConfig | InlineScrapeConfig allows for an array of scrape jobs to be passed as a YAML formatted string (it is recommended to use the literal block style indicator). As scrape configs are appended,the user is responsible to make sure it is valid. Note that using this feature may expose the possibility to break upgrades of VMAgent. It is advised to review VMAgent release notes to ensure that no incompatible scrape configs are going to break VMAgent after the upgrade. | string | false |
[Back to TOC](#table-of-contents)
@ -714,10 +771,10 @@ VmAgentStatus defines the observed state of VmAgent
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| replicas | ReplicaCount Total number of non-terminated pods targeted by this VMAgent cluster (their labels match the selector). | int32 | true |
| updatedReplicas | UpdatedReplicas Total number of non-terminated pods targeted by this VMAgent cluster that have the desired version spec. | int32 | true |
| availableReplicas | AvailableReplicas Total number of available pods (ready for at least minReadySeconds) targeted by this VMAgent cluster. | int32 | true |
| unavailableReplicas | UnavailableReplicas Total number of unavailable pods targeted by this VMAgent cluster. | int32 | true |
| replicas | ReplicaCount Total number of non-terminated pods targeted by this VMAlert cluster (their labels match the selector). | int32 | true |
| updatedReplicas | UpdatedReplicas Total number of non-terminated pods targeted by this VMAlert cluster that have the desired version spec. | int32 | true |
| availableReplicas | AvailableReplicas Total number of available pods (ready for at least minReadySeconds) targeted by this VMAlert cluster. | int32 | true |
| unavailableReplicas | UnavailableReplicas Total number of unavailable pods targeted by this VMAlert cluster. | int32 | true |
[Back to TOC](#table-of-contents)
@ -752,8 +809,8 @@ EmbeddedHPA embeds HorizontalPodAutoScaler spec v2. https://kubernetes.io/docs/r
| ----- | ----------- | ------ | -------- |
| minReplicas | | *int32 | false |
| maxReplicas | | int32 | false |
| metrics | | [][v2beta2.MetricSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#metricspec-v2beta2-autoscaling) | false |
| behaviour | | *[v2beta2.HorizontalPodAutoscalerBehavior](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#horizontalpodautoscalerbehavior-v2beta2-autoscaling) | false |
| metrics | | []v2beta2.MetricSpec | false |
| behaviour | | *v2beta2.HorizontalPodAutoscalerBehavior | false |
[Back to TOC](#table-of-contents)
@ -950,10 +1007,13 @@ VMAlertSpec defines the desired state of VMAlert
| extraEnvs | ExtraEnvs that will be added to VMAlert pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| externalLabels | ExternalLabels in the form 'name: value' to add to all generated recording rules and alerts. | map[string]string | false |
| serviceSpec | ServiceSpec that will be added to vmalert service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| updateStrategy | UpdateStrategy - overrides default update strategy. | *[appsv1.DeploymentStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#deploymentstrategy-v1-apps) | false |
| rollingUpdate | RollingUpdate - overrides deployment update params. | *[appsv1.RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#rollingupdatedeployment-v1-apps) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
[Back to TOC](#table-of-contents)
@ -1009,6 +1069,7 @@ VMSingleSpec defines the desired state of VMSingle
| replicaCount | ReplicaCount is the expected size of the VMSingle it can be 0 or 1 if you need more - use vm cluster | *int32 | false |
| storageDataPath | StorageDataPath disables spec.storage option and overrides arg for victoria-metrics binary --storageDataPath, its users responsibility to mount proper device into given path. | string | false |
| storage | Storage is the definition of how storage will be used by the VMSingle by default it`s empty dir | *[v1.PersistentVolumeClaimSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#persistentvolumeclaimspec-v1-core) | false |
| storageMetadata | StorageMeta defines annotations and labels attached to PVC for given vmsingle CR | [EmbeddedObjectMetadata](#embeddedobjectmetadata) | false |
| volumes | Volumes allows configuration of additional volumes on the output deploy definition. Volumes specified will be appended to other volumes that are generated as a result of StorageSpec objects. | [][v1.Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#volume-v1-core) | false |
| volumeMounts | VolumeMounts allows configuration of additional VolumeMounts on the output Deployment definition. VolumeMounts specified will be appended to other VolumeMounts in the VMSingle container, that are generated as a result of StorageSpec objects. | [][v1.VolumeMount](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#volumemount-v1-core) | false |
| resources | Resources container resource request and limits, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ if not defined default resources from operator config will be used | [v1.ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#resourcerequirements-v1-core) | false |
@ -1025,16 +1086,19 @@ VMSingleSpec defines the desired state of VMSingle
| priorityClassName | PriorityClassName assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| insertPorts | InsertPorts - additional listen ports for data ingestion. | *[InsertPorts](#insertports) | false |
| port | Port listen port | string | false |
| removePvcAfterDelete | RemovePvcAfterDelete - if true, controller adds ownership to pvc and after VMSingle objest deletion - pvc will be garbage collected by controller manager | bool | false |
| retentionPeriod | RetentionPeriod in months | string | true |
| retentionPeriod | RetentionPeriod for the stored metrics Note VictoriaMetrics has data/ and indexdb/ folders metrics from data/ removed eventually as soon as partition leaves retention period reverse index data at indexdb rotates once at the half of configured retention period https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#retention | string | true |
| vmBackup | VMBackup configuration for backup | *[VMBackup](#vmbackup) | false |
| extraArgs | ExtraArgs that will be passed to VMSingle pod for example remoteWrite.tmpDataPath: /tmp | map[string]string | false |
| extraEnvs | ExtraEnvs that will be added to VMSingle pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| serviceSpec | ServiceSpec that will be added to vmsingle service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
[Back to TOC](#table-of-contents)
@ -1059,7 +1123,7 @@ Rule describes an alerting or recording rule.
| ----- | ----------- | ------ | -------- |
| record | Record represents a query, that will be recorded to dataSource | string | false |
| alert | Alert is a name for alert | string | false |
| expr | Expr is query, that will be evaluated at dataSource | intstr.IntOrString | true |
| expr | Expr is query, that will be evaluated at dataSource | string | true |
| for | For evaluation interval in time.Duration format 30s, 1m, 1h or nanoseconds | string | false |
| labels | Labels will be added to rule configuration | map[string]string | false |
| annotations | Annotations will be added to rule configuration | map[string]string | false |
@ -1077,8 +1141,10 @@ RuleGroup is a list of sequentially evaluated recording and alerting rules.
| rules | Rules list of alert rules | [][Rule](#rule) | true |
| concurrency | Concurrency defines how many rules execute at once. | int | false |
| labels | Labels optional list of labels added to every rule within a group. It has priority over the external labels. Labels are commonly used for adding environment or tenant-specific tag. | map[string]string | false |
| extra_filter_labels | ExtraFilterLabels optional list of label filters applied to every rule's request withing a group. Is compatible only with VM datasource. See more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements | map[string]string | false |
| extra_filter_labels | ExtraFilterLabels optional list of label filters applied to every rule's request withing a group. Is compatible only with VM datasource. See more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements Deprecated, use params instead | map[string]string | false |
| tenant | Tenant id for group, can be used only with enterprise version of vmalert See more details at https://docs.victoriametrics.com/vmalert.html#multitenancy | string | false |
| params | Params optional HTTP URL parameters added to each rule request | url.Values | false |
| type | Type defines datasource type for enterprise version of vmalert possible values - prometheus,graphite | string | false |
[Back to TOC](#table-of-contents)
@ -1479,7 +1545,7 @@ VMClusterSpec defines the desired state of VMCluster
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| retentionPeriod | RetentionPeriod in months | string | true |
| retentionPeriod | RetentionPeriod for the stored metrics Note VictoriaMetrics has data/ and indexdb/ folders metrics from data/ removed eventually as soon as partition leaves retention period reverse index data at indexdb rotates once at the half of configured retention period https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#retention | string | true |
| replicationFactor | ReplicationFactor defines how many copies of data make among distinct storage nodes | *int32 | false |
| podSecurityPolicyName | PodSecurityPolicyName - defines name for podSecurityPolicy in case of empty value, prefixedName will be used. | string | false |
| serviceAccountName | ServiceAccountName is the name of the ServiceAccount to use to run the VMSelect Pods. | string | false |
@ -1529,6 +1595,7 @@ VMClusterStatus defines the observed state of VMCluster
| priorityClassName | Priority class assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| extraArgs | | map[string]string | false |
| insertPorts | InsertPorts - additional listen ports for data ingestion. | *[InsertPorts](#insertports) | false |
@ -1537,11 +1604,13 @@ VMClusterStatus defines the observed state of VMCluster
| runtimeClassName | RuntimeClassName - defines runtime class for kubernetes pod. https://kubernetes.io/docs/concepts/containers/runtime-class/ | *string | false |
| extraEnvs | ExtraEnvs that will be added to VMSelect pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| serviceSpec | ServiceSpec that will be added to vminsert service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| updateStrategy | UpdateStrategy - overrides default update strategy. | *[appsv1.DeploymentStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#deploymentstrategy-v1-apps) | false |
| rollingUpdate | RollingUpdate - overrides deployment update params. | *[appsv1.RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#rollingupdatedeployment-v1-apps) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| hpa | HPA defines kubernetes PodAutoScaling configuration version 2. | *[EmbeddedHPA](#embeddedhpa) | false |
| nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
[Back to TOC](#table-of-contents)
@ -1570,6 +1639,7 @@ VMClusterStatus defines the observed state of VMCluster
| priorityClassName | Priority class assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| cacheMountPath | CacheMountPath allows to add cache persistent for VMSelect | string | false |
| persistentVolume | Storage - add persistent volume for cacheMounthPath its useful for persistent cache use storage instead of persistentVolume. | *[StorageSpec](#storagespec) | false |
@ -1580,10 +1650,12 @@ VMClusterStatus defines the observed state of VMCluster
| schedulerName | SchedulerName - defines kubernetes scheduler name | string | false |
| runtimeClassName | RuntimeClassName - defines runtime class for kubernetes pod. https://kubernetes.io/docs/concepts/containers/runtime-class/ | *string | false |
| serviceSpec | ServiceSpec that will be added to vmselect service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| hpa | | *[EmbeddedHPA](#embeddedhpa) | false |
| nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
| rollingUpdateStrategy | RollingUpdateStrategy defines strategy for application updates Default is OnDelete, in this case operator handles update process Can be changed for RollingUpdate | [appsv1.StatefulSetUpdateStrategyType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#statefulsetupdatestrategy-v1-apps) | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
[Back to TOC](#table-of-contents)
@ -1612,10 +1684,11 @@ VMClusterStatus defines the observed state of VMCluster
| priorityClassName | Priority class assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| storageDataPath | StorageDataPath - path to storage data | string | false |
| storage | Storage - add persistent volume for StorageDataPath its useful for persistent cache | *[StorageSpec](#storagespec) | false |
| terminationGracePeriodSeconds | | int64 | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | int64 | false |
| schedulerName | SchedulerName - defines kubernetes scheduler name | string | false |
| runtimeClassName | RuntimeClassName - defines runtime class for kubernetes pod. https://kubernetes.io/docs/concepts/containers/runtime-class/ | *string | false |
| port | Port for health check connetions | string | false |
@ -1625,6 +1698,7 @@ VMClusterStatus defines the observed state of VMCluster
| extraArgs | | map[string]string | false |
| extraEnvs | ExtraEnvs that will be added to VMSelect pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| serviceSpec | ServiceSpec that will be create additional service for vmstorage | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| maintenanceInsertNodeIDs | MaintenanceInsertNodeIDs - excludes given node ids from insert requests routing, must contain pod suffixes - for pod-0, id will be 0 and etc. lets say, you have pod-0, pod-1, pod-2, pod-3. to exclude pod-0 and pod-3 from insert routing, define nodeIDs: [0,3]. Useful at storage expanding, when you want to rebalance some data at cluster. | []int32 | false |
| maintenanceSelectNodeIDs | MaintenanceInsertNodeIDs - excludes given node ids from select requests routing, must contain pod suffixes - for pod-0, id will be 0 and etc. | []int32 | false |
@ -1753,6 +1827,7 @@ VMUserSpec defines the desired state of VMUser
| Field | Description | Scheme | Required |
| ----- | ----------- | ------ | -------- |
| name | Name of the VMUser object. | *string | false |
| username | UserName basic auth user name for accessing protected endpoint, will be replaced with metadata.name of VMUser if omitted. | *string | false |
| password | Password basic auth password for accessing protected endpoint. | *string | false |
| passwordRef | PasswordRef allows to fetch password from user-create secret by its name and key. | *[v1.SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#secretkeyselector-v1-core) | false |
@ -1829,6 +1904,7 @@ VMAuthSpec defines the desired state of VMAuth
| priorityClassName | PriorityClassName assigned to the Pods | string | false |
| hostNetwork | HostNetwork controls whether the pod may use the node network namespace | bool | false |
| dnsPolicy | DNSPolicy sets DNS policy for the pod | [v1.DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#pod-v1-core) | false |
| dnsConfig | Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. | *v1.PodDNSConfig | false |
| topologySpreadConstraints | TopologySpreadConstraints embedded kubernetes pod configuration option, controls how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | [][v1.TopologySpreadConstraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) | false |
| port | Port listen port | string | false |
| selectAllByDefault | SelectAllByDefault changes default behavior for empty CRD selectors, such userSelector. with selectAllByDefault: true and empty userSelector and userNamespaceSelector Operator selects all exist users with selectAllByDefault: false - selects nothing | bool | false |
@ -1837,9 +1913,11 @@ VMAuthSpec defines the desired state of VMAuth
| extraArgs | ExtraArgs that will be passed to VMAuth pod for example remoteWrite.tmpDataPath: /tmp | map[string]string | false |
| extraEnvs | ExtraEnvs that will be added to VMAuth pod | [][v1.EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#envvar-v1-core) | false |
| serviceSpec | ServiceSpec that will be added to vmsingle service spec | *[ServiceSpec](#servicespec) | false |
| serviceScrapeSpec | ServiceScrapeSpec that will be added to vmselect VMServiceScrape spec | *[VMServiceScrapeSpec](#vmservicescrapespec) | false |
| podDisruptionBudget | PodDisruptionBudget created by operator | *[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec) | false |
| ingress | Ingress enables ingress configuration for VMAuth. | *[EmbeddedIngress](#embeddedingress) | false |
| nodeSelector | NodeSelector Define which Nodes the Pods are scheduled on. | map[string]string | false |
| terminationGracePeriodSeconds | TerminationGracePeriodSeconds period for container graceful termination | *int64 | false |
[Back to TOC](#table-of-contents)

40
docs/operator/configuration_syncronization.MD Normal file
View file

@ -0,0 +1,40 @@
---
sort: 14
---
# Configuration synchronization
## Basic concepts
VictoriaMetrics applications, like many other applications with configuration file deployed at Kubernetes, uses `ConfigMaps` and `Secrets` for configuration files.
Usually, it's out of application scope to watch for configuration on-disk changes.
Applications reload their configuration by a signal from a user or some other tool, that knows how to watch for updates.
At Kubernetes, the most popular design for this case is a sidecar container, that watches for configuration file changes and sends an HTTP request to the application.
`Configmap` or `Secret` that mounted at `Pod` holds a copy of its content.
Kubernetes component `kubelet` is responsible for content synchronization between an object at Kubernetes API and a file served on disk.
It's not efficient to sync its content immediately, and `kubelet` eventually synchronizes it. There is a configuration option, that controls this period.
That's why, `VMAgent`, `VMAlert` and other applications managed by operator don't receive changes immediately. It usually takes 1-2 min, before content will be updated.
It may trigger errors when an application was deleted, but `VMAgent` still tries to scrape it.
## Possible mitigations
The naive solution for this case decrease the synchronization period. But it configures globally and may be hard for operator users.
That's why operator uses a few hacks.
For `ConfigMap` updates, operator changes annotation with a time of `Configmap` content update. It triggers `ConfigMap`'s content synchronization by kubelet immediately.
It's the case for `VMAlert`, it uses `ConfigMap` as a configuration source.
For `Secret` it doesn't work. And operator offers its implementation for side-car container. It can be configured with env variable for operator:
```
- name: VM_USECUSTOMCONFIGRELOADER
value: "true"
```
If it's defined, operator uses own [config-reloader](https://github.com/VictoriaMetrics/operator/tree/master/internal/config-reloader) instead of [prometheus-config-reload](https://github.com/prometheus-operator/prometheus-operator/tree/main/cmd/prometheus-config-reloader).
It watches corresponding `Secret` for changes with Kubernetes API watch call and writes content into emptyDir.
This emptyDir shared with the application.
In case of content changes, `config-reloader` sends HTTP requests to the application.
It greatly reduces the time for configuration synchronization.

41
docs/operator/high-availability.MD
View file

@ -4,7 +4,7 @@ sort: 7
# High Availability
High availability is not only important for customer-facing software but if the monitoring infrastructure is not highly available, then there is a risk that operations people are not notified for alerts. Therefore high availability must be just as thought through for the monitoring stack, as for anything else.
High availability is not only important for customer-facing software but if the monitoring infrastructure is not highly available, then there is a risk that operations people are not notified of alerts. Therefore high availability must be just as thought through for the monitoring stack, as for anything else.
## VMAgent
@ -46,6 +46,7 @@ EOF
```
### Sharding
Sharding for `VMAgent` distributes scraping between multiple deployments of `VMAgent`.
more info https://victoriametrics.github.io/vmagent.html#scraping-big-number-of-targets
@ -71,14 +72,40 @@ spec:
EOF
```
This configuration produces 5 deployments with 2 replicas at each. Each deployment has own shard num
This configuration produces 5 deployments with 2 replicas at each. Each deployment has its own shard num
and scrapes only 1/5 of all targets.
### StatefulMode
In `StatefulMode` `VMAgent` doesn't lose state of the PersistentQueue (file-based buffer size for unsent data) on pod restarts.
Operator creates `StatefulSet` and, with provided `PersistenVolumeClaimTemplate` at `StatefulStorage` configuration param, metrics queue is stored on disk.
Example of configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-persistent
spec:
selectAllByDefault: true
vmAgentExternalLabelName: vmagent-ha
replicaCount: 2
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
statefulMode: true
statefulStorage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
```
## VMAlert
It can be launched with multiple replicas without an additional configuration, alertmanager is responsible for alert deduplication.
Note, if you want to use `VMAlert` with high-available `VMAlertmanager`, which has more then 1 replica. You have to specify all pod fqdns
Note, if you want to use `VMAlert` with high-available `VMAlertmanager`, which has more than 1 replica. You have to specify all pod fqdns
at `VMAlert.spec.notifiers.[url]`. Or you can use service discovery for notifier, examples:
alertmanager:
@ -162,9 +189,9 @@ It doesn't support high availability by default, for such purpose use VMCluster
## VMCluster
Cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling.
The cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling.
For using cluster version you have to create corresponding CRD object:
For using the cluster version you have to create the corresponding CRD object:
```yaml
cat << EOF | kubectl apply -f -
@ -260,7 +287,7 @@ example-vmcluster-persistent 2 2 2 2s
example-vmcluster-persistent 2 2 2 30s operational
```
Get links for connection by executing command:
Get links for connection by executing the command:
```bash
kubectl get svc -l app.kubernetes.io/instance=example-vmcluster-persistent
@ -315,7 +342,7 @@ EOF
## Alertmanager
The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fires alerts against *all* instances of an Alertmanager cluster.
The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fire alerts against *all* instances of an Alertmanager cluster.
The Alertmanager, starting with the `v0.5.0` release, ships with a high availability mode. It implements a gossip protocol to synchronize instances of an Alertmanager cluster regarding notifications that have been sent out, to prevent duplicate notifications. It is an AP (available and partition tolerant) system. Being an AP system means that notifications are guaranteed to be sent at least once.

106
docs/operator/vmalertmanager-configuration.MD Normal file
View file

@ -0,0 +1,106 @@
---
sort: 13
---
# Managing configuration for VMAlertmanager
The operator generates a configuration file for `VMAlertmanager` based on user input at the definition of `CRD`.
Generated config stored at `Secret` created by the operator, it has the following name template `vmalertmanager-CRD_NAME-config`.
This configuration file is mounted at `VMAlertmanager` `Pod`. A special side-car container tracks its changes and sends config-reload signals to `alertmanager` container.
## Using secret
Basically, you can use the global configuration defined at manually created `Secret`. This `Secret` must be created before `VMAlertmanager`.
Name of the `Secret` must be defined at `VMAlertmanager` `spec.configSecret` option.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: vmalertmanager-example-alertmanager
labels:
app: vm-operator
type: Opaque
stringData:
alertmanager.yaml: |
global:
resolve_timeout: 5m
route:
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
replicaCount: 2
configSecret: vmalertmanager-example-alertmanager
```
## Using inline raw config
Also, if there is no secret data at configuration, or you just want to redefine some global variables for `alertmanager`.
You can define configuration at `spec.configRawYaml` section of `VMAlertmanager` configuration:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
replicaCount: 2
configRawYaml: |
global:
resolve_timeout: 5m
route:
receiver: 'default'
group_interval: 5m
repeat_interval: 12h
receivers:
- name: 'default'
```
If both `configSecret` and `configRawYaml` are defined, only configuration from `configRawYaml` will be used. Values from `configRawYaml` will be ignored.
## Using VMAlertmanagerConfig
`VMAlertmanagerConfig` allows delegating notification configuration to the kubernetes cluster users.
The application owner may configure notifications by defining it at `VMAlertmanagerConfig`.
With the combination of `VMRule` and `VMServiceScrape` it allows delegating configuration observability to application owners, and uses popular `GitOps` practice.
Operator combines `VMAlertmanagerConfigs` into a single configuration file for `VMAlertmanager`.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
name: example-email-web
namespace: production
spec:
route:
receiver: email
group_interval: 1m
routes:
- receiver: email
matchers:
- {severity =~ "warning|critical", app_name = "blog"}
receivers:
- name: email
email_configs:
- to: some-email@example.com
from: alerting@example.com
smarthost: example.com:25
text: ALARM
```
#### Special Case
VMAlertmanagerConfig has enforced namespace matcher.
Alerts must have a proper namespace label, with the same value as name of namespace for VMAlertmanagerConfig.
It can be disabled, by setting the following value to the VMAlertmanager: spec.disableNamespaceMatcher: true.
## behavior without provided config
If no configuration is provided, operator configures stub configuration with blackhole route.

19
docs/vmalert.md
View file

@ -1017,6 +1017,25 @@ It is recommended using
* `vmalert` is located in `vmutils-*` archives there.
### Docker image
You can build `vmalert` docker image from source and push it to your own docker repository.
Run the following commands from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics):
```bash
make package-vmalert
docker tag victoria-metrics/vmalert:version my-repo:my-version-name
docker push my-repo:my-version-name
```
To run the built image in `victoria-metrics-k8s-stack` or `VMAlert` CR object apply the following config change:
```yaml
kind: VMAlert
spec:
image:
repository: my-repo
tag: my-version-name
```
### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.17.