github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/docs/operator/quick-start.MD
Roman Khavronenko 284bda8746
docs: fix liquid syntax errors (#2592) 
For liquid text processor double braces `{{` `}}`
are special chars for templating.
Since we use them in some of our docs with different purpose,
we must escape them to avoid syntax errors from liquid.

For escaping curly braces we use bult-in plugin which helps
to enclose sections of text via `{% raw %}` and `{% endraw %}`.
This approach prevents liquid syntax errors and makes render correct.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-05-16 09:27:19 +02:00

1525 lines
49 KiB
Markdown
Raw Blame History

---
sort: 9
---
# Quick start
Operator serves to make running VictoriaMetrics applications on top of Kubernetes as easy as possible while preserving Kubernetes-native configuration options.
## Installing by Manifest
Obtain release from releases page:
[https://github.com/VictoriaMetrics/operator/releases](https://github.com/VictoriaMetrics/operator/releases)
We suggest use the latest release.
```bash
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
wget https://github.com/VictoriaMetrics/operator/releases/download/$VM_VERSION/bundle_crd.zip
unzip bundle_crd.zip
```
> TIP, operator use monitoring-system namespace, but you can install it to specific namespace with command
> sed -i "s/namespace: monitoring-system/namespace: YOUR_NAMESPACE/g" release/operator/*
First of all, you have to create [custom resource definitions](https://github.com/VictoriaMetrics/operator)
```bash
kubectl apply -f release/crds
```
Then you need RBAC for operator, relevant configuration for the release can be found at release/operator/rbac.yaml
Change configuration for operator at `release/operator/manager.yaml`, possible settings: [operator-settings](/vars.MD)
and apply it:
```bash
kubectl apply -f release/operator/
```
Check the status of operator
```bash
kubectl get pods -n monitoring-system
#NAME READY STATUS RESTARTS AGE
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
```
## Installing by Kustomize
You can install operator using [Kustomize](https://kustomize.io/) by pointing to the remote kustomization file.
```yaml
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
cat << EOF > kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/VictoriaMetrics/operator/config/default?ref=${VM_VERSION}
images:
- name: victoriametrics/operator
newTag: ${VM_VERSION}
EOF
```
You can change [operator-settings](/vars.MD), or use your custom namespace see [kustomize-example](https://github.com/YuriKravetc/yurikravetc.github.io/tree/main/Operator/kustomize-example).
Build template
```bash
kustomize build . -o monitoring.yaml
```
Apply manifests
```bash
kubectl apply -f monitoring.yaml
```
Check the status of operator
```bash
kubectl get pods -n monitoring-system
#NAME READY STATUS RESTARTS AGE
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
```
## Installing to ARM
There is no need in an additional configuration for ARM. Operator and VictoriaMetrics have full support for it.
## Create related resources
The VictoriaMetrics Operator introduces additional resources in Kubernetes to declare the desired state of a Victoria Metrics applications and Alertmanager cluster as well as the Prometheus resources configuration. The resources it introduces are:
* [VMSingle](#vmsingle)
* [VMCluster](#vmcluster)
* [VMAgent](#vmagent)
* [VMAlert](#vmalert)
* [VMAlertmanager](#vmalertmanager)
* [VMServiceScrape](#vmservicescrape)
* [VMRule](#vmrule)
* [VMPodScrape](#vmpodscrape)
* [VMProbe](#vmprobe)
* [VMStaticScrape](#vmstaticscrape)
* [VMAuth](#vmauth)
* [VMUser](#vmuser)
* [Selectors](#object-selectors)
## VMSingle
[VMSingle](https://github.com/VictoriaMetrics/VictoriaMetrics/) represents database for storing metrics, for all possible config options check api [doc](https://docs.victoriametrics.com/operator/api.html#vmsingle):
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
removePvcAfterDelete: true
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
```
Configuration can be extended with extraFlags and extraEnv, check flag documentation at [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/Single-server-VictoriaMetrics.md)
for instance:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
extraArgs:
dedup.minScrapeInterval: 60s
EOF
```
Flag can be replaced with envVar, it's useful for retrieving value from secret:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
extraEnvs:
- name: DEDUP_MINSCRAPEINTERVAL
valueFrom:
secretKeyRef:
name: vm-secret
key: dedup
EOF
```
## VMCluster
[VMCluster](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) represent a high-available and fault-tolerant version of VictoriaMetrics database.
For minimal version without persistent create simple custom resource definition:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster
spec:
# Add fields here
retentionPeriod: "1"
vmstorage:
replicaCount: 2
vmselect:
replicaCount: 2
vminsert:
replicaCount: 2
EOF
```
For persistent it recommends adding persistent storage to `vmstorage` and `vmselect`:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster-persistent
spec:
# Add fields here
retentionPeriod: "4"
replicationFactor: 2
vmstorage:
replicaCount: 2
storageDataPath: "/vm-data"
storage:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 10Gi
resources:
limits:
cpu: "0.5"
memory: 500Mi
vmselect:
replicaCount: 2
cacheMountPath: "/select-cache"
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 2Gi
resources:
limits:
cpu: "0.3"
memory: "300Mi"
vminsert:
replicaCount: 2
EOF
```
You may also create cluster version with external vmstorage
```yaml
cat << EOF | kubectl apply -f -
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
name: example-vmcluster
spec:
# Add fields here
retentionPeriod: "1"
vmselect:
replicaCount: 2
extraArgs:
storageNode: "node-1:8401,node-2:8401"
vminsert:
replicaCount: 2
extraArgs:
storageNode: "node-1:8401,node-2:8401"
EOF
```
## VMAgent
[VMAgent](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent) - is a tiny but brave agent, which helps you collect metrics from various sources and stores them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics).
It requires access to Kubernetes API and you can create RBAC for it first, it can be found at `release/examples/VMAgent_rbac.yaml`
Or you can use default rbac account, that will be created for `VMAgent` by operator automatically.
```bash
kubectl apply -f release/examples/vmagent_rbac.yaml
```
Modify `VMAgent` config parameters at `release/examples/vmagent.yaml` and apply it, config options [doc](https://docs.victoriametrics.com/operator/api.html#vmagent)
Example:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
selectAllByDefault: true
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
For connecting `VMAgent` to cluster, you have to specify url for it, check url building [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#url-format)
Example:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
selectAllByDefault: true
replicaCount: 1
remoteWrite:
- url: "http://vminsert-example-vmcluster.default.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"
EOF
```
## VMAlertmanager
`VMAlertmanager` - represents alertmanager configuration, first, you have to create secret with a configuration for it.
If there is no secret, default secret will be created with predefined content. It is possible to provide raw yaml config
to the alertmanager.
`VMAlertmanager` has exactly the same logic for filtering `VMAlertmanagerConfig` with `spec.configSelector` and `spec.configNamespaceSelector`
as `VMAgent` does:
- `spec.configNamespaceSelector` - if its nil, only `VMAlertmanagerConfig`s in the `VMAlertmanager` namespace can be selected.
If it's defined with empty value: `spec.configNamespaceSelector: {}` - all namespaces are matched.
You may also add matchLabels or matchExpression syntax, but your namespace must be tagged with specified labels first:
```yaml
spec.configNamespaceSelector:
matchLabels:
name: default
```
- `spec.configSelector` - it selects `VMAlertmanagerConfig` objects by its labels.
If it's nil, but `spec.configNamespaceSelector` isn't, the last one will be used for selecting.
If it's defined with empty value: `spec.configSelector: {}`, all objects depending on NamespaceSelector.
```yaml
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: alertmanager-config
stringData:
alertmanager.yaml: |
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
EOF
```
Then add `Alertmanager` object, other config options at [doc](https://docs.victoriametrics.com/operator/api.html#alertmanager)
you have to set configSecret with name of secret, that we created before - `alertmanager-config`.
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager
spec:
# Add fields here
replicaCount: 1
configSecret: alertmanager-config
selectAllByDefault: true
EOF
```
Alertmanager config with raw yaml configuration, use it with care about secret information:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
name: example-alertmanager-raw-config
spec:
# Add fields here
replicaCount: 1
configSecret: alertmanager-config
configRawYaml: |
global:
resolve_timeout: 5m
route:
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://localhost:30502/'
EOF
```
## VMAlertmanagerConfig
`VMAlertmanagerConfig` allows managing `VMAlertmanager` configuration.
```yaml
cat << EOF | kubectl apply -f
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
name: example
namespace: default
spec:
inhibit_rules:
- equals: []
target_matchers: []
source_matchers: []
route:
routes:
- receiver: webhook
continue: true
receiver: email
group_by: []
continue: false
matchers:
- job = "alertmanager"
group_wait: 30s
group_interval: 45s
repeat_interval: 1h
mute_time_intervals:
- name: base
time_intervals:
- times:
- start_time: ""
end_time: ""
weekdays: []
days_of_month: []
months: []
years: []
receivers:
email_configs: []
webhook_configs:
- url: http://some-other-wh
pagerduty_configs: []
pushover_configs: []
slack_configs: []
opsgenie_configs: []
victorops_configs: []
wechat_configs: []
EOF
```
## VMAlert
[VMAlert](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert) - executes a list of given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) rules against configured address. It
has few required config options - `datasource` and `notifier` are required, for other config parameters check [doc](https://docs.victoriametrics.com/operator/api.html#vmalert).
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
replicaCount: 1
datasource:
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
notifier:
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
evaluationInterval: "30s"
selectAllByDefault: true
EOF
```
## VMServiceScrape
It generates part of `VMAgent` configuration with `Endpoint` kubernetes_sd role for service discovery targets
by corresponding `Service` and it's `Endpoint`s.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
Let's make some demo, you have to deploy [VMAgent](#vmagent) and [VMSingle](#vmsingle) from previous step with match any selectors:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
removePvcAfterDelete: true
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
selectAllByDefault: true
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
Then deploy three instances of a simple example application, which listens and exposes metrics on port `8080`.
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-app
spec:
replicas: 3
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: example-app
image: fabxc/instrumented_app
ports:
- name: web
containerPort: 8080
EOF
```
Check status for pods:
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
example-app-594f97677c-g72v8 1/1 Running 0 23s
example-app-594f97677c-kn66h 1/1 Running 0 20s
example-app-594f97677c-l8xfd 1/1 Running 0 17s
vm-operator-646df47888-jvfvj 1/1 Running 0 63m
vmagent-example-vmagent-5777fdf7bf-tctcv 2/2 Running 1 34m
vmalertmanager-example-alertmanager-0 2/2 Running 0 11m
vmsingle-example-vmsingle-persisted-794b59ccc6-fnkpt 1/1 Running 0 36m
```
Checking logs for `VMAgent`:
```bash
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent
2020-08-02T18:18:17.226Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
2020-08-02T18:18:17.229Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
2020-08-02T18:18:17.289Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 2, removed targets: 0; total targets: 2
```
By default, operator creates `VMServiceScrape` for its components, so we have 2 targets - `VMAgent` and `VMSingle`.
Let's add monitoring to our application:
First we add the `Service` for application:
```yaml
cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
name: example-app
labels:
app: example-app
spec:
selector:
app: example-app
ports:
- name: web
port: 8080
EOF
```
`VMServiceScrape` has a label selector to select `Services` and their underlying Endpoint objects.
`Service` object for the example application selects the Pods by the `app` label having the `example-app` value.
`Service` object also specifies the port on which the metrics exposed.
The final part, add `VMServiceScrape`:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: example-app
labels:
team: frontend
spec:
selector:
matchLabels:
app: example-app
endpoints:
- port: web
EOF
```
The `VMServiceScrape` has no labels, but it will be selected by `VMAgent`, because we are launched `VMAgent` with empty selector.
Let's check `VMAgent` logs (you have to wait some time for config sync, usually its around 1 min):
```bash
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent --tail 100
2020-08-03T08:24:13.312Z info VictoriaMetrics/lib/promscrape/scraper.go:106 SIGHUP received; reloading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
2020-08-03T08:24:13.312Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
2020-08-03T08:24:13.315Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
2020-08-03T08:24:13.418Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 3, removed targets: 0; total targets: 5
```
3 new targets added by a config reload.
## VMPodScrape
It generates config part of `VMAgent` with kubernetes_sd role `pod`, that would match all `pods` having specific labels and ports.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
Add `VMAgent` and Example app from step above and continue this step.
Let's add 2 new pods:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
app: example-pod
owner: dev
env: test
name: example-app-pod-0
spec:
containers:
- image: fabxc/instrumented_app
imagePullPolicy: IfNotPresent
name: example-app
ports:
- containerPort: 8080
name: web
protocol: TCP
resources: {}
---
apiVersion: v1
kind: Pod
metadata:
labels:
app: example-pod-scrape
owner: dev
env: dev
name: example-app-pod-1
spec:
containers:
- image: fabxc/instrumented_app
imagePullPolicy: IfNotPresent
name: example-app
ports:
- containerPort: 8080
name: web
protocol: TCP
resources: {}
EOF
```
Ensure, that pods started:
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
example-app-594f97677c-g72v8 1/1 Running 0 3m40s
example-app-594f97677c-kn66h 1/1 Running 0 3m37s
example-app-594f97677c-l8xfd 1/1 Running 0 3m34s
example-app-pod-0 1/1 Running 0 14s
example-app-pod-1 1/1 Running 0 14s
vm-operator-646df47888-jvfvj 1/1 Running 0 67m
vmagent-example-vmagent-5777fdf7bf-tctcv 2/2 Running 1 37m
vmalertmanager-example-alertmanager-0 2/2 Running 0 14m
vmsingle-example-vmsingle-persisted-794b59ccc6-fnkpt 1/1 Running 0 40m
```
These pods have the same label `owner: dev` and port named `web`, so we can select them by `VMPodScrape` with matchLabels expression:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: example-pod-scrape
spec:
podMetricsEndpoints:
- port: web
scheme: http
selector:
matchLabels:
owner: dev
EOF
```
Lets check `VMAgent` logs:
```bash
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent --tail 100
2020-08-03T08:51:13.582Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
2020-08-03T08:51:13.585Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
2020-08-03T08:51:13.652Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 2, removed targets: 0; total targets: 7
```
2 new target was added.
## VMRule
It generates `VMAlert` config with ruleset defined at `VMRule` spec.
Lets create `VMAlert` with selector for `VMRule` with label project=devops.
You also need datasource from previous step [VMSingle](#vmsingle) and [VMAgent](#vmagent) connected to it.
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert
spec:
replicaCount: 1
datasource:
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
notifier:
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
evaluationInterval: "30s"
ruleSelector:
matchLabels:
project: devops
EOF
```
Ensure, that it started and ready:
```bash
kubectl get pods -l app.kubernetes.io/name=vmalert
NAME READY STATUS RESTARTS AGE
vmalert-example-vmalert-6f8748c6f9-hcfrr 2/2 Running 0 2m26s
kubectl logs vmalert-example-vmalert-6f8748c6f9-hcfrr vmalert
2020-08-03T08:52:21.990Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
2020-08-03T08:52:21.992Z info VictoriaMetrics/app/vmalert/group.go:153 group "vmAlertGroup" started with interval 30s
2020-08-03T08:52:21.992Z info VictoriaMetrics/lib/httpserver/httpserver.go:76 starting http server at http://:8080/
2020-08-03T08:52:21.993Z info VictoriaMetrics/lib/httpserver/httpserver.go:77 pprof handlers are exposed at http://:8080/debug/pprof/
```
`VMAlert` shipped with default rule `vmAlertGroup`
Let's add simple rule for `VMAlert` itself, `delta(vmalert_config_last_reload_errors_total[5m]) > 1`
{% raw %}
```yaml
cat << 'EOF' | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: example-vmrule-reload-config
labels:
project: devops
spec:
groups:
- name: vmalert
rules:
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
job: "{{ $labels.job }}"
annotations:
value: "{{ $value }}"
description: 'error reloading vmalert config, reload count for 5 min {{ $value }}'
EOF
```
{% endraw %}
Ensure, that new alert was started:
```bash
kubectl logs vmalert-example-vmalert-6f8748c6f9-hcfrr vmalert
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/web.go:45 api config reload was called, sending sighup
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/main.go:115 SIGHUP received. Going to reload rules ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"] ...
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/group.go:169 group "vmAlertGroup": received stop signal
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/main.go:124 Rules reloaded successfully from ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"]
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/group.go:153 group "vmalert" started with interval 30s
```
Let's trigger it by adding some incorrect rule
{% raw %}
```yaml
cat << 'EOF' | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: example-vmrule-incorrect-rule
labels:
project: devops
spec:
groups:
- name: incorrect rule
rules:
- alert: vmalert bad config
expr: bad expression
for: 10s
labels:
severity: major
annotations:
value: "{{ $badValue | bad function }}"
EOF
```
{% endraw %}
`VMAlert` will report incorrect rule config and fire alert:
```bash
2020-08-03T09:11:40.672Z info VictoriaMetrics/app/vmalert/main.go:115 SIGHUP received. Going to reload rules ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"] ...
2020-08-03T09:11:40.672Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
2020-08-03T09:11:40.673Z error VictoriaMetrics/app/vmalert/main.go:119 error while reloading rules: cannot parse configuration file: invalid group "incorrect rule" in file "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/default-example-vmrule-incorrect-rule.yaml": invalid rule "incorrect rule"."vmalert bad config": invalid expression: unparsed data left: "expression"
```
Clean up incorrect rule:
```bash
kubectl delete vmrule example-vmrule-incorrect-rule
```
## VMNodeScrape
`VMNodeScrape` is useful for node exporters monitoring, lets create scraper for cadvisor metrics:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
name: cadvisor-metrics
spec:
scheme: "https"
tlsConfig:
insecureSkipVerify: true
caFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token"
relabelConfigs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- targetLabel: __address__
replacement: kubernetes.default.svc:443
- sourceLabels: [__meta_kubernetes_node_name]
regex: (.+)
targetLabel: __metrics_path__
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
EOF
```
## VMProbe
`VMProbe` required `VMAgent` and some external prober, blackbox exporter in our case. Ensure that you have `VMAgent` and `VMSingle`:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example-vmsingle-persisted
spec:
retentionPeriod: "1"
removePvcAfterDelete: true
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
selectAllByDefault: true
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
Then add `BlackBox` exporter with simple config:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
data:
blackbox.yaml: |
modules:
http_2xx:
http:
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/1.1
- HTTP/2.0
valid_status_codes: []
prober: http
timeout: 5s
---
kind: Service
apiVersion: v1
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
type: ClusterIP
ports:
- name: http
port: 9115
protocol: TCP
selector:
app: prometheus-blackbox-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-blackbox-exporter
labels:
app: prometheus-blackbox-exporter
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-blackbox-exporter
template:
metadata:
labels:
app: prometheus-blackbox-exporter
spec:
containers:
- name: blackbox-exporter
image: "prom/blackbox-exporter:v0.17.0"
args:
- "--config.file=/config/blackbox.yaml"
resources:
{}
ports:
- containerPort: 9115
name: http
livenessProbe:
httpGet:
path: /health
port: http
readinessProbe:
httpGet:
path: /health
port: http
volumeMounts:
- mountPath: /config
name: config
volumes:
- name: config
configMap:
name: prometheus-blackbox-exporter
EOF
```
Ensure, that pods are ready:
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
prometheus-blackbox-exporter-5b5f44bd9c-2szdj 1/1 Running 0 3m3s
vmagent-example-vmagent-84886cd4d9-x8ml6 2/2 Running 1 9m22s
vmsingle-example-vmsingle-persisted-8584486b68-mqg6b 1/1 Running 0 11m
```
Now define some `VMProbe`, lets start with basic static target and probe `VMAgent` with its service address, for accessing
blackbox exporter, you have to specify its url at `VMProbe` config. Lets get both services names:
```bash
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h21m
prometheus-blackbox-exporter ClusterIP 10.105.251.80 <none> 9115/TCP 4m36s
vmagent-example-vmagent ClusterIP 10.102.31.47 <none> 8429/TCP 12m
vmsingle-example-vmsingle-persisted ClusterIP 10.107.69.7 <none> 8429/TCP 12m
```
So, we will probe `VMAgent` with url - `vmagent-example-vmagent.default.svc:9115/heath` with blackbox url:
`prometheus-blackbox-exporter.default.svc:9115` and module: `http_2xx` it was specified at blackbox configmap.
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
name: probe-agent
spec:
jobName: static-probe
vmProberSpec:
# by default scheme http, and path is /probe
url: prometheus-blackbox-exporter.default.svc:9115
module: http_2xx
targets:
staticConfig:
targets:
- vmagent-example-vmagent.default.svc:8429/health
interval: 2s
EOF
```
Now new target must be added to `VMAgent` configuration, and it starts probing itself throw blackbox exporter.
Let's try another target probe type - `Ingress`. Create ingress rule for `VMSingle` and create `VMProbe` for it:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
labels:
app: victoria-metrics-single
name: victoria-metrics-single
spec:
rules:
- host: vmsingle.example.com
http:
paths:
- backend:
serviceName: vmsingle-example-vmsingle-persisted
servicePort: 8428
path: /
- host: vmsingle2.example.com
http:
paths:
- backend:
serviceName: vmsingle-example-vmsingle-persisted
servicePort: 8428
path: /
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMProbe
metadata:
name: probe-single-ingress
spec:
vmProberSpec:
# by default scheme http, and path is /probe
url: prometheus-blackbox-exporter.default.svc:9115
module: http_2xx
targets:
ingress:
selector:
matchLabels:
app: victoria-metrics-single
interval: 10s
EOF
```
This configuration will add 2 additional targets for probing: `vmsingle2.example.com` and `vmsingle.example.com`.
But probes will be unsuccessful, coz there is no such hosts.
## VMStaticScrape
It generates config part of `VMAgent` with static_configs, targets for targetEndpoint is a required parameter.
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
Add `VMAgent` and Example app from step above and continue this step.
With simple configuration:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMStaticScrape
metadata:
name: vmstaticscrape-sample
spec:
jobName: static
targetEndpoints:
- targets: ["192.168.0.1:9100","196.168.0.50:9100"]
labels:
env: dev
project: operator
EOF
```
2 targets must be added to `VMAgent` scrape config:
```bash
static_configs: added targets: 2, removed targets: 0; total targets: 2
```
## VMAuth
[VMAuth](https://docs.victoriametrics.com/vmauth.html) allows protecting application with authentication and route traffic by rules.
api docs [link](https://docs.victoriametrics.com/operator/api.html#vmauthspec)
First create `VMAuth` configuration:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
name: example
namespace: default
spec:
ingress: {}
selectAllByDefault: true
EOF
```
It will catch all `VMUser` at any kubernetes namespace and create `Ingress` record for it.
```text
kubectl get pods
NAME READY STATUS RESTARTS AGE
vmauth-example-ffcc78fcc-xddk7 2/2 Running 0 84s
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
vmauth-example <none> * 80 106s
kubectl get secret -l app.kubernetes.io/name=vmauth
NAME TYPE DATA AGE
vmauth-config-example Opaque 1 2m32s
```
Generated configuration can be retrieved with command:
{% raw %}
```text
kubectl get secrets/vmauth-config-example -o=go-template='{{index .data "config.yaml.gz"}}' | base64 -d | gunzip
users:
- url_prefix: http://localhost:8428
bearer_token: some-default-token
```
{% endraw %}
Operator generates default config, if `VMUser`s for given `VMAuth` wasn't found.
## VMUser
`VMUser` configures `VMAuth`. api doc [link](https://docs.victoriametrics.com/operator/api.html#vmuserspec)
There are two authentication mechanisms: `bearerToken` and `basicAuth` with `username` and `password`. Only one of them can be used with `VMUser` at one time.
If you need to provide access with different mechanisms for single endpoint, create multiple `VMUsers`.
If `username` is empty, metadata.name from `VMUser` used as `username`.
If `password` is empty, operator generates random password for `VMUser`. This password added to the `Secret` for this `VMUser` at `data.password` field.
Operator creates `Secret` for every `VMUser` with name - `vmuser-{VMUser.metadata.name}`. It places `username` + `password` or `bearerToken` into `data` section.
`TargetRefs` is required field for `VMUser`, it allows to configure routing with:
- `static` ref:
```yaml
- static:
url: http://vmalertmanager.service.svc:9093
```
- `crd` ref, allows to target CRD kind of operator, this `CRDObject` must exist.
```yaml
- crd:
kind: VMAgent
name: example
namespace: default
```
Supported kinds are: `VMAgent, VMSingle, VMAlert, VMAlertmanager, VMCluster/vminsert, VMCluster/vmselect, VMCluster/vmstorage`
`paths` - configures allowed routing paths for given `targetRef`.
Let's create example, with access to `VMSingle` and `VMAlert` as static target:
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
name: example
namespace: default
spec:
retentionPeriod: "2d"
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example
spec:
replicaCount: 1
datasource:
url: "http://vmsingle-example.default.svc:8429"
notifier:
url: "http://vmalertmanager-example.default.svc:9093"
evaluationInterval: "20s"
ruleSelector: {}
EOF
```
Check its status
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
vmalert-example-775b8dfbc9-vzlnv 1/2 Running 0 3s
vmauth-example-ffcc78fcc-xddk7 2/2 Running 0 29m
vmsingle-example-6496b5c95d-k6hhp 1/1 Running 0 3s
```
Then create `VMUser`
```yaml
cat << EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMUser
metadata:
name: example
spec:
username: simple-user
password: simple-password
targetRefs:
- crd:
kind: VMSingle
name: example
namespace: default
paths: ["/.*"]
- static:
url: http://vmalert-example.default.svc:8080
paths: ["/api/v1/groups","/api/v1/alerts"]
EOF
```
Configuration changes for `VMAuth` takes some time, coz of mounted secret, its eventually updated by kubelet. Check vmauth log for changes:
```bash
kubectl logs vmauth-example-ffcc78fcc-xddk7 vmauth -f --tail 10
2021-05-31T10:46:40.171Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
2021-05-31T10:46:40.171Z info VictoriaMetrics/app/vmauth/main.go:37 started vmauth in 0.000 seconds
2021-05-31T10:46:40.171Z info VictoriaMetrics/lib/httpserver/httpserver.go:82 starting http server at http://:8427/
2021-05-31T10:46:40.171Z info VictoriaMetrics/lib/httpserver/httpserver.go:83 pprof handlers are exposed at http://:8427/debug/pprof/
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:143 SIGHUP received; loading -auth.config="/opt/vmauth/config.yaml"
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:150 Successfully reloaded -auth.config="/opt/vmauth/config.yaml"
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:143 SIGHUP received; loading -auth.config="/opt/vmauth/config.yaml"
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:150 Successfully reloaded -auth.config="/opt/vmauth/config.yaml"
```
Now lets try to access protected endpoints, i will use port-forward for that:
```bash
kubectl port-forward vmauth-example-ffcc78fcc-xddk7 8427
# at separate terminal execute:
# vmsingle response
curl http://localhost:8427 -u 'simple-user:simple-password'
# vmalert response
curl localhost:8427/api/v1/groups -u 'simple-user:simple-password'
```
Check create secret for application access:
```bash
kubectl get secrets vmuser-example
NAME TYPE DATA AGE
vmuser-example Opaque 2 6m33s
```
## Migration from prometheus-operator objects
By default, the operator converts all existing prometheus-operator API objects into corresponding VictoriaMetrics Operator objects
You can control this behaviour by setting env variable for operator:
```bash
#disable convertion for each object
VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR=false
VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE=false
VM_ENABLEDPROMETHEUSCONVERTER_PROBE=false
```
Otherwise, VictoriaMetrics Operator would try to discover prometheus-operator API and convert it.
Conversion of api objects can be controlled by annotations, added to `VMObject`s, there are following annotations:
- `operator.victoriametrics.com/merge-meta-strategy` - it controls syncing of metadata labels and annotations between
`VMObject`s and `Prometheus` api objects during updates to `Prometheus` objects. By default, it has `prefer-prometheus`.
And annotations and labels will be used from `Prometheus` objects, manually set values will be dropped.
You can set it to `prefer-victoriametrics`. In this case all labels and annotations applied to `Prometheus` object
will be ignored and `VMObject` will use own values.
Two additional strategies annotations -`merge-victoriametrics-priority` and `merge-prometheus-priority` merges labelSets into one combined labelSet, with priority.
Example:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
- `operator.victoriametrics.com/ignore-prometheus-updates` - it controls updates from Prometheus api objects.
By default, it set to `disabled`. You define it to `enabled` state and all updates from Prometheus api objects will be
ignored.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
annotations:
meta.helm.sh/release-name: prometheus
operator.victoriametrics.com/ignore-prometheus-updates: enabled
labels:
release: prometheus
name: prometheus-monitor
spec:
endpoints: []
```
By default the operator doesn't make converted objects disappear after original ones are deleted. To change this behaviour
configure adding `OwnerReferences` to converted objects:
```bash
VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true
```
Converted objects will be linked to the original ones and will be deleted by kubernetes after the original ones are deleted.
### prometheus Rule duplication
`Prometheus` allows to specify rules with the same content with-in one group at Rule spec, but its forbidden by vmalert.
You can tell operator to deduplicate this rules by adding annotation to the `VMAlert` crd definition. In this case operator
skips rule with the same values, see example below.
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: example-vmalert-with-dedup
annotations:
operator.victoriametrics.com/vmalert-deduplicate-rules: "true"
spec:
replicaCount: 1
datasource:
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
notifier:
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
evaluationInterval: "30s"
ruleNamespaceSelector: {}
ruleSelector: {}
```
Now operator will transform this `VMRule`:
```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
name: example-vmrule-reload-config
labels:
project: devops
spec:
groups:
- name: vmalert
rules:
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 2m
labels:
severity: critical
```
to the rule config:
```yaml
groups:
- name: vmalert
rules:
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 10s
labels:
severity: major
- alert: vmalert config reload error
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
for: 2m
labels:
severity: critical
```
## Expose the VMSingle API
> WARNING: Please protect delete endpoint before exposing it [doc](https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-delete-time-series)
Example for Kubernetes Nginx ingress [doc](https://kubernetes.github.io/ingress-nginx/examples/auth/basic/)
```bash
#generate creds
htpasswd -c auth foo
#create basic auth secret
cat <<EOF | kubectl apply -f -
apiVersion: v1
stringData:
auth: foo:$apr1$wQ0fkxnl$KKOA/OqIZy48CuCB/A1Rc.
kind: Secret
metadata:
name: vm-auth
type: Opaque
EOF
#create ingress rule
cat << EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
nginx.ingress.kubernetes.io/auth-secret: vm-auth
nginx.ingress.kubernetes.io/auth-type: basic
labels:
app: victoria-metrics-single
name: victoria-metrics-single
spec:
rules:
- host: vmsingle.example.com
http:
paths:
- backend:
serviceName: vmsingle-example-vmsingle-persisted
servicePort: 8428
path: /
tls:
- hosts:
- vmsingle.example.com
secretName: ssl-name
EOF
#now you can access victoria metrics
curl -u foo:bar https://vmsingle.example/com/metrics
```
## Object selectors
`VMAgent` has four selectors specification for filtering `VMServiceScrape`, `VMProbe`, `VMNodeScrape` and `VMPodScrape` objects.
`VMAlert` has selectors for `VMRule`.
`VMAuth` has selectors for `VMUsers`.
`VMAlertmanager` has selectors for `VMAlertmanagerConfig`.
Selectors are defined with suffixes - `NamespaceSelector` and `Selector`. It allows configuring objects access control across namespaces and different environments.
Following rules are applied:
- `NamespaceSelector` and `Selector` - undefined, by default select nothing. With option set - `spec.selectAllByDefault: true`, select all objects of given type.
- `NamespaceSelector` defined, `Selector` undefined. All objects are matching at namespaces for given `NamespaceSelector`.
- `NamespaceSelector` undefined, `Selector` defined. All objects at `VMAgent`'s namespaces are matching for given `Selector`.
- `NamespaceSelector` defined, `Selector` defined. Only objects at namespaces matched `NamespaceSelector` for given `Selector` are matching.
### Object selector Examples
This configuration will match only namespace with the label - name=default and all `VMServiceScrape` objects at this namespace will be
selected:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeSelector: {}
serviceScrapeNamespaceSelector:
matchLabels:
name: default
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
This configuration will select `VMServiceScrape` and `VMPodScrape` with labels - team=managers,project=cloud at any namespace:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeNamespaceSelector: {}
serviceScrapeSelector:
matchLabels:
team: managers
project: cloud
podScrapeSelector:
matchLabels:
team: managers
project: clo
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
The last one will select any `VMServiceScrape`,`VMProbe` and `VMPodScrape` at `VMAgent` namespace:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: example-vmagent
spec:
serviceScrapeSelector: {}
podScrapeSelector: {}
probeSelector: {}
replicaCount: 1
remoteWrite:
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
EOF
```
Reference in a new issue View git blame Copy permalink