mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-01 14:47:38 +00:00
284bda8746
For liquid text processor double braces `{{` `}}` are special chars for templating. Since we use them in some of our docs with different purpose, we must escape them to avoid syntax errors from liquid. For escaping curly braces we use bult-in plugin which helps to enclose sections of text via `{% raw %}` and `{% endraw %}`. This approach prevents liquid syntax errors and makes render correct. Signed-off-by: hagen1778 <roman@victoriametrics.com>
1525 lines
49 KiB
Markdown
1525 lines
49 KiB
Markdown
---
|
|
sort: 9
|
|
---
|
|
|
|
# Quick start
|
|
|
|
Operator serves to make running VictoriaMetrics applications on top of Kubernetes as easy as possible while preserving Kubernetes-native configuration options.
|
|
|
|
## Installing by Manifest
|
|
|
|
Obtain release from releases page:
|
|
[https://github.com/VictoriaMetrics/operator/releases](https://github.com/VictoriaMetrics/operator/releases)
|
|
|
|
We suggest use the latest release.
|
|
|
|
```bash
|
|
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
|
|
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
|
|
wget https://github.com/VictoriaMetrics/operator/releases/download/$VM_VERSION/bundle_crd.zip
|
|
unzip bundle_crd.zip
|
|
```
|
|
|
|
> TIP, operator use monitoring-system namespace, but you can install it to specific namespace with command
|
|
> sed -i "s/namespace: monitoring-system/namespace: YOUR_NAMESPACE/g" release/operator/*
|
|
|
|
First of all, you have to create [custom resource definitions](https://github.com/VictoriaMetrics/operator)
|
|
```bash
|
|
kubectl apply -f release/crds
|
|
```
|
|
|
|
Then you need RBAC for operator, relevant configuration for the release can be found at release/operator/rbac.yaml
|
|
|
|
Change configuration for operator at `release/operator/manager.yaml`, possible settings: [operator-settings](/vars.MD)
|
|
and apply it:
|
|
```bash
|
|
kubectl apply -f release/operator/
|
|
```
|
|
|
|
Check the status of operator
|
|
|
|
```bash
|
|
kubectl get pods -n monitoring-system
|
|
|
|
#NAME READY STATUS RESTARTS AGE
|
|
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
|
|
|
|
```
|
|
|
|
|
|
## Installing by Kustomize
|
|
|
|
You can install operator using [Kustomize](https://kustomize.io/) by pointing to the remote kustomization file.
|
|
|
|
```yaml
|
|
# Get latest release version from https://github.com/VictoriaMetrics/operator/releases/latest
|
|
export VM_VERSION=`basename $(curl -fs -o/dev/null -w %{redirect_url} https://github.com/VictoriaMetrics/operator/releases/latest)`
|
|
|
|
cat << EOF > kustomization.yaml
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- github.com/VictoriaMetrics/operator/config/default?ref=${VM_VERSION}
|
|
|
|
images:
|
|
- name: victoriametrics/operator
|
|
newTag: ${VM_VERSION}
|
|
EOF
|
|
```
|
|
|
|
|
|
You can change [operator-settings](/vars.MD), or use your custom namespace see [kustomize-example](https://github.com/YuriKravetc/yurikravetc.github.io/tree/main/Operator/kustomize-example).
|
|
|
|
|
|
|
|
Build template
|
|
|
|
```bash
|
|
kustomize build . -o monitoring.yaml
|
|
```
|
|
|
|
Apply manifests
|
|
|
|
```bash
|
|
kubectl apply -f monitoring.yaml
|
|
```
|
|
|
|
Check the status of operator
|
|
|
|
```bash
|
|
kubectl get pods -n monitoring-system
|
|
|
|
#NAME READY STATUS RESTARTS AGE
|
|
#vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s
|
|
|
|
```
|
|
|
|
## Installing to ARM
|
|
|
|
There is no need in an additional configuration for ARM. Operator and VictoriaMetrics have full support for it.
|
|
|
|
## Create related resources
|
|
|
|
The VictoriaMetrics Operator introduces additional resources in Kubernetes to declare the desired state of a Victoria Metrics applications and Alertmanager cluster as well as the Prometheus resources configuration. The resources it introduces are:
|
|
|
|
* [VMSingle](#vmsingle)
|
|
* [VMCluster](#vmcluster)
|
|
* [VMAgent](#vmagent)
|
|
* [VMAlert](#vmalert)
|
|
* [VMAlertmanager](#vmalertmanager)
|
|
* [VMServiceScrape](#vmservicescrape)
|
|
* [VMRule](#vmrule)
|
|
* [VMPodScrape](#vmpodscrape)
|
|
* [VMProbe](#vmprobe)
|
|
* [VMStaticScrape](#vmstaticscrape)
|
|
* [VMAuth](#vmauth)
|
|
* [VMUser](#vmuser)
|
|
* [Selectors](#object-selectors)
|
|
|
|
## VMSingle
|
|
|
|
[VMSingle](https://github.com/VictoriaMetrics/VictoriaMetrics/) represents database for storing metrics, for all possible config options check api [doc](https://docs.victoriametrics.com/operator/api.html#vmsingle):
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example-vmsingle-persisted
|
|
spec:
|
|
retentionPeriod: "1"
|
|
removePvcAfterDelete: true
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
EOF
|
|
```
|
|
|
|
Configuration can be extended with extraFlags and extraEnv, check flag documentation at [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/Single-server-VictoriaMetrics.md)
|
|
for instance:
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example-vmsingle-persisted
|
|
spec:
|
|
retentionPeriod: "1"
|
|
extraArgs:
|
|
dedup.minScrapeInterval: 60s
|
|
EOF
|
|
```
|
|
|
|
Flag can be replaced with envVar, it's useful for retrieving value from secret:
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example-vmsingle-persisted
|
|
spec:
|
|
retentionPeriod: "1"
|
|
extraEnvs:
|
|
- name: DEDUP_MINSCRAPEINTERVAL
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: vm-secret
|
|
key: dedup
|
|
EOF
|
|
```
|
|
|
|
## VMCluster
|
|
|
|
[VMCluster](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) represent a high-available and fault-tolerant version of VictoriaMetrics database.
|
|
For minimal version without persistent create simple custom resource definition:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMCluster
|
|
metadata:
|
|
name: example-vmcluster
|
|
spec:
|
|
# Add fields here
|
|
retentionPeriod: "1"
|
|
vmstorage:
|
|
replicaCount: 2
|
|
vmselect:
|
|
replicaCount: 2
|
|
vminsert:
|
|
replicaCount: 2
|
|
EOF
|
|
```
|
|
|
|
For persistent it recommends adding persistent storage to `vmstorage` and `vmselect`:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMCluster
|
|
metadata:
|
|
name: example-vmcluster-persistent
|
|
spec:
|
|
# Add fields here
|
|
retentionPeriod: "4"
|
|
replicationFactor: 2
|
|
vmstorage:
|
|
replicaCount: 2
|
|
storageDataPath: "/vm-data"
|
|
storage:
|
|
volumeClaimTemplate:
|
|
spec:
|
|
storageClassName: standard
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|
|
resources:
|
|
limits:
|
|
cpu: "0.5"
|
|
memory: 500Mi
|
|
vmselect:
|
|
replicaCount: 2
|
|
cacheMountPath: "/select-cache"
|
|
storage:
|
|
volumeClaimTemplate:
|
|
spec:
|
|
resources:
|
|
requests:
|
|
storage: 2Gi
|
|
resources:
|
|
limits:
|
|
cpu: "0.3"
|
|
memory: "300Mi"
|
|
vminsert:
|
|
replicaCount: 2
|
|
EOF
|
|
```
|
|
|
|
You may also create cluster version with external vmstorage
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMCluster
|
|
metadata:
|
|
name: example-vmcluster
|
|
spec:
|
|
# Add fields here
|
|
retentionPeriod: "1"
|
|
vmselect:
|
|
replicaCount: 2
|
|
extraArgs:
|
|
storageNode: "node-1:8401,node-2:8401"
|
|
vminsert:
|
|
replicaCount: 2
|
|
extraArgs:
|
|
storageNode: "node-1:8401,node-2:8401"
|
|
EOF
|
|
```
|
|
|
|
## VMAgent
|
|
|
|
[VMAgent](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent) - is a tiny but brave agent, which helps you collect metrics from various sources and stores them in [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics).
|
|
It requires access to Kubernetes API and you can create RBAC for it first, it can be found at `release/examples/VMAgent_rbac.yaml`
|
|
Or you can use default rbac account, that will be created for `VMAgent` by operator automatically.
|
|
|
|
```bash
|
|
kubectl apply -f release/examples/vmagent_rbac.yaml
|
|
```
|
|
|
|
Modify `VMAgent` config parameters at `release/examples/vmagent.yaml` and apply it, config options [doc](https://docs.victoriametrics.com/operator/api.html#vmagent)
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
selectAllByDefault: true
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|
|
|
|
For connecting `VMAgent` to cluster, you have to specify url for it, check url building [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#url-format)
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
selectAllByDefault: true
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vminsert-example-vmcluster.default.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"
|
|
EOF
|
|
|
|
```
|
|
|
|
|
|
## VMAlertmanager
|
|
|
|
`VMAlertmanager` - represents alertmanager configuration, first, you have to create secret with a configuration for it.
|
|
If there is no secret, default secret will be created with predefined content. It is possible to provide raw yaml config
|
|
to the alertmanager.
|
|
|
|
`VMAlertmanager` has exactly the same logic for filtering `VMAlertmanagerConfig` with `spec.configSelector` and `spec.configNamespaceSelector`
|
|
as `VMAgent` does:
|
|
- `spec.configNamespaceSelector` - if its nil, only `VMAlertmanagerConfig`s in the `VMAlertmanager` namespace can be selected.
|
|
If it's defined with empty value: `spec.configNamespaceSelector: {}` - all namespaces are matched.
|
|
You may also add matchLabels or matchExpression syntax, but your namespace must be tagged with specified labels first:
|
|
```yaml
|
|
spec.configNamespaceSelector:
|
|
matchLabels:
|
|
name: default
|
|
```
|
|
- `spec.configSelector` - it selects `VMAlertmanagerConfig` objects by its labels.
|
|
If it's nil, but `spec.configNamespaceSelector` isn't, the last one will be used for selecting.
|
|
If it's defined with empty value: `spec.configSelector: {}`, all objects depending on NamespaceSelector.
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: alertmanager-config
|
|
stringData:
|
|
alertmanager.yaml: |
|
|
global:
|
|
resolve_timeout: 5m
|
|
route:
|
|
group_by: ['job']
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 12h
|
|
receiver: 'webhook'
|
|
receivers:
|
|
- name: 'webhook'
|
|
webhook_configs:
|
|
- url: 'http://alertmanagerwh:30500/'
|
|
EOF
|
|
```
|
|
|
|
Then add `Alertmanager` object, other config options at [doc](https://docs.victoriametrics.com/operator/api.html#alertmanager)
|
|
you have to set configSecret with name of secret, that we created before - `alertmanager-config`.
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlertmanager
|
|
metadata:
|
|
name: example-alertmanager
|
|
spec:
|
|
# Add fields here
|
|
replicaCount: 1
|
|
configSecret: alertmanager-config
|
|
selectAllByDefault: true
|
|
EOF
|
|
```
|
|
|
|
Alertmanager config with raw yaml configuration, use it with care about secret information:
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlertmanager
|
|
metadata:
|
|
name: example-alertmanager-raw-config
|
|
spec:
|
|
# Add fields here
|
|
replicaCount: 1
|
|
configSecret: alertmanager-config
|
|
configRawYaml: |
|
|
global:
|
|
resolve_timeout: 5m
|
|
route:
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 12h
|
|
receiver: 'webhook'
|
|
receivers:
|
|
- name: 'webhook'
|
|
webhook_configs:
|
|
- url: 'http://localhost:30502/'
|
|
EOF
|
|
```
|
|
|
|
|
|
## VMAlertmanagerConfig
|
|
|
|
`VMAlertmanagerConfig` allows managing `VMAlertmanager` configuration.
|
|
|
|
```yaml
|
|
|
|
cat << EOF | kubectl apply -f
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlertmanagerConfig
|
|
metadata:
|
|
name: example
|
|
namespace: default
|
|
spec:
|
|
inhibit_rules:
|
|
- equals: []
|
|
target_matchers: []
|
|
source_matchers: []
|
|
route:
|
|
routes:
|
|
- receiver: webhook
|
|
continue: true
|
|
receiver: email
|
|
group_by: []
|
|
continue: false
|
|
matchers:
|
|
- job = "alertmanager"
|
|
group_wait: 30s
|
|
group_interval: 45s
|
|
repeat_interval: 1h
|
|
mute_time_intervals:
|
|
- name: base
|
|
time_intervals:
|
|
- times:
|
|
- start_time: ""
|
|
end_time: ""
|
|
weekdays: []
|
|
days_of_month: []
|
|
months: []
|
|
years: []
|
|
receivers:
|
|
email_configs: []
|
|
webhook_configs:
|
|
- url: http://some-other-wh
|
|
pagerduty_configs: []
|
|
pushover_configs: []
|
|
slack_configs: []
|
|
opsgenie_configs: []
|
|
victorops_configs: []
|
|
wechat_configs: []
|
|
EOF
|
|
```
|
|
|
|
## VMAlert
|
|
|
|
[VMAlert](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert) - executes a list of given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) rules against configured address. It
|
|
has few required config options - `datasource` and `notifier` are required, for other config parameters check [doc](https://docs.victoriametrics.com/operator/api.html#vmalert).
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlert
|
|
metadata:
|
|
name: example-vmalert
|
|
spec:
|
|
replicaCount: 1
|
|
datasource:
|
|
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
|
|
notifier:
|
|
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
|
|
evaluationInterval: "30s"
|
|
selectAllByDefault: true
|
|
|
|
EOF
|
|
```
|
|
|
|
## VMServiceScrape
|
|
|
|
It generates part of `VMAgent` configuration with `Endpoint` kubernetes_sd role for service discovery targets
|
|
by corresponding `Service` and it's `Endpoint`s.
|
|
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
|
|
|
|
Let's make some demo, you have to deploy [VMAgent](#vmagent) and [VMSingle](#vmsingle) from previous step with match any selectors:
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example-vmsingle-persisted
|
|
spec:
|
|
retentionPeriod: "1"
|
|
removePvcAfterDelete: true
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
selectAllByDefault: true
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|
|
|
|
|
|
|
|
Then deploy three instances of a simple example application, which listens and exposes metrics on port `8080`.
|
|
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: example-app
|
|
spec:
|
|
replicas: 3
|
|
selector:
|
|
matchLabels:
|
|
app: example-app
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: example-app
|
|
spec:
|
|
containers:
|
|
- name: example-app
|
|
image: fabxc/instrumented_app
|
|
ports:
|
|
- name: web
|
|
containerPort: 8080
|
|
EOF
|
|
```
|
|
|
|
Check status for pods:
|
|
```bash
|
|
kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
example-app-594f97677c-g72v8 1/1 Running 0 23s
|
|
example-app-594f97677c-kn66h 1/1 Running 0 20s
|
|
example-app-594f97677c-l8xfd 1/1 Running 0 17s
|
|
vm-operator-646df47888-jvfvj 1/1 Running 0 63m
|
|
vmagent-example-vmagent-5777fdf7bf-tctcv 2/2 Running 1 34m
|
|
vmalertmanager-example-alertmanager-0 2/2 Running 0 11m
|
|
vmsingle-example-vmsingle-persisted-794b59ccc6-fnkpt 1/1 Running 0 36m
|
|
|
|
```
|
|
|
|
Checking logs for `VMAgent`:
|
|
```bash
|
|
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent
|
|
2020-08-02T18:18:17.226Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
|
|
2020-08-02T18:18:17.229Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
|
|
2020-08-02T18:18:17.289Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 2, removed targets: 0; total targets: 2
|
|
```
|
|
By default, operator creates `VMServiceScrape` for its components, so we have 2 targets - `VMAgent` and `VMSingle`.
|
|
|
|
Let's add monitoring to our application:
|
|
|
|
First we add the `Service` for application:
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
kind: Service
|
|
apiVersion: v1
|
|
metadata:
|
|
name: example-app
|
|
labels:
|
|
app: example-app
|
|
spec:
|
|
selector:
|
|
app: example-app
|
|
ports:
|
|
- name: web
|
|
port: 8080
|
|
EOF
|
|
```
|
|
|
|
`VMServiceScrape` has a label selector to select `Services` and their underlying Endpoint objects.
|
|
`Service` object for the example application selects the Pods by the `app` label having the `example-app` value.
|
|
`Service` object also specifies the port on which the metrics exposed.
|
|
|
|
The final part, add `VMServiceScrape`:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMServiceScrape
|
|
metadata:
|
|
name: example-app
|
|
labels:
|
|
team: frontend
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app: example-app
|
|
endpoints:
|
|
- port: web
|
|
EOF
|
|
```
|
|
The `VMServiceScrape` has no labels, but it will be selected by `VMAgent`, because we are launched `VMAgent` with empty selector.
|
|
|
|
Let's check `VMAgent` logs (you have to wait some time for config sync, usually its around 1 min):
|
|
|
|
```bash
|
|
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent --tail 100
|
|
2020-08-03T08:24:13.312Z info VictoriaMetrics/lib/promscrape/scraper.go:106 SIGHUP received; reloading Prometheus configs from "/etc/vmagent/config_out/vmagent.env.yaml"
|
|
2020-08-03T08:24:13.312Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
|
|
2020-08-03T08:24:13.315Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
|
|
2020-08-03T08:24:13.418Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 3, removed targets: 0; total targets: 5
|
|
|
|
```
|
|
|
|
3 new targets added by a config reload.
|
|
|
|
## VMPodScrape
|
|
|
|
It generates config part of `VMAgent` with kubernetes_sd role `pod`, that would match all `pods` having specific labels and ports.
|
|
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
|
|
|
|
Add `VMAgent` and Example app from step above and continue this step.
|
|
|
|
Let's add 2 new pods:
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
labels:
|
|
app: example-pod
|
|
owner: dev
|
|
env: test
|
|
name: example-app-pod-0
|
|
spec:
|
|
containers:
|
|
- image: fabxc/instrumented_app
|
|
imagePullPolicy: IfNotPresent
|
|
name: example-app
|
|
ports:
|
|
- containerPort: 8080
|
|
name: web
|
|
protocol: TCP
|
|
resources: {}
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
labels:
|
|
app: example-pod-scrape
|
|
owner: dev
|
|
env: dev
|
|
name: example-app-pod-1
|
|
spec:
|
|
containers:
|
|
- image: fabxc/instrumented_app
|
|
imagePullPolicy: IfNotPresent
|
|
name: example-app
|
|
ports:
|
|
- containerPort: 8080
|
|
name: web
|
|
protocol: TCP
|
|
resources: {}
|
|
EOF
|
|
```
|
|
|
|
Ensure, that pods started:
|
|
```bash
|
|
kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
example-app-594f97677c-g72v8 1/1 Running 0 3m40s
|
|
example-app-594f97677c-kn66h 1/1 Running 0 3m37s
|
|
example-app-594f97677c-l8xfd 1/1 Running 0 3m34s
|
|
example-app-pod-0 1/1 Running 0 14s
|
|
example-app-pod-1 1/1 Running 0 14s
|
|
vm-operator-646df47888-jvfvj 1/1 Running 0 67m
|
|
vmagent-example-vmagent-5777fdf7bf-tctcv 2/2 Running 1 37m
|
|
vmalertmanager-example-alertmanager-0 2/2 Running 0 14m
|
|
vmsingle-example-vmsingle-persisted-794b59ccc6-fnkpt 1/1 Running 0 40m
|
|
```
|
|
|
|
These pods have the same label `owner: dev` and port named `web`, so we can select them by `VMPodScrape` with matchLabels expression:
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMPodScrape
|
|
metadata:
|
|
name: example-pod-scrape
|
|
spec:
|
|
podMetricsEndpoints:
|
|
- port: web
|
|
scheme: http
|
|
selector:
|
|
matchLabels:
|
|
owner: dev
|
|
EOF
|
|
```
|
|
|
|
Lets check `VMAgent` logs:
|
|
```bash
|
|
kubectl logs vmagent-example-vmagent-5777fdf7bf-tctcv vmagent --tail 100
|
|
2020-08-03T08:51:13.582Z info VictoriaMetrics/app/vmagent/remotewrite/remotewrite.go:98 Successfully reloaded relabel configs
|
|
2020-08-03T08:51:13.585Z info VictoriaMetrics/lib/promscrape/scraper.go:137 found changes in "/etc/vmagent/config_out/vmagent.env.yaml"; applying these changes
|
|
2020-08-03T08:51:13.652Z info VictoriaMetrics/lib/promscrape/scraper.go:303 kubernetes_sd_configs: added targets: 2, removed targets: 0; total targets: 7
|
|
```
|
|
|
|
2 new target was added.
|
|
|
|
## VMRule
|
|
|
|
It generates `VMAlert` config with ruleset defined at `VMRule` spec.
|
|
|
|
Lets create `VMAlert` with selector for `VMRule` with label project=devops.
|
|
You also need datasource from previous step [VMSingle](#vmsingle) and [VMAgent](#vmagent) connected to it.
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlert
|
|
metadata:
|
|
name: example-vmalert
|
|
spec:
|
|
replicaCount: 1
|
|
datasource:
|
|
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
|
|
notifier:
|
|
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
|
|
evaluationInterval: "30s"
|
|
ruleSelector:
|
|
matchLabels:
|
|
project: devops
|
|
EOF
|
|
```
|
|
|
|
Ensure, that it started and ready:
|
|
```bash
|
|
kubectl get pods -l app.kubernetes.io/name=vmalert
|
|
NAME READY STATUS RESTARTS AGE
|
|
vmalert-example-vmalert-6f8748c6f9-hcfrr 2/2 Running 0 2m26s
|
|
|
|
kubectl logs vmalert-example-vmalert-6f8748c6f9-hcfrr vmalert
|
|
2020-08-03T08:52:21.990Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
|
|
2020-08-03T08:52:21.992Z info VictoriaMetrics/app/vmalert/group.go:153 group "vmAlertGroup" started with interval 30s
|
|
2020-08-03T08:52:21.992Z info VictoriaMetrics/lib/httpserver/httpserver.go:76 starting http server at http://:8080/
|
|
2020-08-03T08:52:21.993Z info VictoriaMetrics/lib/httpserver/httpserver.go:77 pprof handlers are exposed at http://:8080/debug/pprof/
|
|
|
|
```
|
|
`VMAlert` shipped with default rule `vmAlertGroup`
|
|
|
|
Let's add simple rule for `VMAlert` itself, `delta(vmalert_config_last_reload_errors_total[5m]) > 1`
|
|
|
|
{% raw %}
|
|
```yaml
|
|
cat << 'EOF' | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMRule
|
|
metadata:
|
|
name: example-vmrule-reload-config
|
|
labels:
|
|
project: devops
|
|
spec:
|
|
groups:
|
|
- name: vmalert
|
|
rules:
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 10s
|
|
labels:
|
|
severity: major
|
|
job: "{{ $labels.job }}"
|
|
annotations:
|
|
value: "{{ $value }}"
|
|
description: 'error reloading vmalert config, reload count for 5 min {{ $value }}'
|
|
EOF
|
|
```
|
|
{% endraw %}
|
|
|
|
Ensure, that new alert was started:
|
|
```bash
|
|
kubectl logs vmalert-example-vmalert-6f8748c6f9-hcfrr vmalert
|
|
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/web.go:45 api config reload was called, sending sighup
|
|
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/main.go:115 SIGHUP received. Going to reload rules ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"] ...
|
|
2020-08-03T09:07:49.772Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
|
|
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/group.go:169 group "vmAlertGroup": received stop signal
|
|
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/main.go:124 Rules reloaded successfully from ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"]
|
|
2020-08-03T09:07:49.773Z info VictoriaMetrics/app/vmalert/group.go:153 group "vmalert" started with interval 30s
|
|
|
|
```
|
|
|
|
Let's trigger it by adding some incorrect rule
|
|
|
|
{% raw %}
|
|
```yaml
|
|
cat << 'EOF' | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMRule
|
|
metadata:
|
|
name: example-vmrule-incorrect-rule
|
|
labels:
|
|
project: devops
|
|
spec:
|
|
groups:
|
|
- name: incorrect rule
|
|
rules:
|
|
- alert: vmalert bad config
|
|
expr: bad expression
|
|
for: 10s
|
|
labels:
|
|
severity: major
|
|
annotations:
|
|
value: "{{ $badValue | bad function }}"
|
|
EOF
|
|
```
|
|
{% endraw %}
|
|
|
|
`VMAlert` will report incorrect rule config and fire alert:
|
|
```bash
|
|
2020-08-03T09:11:40.672Z info VictoriaMetrics/app/vmalert/main.go:115 SIGHUP received. Going to reload rules ["/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"] ...
|
|
2020-08-03T09:11:40.672Z info VictoriaMetrics/app/vmalert/manager.go:83 reading rules configuration file from "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/*.yaml"
|
|
2020-08-03T09:11:40.673Z error VictoriaMetrics/app/vmalert/main.go:119 error while reloading rules: cannot parse configuration file: invalid group "incorrect rule" in file "/etc/vmalert/config/vm-example-vmalert-rulefiles-0/default-example-vmrule-incorrect-rule.yaml": invalid rule "incorrect rule"."vmalert bad config": invalid expression: unparsed data left: "expression"
|
|
```
|
|
|
|
Clean up incorrect rule:
|
|
```bash
|
|
kubectl delete vmrule example-vmrule-incorrect-rule
|
|
```
|
|
|
|
## VMNodeScrape
|
|
|
|
`VMNodeScrape` is useful for node exporters monitoring, lets create scraper for cadvisor metrics:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMNodeScrape
|
|
metadata:
|
|
name: cadvisor-metrics
|
|
spec:
|
|
scheme: "https"
|
|
tlsConfig:
|
|
insecureSkipVerify: true
|
|
caFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
|
|
bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token"
|
|
relabelConfigs:
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_node_label_(.+)
|
|
- targetLabel: __address__
|
|
replacement: kubernetes.default.svc:443
|
|
- sourceLabels: [__meta_kubernetes_node_name]
|
|
regex: (.+)
|
|
targetLabel: __metrics_path__
|
|
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
|
|
EOF
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## VMProbe
|
|
|
|
`VMProbe` required `VMAgent` and some external prober, blackbox exporter in our case. Ensure that you have `VMAgent` and `VMSingle`:
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example-vmsingle-persisted
|
|
spec:
|
|
retentionPeriod: "1"
|
|
removePvcAfterDelete: true
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
selectAllByDefault: true
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|
|
|
|
Then add `BlackBox` exporter with simple config:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: prometheus-blackbox-exporter
|
|
labels:
|
|
app: prometheus-blackbox-exporter
|
|
data:
|
|
blackbox.yaml: |
|
|
modules:
|
|
http_2xx:
|
|
http:
|
|
preferred_ip_protocol: ip4
|
|
valid_http_versions:
|
|
- HTTP/1.1
|
|
- HTTP/2.0
|
|
valid_status_codes: []
|
|
prober: http
|
|
timeout: 5s
|
|
---
|
|
kind: Service
|
|
apiVersion: v1
|
|
metadata:
|
|
name: prometheus-blackbox-exporter
|
|
labels:
|
|
app: prometheus-blackbox-exporter
|
|
spec:
|
|
type: ClusterIP
|
|
ports:
|
|
- name: http
|
|
port: 9115
|
|
protocol: TCP
|
|
selector:
|
|
app: prometheus-blackbox-exporter
|
|
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: prometheus-blackbox-exporter
|
|
labels:
|
|
app: prometheus-blackbox-exporter
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: prometheus-blackbox-exporter
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: prometheus-blackbox-exporter
|
|
spec:
|
|
containers:
|
|
- name: blackbox-exporter
|
|
image: "prom/blackbox-exporter:v0.17.0"
|
|
args:
|
|
- "--config.file=/config/blackbox.yaml"
|
|
resources:
|
|
{}
|
|
ports:
|
|
- containerPort: 9115
|
|
name: http
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: http
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: http
|
|
volumeMounts:
|
|
- mountPath: /config
|
|
name: config
|
|
volumes:
|
|
- name: config
|
|
configMap:
|
|
name: prometheus-blackbox-exporter
|
|
EOF
|
|
```
|
|
|
|
Ensure, that pods are ready:
|
|
|
|
```bash
|
|
kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
prometheus-blackbox-exporter-5b5f44bd9c-2szdj 1/1 Running 0 3m3s
|
|
vmagent-example-vmagent-84886cd4d9-x8ml6 2/2 Running 1 9m22s
|
|
vmsingle-example-vmsingle-persisted-8584486b68-mqg6b 1/1 Running 0 11m
|
|
```
|
|
|
|
Now define some `VMProbe`, lets start with basic static target and probe `VMAgent` with its service address, for accessing
|
|
blackbox exporter, you have to specify its url at `VMProbe` config. Lets get both services names:
|
|
```bash
|
|
kubectl get svc
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h21m
|
|
prometheus-blackbox-exporter ClusterIP 10.105.251.80 <none> 9115/TCP 4m36s
|
|
vmagent-example-vmagent ClusterIP 10.102.31.47 <none> 8429/TCP 12m
|
|
vmsingle-example-vmsingle-persisted ClusterIP 10.107.69.7 <none> 8429/TCP 12m
|
|
```
|
|
|
|
So, we will probe `VMAgent` with url - `vmagent-example-vmagent.default.svc:9115/heath` with blackbox url:
|
|
`prometheus-blackbox-exporter.default.svc:9115` and module: `http_2xx` it was specified at blackbox configmap.
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMProbe
|
|
metadata:
|
|
name: probe-agent
|
|
spec:
|
|
jobName: static-probe
|
|
vmProberSpec:
|
|
# by default scheme http, and path is /probe
|
|
url: prometheus-blackbox-exporter.default.svc:9115
|
|
module: http_2xx
|
|
targets:
|
|
staticConfig:
|
|
targets:
|
|
- vmagent-example-vmagent.default.svc:8429/health
|
|
interval: 2s
|
|
EOF
|
|
```
|
|
|
|
Now new target must be added to `VMAgent` configuration, and it starts probing itself throw blackbox exporter.
|
|
|
|
Let's try another target probe type - `Ingress`. Create ingress rule for `VMSingle` and create `VMProbe` for it:
|
|
|
|
```yaml
|
|
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: networking.k8s.io/v1beta1
|
|
kind: Ingress
|
|
metadata:
|
|
labels:
|
|
app: victoria-metrics-single
|
|
name: victoria-metrics-single
|
|
spec:
|
|
rules:
|
|
- host: vmsingle.example.com
|
|
http:
|
|
paths:
|
|
- backend:
|
|
serviceName: vmsingle-example-vmsingle-persisted
|
|
servicePort: 8428
|
|
path: /
|
|
- host: vmsingle2.example.com
|
|
http:
|
|
paths:
|
|
- backend:
|
|
serviceName: vmsingle-example-vmsingle-persisted
|
|
servicePort: 8428
|
|
path: /
|
|
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMProbe
|
|
metadata:
|
|
name: probe-single-ingress
|
|
spec:
|
|
vmProberSpec:
|
|
# by default scheme http, and path is /probe
|
|
url: prometheus-blackbox-exporter.default.svc:9115
|
|
module: http_2xx
|
|
targets:
|
|
ingress:
|
|
selector:
|
|
matchLabels:
|
|
app: victoria-metrics-single
|
|
interval: 10s
|
|
EOF
|
|
```
|
|
|
|
This configuration will add 2 additional targets for probing: `vmsingle2.example.com` and `vmsingle.example.com`.
|
|
|
|
But probes will be unsuccessful, coz there is no such hosts.
|
|
|
|
## VMStaticScrape
|
|
|
|
It generates config part of `VMAgent` with static_configs, targets for targetEndpoint is a required parameter.
|
|
It has various options for scraping configuration of target (with basic auth,tls access, by specific port name etc).
|
|
|
|
Add `VMAgent` and Example app from step above and continue this step.
|
|
|
|
With simple configuration:
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMStaticScrape
|
|
metadata:
|
|
name: vmstaticscrape-sample
|
|
spec:
|
|
jobName: static
|
|
targetEndpoints:
|
|
- targets: ["192.168.0.1:9100","196.168.0.50:9100"]
|
|
labels:
|
|
env: dev
|
|
project: operator
|
|
EOF
|
|
```
|
|
2 targets must be added to `VMAgent` scrape config:
|
|
```bash
|
|
static_configs: added targets: 2, removed targets: 0; total targets: 2
|
|
```
|
|
|
|
|
|
## VMAuth
|
|
|
|
[VMAuth](https://docs.victoriametrics.com/vmauth.html) allows protecting application with authentication and route traffic by rules.
|
|
|
|
api docs [link](https://docs.victoriametrics.com/operator/api.html#vmauthspec)
|
|
|
|
First create `VMAuth` configuration:
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAuth
|
|
metadata:
|
|
name: example
|
|
namespace: default
|
|
spec:
|
|
ingress: {}
|
|
selectAllByDefault: true
|
|
EOF
|
|
```
|
|
It will catch all `VMUser` at any kubernetes namespace and create `Ingress` record for it.
|
|
```text
|
|
kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
vmauth-example-ffcc78fcc-xddk7 2/2 Running 0 84s
|
|
kubectl get ingress
|
|
NAME CLASS HOSTS ADDRESS PORTS AGE
|
|
vmauth-example <none> * 80 106s
|
|
kubectl get secret -l app.kubernetes.io/name=vmauth
|
|
NAME TYPE DATA AGE
|
|
vmauth-config-example Opaque 1 2m32s
|
|
```
|
|
|
|
Generated configuration can be retrieved with command:
|
|
{% raw %}
|
|
```text
|
|
kubectl get secrets/vmauth-config-example -o=go-template='{{index .data "config.yaml.gz"}}' | base64 -d | gunzip
|
|
|
|
users:
|
|
- url_prefix: http://localhost:8428
|
|
bearer_token: some-default-token
|
|
```
|
|
{% endraw %}
|
|
|
|
Operator generates default config, if `VMUser`s for given `VMAuth` wasn't found.
|
|
|
|
## VMUser
|
|
|
|
`VMUser` configures `VMAuth`. api doc [link](https://docs.victoriametrics.com/operator/api.html#vmuserspec)
|
|
|
|
There are two authentication mechanisms: `bearerToken` and `basicAuth` with `username` and `password`. Only one of them can be used with `VMUser` at one time.
|
|
If you need to provide access with different mechanisms for single endpoint, create multiple `VMUsers`.
|
|
If `username` is empty, metadata.name from `VMUser` used as `username`.
|
|
If `password` is empty, operator generates random password for `VMUser`. This password added to the `Secret` for this `VMUser` at `data.password` field.
|
|
Operator creates `Secret` for every `VMUser` with name - `vmuser-{VMUser.metadata.name}`. It places `username` + `password` or `bearerToken` into `data` section.
|
|
|
|
`TargetRefs` is required field for `VMUser`, it allows to configure routing with:
|
|
- `static` ref:
|
|
```yaml
|
|
- static:
|
|
url: http://vmalertmanager.service.svc:9093
|
|
```
|
|
- `crd` ref, allows to target CRD kind of operator, this `CRDObject` must exist.
|
|
```yaml
|
|
- crd:
|
|
kind: VMAgent
|
|
name: example
|
|
namespace: default
|
|
```
|
|
Supported kinds are: `VMAgent, VMSingle, VMAlert, VMAlertmanager, VMCluster/vminsert, VMCluster/vmselect, VMCluster/vmstorage`
|
|
|
|
`paths` - configures allowed routing paths for given `targetRef`.
|
|
|
|
Let's create example, with access to `VMSingle` and `VMAlert` as static target:
|
|
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMSingle
|
|
metadata:
|
|
name: example
|
|
namespace: default
|
|
spec:
|
|
retentionPeriod: "2d"
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlert
|
|
metadata:
|
|
name: example
|
|
spec:
|
|
replicaCount: 1
|
|
datasource:
|
|
url: "http://vmsingle-example.default.svc:8429"
|
|
notifier:
|
|
url: "http://vmalertmanager-example.default.svc:9093"
|
|
evaluationInterval: "20s"
|
|
ruleSelector: {}
|
|
EOF
|
|
```
|
|
|
|
Check its status
|
|
```bash
|
|
|
|
kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
vmalert-example-775b8dfbc9-vzlnv 1/2 Running 0 3s
|
|
vmauth-example-ffcc78fcc-xddk7 2/2 Running 0 29m
|
|
vmsingle-example-6496b5c95d-k6hhp 1/1 Running 0 3s
|
|
```
|
|
|
|
Then create `VMUser`
|
|
```yaml
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMUser
|
|
metadata:
|
|
name: example
|
|
spec:
|
|
username: simple-user
|
|
password: simple-password
|
|
targetRefs:
|
|
- crd:
|
|
kind: VMSingle
|
|
name: example
|
|
namespace: default
|
|
paths: ["/.*"]
|
|
- static:
|
|
url: http://vmalert-example.default.svc:8080
|
|
paths: ["/api/v1/groups","/api/v1/alerts"]
|
|
EOF
|
|
```
|
|
|
|
Configuration changes for `VMAuth` takes some time, coz of mounted secret, its eventually updated by kubelet. Check vmauth log for changes:
|
|
|
|
```bash
|
|
kubectl logs vmauth-example-ffcc78fcc-xddk7 vmauth -f --tail 10
|
|
2021-05-31T10:46:40.171Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
|
|
2021-05-31T10:46:40.171Z info VictoriaMetrics/app/vmauth/main.go:37 started vmauth in 0.000 seconds
|
|
2021-05-31T10:46:40.171Z info VictoriaMetrics/lib/httpserver/httpserver.go:82 starting http server at http://:8427/
|
|
2021-05-31T10:46:40.171Z info VictoriaMetrics/lib/httpserver/httpserver.go:83 pprof handlers are exposed at http://:8427/debug/pprof/
|
|
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:143 SIGHUP received; loading -auth.config="/opt/vmauth/config.yaml"
|
|
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
|
|
2021-05-31T10:46:45.077Z info VictoriaMetrics/app/vmauth/auth_config.go:150 Successfully reloaded -auth.config="/opt/vmauth/config.yaml"
|
|
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:143 SIGHUP received; loading -auth.config="/opt/vmauth/config.yaml"
|
|
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:168 Loaded information about 1 users from "/opt/vmauth/config.yaml"
|
|
2021-05-31T11:18:21.313Z info VictoriaMetrics/app/vmauth/auth_config.go:150 Successfully reloaded -auth.config="/opt/vmauth/config.yaml"
|
|
```
|
|
|
|
Now lets try to access protected endpoints, i will use port-forward for that:
|
|
|
|
```bash
|
|
kubectl port-forward vmauth-example-ffcc78fcc-xddk7 8427
|
|
|
|
# at separate terminal execute:
|
|
|
|
# vmsingle response
|
|
curl http://localhost:8427 -u 'simple-user:simple-password'
|
|
|
|
# vmalert response
|
|
curl localhost:8427/api/v1/groups -u 'simple-user:simple-password'
|
|
```
|
|
|
|
Check create secret for application access:
|
|
|
|
```bash
|
|
kubectl get secrets vmuser-example
|
|
NAME TYPE DATA AGE
|
|
vmuser-example Opaque 2 6m33s
|
|
```
|
|
|
|
## Migration from prometheus-operator objects
|
|
|
|
By default, the operator converts all existing prometheus-operator API objects into corresponding VictoriaMetrics Operator objects
|
|
|
|
You can control this behaviour by setting env variable for operator:
|
|
|
|
```bash
|
|
#disable convertion for each object
|
|
VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR=false
|
|
VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE=false
|
|
VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE=false
|
|
VM_ENABLEDPROMETHEUSCONVERTER_PROBE=false
|
|
```
|
|
Otherwise, VictoriaMetrics Operator would try to discover prometheus-operator API and convert it.
|
|
|
|
|
|
Conversion of api objects can be controlled by annotations, added to `VMObject`s, there are following annotations:
|
|
- `operator.victoriametrics.com/merge-meta-strategy` - it controls syncing of metadata labels and annotations between
|
|
`VMObject`s and `Prometheus` api objects during updates to `Prometheus` objects. By default, it has `prefer-prometheus`.
|
|
And annotations and labels will be used from `Prometheus` objects, manually set values will be dropped.
|
|
You can set it to `prefer-victoriametrics`. In this case all labels and annotations applied to `Prometheus` object
|
|
will be ignored and `VMObject` will use own values.
|
|
Two additional strategies annotations -`merge-victoriametrics-priority` and `merge-prometheus-priority` merges labelSets into one combined labelSet, with priority.
|
|
Example:
|
|
```yaml
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMServiceScrape
|
|
metadata:
|
|
annotations:
|
|
meta.helm.sh/release-name: prometheus
|
|
operator.victoriametrics.com/merge-meta-strategy: prefer-victoriametrics
|
|
labels:
|
|
release: prometheus
|
|
name: prometheus-monitor
|
|
spec:
|
|
endpoints: []
|
|
```
|
|
|
|
- `operator.victoriametrics.com/ignore-prometheus-updates` - it controls updates from Prometheus api objects.
|
|
By default, it set to `disabled`. You define it to `enabled` state and all updates from Prometheus api objects will be
|
|
ignored.
|
|
```yaml
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMServiceScrape
|
|
metadata:
|
|
annotations:
|
|
meta.helm.sh/release-name: prometheus
|
|
operator.victoriametrics.com/ignore-prometheus-updates: enabled
|
|
labels:
|
|
release: prometheus
|
|
name: prometheus-monitor
|
|
spec:
|
|
endpoints: []
|
|
```
|
|
|
|
By default the operator doesn't make converted objects disappear after original ones are deleted. To change this behaviour
|
|
configure adding `OwnerReferences` to converted objects:
|
|
```bash
|
|
VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true
|
|
```
|
|
Converted objects will be linked to the original ones and will be deleted by kubernetes after the original ones are deleted.
|
|
|
|
### prometheus Rule duplication
|
|
`Prometheus` allows to specify rules with the same content with-in one group at Rule spec, but its forbidden by vmalert.
|
|
You can tell operator to deduplicate this rules by adding annotation to the `VMAlert` crd definition. In this case operator
|
|
skips rule with the same values, see example below.
|
|
```yaml
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlert
|
|
metadata:
|
|
name: example-vmalert-with-dedup
|
|
annotations:
|
|
operator.victoriametrics.com/vmalert-deduplicate-rules: "true"
|
|
spec:
|
|
replicaCount: 1
|
|
datasource:
|
|
url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429"
|
|
notifier:
|
|
url: "http://vmalertmanager-example-alertmanager.default.svc:9093"
|
|
evaluationInterval: "30s"
|
|
ruleNamespaceSelector: {}
|
|
ruleSelector: {}
|
|
```
|
|
Now operator will transform this `VMRule`:
|
|
```yaml
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMRule
|
|
metadata:
|
|
name: example-vmrule-reload-config
|
|
labels:
|
|
project: devops
|
|
spec:
|
|
groups:
|
|
- name: vmalert
|
|
rules:
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 10s
|
|
labels:
|
|
severity: major
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 10s
|
|
labels:
|
|
severity: major
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 2m
|
|
labels:
|
|
severity: critical
|
|
```
|
|
to the rule config:
|
|
|
|
```yaml
|
|
groups:
|
|
- name: vmalert
|
|
rules:
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 10s
|
|
labels:
|
|
severity: major
|
|
- alert: vmalert config reload error
|
|
expr: delta(vmalert_config_last_reload_errors_total[5m]) > 0
|
|
for: 2m
|
|
labels:
|
|
severity: critical
|
|
```
|
|
## Expose the VMSingle API
|
|
|
|
|
|
> WARNING: Please protect delete endpoint before exposing it [doc](https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-delete-time-series)
|
|
|
|
Example for Kubernetes Nginx ingress [doc](https://kubernetes.github.io/ingress-nginx/examples/auth/basic/)
|
|
|
|
```bash
|
|
#generate creds
|
|
htpasswd -c auth foo
|
|
|
|
#create basic auth secret
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: v1
|
|
stringData:
|
|
auth: foo:$apr1$wQ0fkxnl$KKOA/OqIZy48CuCB/A1Rc.
|
|
kind: Secret
|
|
metadata:
|
|
name: vm-auth
|
|
type: Opaque
|
|
EOF
|
|
|
|
#create ingress rule
|
|
cat << EOF | kubectl apply -f -
|
|
apiVersion: networking.k8s.io/v1beta1
|
|
kind: Ingress
|
|
metadata:
|
|
annotations:
|
|
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
|
|
nginx.ingress.kubernetes.io/auth-secret: vm-auth
|
|
nginx.ingress.kubernetes.io/auth-type: basic
|
|
labels:
|
|
app: victoria-metrics-single
|
|
name: victoria-metrics-single
|
|
spec:
|
|
rules:
|
|
- host: vmsingle.example.com
|
|
http:
|
|
paths:
|
|
- backend:
|
|
serviceName: vmsingle-example-vmsingle-persisted
|
|
servicePort: 8428
|
|
path: /
|
|
tls:
|
|
- hosts:
|
|
- vmsingle.example.com
|
|
secretName: ssl-name
|
|
|
|
EOF
|
|
|
|
#now you can access victoria metrics
|
|
curl -u foo:bar https://vmsingle.example/com/metrics
|
|
```
|
|
|
|
## Object selectors
|
|
`VMAgent` has four selectors specification for filtering `VMServiceScrape`, `VMProbe`, `VMNodeScrape` and `VMPodScrape` objects.
|
|
`VMAlert` has selectors for `VMRule`.
|
|
`VMAuth` has selectors for `VMUsers`.
|
|
`VMAlertmanager` has selectors for `VMAlertmanagerConfig`.
|
|
|
|
Selectors are defined with suffixes - `NamespaceSelector` and `Selector`. It allows configuring objects access control across namespaces and different environments.
|
|
Following rules are applied:
|
|
- `NamespaceSelector` and `Selector` - undefined, by default select nothing. With option set - `spec.selectAllByDefault: true`, select all objects of given type.
|
|
- `NamespaceSelector` defined, `Selector` undefined. All objects are matching at namespaces for given `NamespaceSelector`.
|
|
- `NamespaceSelector` undefined, `Selector` defined. All objects at `VMAgent`'s namespaces are matching for given `Selector`.
|
|
- `NamespaceSelector` defined, `Selector` defined. Only objects at namespaces matched `NamespaceSelector` for given `Selector` are matching.
|
|
|
|
### Object selector Examples
|
|
This configuration will match only namespace with the label - name=default and all `VMServiceScrape` objects at this namespace will be
|
|
selected:
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
serviceScrapeSelector: {}
|
|
serviceScrapeNamespaceSelector:
|
|
matchLabels:
|
|
name: default
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|
|
|
|
This configuration will select `VMServiceScrape` and `VMPodScrape` with labels - team=managers,project=cloud at any namespace:
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
serviceScrapeNamespaceSelector: {}
|
|
serviceScrapeSelector:
|
|
matchLabels:
|
|
team: managers
|
|
project: cloud
|
|
podScrapeSelector:
|
|
matchLabels:
|
|
team: managers
|
|
project: clo
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|
|
|
|
The last one will select any `VMServiceScrape`,`VMProbe` and `VMPodScrape` at `VMAgent` namespace:
|
|
```yaml
|
|
cat <<EOF | kubectl apply -f -
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: example-vmagent
|
|
spec:
|
|
serviceScrapeSelector: {}
|
|
podScrapeSelector: {}
|
|
probeSelector: {}
|
|
replicaCount: 1
|
|
remoteWrite:
|
|
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
|
EOF
|
|
```
|