mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
0a5ffb3bc1
Each Grafana dashboard has unique ID which can be used to fetch the dashboard
from grafana.com: https://grafana.com/grafana/dashboards/11176
The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/
But using slug implies that any change to dashboard name will break the link.
So it is better to just use ID, so the dashboard URL will never break.
This is follow-up for ff33e60a3d
Signed-off-by: hagen1778 <roman@victoriametrics.com>
361 lines
12 KiB
Markdown
361 lines
12 KiB
Markdown
---
|
||
weight: 2
|
||
title: Kubernetes monitoring via VictoriaMetrics Single
|
||
menu:
|
||
docs:
|
||
parent: "guides"
|
||
weight: 2
|
||
aliases:
|
||
- /guides/k8s-monitoring-via-vm-single.html
|
||
---
|
||
# Kubernetes monitoring via VictoriaMetrics Single
|
||
|
||
|
||
**This guide covers:**
|
||
|
||
* The setup of a [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html) in [Kubernetes](https://kubernetes.io/) via Helm charts
|
||
* How to scrape metrics from k8s components using service discovery
|
||
* How to visualize stored data
|
||
* How to store metrics in [VictoriaMetrics](https://victoriametrics.com) tsdb
|
||
|
||
**Precondition**
|
||
|
||
We will use:
|
||
* [Kubernetes cluster 1.19.9-gke.1900](https://cloud.google.com/kubernetes-engine)
|
||
> We use GKE cluster from [GCP](https://cloud.google.com/) but this guide is also applied on any Kubernetes cluster. For example [Amazon EKS](https://aws.amazon.com/ru/eks/).
|
||
* [Helm 3 ](https://helm.sh/docs/intro/install)
|
||
* [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl)
|
||
|
||
<p align="center">
|
||
<img src="k8s-monitoring-via-vm-single_k8s-scheme.webp" width="800" alt="VictoriaMetrics Single on Kubernetes cluster">
|
||
</p>
|
||
|
||
## 1. VictoriaMetrics Helm repository
|
||
|
||
> For this guide we will use Helm 3 but if you already use Helm 2 please see this [https://github.com/VictoriaMetrics/helm-charts#for-helm-v2](https://github.com/VictoriaMetrics/helm-charts#for-helm-v2)
|
||
|
||
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html). You can do this by running the following command:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
helm repo add vm https://victoriametrics.github.io/helm-charts/
|
||
```
|
||
|
||
</div>
|
||
|
||
Update Helm repositories:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
helm repo update
|
||
```
|
||
|
||
</div>
|
||
|
||
To verify that everything is set up correctly you may run this command:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
helm search repo vm/
|
||
```
|
||
|
||
</div>
|
||
|
||
The expected output is:
|
||
|
||
```console
|
||
NAME CHART VERSION APP VERSION DESCRIPTION
|
||
vm/victoria-metrics-agent 0.7.20 v1.62.0 Victoria Metrics Agent - collects metrics from ...
|
||
vm/victoria-metrics-alert 0.3.34 v1.62.0 Victoria Metrics Alert - executes a list of giv...
|
||
vm/victoria-metrics-auth 0.2.23 1.62.0 Victoria Metrics Auth - is a simple auth proxy ...
|
||
vm/victoria-metrics-cluster 0.8.32 1.62.0 Victoria Metrics Cluster version - high-perform...
|
||
vm/victoria-metrics-k8s-stack 0.2.9 1.16.0 Kubernetes monitoring on VictoriaMetrics stack....
|
||
vm/victoria-metrics-operator 0.1.17 0.16.0 Victoria Metrics Operator
|
||
vm/victoria-metrics-single 0.7.5 1.62.0 Victoria Metrics Single version - high-performa...
|
||
```
|
||
|
||
|
||
## 2. Install [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html) from Helm Chart
|
||
|
||
Run this command in your terminal:
|
||
|
||
<div class="with-copy" markdown="1">.html
|
||
|
||
```console
|
||
helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml
|
||
```
|
||
|
||
Here is full file content `guide-vmsingle-values.yaml`
|
||
|
||
```yaml
|
||
server:
|
||
scrape:
|
||
enabled: true
|
||
configMap: ""
|
||
config:
|
||
global:
|
||
scrape_interval: 15s
|
||
scrape_configs:
|
||
- job_name: victoriametrics
|
||
static_configs:
|
||
- targets: [ "localhost:8428" ]
|
||
- job_name: "kubernetes-apiservers"
|
||
kubernetes_sd_configs:
|
||
- role: endpoints
|
||
scheme: https
|
||
tls_config:
|
||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||
insecure_skip_verify: true
|
||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||
relabel_configs:
|
||
- source_labels:
|
||
[
|
||
__meta_kubernetes_namespace,
|
||
__meta_kubernetes_service_name,
|
||
__meta_kubernetes_endpoint_port_name,
|
||
]
|
||
action: keep
|
||
regex: default;kubernetes;https
|
||
- job_name: "kubernetes-nodes"
|
||
scheme: https
|
||
tls_config:
|
||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||
insecure_skip_verify: true
|
||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||
kubernetes_sd_configs:
|
||
- role: node
|
||
relabel_configs:
|
||
- action: labelmap
|
||
regex: __meta_kubernetes_node_label_(.+)
|
||
- target_label: __address__
|
||
replacement: kubernetes.default.svc:443
|
||
- source_labels: [ __meta_kubernetes_node_name ]
|
||
regex: (.+)
|
||
target_label: __metrics_path__
|
||
replacement: /api/v1/nodes/$1/proxy/metrics
|
||
- job_name: "kubernetes-nodes-cadvisor"
|
||
scheme: https
|
||
tls_config:
|
||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||
insecure_skip_verify: true
|
||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||
kubernetes_sd_configs:
|
||
- role: node
|
||
relabel_configs:
|
||
- action: labelmap
|
||
regex: __meta_kubernetes_node_label_(.+)
|
||
- target_label: __address__
|
||
replacement: kubernetes.default.svc:443
|
||
- source_labels: [ __meta_kubernetes_node_name ]
|
||
regex: (.+)
|
||
target_label: __metrics_path__
|
||
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
|
||
metric_relabel_configs:
|
||
- action: replace
|
||
source_labels: [pod]
|
||
regex: '(.+)'
|
||
target_label: pod_name
|
||
replacement: '${1}'
|
||
- action: replace
|
||
source_labels: [container]
|
||
regex: '(.+)'
|
||
target_label: container_name
|
||
replacement: '${1}'
|
||
- action: replace
|
||
target_label: name
|
||
replacement: k8s_stub
|
||
- action: replace
|
||
source_labels: [id]
|
||
regex: '^/system\.slice/(.+)\.service$'
|
||
target_label: systemd_service_name
|
||
replacement: '${1}'
|
||
|
||
```
|
||
|
||
|
||
</div>
|
||
|
||
* By running `helm install vmsingle vm/victoria-metrics-single` we install [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster
|
||
* By adding `scrape: enable: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
|
||
* On line 166 from [https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml](https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml) we added `metric_relabel_configs` section that will help us to show Kubernetes metrics on Grafana dashboard.
|
||
|
||
|
||
As a result of the command you will see the following output:
|
||
|
||
```console
|
||
NAME: victoria-metrics
|
||
LAST DEPLOYED: Fri Jun 25 12:06:13 2021
|
||
NAMESPACE: default
|
||
STATUS: deployed
|
||
REVISION: 1
|
||
TEST SUITE: None
|
||
NOTES:
|
||
The VictoriaMetrics write api can be accessed via port 8428 on the following DNS name from within your cluster:
|
||
vmsingle-victoria-metrics-single-server.default.svc.cluster.local
|
||
|
||
|
||
Metrics Ingestion:
|
||
Get the Victoria Metrics service URL by running these commands in the same shell:
|
||
export POD_NAME=$(kubectl get pods --namespace default -l "app=server" -o jsonpath="{.items[0].metadata.name}")
|
||
kubectl --namespace default port-forward $POD_NAME 8428
|
||
|
||
Write url inside the kubernetes cluster:
|
||
http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write
|
||
|
||
Metrics Scrape:
|
||
Pull-based scrapes are enabled
|
||
Scrape config can be displayed by running this command::
|
||
kubectl get cm vmsingle-victoria-metrics-single-server-scrapeconfig -n default
|
||
|
||
The target’s information is accessible via api:
|
||
Inside cluster:
|
||
http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428/targets
|
||
Outside cluster:
|
||
You need to port-forward service (see instructions above) and call
|
||
http://<service-host-port>/targets
|
||
|
||
Read Data:
|
||
The following url can be used as the datasource url in Grafana::
|
||
http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428
|
||
|
||
```
|
||
|
||
For us it’s important to remember the url for the datasource (copy lines from output).
|
||
|
||
Verify that VictoriaMetrics pod is up and running by executing the following command:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
kubectl get pods
|
||
```
|
||
|
||
</div>
|
||
|
||
The expected output is:
|
||
|
||
```console
|
||
NAME READY STATUS RESTARTS AGE
|
||
vmsingle-victoria-metrics-single-server-0 1/1 Running 0 68s
|
||
```
|
||
|
||
|
||
## 3. Install and connect Grafana to VictoriaMetrics with Helm
|
||
|
||
Add the Grafana Helm repository.
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
helm repo add grafana https://grafana.github.io/helm-charts
|
||
helm repo update
|
||
```
|
||
|
||
</div>
|
||
|
||
By installing the Chart with the release name `my-grafana`, you add the VictoriaMetrics datasource with official dashboard and kubernetes dashboard:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```yaml
|
||
cat <<EOF | helm install my-grafana grafana/grafana -f -
|
||
datasources:
|
||
datasources.yaml:
|
||
apiVersion: 1
|
||
datasources:
|
||
- name: victoriametrics
|
||
type: prometheus
|
||
orgId: 1
|
||
url: http://vmsingle-victoria-metrics-single-server.default.svc.cluster.local:8428
|
||
access: proxy
|
||
isDefault: true
|
||
updateIntervalSeconds: 10
|
||
editable: true
|
||
|
||
dashboardProviders:
|
||
dashboardproviders.yaml:
|
||
apiVersion: 1
|
||
providers:
|
||
- name: 'default'
|
||
orgId: 1
|
||
folder: ''
|
||
type: file
|
||
disableDeletion: true
|
||
editable: true
|
||
options:
|
||
path: /var/lib/grafana/dashboards/default
|
||
|
||
dashboards:
|
||
default:
|
||
victoriametrics:
|
||
gnetId: 10229
|
||
revision: 22
|
||
datasource: victoriametrics
|
||
kubernetes:
|
||
gnetId: 14205
|
||
revision: 1
|
||
datasource: victoriametrics
|
||
EOF
|
||
```
|
||
|
||
</div>
|
||
|
||
By running this command we:
|
||
* Install Grafana from Helm repository.
|
||
* Provision VictoriaMetrics datasource with the url from the output above which we copied before.
|
||
* Add this [https://grafana.com/grafana/dashboards/10229-victoriametrics/](https://grafana.com/grafana/dashboards/10229/) dashboard for VictoriaMetrics.
|
||
* Add this [https://grafana.com/grafana/dashboards/14205-kubernetes-cluster-monitoring-via-prometheus/](https://grafana.com/grafana/dashboards/14205-kubernetes-cluster-monitoring-via-prometheus/) dashboard to see Kubernetes cluster metrics.
|
||
|
||
|
||
Check the output log in your terminal.
|
||
To see the password for Grafana `admin` user use the following command:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
kubectl get secret --namespace default my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
|
||
```
|
||
|
||
</div>
|
||
|
||
Expose Grafana service on `127.0.0.1:3000`:
|
||
|
||
<div class="with-copy" markdown="1">
|
||
|
||
```console
|
||
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=my-grafana" -o jsonpath="{.items[0].metadata.name}")
|
||
|
||
kubectl --namespace default port-forward $POD_NAME 3000
|
||
```
|
||
|
||
</div>
|
||
|
||
Now Grafana should be accessible on the [http://127.0.0.1:3000](http://127.0.0.1:3000) address.
|
||
|
||
|
||
## 4. Check the obtained result in your browser
|
||
|
||
To check that VictoriaMetrics has collects metrics from the k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously obtained from kubectl.
|
||
|
||
<p align="center">
|
||
<img src="k8s-monitoring-via-vm-single_grafana-dashboards.webp" width="800" alt="">
|
||
</p>
|
||
|
||
You will see something like this:
|
||
<p align="center">
|
||
<img src="k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp" width="800" alt="">
|
||
</p>
|
||
|
||
VictoriaMetrics dashboard also available to use:
|
||
<p align="center">
|
||
<img src="k8s-monitoring-via-vm-single_grafana.webp" width="800" alt="">
|
||
</p>
|
||
|
||
## 5. Final thoughts
|
||
|
||
* We have set up TimeSeries Database for your k8s cluster.
|
||
* Collected metrics from all running pods,nodes, … and store them in VictoriaMetrics database.
|
||
* Visualize resources used in Kubernetes cluster by Grafana dashboards.
|