docs: remove witdh from images, remove <p>, remove <div> (#5705)

* docs: remove witdh from images, remove <p>, remove <div>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* docs: remove <div> clarify language in code blocks

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
This commit is contained in:
Artem Navoiev 2024-01-27 10:08:07 -08:00 committed by GitHub
parent 4a9f8f4cb0
commit 491287ed15
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
21 changed files with 267 additions and 487 deletions

View file

@ -517,14 +517,12 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
<p align="center">
<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp" width="800">
</p>
<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp">
To configure DataDog agent via ENV variable add the following prefix:
```
```sh
DD_DD_URL=http://victoriametrics:8428/datadog
```
@ -534,7 +532,7 @@ _Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/
To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33)
add the following line:
```
```yaml
dd_url: http://victoriametrics:8428/datadog
```
@ -546,9 +544,7 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples.
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
<p align="center">
<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp" width="800">
</p>
<img src="docs/Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp">
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:

View file

@ -56,7 +56,7 @@ Each service may scale independently and may run on the most suitable hardware.
This is a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture).
It increases cluster availability, and simplifies cluster maintenance as well as cluster scaling.
<img src="Cluster-VictoriaMetrics_cluster-scheme.webp" width="800">
<img src="Cluster-VictoriaMetrics_cluster-scheme.webp">
## Multitenancy

View file

@ -520,12 +520,12 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp" width="800">
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp">
To configure DataDog agent via ENV variable add the following prefix:
```
```text
DD_DD_URL=http://victoriametrics:8428/datadog
```
@ -535,7 +535,7 @@ _Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/
To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33)
add the following line:
```
```text
dd_url: http://victoriametrics:8428/datadog
```
@ -547,12 +547,12 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples.
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp" width="800">
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp" >
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:
```
```text
DD_ADDITIONAL_ENDPOINTS='{\"http://victoriametrics:8428/datadog\": [\"apikey\"]}'
```
@ -565,7 +565,7 @@ To configure DataDog Dual Shipping via [configuration file](https://docs.datadog
add the following line:
```
```yaml
additional_endpoints:
"http://victoriametrics:8428/datadog":
- apikey

View file

@ -528,12 +528,12 @@ via ["submit metrics" API](https://docs.datadoghq.com/api/latest/metrics/#submit
DataDog agent allows configuring destinations for metrics sending via ENV variable `DD_DD_URL`
or via [configuration file](https://docs.datadoghq.com/agent/guide/agent-configuration-files/) in section `dd_url`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp" width="800">
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM.webp">
To configure DataDog agent via ENV variable add the following prefix:
```
```text
DD_DD_URL=http://victoriametrics:8428/datadog
```
@ -543,7 +543,7 @@ _Choose correct URL for VictoriaMetrics [here](https://docs.victoriametrics.com/
To configure DataDog agent via [configuration file](https://github.com/DataDog/datadog-agent/blob/878600ef7a55c5ef0efb41ed0915f020cf7e3bd0/pkg/config/config_template.yaml#L33)
add the following line:
```
```text
dd_url: http://victoriametrics:8428/datadog
```
@ -555,12 +555,12 @@ pick [single-node or cluster URL](https://docs.victoriametrics.com/url-examples.
DataDog allows configuring [Dual Shipping](https://docs.datadoghq.com/agent/guide/dual-shipping/) for metrics
sending via ENV variable `DD_ADDITIONAL_ENDPOINTS` or via configuration file `additional_endpoints`.
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp" width="800">
<img src="Single-server-VictoriaMetrics-sending_DD_metrics_to_VM_and_DD.webp">
Run DataDog using the following ENV variable with VictoriaMetrics as additional metrics receiver:
```
```text
DD_ADDITIONAL_ENDPOINTS='{\"http://victoriametrics:8428/datadog\": [\"apikey\"]}'
```
@ -573,7 +573,7 @@ To configure DataDog Dual Shipping via [configuration file](https://docs.datadog
add the following line:
```
```yaml
additional_endpoints:
"http://victoriametrics:8428/datadog":
- apikey

View file

@ -81,7 +81,7 @@ with `vl_http_requests_total{path="/select/logsql/query"}` metric.
VictoriaLogs provides a simple Web UI for logs [querying](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) and exploration
at `http://localhost:9428/select/vmui`. The UI allows exploring query results:
<img src="vmui.webp" width="800" />
<img src="vmui.webp" />
There are three modes of displaying query results:

View file

@ -66,7 +66,6 @@ Depending on chosen `seasonality` parameter FB Prophet can return additional met
- and a number of columns for each holiday if `holidays` param is set
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -83,7 +82,6 @@ model:
country_holidays: 'US'
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output)
@ -95,7 +93,6 @@ Resulting metrics of the model are described [here](#vmanomaly-output)
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -103,7 +100,6 @@ model:
z_threshold: 2.5
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -133,7 +129,6 @@ Used to compute "seasonal_periods" param for the model (e.g. '1D' or '1W').
* `args` (dict, optional) - Inner model args (key-value pairs). See accepted params in [model documentation](https://www.statsmodels.org/dev/generated/statsmodels.tsa.holtwinters.ExponentialSmoothing.html#statsmodels.tsa.holtwinters.ExponentialSmoothing-parameters). Defaults to empty (not provided). Example: {"seasonal": "add", "initialization_method": "estimated"}
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -146,7 +141,6 @@ model:
initialization_method: 'estimated'
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -160,7 +154,6 @@ The MAD model is a robust method for anomaly detection that is *less sensitive*
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -168,7 +161,6 @@ model:
threshold: 2.5
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -181,7 +173,6 @@ Resulting metrics of the model are described [here](#vmanomaly-output).
* `window_steps` (integer) - size of the moving window. (see 'sampling_period')
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -190,7 +181,6 @@ model:
window_steps: 96
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -206,7 +196,6 @@ Here we use Seasonal Decompose implementation from `statsmodels` [library](https
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -214,7 +203,6 @@ model:
period: 2
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -243,7 +231,6 @@ Here we use ARIMA implementation from `statsmodels` [library](https://www.statsm
* `args` (dict, optional) - Inner model args (key-value pairs). See accepted params in [model documentation](https://www.statsmodels.org/dev/generated/statsmodels.tsa.arima.model.ARIMA.html). Defaults to empty (not provided). Example: {"trend": "c"}
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -257,7 +244,6 @@ model:
trend: 'c'
```
</div>
### [Isolation forest](https://en.wikipedia.org/wiki/Isolation_forest) (Multivariate)
Detects anomalies using binary trees. The algorithm has a linear time complexity and a low memory requirement, which works well with high-volume data. It can be used on both univatiate and multivariate data, but it is more effective in multivariate case.
@ -276,7 +262,6 @@ Here we use Isolation Forest implementation from `scikit-learn` [library](https:
*Config Example*
<div class="with-copy" markdown="1">
```yaml
model:
@ -289,7 +274,6 @@ model:
random_state: 42
```
</div>
Resulting metrics of the model are described [here](#vmanomaly-output).
@ -348,7 +332,6 @@ In the `CustomModel` class there should be three required methods - `__init__`,
For the sake of simplicity, the model in this example will return one of two values of `anomaly_score` - 0 or 1 depending on input parameter `percentage`.
<div class="with-copy" markdown="1">
```python
import numpy as np
@ -393,7 +376,6 @@ class CustomModel(Model):
```
</div>
### 2. Configuration file
@ -402,7 +384,6 @@ Next, we need to create `config.yaml` file with VM Anomaly Detection configurati
In the config file `model` section we need to put our model class `model.custom.CustomModel` and all parameters used in `__init__` method.
You can find out more about configuration parameters in vmanomaly docs.
<div class="with-copy" markdown="1">
```yaml
scheduler:
@ -441,23 +422,19 @@ monitoring:
config: "custom.yaml"
```
</div>
### 3. Running custom model
Let's pull the docker image for vmanomaly:
<div class="with-copy" markdown="1">
```sh
docker pull us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial:latest
```
</div>
Now we can run the docker container putting as volumes both config and model file:
**Note**: place the model file to `/model/custom.py` path when copying
<div class="with-copy" markdown="1">
```sh
docker run -it \
@ -469,7 +446,6 @@ us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial:latest /config.yam
--license-file=/license.txt
```
</div>
Please find more detailed instructions (license, etc.) [here](/anomaly-detection/overview.html#run-vmanomaly-docker-container)

View file

@ -90,8 +90,6 @@ There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - [push
## Monitoring section config example
<div class="with-copy" markdown="1">
``` yaml
monitoring:
pull: # Enable /metrics endpoint.
@ -107,7 +105,6 @@ monitoring:
job: "vmanomaly-push"
test: "test-1"
```
</div>
## Metrics generated by vmanomaly

View file

@ -26,7 +26,7 @@ aliases:
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/)
- [Node exporter](https://github.com/prometheus/node_exporter#node-exporter)(v1.7.0) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)(v0.25.0)
<img max-width="1000" alt="vmanomaly typical setup diagramm" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp">
<img alt="vmanomaly typical setup diagramm" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_overview.webp">
> **Note: Configurations used throughout this guide can be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/vmanomaly/vmanomaly-integration/)**
@ -103,7 +103,7 @@ These modes are mutually exclusive. A high _iowait_ means that you are disk or n
The metric `node_cpu_seconds_total` is a [counter](https://docs.victoriametrics.com/keyConcepts.html#counter) type of metric. If we'd like to see how much time CPU spent in each of the nodes, we need to calculate the per-second values change via [rate function](https://docs.victoriametrics.com/MetricsQL.html#rate): `rate(node_cpu_seconds_total)`. To aggregate data by mode we'll use median or 50% quantile function. Resulting query will look likt this: `quantile by (mode) (0.5, rate(node_cpu_seconds_total[5m])`
Here is how this query may look like in Grafana:
<img max-width="1000" alt="node_cpu_rate_graph" src="guide-vmanomaly-vmalert-query.webp">
<img alt="node_cpu_rate_graph" src="guide-vmanomaly-vmalert-query.webp">
This query result will generate 8 time series per each cpu, and we will use them as an input for our VM Anomaly Detection. vmanomaly will start learning configured model type separately for each of the time series.
@ -141,7 +141,6 @@ Let's look into parameters in each section:
Here is an example of the config file `vmanomaly_config.yml`.
<div class="with-copy" markdown="1">
``` yaml
scheduler:
@ -169,7 +168,6 @@ monitoring:
port: 8490
```
</div>
## 6. vmanomaly output
@ -187,7 +185,6 @@ Here is an example of how output metric will be written into VictoriaMetrics:
## 7. vmalert configuration
Here we provide an example of the config for vmalert `vmalert_config.yml`.
<div class="with-copy" markdown="1">
``` yaml
groups:
@ -201,7 +198,6 @@ groups:
summary: Anomaly Score exceeded 1.0. `rate(node_cpu_seconds_total)` is showing abnormal behavior.
```
</div>
In the query expression we need to put a condition on the generated anomaly scores. Usually if the anomaly score is between 0.0 and 1.0, the analyzed value is not abnormal. The more anomaly score exceeded 1 the more our model is sure that value is an anomaly.
You can choose your threshold value that you consider reasonable based on the anomaly score metric, generated by vmanomaly. One of the best ways is to estimate it visually, by plotting the `anomaly_score` metric, along with predicted "expected" range of `yhat_lower` and `yhat_upper`. Later in this tutorial we will show an example
@ -229,7 +225,6 @@ In the `provisioning/datasources/` directory, create a file called `datasource.y
> The default username/password pair is `admin:admin`
<div class="with-copy" markdown="1">
``` yaml
apiVersion: 1
@ -246,12 +241,10 @@ datasources:
```
</div>
#### Define a dashboard provider
In the` provisioning/dashboards/` directory, create a file called `dashboard.yml` with the following content:
<div class="with-copy" markdown="1">
``` yaml
apiVersion: 1
@ -266,14 +259,12 @@ providers:
```
</div>
### Scrape config
Let's create `prometheus.yml` file for `vmagent` configuration.
<div class="with-copy" markdown="1">
``` yaml
global:
@ -297,7 +288,6 @@ scrape_configs:
- targets: [ 'vmanomaly:8490' ]
```
</div>
### vmanomaly licensing
@ -322,7 +312,6 @@ receivers:
Let's wrap it all up together into the `docker-compose.yml` file.
<div class="with-copy" markdown="1">
``` yaml
services:
@ -447,32 +436,27 @@ networks:
```
</div>
Before running our docker-compose make sure that your directory contains all required files:
<img src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_files.webp" width="800" alt="all files">
<img src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_files.webp" alt="all files">
This docker-compose file will pull docker images, set up each service and run them all together with the command:
<div class="with-copy" markdown="1">
```
```sh
docker-compose up -d
```
</div>
To check if vmanomaly is up and running you can check docker logs:
<div class="with-copy" markdown="1">
```
```sh
docker logs vmanomaly
```
</div>
## 9. Model results
@ -492,7 +476,7 @@ Each of these metrics will contain same labels our query `quantile by (mode) (0.
### Anomaly scores for each metric with its according labels.
Query: `anomaly_score`
<img max-width="1000" alt="Anomaly score graph" src="guide-vmanomaly-vmalert-anomaly-score.webp">
<img alt="Anomaly score graph" src="guide-vmanomaly-vmalert-anomaly-score.webp">
<br>Check out if the anomaly score is high for datapoints you think are anomalies. If not, you can try other parameters in the config file or try other model type.
@ -503,7 +487,7 @@ As you may notice a lot of data shows anomaly score greater than 1. It is expect
Queries: `yhat_lower`, `yhat_upper` and `yhat`
<img max-width="1000" alt="yhat lower and yhat upper" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert-boundaries.webp">
<img alt="yhat lower and yhat upper" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert-boundaries.webp">
Boundaries of 'normal' metric values according to model inference.
@ -511,10 +495,10 @@ Boundaries of 'normal' metric values according to model inference.
On the page `http://localhost:8880/vmalert/groups` you can find our configured Alerting rule:
<img max-width="1000" alt="alert rule" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_alert-rule.webp">
<img alt="alert rule" src="guide-vmanomaly-vmalert/guide-vmanomaly-vmalert_alert-rule.webp">
According to the rule configured for vmalert we will see Alert when anomaly score exceed 1. You will see an alert on Alert tab. `http://localhost:8880/vmalert/alerts`:
<img max-width="1000" alt="alerts firing" src="guide-vmanomaly-vmalert_alerts-firing.webp">
<img alt="alerts firing" src="guide-vmanomaly-vmalert_alerts-firing.webp">
## 10. Conclusion

View file

@ -147,7 +147,7 @@ docker-compose -f docker-compose.yaml up
When Grafana starts successfully datasources should be present on the datasources tab
<img src="provision_datasources.webp" width="800" alt="Configuration">
<img src="provision_datasources.webp" alt="Configuration">
### Install in Kubernetes

View file

@ -30,13 +30,11 @@ See how to work with a [VictoriaMetrics Helm repository in previous guide](https
## 2. Install the VM Operator from the Helm chart
<div class="with-copy" markdown="1">
```console
helm install vmoperator vm/victoria-metrics-operator
```
</div>
The expected output is:
@ -57,12 +55,10 @@ See "Getting started guide for VM Operator" on https://docs.victoriametrics.com/
Run the following command to check that VM Operator is up and running:
<div class="with-copy" markdown="1">
```console
kubectl --namespace default get pods -l "app.kubernetes.io/instance=vmoperator"
```
</div>
The expected output:
```console
@ -76,7 +72,7 @@ vmoperator-victoria-metrics-operator-67cff44cd6-s47n6 1/1 Running 0
Run the following command to install [VictoriaMetrics Cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) via [VM Operator](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-operator):
<div class="with-copy" markdown="1" id="example-cluster-config">
<p id="example-cluster-config"></p>
```console
cat << EOF | kubectl apply -f -
@ -95,7 +91,6 @@ spec:
replicaCount: 2
EOF
```
</div>
The expected output:
@ -108,15 +103,14 @@ vmcluster.operator.victoriametrics.com/example-vmcluster-persistent created
* `replicaCount: 2` creates two replicas of vmselect, vminsert and vmstorage.
Please note that it may take some time for the pods to start. To check that the pods are started, run the following command:
<div class="with-copy" markdown="1" id="example-cluster-config">
<p id="example-cluster-config"></p>
```console
```sh
kubectl get pods | grep vmcluster
```
</div>
The expected output:
```console
```sh
NAME READY STATUS RESTARTS AGE
vminsert-example-vmcluster-persistent-845849cb84-9vb6f 1/1 Running 0 5m15s
vminsert-example-vmcluster-persistent-845849cb84-r7mmk 1/1 Running 0 5m15s
@ -127,15 +121,13 @@ vmstorage-example-vmcluster-persistent-1 1/1 Running 0
```
There is an extra command to get information about the cluster state:
<div class="with-copy" markdown="1" id="services">
```console
```sh
kubectl get vmclusters
```
</div>
The expected output:
```console
```text
NAME INSERT COUNT STORAGE COUNT SELECT COUNT AGE STATUS
example-vmcluster-persistent 2 2 2 5m53s operational
```
@ -144,12 +136,10 @@ Internet traffic goes through the Kubernetes Load balancer which use the set of
To get the name of `vminsert` services, please run the following command:
<div class="with-copy" markdown="1" id="services">
```console
```sh
kubectl get svc | grep vminsert
```
</div>
The expected output:
@ -161,9 +151,8 @@ To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to
Copy `vminsert-example-vmcluster-persistent` (or whatever user put into metadata.name field [https://docs.victoriametrics.com/guides/getting-started-with-vm-operator.html#example-cluster-config](https://docs.victoriametrics.com/guides/getting-started-with-vm-operator.html#example-cluster-config)) service name and add it to the `remoteWrite` URL from [quick-start example](https://github.com/VictoriaMetrics/operator/blob/master/docs/quick-start.MD#vmagent).
Here is an example of the full configuration that we need to apply:
<div class="with-copy" markdown="1">
```console
```sh
cat <<EOF | kubectl apply -f -
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
@ -183,11 +172,11 @@ spec:
- url: "http://vminsert-example-vmcluster-persistent.default.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"
EOF
```
</div>
The expected output:
```console
```text
vmagent.operator.victoriametrics.com/example-vmagent created
```
@ -196,16 +185,14 @@ vmagent.operator.victoriametrics.com/example-vmagent created
Verify that `VMAgent` is up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
```sh
kubectl get pods | grep vmagent
```
</div>
The expected output is:
```console
```text
vmagent-example-vmagent-7996844b5f-b5rzs 2/2 Running 0 9s
```
@ -213,17 +200,14 @@ vmagent-example-vmagent-7996844b5f-b5rzs 2/2 Running 0
Run the following command to make `VMAgent`'s port accessible from the local machine:
<div class="with-copy" markdown="1">
</div>
```console
```sh
kubectl port-forward svc/vmagent-example-vmagent 8429:8429
```
The expected output is:
```console
```text
Forwarding from 127.0.0.1:8429 -> 8429
Forwarding from [::1]:8429 -> 8429
```
@ -231,7 +215,7 @@ Forwarding from [::1]:8429 -> 8429
To check that `VMAgent` collects metrics from the k8s cluster open in the browser [http://127.0.0.1:8429/targets](http://127.0.0.1:8429/targets) .
You will see something like this:
<img src="getting-started-with-vm-operator_vmcluster.webp" width="800" alt="">
<img src="getting-started-with-vm-operator_vmcluster.webp">
`VMAgent` connects to [kubernetes service discovery](https://kubernetes.io/docs/concepts/services-networking/service/) and gets targets which needs to be scraped. This service discovery is controlled by [VictoriaMetrics Operator](https://github.com/VictoriaMetrics/operator)
@ -241,12 +225,10 @@ See [how to install and connect Grafana to VictoriaMetrics](https://docs.victori
To get the new service name, please run the following command:
<div class="with-copy" markdown="1" id="services">
```console
kubectl get svc | grep vmselect
```
</div>
The expected output:
@ -256,9 +238,8 @@ vmselect-example-vmcluster-persistent ClusterIP None <none>
The final config will look like this:
<div class="with-copy" markdown="1">
```yaml
```sh
cat <<EOF | helm install my-grafana grafana/grafana -f -
datasources:
datasources.yaml:
@ -302,18 +283,17 @@ cat <<EOF | helm install my-grafana grafana/grafana -f -
datasource: victoriametrics
EOF
```
</div>
## 5. Check the result you obtained in your browser
To check that [VictoriaMetrics](https://victoriametrics.com) collecting metrics from the k8s cluster open in your browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `VictoriaMetrics - cluster` dashboard. Use `admin` for login and the `password` that you previously got from kubectl.
<img src="getting-started-with-vm-operator_vmcluster-grafana1.webp" width="800" alt="grafana dashboards">
<img src="getting-started-with-vm-operator_vmcluster-grafana1.webp" alt="grafana dashboards">
The expected output is:
<img src="getting-started-with-vm-operator_vmcluster-grafana2.webp" width="800" alt="grafana dashboards">
<img src="getting-started-with-vm-operator_vmcluster-grafana2.webp" alt="grafana dashboards">
## 6. Summary

View file

@ -48,21 +48,21 @@ See details about all supported options in the [vmgateway documentation](https:/
Use `OpenID Connect` as `Client Type`.<br>
Specify `grafana` as `Client ID`.<br>
Click `Next`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-1.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/create-client-1.webp" >
1. Enable `Client authentication`.<br>
Enable `Authorization`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-2.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/create-client-2.webp" ><br>
Click `Next`.<br>
1. Add Grafana URL as `Root URL`. For example, `http://localhost:3000/`.<br>
<img src="grafana-vmgateway-openid-configuration/create-client-3.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/create-client-3.webp" ><br>
Click `Save`.<br>
1. Go to `Clients` -> `grafana` -> `Credentials`.<br>
<img src="grafana-vmgateway-openid-configuration/client-secret.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/client-secret.webp" ><br>
Copy the value of `Client secret`. It will be used later in Grafana configuration.<br>
1. Go to `Clients` -> `grafana` -> `Client scopes`.<br>
Click at `grafana-dedicated` -> `Add mapper` -> `By configuration` -> `User attribute`.<br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-1.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-2.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-1.webp" ><br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-2.webp" ><br>
Configure the mapper as follows<br>
- `Name` as `vm_access`.
- `Token Claim Name` as `vm_access`.
@ -70,7 +70,7 @@ See details about all supported options in the [vmgateway documentation](https:/
- `Claim JSON Type` as `JSON`.
Enable `Add to ID token` and `Add to access token`.<br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-3.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/create-mapper-3.webp" ><br>
Click `Save`.<br>
1. Go to `Users` -> select user to configure claims -> `Attributes`.<br>
Specify `vm_access` as `Key`.<br>
@ -78,7 +78,7 @@ See details about all supported options in the [vmgateway documentation](https:/
- for the first user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 0 },"extra_labels":{ "team": "admin" }}` as `Value`.
- for the second user we will specify `{"tenant_id" : {"account_id": 0, "project_id": 1 },"extra_labels":{ "team": "dev" }}` as `Value`.
<br>
<img src="grafana-vmgateway-openid-configuration/user-attributes.webp" width="800"><br>
<img src="grafana-vmgateway-openid-configuration/user-attributes.webp" ><br>
Click `Save`.
## Configure grafana
@ -207,7 +207,7 @@ You can also use VictoriaMetrics [Grafana datasource](https://github.com/Victori
See installation instructions [here](https://github.com/VictoriaMetrics/grafana-datasource#installation).
Enable `Forward OAuth identity` flag.<br>
<img src="grafana-vmgateway-openid-configuration/grafana-ds.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/grafana-ds.webp" >
Now you can use Grafana to query metrics from the specified tenant.
Users with `vm_access` claim will be able to query metrics from the specified tenant.
@ -330,22 +330,22 @@ vmagent will write data into VictoriaMetrics single-node and cluster(with tenant
Grafana datasources configuration will be the following:
<img src="grafana-vmgateway-openid-configuration/grafana-test-datasources.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/grafana-test-datasources.webp" >
Let's login as user with `team=dev` labels limitation set via claims.
Using `vmgateway-cluster` results into `No data` response as proxied request will go to tenant `0:1`.
Since vmagent is only configured to write to `0:0` `No data` is an expected response.
<img src="grafana-vmgateway-openid-configuration/dev-cluster-nodata.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/dev-cluster-nodata.webp" >
Switching to `vmgateway-single` does have data. Note that it is limited to metrics with `team=dev` label.
<img src="grafana-vmgateway-openid-configuration/dev-single-data.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/dev-single-data.webp" >
Now lets login as user with `team=admin`.
Both cluster and single node datasources now return metrics for `team=admin`.
<img src="grafana-vmgateway-openid-configuration/admin-cluster-data.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/admin-single-data.webp" width="800">
<img src="grafana-vmgateway-openid-configuration/admin-cluster-data.webp" >
<img src="grafana-vmgateway-openid-configuration/admin-single-data.webp" >

View file

@ -30,13 +30,11 @@ To check that metrics are present in **VictoriaMetrics Cluster** run the followi
_Warning: response can return many metrics, so be careful with series selector._
<div class="with-copy" markdown="1">
```console
```curl
curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/series?match[]=process_cpu_cores_available' | jq
```
</div>
The expected output:
@ -82,24 +80,20 @@ The expected output:
When you're sure [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) is correct, send a POST request to [delete API](https://docs.victoriametrics.com/url-examples.html#apiv1admintsdbdelete_series) with [`match[]=<time-series-selector>`](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) argument. For example:
<div class="with-copy" markdown="1">
```console
```curl
curl -s 'http://vmselect:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=process_cpu_cores_available'
```
</div>
If operation was successful, the deleted series will stop being [queryable](https://docs.victoriametrics.com/keyConcepts.html#query-data). Storage space for the deleted time series isn't freed instantly - it is freed during subsequent [background merges of data files](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). The background merges may never occur for data from previous months, so storage space won't be freed for historical data. In this case [forced merge](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#forced-merge) may help freeing up storage space.
To trigger [forced merge](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#forced-merge) on VictoriaMetrics Cluster run the following command:
<div class="with-copy" markdown="1">
```console
```curl
curl -v -X POST http://vmstorage:8482/internal/force_merge
```
</div>
After the merge is complete, the data will be permanently deleted from the disk.
@ -116,23 +110,19 @@ By default, VictoriaMetrics doesn't provide a mechanism for replacing or updatin
For example, let's export metric for `node_memory_MemTotal_bytes` with labels `instance="node-exporter:9100"` and `job="hostname.com"`:
<div class="with-copy" markdown="1">
```console
```curl
curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d 'match[]=node_memory_MemTotal_bytes{instance="node-exporter:9100", job="hostname.com"}' > data.jsonl
```
</div>
To check that exported file contains time series we can use [cat](https://man7.org/linux/man-pages/man1/cat.1.html) and [jq](https://stedolan.github.io/jq/download/)
<div class="with-copy" markdown="1">
```console
```curl
cat data.jsonl | jq
```
</div>
The expected output will look like:
@ -161,13 +151,13 @@ The expected output will look like:
In this example, we will replace the values of `node_memory_MemTotal_bytes` from `33604390912` to `17179869184` (from 32Gb to 16Gb) via [sed](https://linux.die.net/man/1/sed), but it can be done in any of the available ways.
```console
```sh
sed -i 's/33604390912/17179869184/g' data.jsonl
```
Let's check the changes in data.jsonl with `cat`:
```console
```sh
cat data.jsonl | jq
```
@ -205,22 +195,18 @@ Victoriametrics supports a lot of [ingestion protocols](https://docs.victoriamet
The next command will import metrics from `data.jsonl` to VictoriaMetrics:
<div class="with-copy" markdown="1">
```console
```curl
curl -v -X POST http://vminsert:8480/insert/0/prometheus/api/v1/import -T data.jsonl
```
</div>
### Check imported metrics
<div class="with-copy" markdown="1">
```console
```curk
curl -X POST -g http://vmselect:8481/select/0/prometheus/api/v1/export -d match[]=node_memory_MemTotal_bytes
```
</div>
The expected output will look like:

View file

@ -34,7 +34,7 @@ The [-retentionPeriod](https://docs.victoriametrics.com/#retention) sets how lon
The diagram below shows a proposed solution
<img src="guide-vmcluster-multiple-retention-setup.webp" width="800">
<img src="guide-vmcluster-multiple-retention-setup.webp">
**Implementation Details**

View file

@ -35,9 +35,8 @@ Please see the relevant [VictoriaMetrics Helm repository](https://docs.victoriam
Execute the following command in your terminal:
<div class="with-copy" markdown="1">
```yaml
```sh
cat <<EOF | helm install vmcluster vm/victoria-metrics-cluster -f -
vmselect:
extraArgs:
@ -63,7 +62,6 @@ vmstorage:
replicaCount: 3
EOF
```
</div>
* The `Helm install vmcluster vm/victoria-metrics-cluster` command installs [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) to the default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
* `dedup.minScrapeInterval: 1ms` configures [de-duplication](https://docs.victoriametrics.com/#deduplication) for the cluster that de-duplicates data points in the same time series if they fall within the same discrete 1ms bucket. The earliest data point will be kept. In the case of equal timestamps, an arbitrary data point will be kept.
@ -75,7 +73,7 @@ EOF
The expected result of the command execution is the following:
```console
```text
NAME: vmcluster
LAST DEPLOYED: Thu Jul 29 13:33:51 2021
NAMESPACE: default
@ -129,16 +127,14 @@ for example - inside the Kubernetes cluster:
Verify that the VictoriaMetrics cluster pods are up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
```sh
kubectl get pods | grep vmcluster
```
</div>
The expected output is:
```console
```text
vmcluster-victoria-metrics-cluster-vminsert-78b84d8cd9-4mh9d 1/1 Running 0 2m28s
vmcluster-victoria-metrics-cluster-vminsert-78b84d8cd9-4ppl7 1/1 Running 0 2m28s
vmcluster-victoria-metrics-cluster-vminsert-78b84d8cd9-782qk 1/1 Running 0 2m28s
@ -154,12 +150,9 @@ vmcluster-victoria-metrics-cluster-vmstorage-2 1/1 Running
To scrape metrics from Kubernetes with a VictoriaMetrics Cluster we will need to install [vmagent](https://docs.victoriametrics.com/vmagent.html) with some additional configurations. To do so, please run the following command:
<div class="with-copy" markdown="1">
```yaml
helm install vmagent vm/victoria-metrics-agent -f https://docs.victoriametrics.com/guides/guide-vmcluster-vmagent-values.yaml
```
</div>
Here is full file content `guide-vmcluster-vmagent-values.yaml`
@ -249,33 +242,29 @@ scrape_configs:
Verify that `vmagent`'s pod is up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get pods | grep vmagent
```
</div>
The expected output is:
```console
```text
vmagent-victoria-metrics-agent-57ddbdc55d-h4ljb 1/1 Running 0 13s
```
## 4. Verifying HA of VictoriaMetrics Cluster
Run the following command to check that VictoriaMetrics services are up and running:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get pods | grep victoria-metrics
```
</div>
The expected output is:
```console
```text
vmagent-victoria-metrics-agent-57ddbdc55d-h4ljb 1/1 Running 0 75s
vmcluster-victoria-metrics-cluster-vminsert-78b84d8cd9-s8v7x 1/1 Running 0 89s
vmcluster-victoria-metrics-cluster-vminsert-78b84d8cd9-xlm9d 1/1 Running 0 89s
@ -291,37 +280,33 @@ vmcluster-victoria-metrics-cluster-vmstorage-2 1/1 Running
To verify that metrics are present in the VictoriaMetrics send a curl request to the `vmselect` service from kubernetes or setup Grafana and check it via the web interface.
Run the following command to see the list of services:
<div class="with-copy" markdown="1">
```console
k get svc | grep vmselect
```shell
kubectl get svc | grep vmselect
```
</div>
The expected output:
```console
```text
vmcluster-victoria-metrics-cluster-vmselect ClusterIP 10.88.2.69 <none> 8481/TCP 1m
```
Run the following command to make `vmselect`'s port accessible from the local machine:
<div class="with-copy" markdown="1">
```console
```shell
kubectl port-forward svc/vmcluster-victoria-metrics-cluster-vmselect 8481:8481
```
</div>
Execute the following command to get metrics via `curl`:
```console
```curl
curl -sg 'http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range?query=count(up{kubernetes_pod_name=~".*vmselect.*"})&start=-10m&step=1m' | jq
```
The expected output is:
```console
```json
{
"status": "success",
"isPartial": false,
@ -377,38 +362,34 @@ The expected result of the query `count(up{kubernetes_pod_name=~".*vmselect.*"})
To test via Grafana, we need to install it first. [Install and connect Grafana to VictoriaMetrics](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster.html#4-install-and-connect-grafana-to-victoriametrics-with-helm), login into Grafana and open the metrics [Explore](http://127.0.0.1:3000/explore) page.
<img src="k8s-ha-monitoring-via-vm-cluster_explore.webp" width="800" alt="grafana explore">
<img src="k8s-ha-monitoring-via-vm-cluster_explore.webp" alt="grafana explore">
Choose `victoriametrics` from the list of datasources and enter `count(up{kubernetes_pod_name=~".*vmselect.*"})` to the **Metric browser** field as shown on the screenshot, then press **Run query** button:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up.webp" width="800" alt="">
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up.webp">
The expected output is:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp" width="800" alt="">
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp">
## 5. High Availability
To test if High Availability works, we need to shutdown one of the `vmstorages`. To do this, run the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl scale sts vmcluster-victoria-metrics-cluster-vmstorage --replicas=2
```
</div>
Verify that now we have two running `vmstorages` in the cluster by executing the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get pods | grep vmstorage
```
</div>
The expected output is:
```console
```text
vmcluster-victoria-metrics-cluster-vmstorage-0 1/1 Running 0 44m
vmcluster-victoria-metrics-cluster-vmstorage-1 1/1 Running 0 43m
```
@ -417,13 +398,13 @@ Return to Grafana Explore and press the **Run query** button again.
The expected output is:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp" width="800" alt="">
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph.webp" >
As you can see, after we scaled down the `vmstorage` replicas number from three to two pods, metrics are still available and correct. The response is not partial as it was before scaling. Also we see that query `count(up{kubernetes_pod_name=~".*vmselect.*"})` returns the same value as before.
To confirm that the number of `vmstorage` pods is equivalent to two, execute the following request in Grafana Explore:
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph2.webp" width="800" alt="">
<img src="k8s-ha-monitoring-via-vm-cluster_explore-count-up-graph2.webp" >
## 6. Final thoughts

View file

@ -26,7 +26,7 @@ We will use:
* [Helm 3 ](https://helm.sh/docs/intro/install)
* [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl)
<img src="k8s-monitoring-via-vm-cluster_scheme.webp" width="800" alt="VictoriaMetrics Cluster on Kubernetes cluster">
<img src="k8s-monitoring-via-vm-cluster_scheme.webp" alt="VictoriaMetrics Cluster on Kubernetes cluster">
## 1. VictoriaMetrics Helm repository
@ -34,37 +34,25 @@ We will use:
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. Were going to use [VictoriaMetrics Cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html). You can do this by running the following command:
<div class="with-copy" markdown="1">
```console
```shell
helm repo add vm https://victoriametrics.github.io/helm-charts/
```
</div>
Update Helm repositories:
<div class="with-copy" markdown="1">
```console
```shell
helm repo update
```
</div>
To verify that everything is set up correctly you may run this command:
<div class="with-copy" markdown="1">
```console
```shell
helm search repo vm/
```
</div>
The expected output is:
```console
```text
NAME CHART VERSION APP VERSION DESCRIPTION
vm/victoria-metrics-agent 0.7.20 v1.62.0 Victoria Metrics Agent - collects metrics from ...
vm/victoria-metrics-alert 0.3.34 v1.62.0 Victoria Metrics Alert - executes a list of giv...
@ -79,9 +67,7 @@ vm/victoria-metrics-single 0.7.5 1.62.0 Victoria Metrics Single
Run this command in your terminal:
<div class="with-copy" markdown="1">
```yaml
```sh
cat <<EOF | helm install vmcluster vm/victoria-metrics-cluster -f -
vmselect:
podAnnotations:
@ -99,7 +85,6 @@ vmstorage:
prometheus.io/port: "8482"
EOF
```
</div>
* By running `Helm install vmcluster vm/victoria-metrics-cluster` we install [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster.
* By adding `podAnnotations: prometheus.io/scrape: "true"` we enable the scraping of metrics from the vmselect, vminsert and vmstorage pods.
@ -108,7 +93,7 @@ EOF
As a result of this command you will see the following output:
```console
```text
NAME: vmcluster
LAST DEPLOYED: Thu Jul 1 09:41:57 2021
NAMESPACE: default
@ -165,16 +150,14 @@ For us its important to remember the url for the datasource (copy lines from
Verify that [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) pods are up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
kubectl get pods
```
</div>
The expected output is:
```console
```text
NAME READY STATUS RESTARTS AGE
vmcluster-victoria-metrics-cluster-vminsert-689cbc8f55-95szg 1/1 Running 0 16m
vmcluster-victoria-metrics-cluster-vminsert-689cbc8f55-f852l 1/1 Running 0 16m
@ -188,12 +171,10 @@ vmcluster-victoria-metrics-cluster-vmstorage-1 1/1 Running
To scrape metrics from Kubernetes with a [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) we need to install [vmagent](https://docs.victoriametrics.com/vmagent.html) with additional configuration. To do so, please run these commands in your terminal:
<div class="with-copy" markdown="1">
```yaml
```shell
helm install vmagent vm/victoria-metrics-agent -f https://docs.victoriametrics.com/guides/guide-vmcluster-vmagent-values.yaml
```
</div>
Here is full file content `guide-vmcluster-vmagent-values.yaml`
@ -428,16 +409,14 @@ config:
Verify that `vmagent`'s pod is up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get pods | grep vmagent
```
</div>
The expected output is:
```console
```text
vmagent-victoria-metrics-agent-69974b95b4-mhjph 1/1 Running 0 11m
```
@ -446,21 +425,18 @@ vmagent-victoria-metrics-agent-69974b95b4-mhjph 1/1 Running
Add the Grafana Helm repository.
<div class="with-copy" markdown="1">
```console
```shell
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
```
</div>
See more information on Grafana ArtifactHUB [https://artifacthub.io/packages/helm/grafana/grafana](https://artifacthub.io/packages/helm/grafana/grafana)
To install the chart with the release name `my-grafana`, add the VictoriaMetrics datasource with official dashboard and the Kubernetes dashboard:
<div class="with-copy" markdown="1">
```yaml
```sh
cat <<EOF | helm install my-grafana grafana/grafana -f -
datasources:
datasources.yaml:
@ -504,7 +480,6 @@ cat <<EOF | helm install my-grafana grafana/grafana -f -
datasource: victoriametrics
EOF
```
</div>
By running this command we:
* Install Grafana from the Helm repository.
@ -518,34 +493,32 @@ Please see the output log in your terminal. Copy, paste and run these commands.
The first one will show `admin` password for the Grafana admin.
The second and the third will forward Grafana to `127.0.0.1:3000`:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get secret --namespace default my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=my-grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000
```
</div>
## 5. Check the result you obtained in your browser
To check that [VictoriaMetrics](https://victoriametrics.com) collects metrics from k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose the `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously got from kubectl.
<img src="k8s-monitoring-via-vm-cluster_dashes-agent.webp" width="800" alt="grafana dashboards">
<img src="k8s-monitoring-via-vm-cluster_dashes-agent.webp" alt="grafana dashboards">
You will see something like this:
<img src="k8s-monitoring-via-vm-cluster_dashboard.webp" width="800" alt="Kubernetes metrics provided by vmcluster">
<img src="k8s-monitoring-via-vm-cluster_dashboard.webp" alt="Kubernetes metrics provided by vmcluster">
The VictoriaMetrics dashboard is also available to use:
<img src="k8s-monitoring-via-vm-cluster_grafana-dash.webp" width="800" alt="VictoriaMetrics cluster dashboard">
<img src="k8s-monitoring-via-vm-cluster_grafana-dash.webp" alt="VictoriaMetrics cluster dashboard">
vmagent has its own dashboard:
<img src="k8s-monitoring-via-vm-cluster_vmagent-grafana-dash.webp" width="800" alt="vmagent dashboard">
<img src="k8s-monitoring-via-vm-cluster_vmagent-grafana-dash.webp" alt="vmagent dashboard">
## 6. Final thoughts

View file

@ -26,7 +26,7 @@ We will use:
* [Helm 3 ](https://helm.sh/docs/intro/install)
* [kubectl 1.21](https://kubernetes.io/docs/tasks/tools/install-kubectl)
<img src="k8s-monitoring-via-vm-single_k8s-scheme.webp" width="800" alt="VictoriaMetrics Single on Kubernetes cluster">
<img src="k8s-monitoring-via-vm-single_k8s-scheme.webp" alt="VictoriaMetrics Single on Kubernetes cluster">
## 1. VictoriaMetrics Helm repository
@ -34,37 +34,27 @@ We will use:
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. Were going to use [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html). You can do this by running the following command:
<div class="with-copy" markdown="1">
```console
```shell
helm repo add vm https://victoriametrics.github.io/helm-charts/
```
</div>
Update Helm repositories:
<div class="with-copy" markdown="1">
```console
```shell
helm repo update
```
</div>
To verify that everything is set up correctly you may run this command:
<div class="with-copy" markdown="1">
```console
```shell
helm search repo vm/
```
</div>
The expected output is:
```console
```text
NAME CHART VERSION APP VERSION DESCRIPTION
vm/victoria-metrics-agent 0.7.20 v1.62.0 Victoria Metrics Agent - collects metrics from ...
vm/victoria-metrics-alert 0.3.34 v1.62.0 Victoria Metrics Alert - executes a list of giv...
@ -80,9 +70,7 @@ vm/victoria-metrics-single 0.7.5 1.62.0 Victoria Metrics Single
Run this command in your terminal:
<div class="with-copy" markdown="1">.html
```console
```text
helm install vmsingle vm/victoria-metrics-single -f https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml
```
@ -174,8 +162,6 @@ server:
```
</div>
* By running `helm install vmsingle vm/victoria-metrics-single` we install [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html) to default [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) inside your cluster
* By adding `scrape: enable: true` we add and enable autodiscovery scraping from kubernetes cluster to [VictoriaMetrics Single](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html)
* On line 166 from [https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml](https://docs.victoriametrics.com/guides/guide-vmsingle-values.yaml) we added `metric_relabel_configs` section that will help us to show Kubernetes metrics on Grafana dashboard.
@ -183,7 +169,7 @@ server:
As a result of the command you will see the following output:
```console
```text
NAME: victoria-metrics
LAST DEPLOYED: Fri Jun 25 12:06:13 2021
NAMESPACE: default
@ -225,17 +211,14 @@ For us its important to remember the url for the datasource (copy lines from
Verify that VictoriaMetrics pod is up and running by executing the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get pods
```
</div>
The expected output is:
```console
```text
NAME READY STATUS RESTARTS AGE
vmsingle-victoria-metrics-single-server-0 1/1 Running 0 68s
```
@ -245,19 +228,15 @@ vmsingle-victoria-metrics-single-server-0 1/1 Running 0 68s
Add the Grafana Helm repository.
<div class="with-copy" markdown="1">
```console
```shell
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
```
</div>
By installing the Chart with the release name `my-grafana`, you add the VictoriaMetrics datasource with official dashboard and kubernetes dashboard:
<div class="with-copy" markdown="1">
```yaml
cat <<EOF | helm install my-grafana grafana/grafana -f -
datasources:
@ -299,7 +278,6 @@ cat <<EOF | helm install my-grafana grafana/grafana -f -
EOF
```
</div>
By running this command we:
* Install Grafana from Helm repository.
@ -311,26 +289,18 @@ By running this command we:
Check the output log in your terminal.
To see the password for Grafana `admin` user use the following command:
<div class="with-copy" markdown="1">
```console
```shell
kubectl get secret --namespace default my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
</div>
Expose Grafana service on `127.0.0.1:3000`:
<div class="with-copy" markdown="1">
```console
```shell
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=my-grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000
```
</div>
Now Grafana should be accessible on the [http://127.0.0.1:3000](http://127.0.0.1:3000) address.
@ -338,15 +308,15 @@ Now Grafana should be accessible on the [http://127.0.0.1:3000](http://127.0.0.1
To check that VictoriaMetrics has collects metrics from the k8s cluster open in browser [http://127.0.0.1:3000/dashboards](http://127.0.0.1:3000/dashboards) and choose `Kubernetes Cluster Monitoring (via Prometheus)` dashboard. Use `admin` for login and `password` that you previously obtained from kubectl.
<img src="k8s-monitoring-via-vm-single_grafana-dashboards.webp" width="800" alt="">
<img src="k8s-monitoring-via-vm-single_grafana-dashboards.webp" alt="">
You will see something like this:
<img src="k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp" width="800" alt="">
<img src="k8s-monitoring-via-vm-single_grafana-k8s-dashboard.webp" alt="">
VictoriaMetrics dashboard also available to use:
<img src="k8s-monitoring-via-vm-single_grafana.webp" width="800" alt="">
<img src="k8s-monitoring-via-vm-single_grafana.webp" alt="">
## 5. Final thoughts

View file

@ -16,7 +16,7 @@ Let's cover the case. You have multiple regions with workloads and want to colle
The monitoring setup is in the dedicated regions as shown below:
<img src="multi-regional-setup-dedicated-regions.webp" width="800" alt="Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring">
<img src="multi-regional-setup-dedicated-regions.webp" alt="Multi-regional setup with VictoriaMetrics: Dedicated regions for monitoring">
Every workload region (Earth, Mars, Venus) has a vmagent that sends data to multiple regions with a monitoring setup.
The monitoring setup (Ground Control 1,2) contains VictoriaMetrics Time Series Database(TSDB) cluster or single.

View file

@ -34,37 +34,30 @@ This chart will install `VMOperator`, `VMAgent`, `NodeExporter`, `kube-state-met
Install the Helm chart in a custom namespace
1. Create a unique Kubernetes namespace, for example `monitoring`
<div class="with-copy" markdown="1">
```bash
```shell
kubectl create namespace monitoring
```
</div>
1. Create kubernetes-secrets with token to access your dbaas deployment
<div class="with-copy" markdown="1">
```bash
```shell
kubectl --namespace monitoring create secret generic dbaas-write-access-token --from-literal=bearerToken=your-token
kubectl --namespace monitoring create secret generic dbaas-read-access-token --from-literal=bearerToken=your-token
```
</div>
You can find your access token on the "Access" tab of your deployment
<img src="kubernetes_monitoring.webp" width="800">
<img src="kubernetes_monitoring.webp">
1. Set up a Helm repository using the following commands:
<div class="with-copy" markdown="1">
```bash
```shell
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add vm https://victoriametrics.github.io/helm-charts
helm repo update
```
</div>
1. Create a YAML file of Helm values called dbaas.yaml with following content
<div class="with-copy" markdown="1">
```yaml
externalVM:
@ -104,15 +97,12 @@ Install the Helm chart in a custom namespace
enabled: true
```
</div>
1. Install VictoriaMetrics-k8s-stack helm chart
<div class="with-copy" markdown="1">
```bash
```shell
helm --namespace monitoring install vm vm/victoria-metrics-k8s-stack -f dbaas.yaml -n monitoring
```
</div>
## Connect grafana
@ -121,21 +111,17 @@ Connect to grafana and create your datasource
> If you are using external grafana, you can skip steps 1-3 and you will need to import dashboards that can be found here manually
1. Get grafana password
<div class="with-copy" markdown="1">
```bash
```shell
kubectl --namespace monitoring get secret vm-grafana -o jsonpath="{.data.admin-password}" | base64 -d
```
</div>
1. Connect to grafana
<div class="with-copy" markdown="1">
```bash
```shell
kubectl --namespace monitoring port-forward service/vm-grafana 3000:80
```
</div>
1. Open grafana in your browser [http://localhost:3000/datasources](http://localhost:3000/datasources)
Use admin as username and password from previous step
@ -143,7 +129,7 @@ Connect to grafana and create your datasource
Choose VictoriaMetrics or Prometheus as datasource type. Make sure you made this datasource as default for dashboards to work.
> You can find token and URL in your deployment, on Access tab
<img src="how-to-monitor-k8s_datasource.webp" width="800">
<img src="how-to-monitor-k8s_datasource.webp">
## Test it

View file

@ -34,157 +34,149 @@ There are two different methods to create an account:
### Create an account via Google Auth service:
1. Click `Continue with Google` button on the [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart)
<p>
<img src="quick_start_signup_google_click.webp" width="800">
</p>
<img src="quick_start_signup_google_click.webp" >
1. Choose Google account you want to use for registration
<p>
<img src="quick_start_signup_choose_google_account.webp" width="800">
</p>
<img src="quick_start_signup_choose_google_account.webp" >
1. You will be automatically redirected to the main page of the Managed VictoriaMetrics
<p>
<img src="quick_start_signup_success.webp" width="800">
</p>
<img src="quick_start_signup_success.webp" >
### Create an account by filling in a registration form:
1. Fill in your email, password and password confirmation on [Sign Up page](https://cloud.victoriametrics.com/signUp?utm_source=website&utm_campaign=docs_quickstart).
<p>
<img src="quick_start_signup.webp" width="800">
</p>
1. All fields are required. Any errors will be shown in the interface, so it is easy to understand what should be adjusted.
<p>
<img src="quick_start_signup_errors.webp" width="800">
</p>
<img src="quick_start_signup.webp" >
1.All fields are required. Any errors will be shown in the interface, so it is easy to understand what should be adjusted.
<img src="quick_start_signup_errors.webp" >
1. Press `Create account` button when all fields are filled in.
<p>
<img src="quick_start_signup_create_account_click.webp" width="800">
</p>
<img src="quick_start_signup_create_account_click.webp" >
You will be redirected to the main page with a notification message to confirm your email.
<p>
<img src="quick_start_signup_success.webp" width="800">
</p>
<img src="quick_start_signup_success.webp" >
You will also receive an email with a confirmation link as shown on the picture below:
<p>
<img src="quick_start_signup_email_confirm.webp" width="800">
</p>
<img src="quick_start_signup_email_confirm.webp" >
It is necessary to confirm your email address. Otherwise, you won't be able to create a deployment.
After successful confirmation of your email address, you'll be able to [create your first deployment](#creating-deployments) or [add a payment method](#adding-a-payment-method).
<p>
<img src="quick_start_signup_email_confirmed.webp" width="800">
</p>
<img src="quick_start_signup_email_confirmed.webp" >
## Adding a payment method
1. Navigate to a [Billing](https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart) page or click on `Upgrade` button as shown below:
<p>
<img src="how_to_add_payment_method_upgrade.webp" width="800">
</p>
<img src="how_to_add_payment_method_upgrade.webp" >
1. Choose a payment method
<p>
<img src="how_to_add_payment_method_choose_method.webp" width="800">
</p>
<img src="how_to_add_payment_method_choose_method.webp" >
### Pay with a card
1. Click on an `Add card` panel and fill in all the fields in the form and press `Add card` button
<p>
<img src="how_to_add_payment_method_add_card.webp" width="800">
</p>
<img src="how_to_add_payment_method_add_card.webp" >
1. An error message will appear if a card us invalid
<p>
<img src="how_to_add_payment_method_invalid_card.webp" width="800">
</p>
<img src="how_to_add_payment_method_invalid_card.webp" >
1. Successfully added card will be shown on the page as follows:
<p>
<img src="how_to_add_payment_method_card_added.webp" width="800">
</p>
<img src="how_to_add_payment_method_card_added.webp" >
### Link your AWS billing account via AWS Marketplace
When you need to unify your AWS billing, you can start a subscription on AWS Marketplace.
1. Click on the `Buy on AWS Marketplace` panel:
<p>
<img src="how_to_add_payment_method_aws_click.webp" width="800">
</p>
<img src="how_to_add_payment_method_aws_click.webp" >
1. You will be redirected to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc" target="_blank">Managed VictoriaMetrics</a> product page.
1. Click on `View purchase option` button, and you will be redirected to an AWS login page or to a subscribe page on AWS Marketplace.
<p>
<img src="quickstart_aws-purchase-click.webp" width="800">
</p>
<img src="quickstart_aws-purchase-click.webp" >
1. Go to the <a href="https://aws.amazon.com/marketplace/pp/prodview-4tbfq5icmbmyc">Managed VictoriaMetrics</a> product page and click `Continue to Subscribe` button:
<p>
<img src="quickstart_continue-subscribe.webp" width="800">
</p>
<img src="quickstart_continue-subscribe.webp" >
1. Press the `Subscribe` button:
<p>
<img src="quickstart_subscribe.webp" width="800">
</p>
<img src="quickstart_subscribe.webp" >
1. After that you will see a success message where you should click `Set up your account` button:
<p>
<img src="quickstart_setup-your-account.webp" width="800">
</p>
<img src="quickstart_setup-your-account.webp" >
1. You'll be redirected back to Managed VictoriaMetrics <a href="https://cloud.victoriametrics.com/billing?utm_source=website&utm_campaign=docs_quickstart" target="_blank">billing page</a>:
<p>
<img src="how_to_add_payment_method_aws_finish.webp" width="800">
</p>
<img src="how_to_add_payment_method_aws_finish.webp" >
### Switching between payment methods
If both payment methods are added, it is possible to easily switch between them.
Click on the radio button like on the picture below and confirm the change:
<p>
<img src="change_payment_method.webp" width="800">
</p>
<p>
<img src="change_payment_confirmation.webp" width="800">
</p>
<img src="change_payment_method.webp" >
<img src="change_payment_confirmation.webp" >
If the payment method was changed successfully, the following message will appear:
<p>
<img src="change_payment_method_success.webp" width="800">
</p>
<img src="change_payment_method_success.webp" >
## Password restoration
If you forgot your password, it can be restored in the following way:
1. Click `Forgot password?` link on the [Sign In](https://cloud.victoriametrics.com/signIn?utm_source=website&utm_campaign=docs_quickstart) page:
<p>
<img src="quick_start_restore_password.webp" width="800">
</p>
<img src="quick_start_restore_password.webp" >
1. Enter your email and click `Reset password` button:
<p>
<img src="quick_start_restore_password_email_field.webp" width="800">
</p>
<img src="quick_start_restore_password_email_field.webp" >
<p>
<img src="quick_start_restore_password_message.webp" width="800">
</p>
<img src="quick_start_restore_password_message.webp" >
1. Follow the instructions sent to your email in order to get access to your Managed VictoriaMetrics account:
<p>
<img src="quick_start_restore_password_email.webp" width="800">
</p>
<img src="quick_start_restore_password_email.webp" >
1. Navigate to the Profile page by clicking the corresponding link in the top right corner:
<p>
<img src="quick_start_restore_password_profile_click.webp" width="800">
</p>
<img src="quick_start_restore_password_profile_click.webp" >
1. Enter a new password on the Profile page and press `Save`:
<p>
<img src="quick_start_restore_password_profile_fields.webp" width="800">
</p>
<img src="quick_start_restore_password_profile_fields.webp" >
## Creating deployments
@ -193,9 +185,9 @@ will see a list of your existing deployments and will be able to manage them.
To create a deployment click on the button `Create Deployment` button:
<p>
<img src="create_deployment_start.webp" width="800">
</p>
<img src="create_deployment_start.webp" >
On the opened screen, choose parameters of your new deployment:
@ -207,9 +199,9 @@ On the opened screen, choose parameters of your new deployment:
* `Retention` period for stored metrics.
* `Size` of your deployment [based on your needs](https://docs.victoriametrics.com/guides/understand-your-setup-size.html)
<p>
<img src="create_deployment_form.webp" width="800">
</p>
<img src="create_deployment_form.webp" >
When all parameters are configured, click on the `Create` button, and deployment will be created.
@ -217,13 +209,13 @@ Once created, deployment will remain for a short period of time in `PROVISIONING
while the hardware spins-up, just wait for a couple of minutes and reload the page.
You'll also be notified via email once your deployment is ready to use:
<p>
<img src="create_deployment_created.webp" width="800">
</p>
<p>
<img src="create_deployment_active_email.webp" width="800">
</p>
<img src="create_deployment_created.webp" >
<img src="create_deployment_active_email.webp" >
## Start writing and reading data
@ -232,21 +224,21 @@ and is ready to accept write and read requests.
Click on deployment name and navigate to the Access tab to get the access token:
<p>
<img src="deployment_access.webp" width="800">
</p>
<img src="deployment_access.webp" >
Access tokens are used in token-based authentication to allow an application to access the VictoriaMetrics API.
Supported token types are `Read-Only`, `Write-Only` and `Read-Write`. Click on token created by default
to see usage examples:
<p>
<img src="deployment_access_write_example.webp" width="800">
</p>
<p>
<img src="deployment_access_read_example.webp" width="800">
</p>
<img src="deployment_access_write_example.webp" >
<img src="deployment_access_read_example.webp" >
Follow usage examples in order to configure access to VictoriaMetrics for your Prometheus,
Grafana or any other software.
@ -257,15 +249,15 @@ Remember, you can always add, remove or modify existing deployment by changing i
deployment's page.
It is important to know that downgrade for cluster is currently not available.
<p>
<img src="modify_deployment.webp" width="800">
</p>
<img src="modify_deployment.webp" >
To discover additional configuration options click on `Advanced Settings` button, so you should see the following:
<p>
<img src="modify_deployment_additional_settings.webp" width="800">
</p>
<img src="modify_deployment_additional_settings.webp" >
In that section, additional params can be set:

View file

@ -21,9 +21,7 @@ Table of content:
When you enter the notification section, you will be able to fill in the channels in which you
want to receive notifications
<p>
<img src="notifications_view.webp" width="800">
</p>
<img src="notifications_view.webp" >
## Setup Slack notifications
@ -31,34 +29,23 @@ want to receive notifications
How to do this is indicated on the following link
<a href="https://api.slack.com/messaging/webhooks" target="_blank">https://api.slack.com/messaging/webhooks</a>
<p>
<img src="notifications_view.webp" width="800">
</p>
<img src="notifications_view.webp" >
1. Specify Slack channels
Enter one or more channels into input and press enter or choose it after each input.
<p>
<img src="notifications_setup_slack.webp" width="800">
</p>
<p>
<img src="notifications_setup_slack_enter_channel.webp" width="800">
</p>
<img src="notifications_setup_slack.webp" >
<img src="notifications_setup_slack_enter_channel.webp">
## Setup emails notifications
You can specify one or multiple emails for notifications in the input field. By default,
email notifications are enabled for the account owner
<p>
<img src="notifications_setup_emails.webp" width="800">
</p>
<img src="notifications_setup_emails.webp" >
<img src="notifications_setup_emails_input.webp" >
<p>
<img src="notifications_setup_emails_input.webp" width="800">
</p>
## Send test notification
@ -68,26 +55,18 @@ If only Slack channels and webhook are specified correctly, you will receive the
If only the emails are specified, you will receive notifications to those emails.
When both notifications are specified, all notifications will be sent to Slack channels and emails.
<p>
<img src="notifications_save_and_test.webp" width="800">
</p>
<img src="notifications_save_and_test.webp" >
If the Save button is pressed, then entered channels will be only saved, and you get a success message.
If the Save and Test button is pressed, then all entered information will be saved,
and test notifications will be sent to the entered channels
<p>
<img src="notifications_save_success.webp" width="800">
</p>
<img src="notifications_save_success.webp" >
Examples of the test notification messages:
<p>
<img src="notifications_slack_test.webp" width="800">
</p>
<img src="notifications_slack_test.webp" >
<p>
<img src="notifications_email_test.webp" width="800">
</p>
<img src="notifications_email_test.webp" >

View file

@ -105,9 +105,7 @@ You assign the role to the user during the user creation procedure. You can chan
It shows all users with different roles, and you can apply provided actions with them.
<p>
<img src="user_management_list.webp" width="800">
</p>
<img src="user_management_list.webp">
In the table, there is additional information about the users:
@ -140,9 +138,7 @@ In the table, there is additional information about the users:
Click on `Invite user` button
the user invitation button and fill out the form in the modal, which will appear after you click. All fields are mandatory.
<p>
<img src="user_management_invite_user.webp" width="800">
</p>
<img src="user_management_invite_user.webp" >
After filling out the form, click on the `Invite` button.
The user will be saved, and an invitation email to the provided email address will be sent. As a confirmation, you will see the success message.
@ -151,30 +147,26 @@ The user will be saved, and an invitation email to the provided email address wi
The user will be at Pending Invitation status. After accepting the invitation user status changes to Active.
<p>
<img src="user_management_invite_success.webp" width="800">
</p>
<img src="user_management_invite_success.webp" >
## How to Update User
To edit the user role and details, activate or deactivate a user, and click on
<p>
<img src="user_management_update_user.webp" width="800">
</p>
<img src="user_management_update_user.webp" >
or `user email` to edit the user.
User editing form:
<p>
<img src="user_management_user_update_form.webp" width="800">
</p>
<img src="user_management_user_update_form.webp" >
To save changes, click the `Save changes` button. If changes are saved successfully, you will see a message at the top of the page.
<p>
<img src="user_management_user_updated_success.webp" width="800">
</p>
<img src="user_management_user_updated_success.webp" >
## How to Delete User
@ -182,21 +174,15 @@ You can delete a user from your account. However, you will need to invite them a
Click on `Delete` button to delete the user.
<p>
<img src="user_management_user_delete.webp" width="800">
</p>
<img src="user_management_user_delete.webp" >
To confirm the deletion of a user, you will need to re-enter their email address and press the **Delete** button
<p>
<img src="user_management_delete_user_form.webp" width="800">
</p>
<img src="user_management_delete_user_form.webp" >
You will be redirected to the main page with a success or error message
<p>
<img src="user_management_delete_sucess.webp" width="800">
</p>
<img src="user_management_delete_sucess.webp" >
## How to resend invitation
@ -204,18 +190,12 @@ If the invitation is expired, it is possible to resend email to the user
Click `Resend invitation` button
<p>
<img src="user_management_resend_invitation.webp" width="800">
</p>
<img src="user_management_resend_invitation.webp" >
Confirm resend invitation by clicking `Resend` button in the modal dialog
<p>
<img src="user_management_confirm_resend_invitation.webp" width="800">
</p>
<img src="user_management_confirm_resend_invitation.webp" >
If invitation successfully resented to the user success message will appear
<p>
<img src="user_management_resend_success.webp" width="800">
</p>
<img src="user_management_resend_success.webp" >