a99d26633b
Each Grafana dashboard has unique ID which can be used to fetch the dashboard
from grafana.com: https://grafana.com/grafana/dashboards/11176
The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/
But using slug implies that any change to dashboard name will break the link.
So it is better to just use ID, so the dashboard URL will never break.
This is follow-up for
|
||
|---|---|---|
| .. | ||
| base | ||
| builder | ||
| provisioning | ||
| victorialogs | ||
| vmanomaly/vmanomaly-vmalert-guide | ||
| alertmanager.yml | ||
| alerts-cluster.yml | ||
| alerts-health.yml | ||
| alerts-vmagent.yml | ||
| alerts-vmalert.yml | ||
| alerts.yml | ||
| auth-cluster.yml | ||
| docker-compose-cluster.yml | ||
| docker-compose.yml | ||
| fluent-bit.conf | ||
| Makefile | ||
| prometheus-cluster.yml | ||
| prometheus.yml | ||
| README.md | ||
Docker compose environment for VictoriaMetrics
Docker compose environment for VictoriaMetrics includes VictoriaMetrics components, Alertmanager and Grafana.
For starting the docker-compose environment ensure you have docker installed and running and access to the Internet. All commands should be executed from the root directory of the repo.
To spin-up environment for single server VictoriaMetrics run the following command:
make docker-single-up
To shut down the docker-compose environment for single server run the following command:
make docker-single-down
For cluster version the command will be the following:
make docker-cluster-up
To shut down the docker compose environment for cluster version run the following command:
make docker-cluster-down
VictoriaMetrics single server
VictoriaMetrics will be accessible on the following ports:
--graphiteListenAddr=:2003--opentsdbListenAddr=:4242--httpListenAddr=:8428
The communication scheme between components is the following:
- vmagent sends scraped metrics to
single server VictoriaMetrics; - grafana is configured with datasource pointing to
single server VictoriaMetrics; - vmalert is configured to query
single server VictoriaMetricsand send alerts state and recording rules back to it; - alertmanager is configured to receive notifications from
vmalert.
To access Grafana use link http://localhost:3000.
To access vmui use link http://localhost:8428/vmui.
To access vmalert use link http://localhost:8428/vmalert.
VictoriaMetrics cluster
VictoriaMetrics cluster environment consists of vminsert, vmstorage and vmselect components.
vminsert has exposed port :8480, access to vmselect components goes through vmauth on port :8427,
and the rest of components are available only inside the environment.
The communication scheme between components is the following:
- vmagent sends scraped metrics to
vminsert; vminsertforwards data tovmstorage;vmselects are connected tovmstoragefor querying data;- vmauth balances incoming read requests among
vmselects; - grafana is configured with datasource pointing to
vmauth; - vmalert is configured to query
vmselects viavmauthand send alerts state and recording rules tovminsert; - alertmanager is configured to receive notifications from
vmalert.
To access Grafana use link http://localhost:3000.
To access vmui use link http://localhost:8427/select/0/prometheus/vmui/.
To access vmalert use link http://localhost:8427/select/0/prometheus/vmalert/.
vmagent
vmagent is used for scraping and pushing time series to VictoriaMetrics instance. It accepts Prometheus-compatible configuration prometheus.yml with listed targets for scraping.
vmauth
vmauth acts as a balancer
to spread the load across vmselect's. Grafana and vmalert use vmauth for read queries.
vmauth config is available here
vmalert
vmalert evaluates alerting rules alerts.yml to track VictoriaMetrics health state. It is connected with AlertManager for firing alerts, and with VictoriaMetrics for executing queries and storing alert's state.
alertmanager
AlertManager accepts notifications from vmalert and fires alerts.
All notifications are blackholed according to alertmanager.yml config.
Grafana
To access service open following link.
Default credential:
- login -
admin - password -
admin
Grafana is provisioned by default with following entities:
VictoriaMetricsdatasourceVictoriaMetrics - clusterdatasourceVictoriaMetrics overviewdashboardVictoriaMetrics - clusterdashboardVictoriaMetrics - vmagentdashboardVictoriaMetrics - vmalertdashboard
Remember to pick VictoriaMetrics - cluster datasource when viewing VictoriaMetrics - cluster dashboard.
Alerts
See below a list of recommended alerting rules for various VictoriaMetrics components for running in production. Some of the alerting rules thresholds are just recommendations and could require an adjustment. The list of alerting rules is the following:
- alerts-health.yml: alerting rules related to all VictoriaMetrics components for tracking their "health" state;
- alerts.yml: alerting rules related to single-server VictoriaMetrics installation;
- alerts-cluster.yml: alerting rules related to cluster version of VictoriaMetrics;
- alerts-vmagent.yml: alerting rules related to vmagent component;
- alerts-vmalert.yml: alerting rules related to vmalert component;
Please, also see how to monitor VictoriaMetrics installations.
VictoriaLogs server
VictoriaLogs will be accessible on the following port: --httpListenAddr=:9428
Fluent Bit is used to send logs to VictoriaLogs instance.
Fluent Bit is configured to send logs from running containers to VictoriaLogs instance.
Additionally, it is configured to listen for syslog logs on port 5140 and send them to VictoriaLogs instance.
To access VictoriaLogs UI use link http://localhost:9428/select/vmui/.
Please, also see how to monitor VictoriaLogs installations.