address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.
Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.
Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.
Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68bad22fd2
)
9.7 KiB
Docker compose environment for VictoriaMetrics
Docker compose environment for VictoriaMetrics includes VictoriaMetrics components, Alertmanager and Grafana.
For starting the docker-compose environment ensure you have docker installed and running and access to the Internet. All commands should be executed from the root directory of the repo.
- VictoriaMetrics single server
- VictoriaMetrics cluster
- vmagent
- vmauth
- vmalert
- alertmanager
- Alerts
- Grafana
- VictoriaLogs
VictoriaMetrics single server
To spin-up environment with VictoriaMetrics single server run the following command:
make docker-single-up
VictoriaMetrics will be accessible on the following ports:
--graphiteListenAddr=:2003
--opentsdbListenAddr=:4242
--httpListenAddr=:8428
The communication scheme between components is the following:
- vmagent sends scraped metrics to
single server VictoriaMetrics
; - grafana is configured with datasource pointing to
single server VictoriaMetrics
; - vmalert is configured to query
single server VictoriaMetrics
and send alerts state and recording rules back to it; - alertmanager is configured to receive notifications from
vmalert
.
To access Grafana use link http://localhost:3000.
To access vmui use link http://localhost:8428/vmui.
To access vmalert
use link http://localhost:8428/vmalert.
To shutdown environment execute the following command:
make docker-single-down
VictoriaMetrics cluster
To spin-up environment with VictoriaMetrics cluster run the following command:
make docker-cluster-up
VictoriaMetrics cluster environment consists of vminsert
, vmstorage
and vmselect
components.
vminsert
has exposed port :8480
, access to vmselect
components goes through vmauth
on port :8427
,
and the rest of components are available only inside the environment.
The communication scheme between components is the following:
- vmagent sends scraped metrics to
vminsert
; vminsert
forwards data tovmstorage
;vmselect
s are connected tovmstorage
for querying data;- vmauth balances incoming read requests among
vmselect
s; - grafana is configured with datasource pointing to
vmauth
; - vmalert is configured to query
vmselect
s viavmauth
and send alerts state and recording rules tovminsert
; - alertmanager is configured to receive notifications from
vmalert
.
To access Grafana use link http://localhost:3000.
To access vmui use link http://localhost:8427/select/0/prometheus/vmui/.
To access vmalert
use link http://localhost:8427/select/0/prometheus/vmalert/.
To shutdown environment execute the following command:
make docker-cluster-down
vmagent
vmagent is used for scraping and pushing time series to VictoriaMetrics instance. It accepts Prometheus-compatible configuration prometheus.yml with listed targets for scraping.
vmauth
vmauth acts as a balancer
to spread the load across vmselect
's. Grafana and vmalert use vmauth for read queries.
vmauth config is available here
vmalert
vmalert evaluates alerting rules alerts.yml to track VictoriaMetrics health state. It is connected with AlertManager for firing alerts, and with VictoriaMetrics for executing queries and storing alert's state.
alertmanager
AlertManager accepts notifications from vmalert
and fires alerts.
All notifications are blackholed according to alertmanager.yml config.
Grafana
To access service open following link.
Default credential:
- login -
admin
- password -
admin
Grafana is provisioned by default with following entities:
VictoriaMetrics
datasourceVictoriaMetrics - cluster
datasourceVictoriaMetrics overview
dashboardVictoriaMetrics - cluster
dashboardVictoriaMetrics - vmagent
dashboardVictoriaMetrics - vmalert
dashboard
Remember to pick VictoriaMetrics - cluster
datasource when viewing VictoriaMetrics - cluster
dashboard.
Optionally, environment with VictoriaMetrics Grafana datasource can be started with the following commands:
make docker-single-vm-datasource-up # start single server
make docker-single-vm-datasource-down # shut down single server
make docker-cluster-vm-datasource-up # start cluster
make docker-cluster-vm-datasource-down # shutdown cluster
Alerts
See below a list of recommended alerting rules for various VictoriaMetrics components for running in production. Some alerting rules thresholds are just recommendations and could require an adjustment. The list of alerting rules is the following:
- alerts-health.yml: alerting rules related to all VictoriaMetrics components for tracking their "health" state;
- alerts.yml: alerting rules related to single-server VictoriaMetrics installation;
- alerts-cluster.yml: alerting rules related to cluster version of VictoriaMetrics;
- alerts-vmagent.yml: alerting rules related to vmagent component;
- alerts-vmalert.yml: alerting rules related to vmalert component;
- alerts-vmauth.yml: alerting rules related to vmauth component;
- alerts-vlogs.yml: alerting rules related to VictoriaLogs;
Please, also see how to monitor VictoriaMetrics installations.
VictoriaLogs server
To spin-up environment with VictoriaLogs run the following command:
make docker-victorialogs-up
VictoriaLogs will be accessible on the --httpListenAddr=:9428
port.
In addition to VictoriaLogs server, the docker compose contains the following componetns:
- fluentbit service for collecting docker logs and sending them to VictoriaLogs;
- VictoriaMetrics single server to collect metrics from
VictoriaLogs
andfluentbit
; - grafana is configured with VictoriaLogs datasource.
To access Grafana use link http://localhost:3000.
To access VictoriaLogs UI use link http://localhost:9428/select/vmui.
Please, also see how to monitor VictoriaLogs installations.
To shutdown environment execute the following command:
make docker-victorialogs-down
Please see more examples on integration of VictoriaLogs with other log shippers below: