From 4d1b572f4699a13f8fd8f41d55a06947cbed2da8 Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Tue, 26 Sep 2023 10:50:10 +0200 Subject: [PATCH] Docker add vmauth (#5057) * docker-compose: add vmauth to cluster env vmauth acts as a balancer and used as an example of how to interconnect VM components via vmauth. Signed-off-by: hagen1778 * docker-compose: add vmauth to cluster env vmauth acts as a balancer and used as an example of how to interconnect VM components via vmauth. Signed-off-by: hagen1778 --------- Signed-off-by: hagen1778 Co-authored-by: Nikolay --- app/vmauth/README.md | 1 + deployment/docker/README.md | 31 +++++++--- deployment/docker/auth-cluster.yml | 6 ++ deployment/docker/docker-compose-cluster.yml | 59 +++++++++++++++---- deployment/docker/prometheus-cluster.yml | 2 +- .../provisioning/datasources/datasource.yml | 2 +- docs/CHANGELOG.md | 1 + docs/vmauth.md | 1 + 8 files changed, 79 insertions(+), 24 deletions(-) create mode 100644 deployment/docker/auth-cluster.yml diff --git a/app/vmauth/README.md b/app/vmauth/README.md index 610462412..71a627e01 100644 --- a/app/vmauth/README.md +++ b/app/vmauth/README.md @@ -25,6 +25,7 @@ The auth config can be reloaded via the following ways: and apply new changes every 5 seconds. Docker images for `vmauth` are available [here](https://hub.docker.com/r/victoriametrics/vmauth/tags). +See how `vmauth` used in [docker-compose env](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/README.md#victoriametrics-cluster). Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions. diff --git a/deployment/docker/README.md b/deployment/docker/README.md index ed9364557..fe368e880 100644 --- a/deployment/docker/README.md +++ b/deployment/docker/README.md @@ -42,30 +42,36 @@ The communication scheme between components is the following: and recording rules back to it; * [alertmanager](#alertmanager) is configured to receive notifications from `vmalert`. -To access `vmalert` use link [http://localhost:8428/vmalert](http://localhost:8428/vmalert/). +To access Grafana use link [http://localhost:3000](http://localhost:3000). To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) use link [http://localhost:8428/vmui](http://localhost:8428/vmui). +To access `vmalert` use link [http://localhost:8428/vmalert](http://localhost:8428/vmalert/). + + ## VictoriaMetrics cluster VictoriaMetrics cluster environment consists of `vminsert`, `vmstorage` and `vmselect` components. -`vmselect` has exposed port `:8481`, `vminsert` has exposed port `:8480` and the rest of components -are available only inside the environment. +`vminsert` has exposed port `:8480`, access to `vmselect` components goes through `vmauth` on port `:8427`, +and the rest of components are available only inside the environment. The communication scheme between components is the following: * [vmagent](#vmagent) sends scraped metrics to `vminsert`; * `vminsert` forwards data to `vmstorage`; -* `vmselect` is connected to `vmstorage` for querying data; -* [grafana](#grafana) is configured with datasource pointing to `vmselect`; -* [vmalert](#vmalert) is configured to query `vmselect` and send alerts state +* `vmselect`s are connected to `vmstorage` for querying data; +* [vmauth](#vmauth) balances incoming read requests among `vmselect`s; +* [grafana](#grafana) is configured with datasource pointing to `vmauth`; +* [vmalert](#vmalert) is configured to query `vmselect`s via `vmauth` and send alerts state and recording rules to `vminsert`; * [alertmanager](#alertmanager) is configured to receive notifications from `vmalert`. -To access `vmalert` use link [http://localhost:8481/select/0/prometheus/vmalert](http://localhost:8481/select/0/prometheus/vmalert/). +To access Grafana use link [http://localhost:3000](http://localhost:3000). -To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) -use link [http://localhost:8481/select/0/prometheus/vmui](http://localhost:8481/select/0/prometheus/vmui). +To access [vmui](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#vmui) +use link [http://localhost:8427/select/0/prometheus/vmui/](http://localhost:8427/select/0/prometheus/vmui/). + +To access `vmalert` use link [http://localhost:8427/select/0/prometheus/vmalert/](http://localhost:8427/select/0/prometheus/vmalert/). ## vmagent @@ -75,6 +81,13 @@ with listed targets for scraping. [Web interface link](http://localhost:8429/). +## vmauth + +[vmauth](https://docs.victoriametrics.com/vmauth.html) acts as a [balancer](https://docs.victoriametrics.com/vmauth.html#load-balancing) +to spread the load across `vmselect`'s. [Grafana](#grafana) and [vmalert](#vmalert) use vmauth for read queries. +vmauth config is available [here](ttps://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/auth-cluster.yml) + + ## vmalert vmalert evaluates alerting rules [alerts.yml](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml) diff --git a/deployment/docker/auth-cluster.yml b/deployment/docker/auth-cluster.yml new file mode 100644 index 000000000..820fa0f40 --- /dev/null +++ b/deployment/docker/auth-cluster.yml @@ -0,0 +1,6 @@ +# balance load among vmselects +# see https://docs.victoriametrics.com/vmauth.html#load-balancing +unauthorized_user: + url_prefix: + - http://vmselect-1:8481 + - http://vmselect-2:8481 \ No newline at end of file diff --git a/deployment/docker/docker-compose-cluster.yml b/deployment/docker/docker-compose-cluster.yml index 307765cad..35945cf1c 100644 --- a/deployment/docker/docker-compose-cluster.yml +++ b/deployment/docker/docker-compose-cluster.yml @@ -2,7 +2,7 @@ version: '3.5' services: vmagent: container_name: vmagent - image: victoriametrics/vmagent:v1.93.4 + image: victoriametrics/vmagent:v1.93.5 depends_on: - "vminsert" ports: @@ -19,7 +19,8 @@ services: container_name: grafana image: grafana/grafana:9.2.7 depends_on: - - "vmselect" + - "vmselect-1" + - "vmselect-2" ports: - 3000:3000 restart: always @@ -32,7 +33,7 @@ services: vmstorage-1: container_name: vmstorage-1 - image: victoriametrics/vmstorage:v1.93.4-cluster + image: victoriametrics/vmstorage:v1.93.5-cluster ports: - 8482 - 8400 @@ -44,7 +45,7 @@ services: restart: always vmstorage-2: container_name: vmstorage-2 - image: victoriametrics/vmstorage:v1.93.4-cluster + image: victoriametrics/vmstorage:v1.93.5-cluster ports: - 8482 - 8400 @@ -54,9 +55,10 @@ services: command: - '--storageDataPath=/storage' restart: always + vminsert: container_name: vminsert - image: victoriametrics/vminsert:v1.93.4-cluster + image: victoriametrics/vminsert:v1.93.5-cluster depends_on: - "vmstorage-1" - "vmstorage-2" @@ -66,9 +68,10 @@ services: ports: - 8480:8480 restart: always - vmselect: - container_name: vmselect - image: victoriametrics/vmselect:v1.93.4-cluster + + vmselect-1: + container_name: vmselect-1 + image: victoriametrics/vmselect:v1.93.5-cluster depends_on: - "vmstorage-1" - "vmstorage-2" @@ -77,14 +80,44 @@ services: - '--storageNode=vmstorage-2:8401' - '--vmalert.proxyURL=http://vmalert:8880' ports: - - 8481:8481 + - 8481 + restart: always + + vmselect-2: + container_name: vmselect-2 + image: victoriametrics/vmselect:v1.93.5-cluster + depends_on: + - "vmstorage-1" + - "vmstorage-2" + command: + - '--storageNode=vmstorage-1:8401' + - '--storageNode=vmstorage-2:8401' + - '--vmalert.proxyURL=http://vmalert:8880' + ports: + - 8481 + restart: always + + vmauth: + container_name: vmauth + image: victoriametrics/vmauth:v1.93.5 + depends_on: + - "vmselect-1" + - "vmselect-2" + volumes: + - ./auth-cluster.yml:/etc/auth.yml +# - /var/run/docker.sock:/var/run/docker.sock + command: + - '--auth.config=/etc/auth.yml' + ports: + - 8427:8427 restart: always vmalert: container_name: vmalert - image: victoriametrics/vmalert:v1.93.4 + image: victoriametrics/vmalert:v1.93.5 depends_on: - - "vmselect" + - "vmselect-1" + - "vmselect-2" ports: - 8880:8880 volumes: @@ -93,8 +126,8 @@ services: - ./alerts-vmagent.yml:/etc/alerts/alerts-vmagent.yml - ./alerts-vmalert.yml:/etc/alerts/alerts-vmalert.yml command: - - '--datasource.url=http://vmselect:8481/select/0/prometheus' - - '--remoteRead.url=http://vmselect:8481/select/0/prometheus' + - '--datasource.url=http://vmauth:8427/select/0/prometheus' + - '--remoteRead.url=http://vmauth:8427/select/0/prometheus' - '--remoteWrite.url=http://vminsert:8480/insert/0/prometheus' - '--notifier.url=http://alertmanager:9093/' - '--rule=/etc/alerts/*.yml' diff --git a/deployment/docker/prometheus-cluster.yml b/deployment/docker/prometheus-cluster.yml index 32336929b..e765b0860 100644 --- a/deployment/docker/prometheus-cluster.yml +++ b/deployment/docker/prometheus-cluster.yml @@ -13,7 +13,7 @@ scrape_configs: - targets: ['vminsert:8480'] - job_name: 'vmselect' static_configs: - - targets: ['vmselect:8481'] + - targets: ['vmselect-1:8481', 'vmselect-2:8481'] - job_name: 'vmstorage' static_configs: - targets: ['vmstorage-1:8482', 'vmstorage-2:8482'] \ No newline at end of file diff --git a/deployment/docker/provisioning/datasources/datasource.yml b/deployment/docker/provisioning/datasources/datasource.yml index e16c273c4..c0a7a20c9 100644 --- a/deployment/docker/provisioning/datasources/datasource.yml +++ b/deployment/docker/provisioning/datasources/datasource.yml @@ -10,5 +10,5 @@ datasources: - name: VictoriaMetrics - cluster type: prometheus access: proxy - url: http://vmselect:8481/select/0/prometheus + url: http://vmauth:8427/select/0/prometheus isDefault: false \ No newline at end of file diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 11b56de00..ed800d3e3 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -49,6 +49,7 @@ The sandbox cluster installation is running under the constant load generated by * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `eval_offset` attribute for [Groups](https://docs.victoriametrics.com/vmalert.html#groups). If specified, Group will be evaluated at the exact time offset on the range of [0...evaluationInterval]. The setting might be useful for cron-like rules which must be evaluated at specific moments of time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3409) for details. * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): validate [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) function names in alerting and recording rules when `vmalert` runs with `-dryRun` command-line flag. Previously it was allowed to use unknown (aka invalid) MetricsQL function names there. For example, `foo()` was counted as a valid query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4933). * FEATURE: limit the length of string params in log messages to 500 chars. Longer string params are replaced with the `first_250_chars..last_250_chars`. This prevents from too long log lines, which can be emitted by VictoriaMetrics components. +* FEATURE: [docker compose environment](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker): add `vmauth` component to cluster's docker-compose example for balancing load among multiple `vmselect` components. * FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): make sure that `q2` series are returned after `q1` series in the results of `q1 or q2` query, in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763). * FEATURE: stop exposing `vm_merge_need_free_disk_space` metric, since it has been appeared that it confuses users while doesn't bring any useful information. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686#issuecomment-1733844128). diff --git a/docs/vmauth.md b/docs/vmauth.md index d6e631a7f..65e8d929c 100644 --- a/docs/vmauth.md +++ b/docs/vmauth.md @@ -36,6 +36,7 @@ The auth config can be reloaded via the following ways: and apply new changes every 5 seconds. Docker images for `vmauth` are available [here](https://hub.docker.com/r/victoriametrics/vmauth/tags). +See how `vmauth` used in [docker-compose env](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/README.md#victoriametrics-cluster). Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions.