From 277fed99905967ecfe98de7cef0c11dd36c4e9ca Mon Sep 17 00:00:00 2001 From: Artem Navoiev Date: Mon, 9 Sep 2024 14:33:05 -0700 Subject: [PATCH] victorialogs: add HA example for logstash and fluentbit (#6968) ### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are **mandatory**: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Navoiev --- .../fluentbit-ha-single-node/README.md | 88 +++++++++++++++ .../fluentbit-ha-single-node/auth.yml | 6 + .../fluentbit-ha-single-node/compose.yml | 38 +++++++ .../fluentbit-ha-single-node/fluent-bit.conf | 42 +++++++ .../logstash-ha-single-node/README.md | 71 ++++++++++++ .../logstash-ha-single-node/auth.yml | 6 + .../logstash-ha-single-node/compose.yml | 41 +++++++ .../logstash-ha-single-node/logstash.yml | 2 + .../logstash-ha-single-node/pipeline.conf | 25 +++++ .../vector-ha-single-node/README.md | 104 ++++++++++++++++++ .../vector-ha-single-node/auth.yml | 6 + .../vector-ha-single-node/compose.yml | 48 ++++++++ .../vector-ha-single-node/vector.yaml | 58 ++++++++++ docs/VictoriaLogs/README.md | 7 +- 14 files changed, 541 insertions(+), 1 deletion(-) create mode 100644 deployment/docker/victorialogs/fluentbit-ha-single-node/README.md create mode 100644 deployment/docker/victorialogs/fluentbit-ha-single-node/auth.yml create mode 100644 deployment/docker/victorialogs/fluentbit-ha-single-node/compose.yml create mode 100644 deployment/docker/victorialogs/fluentbit-ha-single-node/fluent-bit.conf create mode 100644 deployment/docker/victorialogs/logstash-ha-single-node/README.md create mode 100644 deployment/docker/victorialogs/logstash-ha-single-node/auth.yml create mode 100644 deployment/docker/victorialogs/logstash-ha-single-node/compose.yml create mode 100644 deployment/docker/victorialogs/logstash-ha-single-node/logstash.yml create mode 100644 deployment/docker/victorialogs/logstash-ha-single-node/pipeline.conf create mode 100644 deployment/docker/victorialogs/vector-ha-single-node/README.md create mode 100644 deployment/docker/victorialogs/vector-ha-single-node/auth.yml create mode 100644 deployment/docker/victorialogs/vector-ha-single-node/compose.yml create mode 100644 deployment/docker/victorialogs/vector-ha-single-node/vector.yaml diff --git a/deployment/docker/victorialogs/fluentbit-ha-single-node/README.md b/deployment/docker/victorialogs/fluentbit-ha-single-node/README.md new file mode 100644 index 000000000..2ef60e207 --- /dev/null +++ b/deployment/docker/victorialogs/fluentbit-ha-single-node/README.md @@ -0,0 +1,88 @@ +# Docker compose Fluentbit integration with VictoriaLogs for docker. High-Availability example + +The folder contains the example of integration of [fluentbit](https://docs.fluentbit.io/manual) with VictoriaLogs Single-Nodes(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability. + +Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components. + +To spin-up environment run the following command: + +```shell +docker compose up -d +``` + +To shut down the docker-compose environment run the following command: + +```shell +docker compose down +docker compose rm -f +``` + +The docker compose file contains the following components: + +* fluentbit - fluentbit is configured to collect logs from the `docker`, you can find configuration in the `fluent-bit.conf`. It writes data in VictoriaLogs +* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol +* vmauth - load balancer for proxying requests to one of VictoriaLogs + +Querying the data + +* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/` +* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) + + +the example of fluentbit configuration(`filebeat.yml`) + +```text +[INPUT] + name tail + path /var/lib/docker/containers/**/*.log + path_key path + multiline.parser docker, cri + Parser docker + Docker_Mode On + +[INPUT] + Name syslog + Listen 0.0.0.0 + Port 5140 + Parser syslog-rfc3164 + Mode tcp + +[SERVICE] + Flush 1 + Parsers_File parsers.conf + +[OUTPUT] + Name http + Match * + host victorialogs-2 + port 9428 + compress gzip + uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + header AccountID 0 + header ProjectID 0 + +[OUTPUT] + Name http + Match * + host victorialogs-1 + port 9428 + compress gzip + uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + header AccountID 0 + header ProjectID 0 +``` + +Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance. + +The example of vmauth configuration (`auth.yml`) + +```yaml +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 +``` \ No newline at end of file diff --git a/deployment/docker/victorialogs/fluentbit-ha-single-node/auth.yml b/deployment/docker/victorialogs/fluentbit-ha-single-node/auth.yml new file mode 100644 index 000000000..eeafbb852 --- /dev/null +++ b/deployment/docker/victorialogs/fluentbit-ha-single-node/auth.yml @@ -0,0 +1,6 @@ +# balance load among victorialogs instances +# see https://docs.victoriametrics.com/vmauth/#load-balancing +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 diff --git a/deployment/docker/victorialogs/fluentbit-ha-single-node/compose.yml b/deployment/docker/victorialogs/fluentbit-ha-single-node/compose.yml new file mode 100644 index 000000000..737be0347 --- /dev/null +++ b/deployment/docker/victorialogs/fluentbit-ha-single-node/compose.yml @@ -0,0 +1,38 @@ +services: + fluentbit: + image: cr.fluentbit.io/fluent/fluent-bit:3.0.7 + volumes: + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf + depends_on: [victorialogs-1,victorialogs-2] + ports: + - "5140:5140" + + victorialogs-1: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-fluentbit-vl-ha-single-1:/vlogs + command: + - -storageDataPath=/vlogs + victorialogs-2: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-fluentbit-vl-ha-single-2:/vlogs + command: + - -storageDataPath=/vlogs + vmauth: + container_name: vmauth + image: victoriametrics/vmauth:v1.103.0 + depends_on: + - "victorialogs-1" + - "victorialogs-2" + volumes: + - ./auth.yml:/etc/auth.yml + command: + - '--auth.config=/etc/auth.yml' + ports: + - 8427:8427 + restart: always +volumes: + victorialogs-fluentbit-vl-ha-single-1: + victorialogs-fluentbit-vl-ha-single-2: diff --git a/deployment/docker/victorialogs/fluentbit-ha-single-node/fluent-bit.conf b/deployment/docker/victorialogs/fluentbit-ha-single-node/fluent-bit.conf new file mode 100644 index 000000000..ce6423898 --- /dev/null +++ b/deployment/docker/victorialogs/fluentbit-ha-single-node/fluent-bit.conf @@ -0,0 +1,42 @@ +[INPUT] + name tail + path /var/lib/docker/containers/**/*.log + path_key path + multiline.parser docker, cri + Parser docker + Docker_Mode On + +[INPUT] + Name syslog + Listen 0.0.0.0 + Port 5140 + Parser syslog-rfc3164 + Mode tcp + +[SERVICE] + Flush 1 + Parsers_File parsers.conf + +[OUTPUT] + Name http + Match * + host victorialogs-2 + port 9428 + compress gzip + uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + header AccountID 0 + header ProjectID 0 + +[OUTPUT] + Name http + Match * + host victorialogs-1 + port 9428 + compress gzip + uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date + format json_lines + json_date_format iso8601 + header AccountID 0 + header ProjectID 0 diff --git a/deployment/docker/victorialogs/logstash-ha-single-node/README.md b/deployment/docker/victorialogs/logstash-ha-single-node/README.md new file mode 100644 index 000000000..7e5805a21 --- /dev/null +++ b/deployment/docker/victorialogs/logstash-ha-single-node/README.md @@ -0,0 +1,71 @@ +# Docker compose Logstash integration with VictoriaLogs for docker. High-Availability example + +The folder contains the example of integration of [logstash](https://www.elastic.co/logstash) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability. + +Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components. + +To spin-up environment run the following command: + +```shell +docker compose up -d +``` + +To shut down the docker-compose environment run the following command: + +```shell +docker compose down +docker compose rm -f +``` + +The docker compose file contains the following components: + +* logstash - logstash is configured to read docker log files, you can find configuration in the `pipeline.conf`. It writes data in two instances of VictoriaLogs +* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol +* vmauth - load balancer for proxying requests to one of VictoriaLogs + +Querying the data + +* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/` +* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) + + +Here is an example of logstash configuration(`pipeline.conf`): + +```text +input { + file { + path => "/var/lib/docker/containers/*/*.log" + start_position => "beginning" + type => "docker" + sincedb_path => "/dev/null" + codec => "json" + add_field => { + "path" => "%{[@metadata][path]}" + } + } +} + +output { + http { + url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time" + format => "json" + http_method => "post" + } + http { + url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time" + format => "json" + http_method => "post" + } +} +``` + +Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance. + +The example of vmauth configuration (`auth.yml`) + +```yaml +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 +``` \ No newline at end of file diff --git a/deployment/docker/victorialogs/logstash-ha-single-node/auth.yml b/deployment/docker/victorialogs/logstash-ha-single-node/auth.yml new file mode 100644 index 000000000..eeafbb852 --- /dev/null +++ b/deployment/docker/victorialogs/logstash-ha-single-node/auth.yml @@ -0,0 +1,6 @@ +# balance load among victorialogs instances +# see https://docs.victoriametrics.com/vmauth/#load-balancing +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 diff --git a/deployment/docker/victorialogs/logstash-ha-single-node/compose.yml b/deployment/docker/victorialogs/logstash-ha-single-node/compose.yml new file mode 100644 index 000000000..6b8153b71 --- /dev/null +++ b/deployment/docker/victorialogs/logstash-ha-single-node/compose.yml @@ -0,0 +1,41 @@ +services: + logstash: + image: docker.elastic.co/logstash/logstash:8.8.1 + user: root + volumes: + - ./pipeline.conf:/usr/share/logstash/pipeline/logstash.conf:ro + - ./logstash.yml:/usr/share/logstash/config/logstash.yml:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + depends_on: [victorialogs-1,victorialogs-2] + ports: + - "5140:5140" + + victorialogs-1: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-logstash-vl-ha-single-1:/vlogs + command: + - -storageDataPath=/vlogs + victorialogs-2: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-logstash-vl-ha-single-2:/vlogs + command: + - -storageDataPath=/vlogs + vmauth: + container_name: vmauth + image: victoriametrics/vmauth:v1.103.0 + depends_on: + - "victorialogs-1" + - "victorialogs-2" + volumes: + - ./auth.yml:/etc/auth.yml + command: + - '--auth.config=/etc/auth.yml' + ports: + - 8427:8427 + restart: always + +volumes: + victorialogs-logstash-vl-ha-single-1: + victorialogs-logstash-vl-ha-single-2: diff --git a/deployment/docker/victorialogs/logstash-ha-single-node/logstash.yml b/deployment/docker/victorialogs/logstash-ha-single-node/logstash.yml new file mode 100644 index 000000000..c223de376 --- /dev/null +++ b/deployment/docker/victorialogs/logstash-ha-single-node/logstash.yml @@ -0,0 +1,2 @@ +http.host: 0.0.0.0 +xpack.monitoring.enabled: false \ No newline at end of file diff --git a/deployment/docker/victorialogs/logstash-ha-single-node/pipeline.conf b/deployment/docker/victorialogs/logstash-ha-single-node/pipeline.conf new file mode 100644 index 000000000..7dacc1d8a --- /dev/null +++ b/deployment/docker/victorialogs/logstash-ha-single-node/pipeline.conf @@ -0,0 +1,25 @@ +input { + file { + path => "/var/lib/docker/containers/*/*.log" + start_position => "beginning" + type => "docker" + sincedb_path => "/dev/null" + codec => "json" + add_field => { + "path" => "%{[@metadata][path]}" + } + } +} + +output { + http { + url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time" + format => "json" + http_method => "post" + } + http { + url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time" + format => "json" + http_method => "post" + } +} \ No newline at end of file diff --git a/deployment/docker/victorialogs/vector-ha-single-node/README.md b/deployment/docker/victorialogs/vector-ha-single-node/README.md new file mode 100644 index 000000000..2a6fd630e --- /dev/null +++ b/deployment/docker/victorialogs/vector-ha-single-node/README.md @@ -0,0 +1,104 @@ +# Docker compose Vector integration with VictoriaLogs for docker. High-Availability example + +The folder contains the example of integration of [vector](https://vector.dev/docs/) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability. + +Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components. + + +To spin-up environment run the following command: + +```shell +docker compose up -d +``` + +To shut down the docker-compose environment run the following command: + +```shell +docker compose down +docker compose rm -f +``` + +The docker compose file contains the following components: + +* vector - vector is configured to collect logs from the `docker`, you can find configuration in the `vector.yaml`. It writes data in two instances of VictoriaLogs +* VictoriaLogs - the two instances of log database, they accept the data from `vector` by json line protocol +* vmauth - load balancer for proxying requests to one of VictoriaLogs + +Querying the data + +* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/` +* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) + + +the example of vector configuration(`vector.yaml`) + +```yaml +api: + enabled: true + address: 0.0.0.0:8686 +sources: + docker: + type: docker_logs +transforms: + msg_parser: + type: remap + inputs: + - docker + source: | + if exists(.message) { + .log, err = parse_json(.message) + if err == null { + del(.message) + } + } +sinks: + console_out: + type: console + inputs: + - msg_parser + encoding: + codec: json + vlogs_http_1: + type: http + inputs: + - msg_parser + uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp + encoding: + codec: json + framing: + method: newline_delimited + compression: gzip + healthcheck: + enabled: false + request: + headers: + AccountID: '0' + ProjectID: '0' + vlogs_http_2: + type: http + inputs: + - msg_parser + uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp + encoding: + codec: json + framing: + method: newline_delimited + compression: gzip + healthcheck: + enabled: false + request: + headers: + AccountID: '0' + ProjectID: '0' +``` + +Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance. + +The example of vmauth configuration (`auth.yml`) + +```yaml +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 +``` \ No newline at end of file diff --git a/deployment/docker/victorialogs/vector-ha-single-node/auth.yml b/deployment/docker/victorialogs/vector-ha-single-node/auth.yml new file mode 100644 index 000000000..eeafbb852 --- /dev/null +++ b/deployment/docker/victorialogs/vector-ha-single-node/auth.yml @@ -0,0 +1,6 @@ +# balance load among victorialogs instances +# see https://docs.victoriametrics.com/vmauth/#load-balancing +unauthorized_user: + url_prefix: + - http://victorialogs-1:9428 + - http://victorialogs-2:9428 diff --git a/deployment/docker/victorialogs/vector-ha-single-node/compose.yml b/deployment/docker/victorialogs/vector-ha-single-node/compose.yml new file mode 100644 index 000000000..ae339f83a --- /dev/null +++ b/deployment/docker/victorialogs/vector-ha-single-node/compose.yml @@ -0,0 +1,48 @@ +services: + vector: + image: docker.io/timberio/vector:0.40.1-distroless-static + restart: on-failure + volumes: + - type: bind + source: /var/run/docker.sock + target: /var/run/docker.sock + - type: bind + source: /var/lib/docker + target: /var/lib/docker + - ./vector.yaml:/etc/vector/vector.yaml:ro + user: root + ports: + - '8686:8686' + depends_on: [victorialogs-1,victorialogs-2] + + victorialogs-1: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-vector-docker-vl-ha-single-1:/vlogs + command: + - -storageDataPath=/vlogs + - -loggerFormat=json + victorialogs-2: + image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs + volumes: + - victorialogs-vector-docker-vl-ha-single-2:/vlogs + command: + - -storageDataPath=/vlogs + - -loggerFormat=json + vmauth: + container_name: vmauth + image: victoriametrics/vmauth:v1.103.0 + depends_on: + - "victorialogs-1" + - "victorialogs-2" + volumes: + - ./auth.yml:/etc/auth.yml + command: + - '--auth.config=/etc/auth.yml' + ports: + - 8427:8427 + restart: always + +volumes: + victorialogs-vector-docker-vl-ha-single-1: + victorialogs-vector-docker-vl-ha-single-2: diff --git a/deployment/docker/victorialogs/vector-ha-single-node/vector.yaml b/deployment/docker/victorialogs/vector-ha-single-node/vector.yaml new file mode 100644 index 000000000..e67320b97 --- /dev/null +++ b/deployment/docker/victorialogs/vector-ha-single-node/vector.yaml @@ -0,0 +1,58 @@ +api: + enabled: true + address: 0.0.0.0:8686 +sources: + docker: + type: docker_logs +transforms: + msg_parser: + type: remap + inputs: + - docker + source: | + if exists(.message) { + .log, err = parse_json(.message) + if err == null { + del(.message) + } + } +sinks: + console_out: + type: console + inputs: + - msg_parser + encoding: + codec: json + vlogs_http_1: + type: http + inputs: + - msg_parser + uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp + encoding: + codec: json + framing: + method: newline_delimited + compression: gzip + healthcheck: + enabled: false + request: + headers: + AccountID: '0' + ProjectID: '0' + vlogs_http_2: + type: http + inputs: + - msg_parser + uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp + encoding: + codec: json + framing: + method: newline_delimited + compression: gzip + healthcheck: + enabled: false + request: + headers: + AccountID: '0' + ProjectID: '0' + diff --git a/docs/VictoriaLogs/README.md b/docs/VictoriaLogs/README.md index 3e8e73f3b..696d9fc50 100644 --- a/docs/VictoriaLogs/README.md +++ b/docs/VictoriaLogs/README.md @@ -135,7 +135,7 @@ VictoriaLogs automatically creates the `-storageDataPath` directory on the first This schema outlines how to configure a High Availability (HA) setup using VictoriaLogs Single-Node instances. The setup consists of the following components: -- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details. +- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Fluent Bit](https://docs.fluentbit.io/manual/concepts/data-pipeline/router), [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details. - **VictoriaLogs Single-Node Instances**: Use two or more instances to achieve HA. @@ -143,6 +143,11 @@ This schema outlines how to configure a High Availability (HA) setup using Victo ![VictoriaLogs Single-Node Instance High-Availability schema](ha-victorialogs-single-node.webp) +Here are the working example of HA configuration for VictoriaLogs using Docker Compose: + +- [Fluent Bit + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-ha-single-node) +- [Logstash + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash-ha-single-node) +- [Vector + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-ha-single-node) ## Backup and restore