victorialogs: add HA example for logstash and fluentbit (#6968)

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 277fed9990)

deployment/docker/victorialogs/fluentbit-ha-single-node/README.md

@@ -0,0 +1,88 @@
+# Docker compose Fluentbit integration with VictoriaLogs for docker. High-Availability example 
+
+The folder contains the example of integration of [fluentbit](https://docs.fluentbit.io/manual) with VictoriaLogs Single-Nodes(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability. 
+
+Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
+
+To spin-up environment  run the following command:
+
+```shell
+docker compose up -d 
+```
+
+To shut down the docker-compose environment run the following command:
+
+```shell
+docker compose down
+docker compose rm -f
+```
+
+The docker compose file contains the following components:
+
+* fluentbit - fluentbit is configured to collect logs from the `docker`, you can find configuration in the `fluent-bit.conf`. It writes data in VictoriaLogs
+* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol
+* vmauth - load balancer for proxying requests to one of VictoriaLogs
+
+Querying the data
+
+* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
+* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
+
+
+the example of fluentbit configuration(`filebeat.yml`)
+
+```text
+[INPUT]
+    name             tail
+    path             /var/lib/docker/containers/**/*.log
+    path_key         path
+    multiline.parser docker, cri
+    Parser           docker
+    Docker_Mode      On
+
+[INPUT]
+    Name     syslog
+    Listen   0.0.0.0
+    Port     5140
+    Parser   syslog-rfc3164
+    Mode     tcp
+
+[SERVICE]
+    Flush        1
+    Parsers_File parsers.conf
+
+[OUTPUT]
+    Name http
+    Match *
+    host victorialogs-2
+    port 9428
+    compress gzip
+    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
+    format json_lines
+    json_date_format iso8601
+    header AccountID 0
+    header ProjectID 0
+
+[OUTPUT]
+    Name http
+    Match *
+    host victorialogs-1
+    port 9428
+    compress gzip
+    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
+    format json_lines
+    json_date_format iso8601
+    header AccountID 0
+    header ProjectID 0
+```
+
+Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
+
+The example of vmauth configuration (`auth.yml`)
+
+```yaml
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428
+```

deployment/docker/victorialogs/fluentbit-ha-single-node/auth.yml

@@ -0,0 +1,6 @@
+# balance load among victorialogs instances
+# see https://docs.victoriametrics.com/vmauth/#load-balancing
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428

deployment/docker/victorialogs/fluentbit-ha-single-node/compose.yml

@@ -0,0 +1,38 @@
+services:
+  fluentbit:
+    image: cr.fluentbit.io/fluent/fluent-bit:3.0.7
+    volumes:
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
+    depends_on: [victorialogs-1,victorialogs-2]
+    ports:
+      - "5140:5140"
+
+  victorialogs-1:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-fluentbit-vl-ha-single-1:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+  victorialogs-2:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-fluentbit-vl-ha-single-2:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+  vmauth:
+    container_name: vmauth
+    image: victoriametrics/vmauth:v1.103.0
+    depends_on:
+      - "victorialogs-1"
+      - "victorialogs-2"
+    volumes:
+      - ./auth.yml:/etc/auth.yml
+    command:
+      - '--auth.config=/etc/auth.yml'
+    ports:
+      - 8427:8427
+    restart: always
+volumes:
+  victorialogs-fluentbit-vl-ha-single-1:
+  victorialogs-fluentbit-vl-ha-single-2:

deployment/docker/victorialogs/fluentbit-ha-single-node/fluent-bit.conf

@@ -0,0 +1,42 @@
+[INPUT]
+    name             tail
+    path             /var/lib/docker/containers/**/*.log
+    path_key         path
+    multiline.parser docker, cri
+    Parser           docker
+    Docker_Mode      On
+
+[INPUT]
+    Name     syslog
+    Listen   0.0.0.0
+    Port     5140
+    Parser   syslog-rfc3164
+    Mode     tcp
+
+[SERVICE]
+    Flush        1
+    Parsers_File parsers.conf
+
+[OUTPUT]
+    Name http
+    Match *
+    host victorialogs-2
+    port 9428
+    compress gzip
+    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
+    format json_lines
+    json_date_format iso8601
+    header AccountID 0
+    header ProjectID 0
+
+[OUTPUT]
+    Name http
+    Match *
+    host victorialogs-1
+    port 9428
+    compress gzip
+    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
+    format json_lines
+    json_date_format iso8601
+    header AccountID 0
+    header ProjectID 0

deployment/docker/victorialogs/logstash-ha-single-node/README.md

@@ -0,0 +1,71 @@
+# Docker compose Logstash integration with VictoriaLogs for docker. High-Availability example
+
+The folder contains the example of integration of [logstash](https://www.elastic.co/logstash) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability.
+
+Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
+
+To spin-up environment  run the following command:
+
+```shell
+docker compose up -d 
+```
+
+To shut down the docker-compose environment run the following command:
+
+```shell
+docker compose down
+docker compose rm -f
+```
+
+The docker compose file contains the following components:
+
+* logstash - logstash is configured to read docker log files, you can find configuration in the `pipeline.conf`. It writes data in two instances of VictoriaLogs
+* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol
+* vmauth - load balancer for proxying requests to one of VictoriaLogs
+
+Querying the data
+
+* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
+* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
+
+
+Here is an example of logstash configuration(`pipeline.conf`):
+
+```text
+input {
+  file {
+    path => "/var/lib/docker/containers/*/*.log"
+    start_position => "beginning"
+    type => "docker"
+    sincedb_path => "/dev/null"
+    codec => "json"
+    add_field => {
+      "path" => "%{[@metadata][path]}"
+    }
+  }
+}
+
+output {
+  http {
+    url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
+    format => "json"
+    http_method => "post"
+  }
+  http {
+    url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
+    format => "json"
+    http_method => "post"
+  }
+}
+```
+
+Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
+
+The example of vmauth configuration (`auth.yml`)
+
+```yaml
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428
+```

deployment/docker/victorialogs/logstash-ha-single-node/auth.yml

@@ -0,0 +1,6 @@
+# balance load among victorialogs instances
+# see https://docs.victoriametrics.com/vmauth/#load-balancing
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428

deployment/docker/victorialogs/logstash-ha-single-node/compose.yml

@@ -0,0 +1,41 @@
+services:
+  logstash:
+    image: docker.elastic.co/logstash/logstash:8.8.1
+    user: root
+    volumes:
+      - ./pipeline.conf:/usr/share/logstash/pipeline/logstash.conf:ro
+      - ./logstash.yml:/usr/share/logstash/config/logstash.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+    depends_on: [victorialogs-1,victorialogs-2]
+    ports:
+      - "5140:5140"
+
+  victorialogs-1:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-logstash-vl-ha-single-1:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+  victorialogs-2:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-logstash-vl-ha-single-2:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+  vmauth:
+    container_name: vmauth
+    image: victoriametrics/vmauth:v1.103.0
+    depends_on:
+      - "victorialogs-1"
+      - "victorialogs-2"
+    volumes:
+      - ./auth.yml:/etc/auth.yml
+    command:
+      - '--auth.config=/etc/auth.yml'
+    ports:
+      - 8427:8427
+    restart: always
+
+volumes:
+  victorialogs-logstash-vl-ha-single-1:
+  victorialogs-logstash-vl-ha-single-2:

deployment/docker/victorialogs/logstash-ha-single-node/logstash.yml

@@ -0,0 +1,2 @@
+http.host: 0.0.0.0
+xpack.monitoring.enabled: false

deployment/docker/victorialogs/logstash-ha-single-node/pipeline.conf

@@ -0,0 +1,25 @@
+input {
+  file {
+    path => "/var/lib/docker/containers/*/*.log"
+    start_position => "beginning"
+    type => "docker"
+    sincedb_path => "/dev/null"
+    codec => "json"
+    add_field => {
+      "path" => "%{[@metadata][path]}"
+    }
+  }
+}
+
+output {
+  http {
+    url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
+    format => "json"
+    http_method => "post"
+  }
+  http {
+    url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
+    format => "json"
+    http_method => "post"
+  }
+}

deployment/docker/victorialogs/vector-ha-single-node/README.md

@@ -0,0 +1,104 @@
+# Docker compose Vector integration with VictoriaLogs for docker. High-Availability example
+
+The folder contains the example of integration of [vector](https://vector.dev/docs/) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability.
+
+Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
+
+
+To spin-up environment  run the following command:
+
+```shell
+docker compose up -d 
+```
+
+To shut down the docker-compose environment run the following command:
+
+```shell
+docker compose down
+docker compose rm -f
+```
+
+The docker compose file contains the following components:
+
+* vector - vector is configured to collect logs from the `docker`, you can find configuration in the `vector.yaml`. It writes data in two instances of VictoriaLogs
+* VictoriaLogs - the two instances of log database, they accept the data from `vector` by json line protocol
+* vmauth - load balancer for proxying requests to one of VictoriaLogs
+
+Querying the data
+
+* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
+* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
+
+
+the example of vector configuration(`vector.yaml`)
+
+```yaml
+api:
+  enabled: true
+  address: 0.0.0.0:8686
+sources:
+  docker:
+    type: docker_logs
+transforms:
+  msg_parser:
+    type: remap
+    inputs:
+      - docker
+    source: |
+      if exists(.message) {
+        .log, err = parse_json(.message)
+        if err == null {
+          del(.message)
+        }
+      }
+sinks:
+  console_out:
+    type: console
+    inputs:
+      - msg_parser
+    encoding:
+      codec: json
+  vlogs_http_1:
+    type: http
+    inputs:
+      - msg_parser
+    uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
+    encoding:
+      codec: json
+    framing:
+      method: newline_delimited
+    compression: gzip
+    healthcheck:
+      enabled: false
+    request:
+      headers:
+        AccountID: '0'
+        ProjectID: '0'
+  vlogs_http_2:
+    type: http
+    inputs:
+      - msg_parser
+    uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
+    encoding:
+      codec: json
+    framing:
+      method: newline_delimited
+    compression: gzip
+    healthcheck:
+      enabled: false
+    request:
+      headers:
+        AccountID: '0'
+        ProjectID: '0'
+```
+
+Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
+
+The example of vmauth configuration (`auth.yml`)
+
+```yaml
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428
+```

deployment/docker/victorialogs/vector-ha-single-node/auth.yml

@@ -0,0 +1,6 @@
+# balance load among victorialogs instances
+# see https://docs.victoriametrics.com/vmauth/#load-balancing
+unauthorized_user:
+  url_prefix:
+    - http://victorialogs-1:9428
+    - http://victorialogs-2:9428

deployment/docker/victorialogs/vector-ha-single-node/compose.yml

@@ -0,0 +1,48 @@
+services:
+  vector:
+    image: docker.io/timberio/vector:0.40.1-distroless-static
+    restart: on-failure
+    volumes:
+      - type: bind
+        source: /var/run/docker.sock
+        target: /var/run/docker.sock
+      - type: bind
+        source: /var/lib/docker
+        target: /var/lib/docker
+      - ./vector.yaml:/etc/vector/vector.yaml:ro
+    user: root
+    ports:
+      - '8686:8686'
+    depends_on: [victorialogs-1,victorialogs-2]
+
+  victorialogs-1:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-vector-docker-vl-ha-single-1:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+      - -loggerFormat=json
+  victorialogs-2:
+    image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
+    volumes:
+      - victorialogs-vector-docker-vl-ha-single-2:/vlogs
+    command:
+      - -storageDataPath=/vlogs
+      - -loggerFormat=json
+  vmauth:
+    container_name: vmauth
+    image: victoriametrics/vmauth:v1.103.0
+    depends_on:
+      - "victorialogs-1"
+      - "victorialogs-2"
+    volumes:
+      - ./auth.yml:/etc/auth.yml
+    command:
+      - '--auth.config=/etc/auth.yml'
+    ports:
+      - 8427:8427
+    restart: always
+
+volumes:
+  victorialogs-vector-docker-vl-ha-single-1:
+  victorialogs-vector-docker-vl-ha-single-2:

deployment/docker/victorialogs/vector-ha-single-node/vector.yaml

@@ -0,0 +1,58 @@
+api:
+  enabled: true
+  address: 0.0.0.0:8686
+sources:
+  docker:
+    type: docker_logs
+transforms:
+  msg_parser:
+    type: remap
+    inputs:
+      - docker
+    source: |
+      if exists(.message) {
+        .log, err = parse_json(.message)
+        if err == null {
+          del(.message)
+        }
+      }
+sinks:
+  console_out:
+    type: console
+    inputs:
+      - msg_parser
+    encoding:
+      codec: json
+  vlogs_http_1:
+    type: http
+    inputs:
+      - msg_parser
+    uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
+    encoding:
+      codec: json
+    framing:
+      method: newline_delimited
+    compression: gzip
+    healthcheck:
+      enabled: false
+    request:
+      headers:
+        AccountID: '0'
+        ProjectID: '0'
+  vlogs_http_2:
+    type: http
+    inputs:
+      - msg_parser
+    uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
+    encoding:
+      codec: json
+    framing:
+      method: newline_delimited
+    compression: gzip
+    healthcheck:
+      enabled: false
+    request:
+      headers:
+        AccountID: '0'
+        ProjectID: '0'
+

docs/VictoriaLogs/README.md

@@ -135,7 +135,7 @@ VictoriaLogs automatically creates the `-storageDataPath` directory on the first
 
 This schema outlines how to configure a High Availability (HA) setup using VictoriaLogs Single-Node instances. The setup consists of the following components:
 
-- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details.
+- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Fluent Bit](https://docs.fluentbit.io/manual/concepts/data-pipeline/router), [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details.
 
 - **VictoriaLogs Single-Node Instances**: Use two or more instances to achieve HA.
 
@@ -143,6 +143,11 @@ This schema outlines how to configure a High Availability (HA) setup using Victo
 
 ![VictoriaLogs Single-Node Instance High-Availability schema](ha-victorialogs-single-node.webp)
 
+Here are the working example of HA configuration for VictoriaLogs using Docker Compose:
+
+- [Fluent Bit + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-ha-single-node)
+- [Logstash + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash-ha-single-node)
+- [Vector + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-ha-single-node)
 
 ## Backup and restore