mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-11 14:53:49 +00:00

victorialogs: add HA example for logstash and fluentbit (#6968 )

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

2024-09-09 14:33:05 -07:00

2.7 KiB

Raw Blame History

Docker compose Fluentbit integration with VictoriaLogs for docker. High-Availability example

The folder contains the example of integration of fluentbit with VictoriaLogs Single-Nodes(s) and vmauth for achieving High Availability.

Check this documentation with a description of the architecture and components.

To spin-up environment run the following command:

docker compose up -d

To shut down the docker-compose environment run the following command:

docker compose down
docker compose rm -f

The docker compose file contains the following components:

fluentbit - fluentbit is configured to collect logs from the docker, you can find configuration in the fluent-bit.conf. It writes data in VictoriaLogs
VictoriaLogs - the two instances of log database, they accept the data from fluentbit by json line protocol
vmauth - load balancer for proxying requests to one of VictoriaLogs

Querying the data

vmui - a web UI is accessible by http://localhost:8427/select/vmui/
for querying the data via command-line please check these docs

the example of fluentbit configuration(filebeat.yml)

[INPUT]
    name             tail
    path             /var/lib/docker/containers/**/*.log
    path_key         path
    multiline.parser docker, cri
    Parser           docker
    Docker_Mode      On

[INPUT]
    Name     syslog
    Listen   0.0.0.0
    Port     5140
    Parser   syslog-rfc3164
    Mode     tcp

[SERVICE]
    Flush        1
    Parsers_File parsers.conf

[OUTPUT]
    Name http
    Match *
    host victorialogs-2
    port 9428
    compress gzip
    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
    format json_lines
    json_date_format iso8601
    header AccountID 0
    header ProjectID 0

[OUTPUT]
    Name http
    Match *
    host victorialogs-1
    port 9428
    compress gzip
    uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
    format json_lines
    json_date_format iso8601
    header AccountID 0
    header ProjectID 0

Please, note that _stream_fields parameter must follow recommended best practices to achieve better performance.

The example of vmauth configuration (auth.yml)

unauthorized_user:
  url_prefix:
    - http://victorialogs-1:9428
    - http://victorialogs-2:9428

2.7 KiB Raw Blame History

Docker compose Fluentbit integration with VictoriaLogs for docker. High-Availability example

2.7 KiB

Raw Blame History