VictoriaMetrics/docs/VictoriaLogs/data-ingestion/Fluentbit.md
Andrii Chubatiuk 1824dd378f
docs/victorialogs/data-ingestion: removed FluentBit Elasticsearch from examples (#7093)
removed FluentBit Elasticsearch example from docs as custom headers are
not supported by elasticsearch output till
https://github.com/fluent/fluent-bit/pull/9416 is merged and released

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6985

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit f934f71708)
2024-09-25 18:47:24 +02:00

4.4 KiB

weight title disableToc menu aliases
2 Fluentbit setup true
docs
parent weight
victorialogs-data-ingestion 2
/VictoriaLogs/data-ingestion/Fluentbit.html
/victorialogs/data-ingestion/fluentbit.html
/victorialogs/data-ingestion/Fluentbit.html

Fluentbit setup

VictoriaLogs supports given below Fluentbit outputs:

Loki

Specify loki output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[OUTPUT]
    name       loki
    match      *
    host       victorialogs
    uri        /insert/loki/api/v1/push
    port       9428
    label_keys $path,$log,$time
    header     VL-Msg-Field log
    header     VL-Time-Field time
    header     VL-Stream-Fields path

HTTP

Specify http output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601

Substitute the host (localhost) and port (9428) with the real TCP address of VictoriaLogs.

See these docs for details on the query args specified in the uri.

It is recommended verifying whether the initial setup generates the needed log fields and uses the correct stream fields. This can be done by specifying debug parameter in the uri and inspecting VictoriaLogs logs then:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1
     format json_lines
     json_date_format iso8601

If some log fields must be skipped during data ingestion, then they can be put into ignore_fields parameter. For example, the following config instructs VictoriaLogs to ignore log.offset and event.original fields in the ingested logs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original
     format json_lines
     json_date_format iso8601

If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via compress gzip option. This usually allows saving network bandwidth and costs by up to 5 times:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     compress gzip

By default, the ingested logs are stored in the (AccountID=0, ProjectID=0) tenant. If you need storing logs in other tenant, then specify the needed tenant via header options. For example, the following fluentbit.conf config instructs Fluentbit to store the data to (AccountID=12, ProjectID=34) tenant:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     header AccountID 12
     header ProjectID 23

See also: