VictoriaMetrics/docs/VictoriaLogs/data-ingestion/Fluentbit.md
Andrii Chubatiuk d5fe4566e5
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 20:24:00 +02:00

5 KiB

weight title disableToc menu aliases
2 Fluentbit setup true
docs
parent weight
victorialogs-data-ingestion 2
/VictoriaLogs/data-ingestion/Fluentbit.html
/victorialogs/data-ingestion/fluentbit.html
/victorialogs/data-ingestion/Fluentbit.html

Fluentbit setup

VictoriaLogs supports given below Fluentbit outputs:

Elasticsearch

Specify elasticsearch output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[Output]
    Name es
    Match *
    host victorialogs
    port 9428
    compress gzip
    path /insert/elasticsearch
    header AccountID 0
    header ProjectID 0
    header VL-Stream-Fields path
    header VL-Msg-Field log
    header VL-Time-Field @timestamp

Loki

Specify loki output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[OUTPUT]
    name       loki
    match      *
    host       victorialogs
    uri        /insert/loki/api/v1/push
    port       9428
    label_keys $path,$log,$time
    header     VL-Msg-Field log
    header     VL-Time-Field time
    header     VL-Stream-Fields path

HTTP

Specify http output section in the fluentbit.conf for sending the collected logs to VictoriaLogs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601

Substitute the host (localhost) and port (9428) with the real TCP address of VictoriaLogs.

See these docs for details on the query args specified in the uri.

It is recommended verifying whether the initial setup generates the needed log fields and uses the correct stream fields. This can be done by specifying debug parameter in the uri and inspecting VictoriaLogs logs then:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1
     format json_lines
     json_date_format iso8601

If some log fields must be skipped during data ingestion, then they can be put into ignore_fields parameter. For example, the following config instructs VictoriaLogs to ignore log.offset and event.original fields in the ingested logs:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original
     format json_lines
     json_date_format iso8601

If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via compress gzip option. This usually allows saving network bandwidth and costs by up to 5 times:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     compress gzip

By default, the ingested logs are stored in the (AccountID=0, ProjectID=0) tenant. If you need storing logs in other tenant, then specify the needed tenant via header options. For example, the following fluentbit.conf config instructs Fluentbit to store the data to (AccountID=12, ProjectID=34) tenant:

[Output]
     Name http
     Match *
     host localhost
     port 9428
     uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
     format json_lines
     json_date_format iso8601
     header AccountID 12
     header ProjectID 23

See also: