removed FluentBit Elasticsearch example from docs as custom headers are
not supported by elasticsearch output till
https://github.com/fluent/fluent-bit/pull/9416 is merged and released
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6985
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
(cherry picked from commit f934f71708
)
4.4 KiB
weight | title | disableToc | menu | aliases | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2 | Fluentbit setup | true |
|
|
Fluentbit setup
VictoriaLogs supports given below Fluentbit outputs:
Loki
Specify loki output section in the fluentbit.conf
for sending the collected logs to VictoriaLogs:
[OUTPUT]
name loki
match *
host victorialogs
uri /insert/loki/api/v1/push
port 9428
label_keys $path,$log,$time
header VL-Msg-Field log
header VL-Time-Field time
header VL-Stream-Fields path
HTTP
Specify http output section in the fluentbit.conf
for sending the collected logs to VictoriaLogs:
[Output]
Name http
Match *
host localhost
port 9428
uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
format json_lines
json_date_format iso8601
Substitute the host (localhost
) and port (9428
) with the real TCP address of VictoriaLogs.
See these docs for details on the query args specified in the uri
.
It is recommended verifying whether the initial setup generates the needed log fields
and uses the correct stream fields.
This can be done by specifying debug
parameter in the uri
and inspecting VictoriaLogs logs then:
[Output]
Name http
Match *
host localhost
port 9428
uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1
format json_lines
json_date_format iso8601
If some log fields must be skipped
during data ingestion, then they can be put into ignore_fields
parameter.
For example, the following config instructs VictoriaLogs to ignore log.offset
and event.original
fields in the ingested logs:
[Output]
Name http
Match *
host localhost
port 9428
uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original
format json_lines
json_date_format iso8601
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via compress gzip
option.
This usually allows saving network bandwidth and costs by up to 5 times:
[Output]
Name http
Match *
host localhost
port 9428
uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
format json_lines
json_date_format iso8601
compress gzip
By default, the ingested logs are stored in the (AccountID=0, ProjectID=0)
tenant.
If you need storing logs in other tenant, then specify the needed tenant via header
options.
For example, the following fluentbit.conf
config instructs Fluentbit to store the data to (AccountID=12, ProjectID=34)
tenant:
[Output]
Name http
Match *
host localhost
port 9428
uri /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
format json_lines
json_date_format iso8601
header AccountID 12
header ProjectID 23
See also: