2023-07-14 20:06:27 +00:00
---
weight: 2
title: Fluentbit setup
2023-07-14 20:56:38 +00:00
disableToc: true
2023-07-14 20:06:27 +00:00
menu:
docs:
parent: "victorialogs-data-ingestion"
weight: 2
2023-07-14 20:56:38 +00:00
aliases:
- /VictoriaLogs/data-ingestion/Fluentbit.html
2024-06-14 10:32:12 +00:00
- /victorialogs/data-ingestion/fluentbit.html
- /victorialogs/data-ingestion/Fluentbit.html
2023-07-14 20:06:27 +00:00
---
# Fluentbit setup
2023-06-21 14:58:43 +00:00
2024-09-03 15:43:26 +00:00
VictoriaLogs supports given below Fluentbit outputs:
- [Loki ](#loki )
- [HTTP JSON ](#http )
## Loki
Specify [loki output ](https://docs.fluentbit.io/manual/pipeline/outputs/loki ) section in the `fluentbit.conf`
for sending the collected logs to [VictoriaLogs ](https://docs.victoriametrics.com/victorialogs/ ):
```conf
[OUTPUT]
name loki
match *
host victorialogs
uri /insert/loki/api/v1/push
port 9428
label_keys $path,$log,$time
header VL-Msg-Field log
header VL-Time-Field time
header VL-Stream-Fields path
```
## HTTP
2023-06-22 01:31:50 +00:00
Specify [http output ](https://docs.fluentbit.io/manual/pipeline/outputs/http ) section in the `fluentbit.conf`
2024-05-24 22:30:58 +00:00
for sending the collected logs to [VictoriaLogs ](https://docs.victoriametrics.com/victorialogs/ ):
2023-06-21 14:58:43 +00:00
2024-07-24 08:00:31 +00:00
```fluentbit
2023-06-21 14:58:43 +00:00
[Output]
Name http
Match *
host localhost
port 9428
2023-06-22 06:48:11 +00:00
uri /insert/jsonline?_stream_fields=stream& _msg_field=log& _time_field=date
2023-06-21 14:58:43 +00:00
format json_lines
json_date_format iso8601
```
2023-06-22 01:31:50 +00:00
Substitute the host (`localhost`) and port (`9428`) with the real TCP address of VictoriaLogs.
2024-05-24 22:30:58 +00:00
See [these docs ](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters ) for details on the query args specified in the `uri` .
2023-06-21 14:58:43 +00:00
2024-05-24 22:30:58 +00:00
It is recommended verifying whether the initial setup generates the needed [log fields ](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model )
and uses the correct [stream fields ](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields ).
This can be done by specifying `debug` [parameter ](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters ) in the `uri`
2023-06-22 01:31:50 +00:00
and inspecting VictoriaLogs logs then:
2023-06-21 14:58:43 +00:00
2024-07-24 08:00:31 +00:00
```fluentbit
2023-06-22 01:31:50 +00:00
[Output]
Name http
Match *
host localhost
port 9428
2023-06-22 06:48:11 +00:00
uri /insert/jsonline?_stream_fields=stream& _msg_field=log& _time_field=date& debug=1
2023-06-22 01:31:50 +00:00
format json_lines
json_date_format iso8601
```
2023-06-21 14:58:43 +00:00
2024-05-24 22:30:58 +00:00
If some [log fields ](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ) must be skipped
during data ingestion, then they can be put into `ignore_fields` [parameter ](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters ).
2023-06-22 01:31:50 +00:00
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
2023-06-21 14:58:43 +00:00
2024-07-24 08:00:31 +00:00
```fluentbit
2023-06-22 01:31:50 +00:00
[Output]
Name http
Match *
host localhost
port 9428
2023-06-22 06:48:11 +00:00
uri /insert/jsonline?_stream_fields=stream& _msg_field=log& _time_field=date& ignore_fields=log.offset,event.original
2023-06-22 01:31:50 +00:00
format json_lines
json_date_format iso8601
```
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option.
2023-06-21 14:58:43 +00:00
This usually allows saving network bandwidth and costs by up to 5 times:
2024-07-24 08:00:31 +00:00
```fluentbit
2023-06-21 14:58:43 +00:00
[Output]
Name http
Match *
host localhost
port 9428
2023-06-22 06:48:11 +00:00
uri /insert/jsonline?_stream_fields=stream& _msg_field=log& _time_field=date
2023-06-21 14:58:43 +00:00
format json_lines
json_date_format iso8601
compress gzip
```
2024-05-24 22:30:58 +00:00
By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant ](https://docs.victoriametrics.com/victorialogs/keyconcepts/#multitenancy ).
2023-06-22 01:31:50 +00:00
If you need storing logs in other tenant, then specify the needed tenant via `header` options.
For example, the following `fluentbit.conf` config instructs Fluentbit to store the data to `(AccountID=12, ProjectID=34)` tenant:
2023-06-21 14:58:43 +00:00
2024-07-24 08:00:31 +00:00
```fluentbit
2023-06-21 14:58:43 +00:00
[Output]
Name http
Match *
host localhost
port 9428
2023-06-22 06:48:11 +00:00
uri /insert/jsonline?_stream_fields=stream& _msg_field=log& _time_field=date
2023-06-21 14:58:43 +00:00
format json_lines
json_date_format iso8601
header AccountID 12
header ProjectID 23
```
2023-06-22 01:31:50 +00:00
See also:
2023-06-21 14:58:43 +00:00
2024-05-24 22:30:58 +00:00
- [Data ingestion troubleshooting ](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting ).
- [How to query VictoriaLogs ](https://docs.victoriametrics.com/victorialogs/querying/ ).
2023-06-22 01:31:50 +00:00
- [Fluentbit HTTP output config docs ](https://docs.fluentbit.io/manual/pipeline/outputs/http ).
2024-09-09 17:56:58 +00:00
- [Docker-compose demo for Fluentbit integration with VictoriaLogs ](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit ).