mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
71 lines
3.3 KiB
Markdown
71 lines
3.3 KiB
Markdown
|
## Fluentbit setup
|
||
|
|
|
||
|
|
[Fluentbit](https://docs.fluentbit.io/manual) log collector supports [HTTP output](https://docs.fluentbit.io/manual/pipeline/outputs/http) compatible with
|
||
|
|
VictoriaMetrics [JSON stream API](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#json-stream-api).
|
||
|
|
|
||
|
|
Specify [`output`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) section with `Name http` in the `fluentbit.conf`
|
||
|
|
for sending the collected logs to VictoriaLogs:
|
||
|
|
|
||
|
|
```conf
|
||
|
|
[Output]
|
||
|
|
Name http
|
||
|
|
Match *
|
||
|
|
host localhost
|
||
|
|
port 9428
|
||
|
|
uri /insert/jsonline/?_stream_fields=stream&_msg_field=log&_time_field=date
|
||
|
|
format json_lines
|
||
|
|
json_date_format iso8601
|
||
|
|
```
|
||
|
|
|
||
|
|
Substitute the address (`localhost`) and port (`9428`) inside `Output` section with the real TCP address of VictoriaLogs.
|
||
|
|
|
||
|
|
The `_msg_field` parameter must contain the field name with the log message generated by Fluentbit. This is usually `message` field.
|
||
|
|
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) for details.
|
||
|
|
|
||
|
|
The `_time_field` parameter must contain the field name with the log timestamp generated by Fluentbit. This is usually `@timestamp` field.
|
||
|
|
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) for details.
|
||
|
|
|
||
|
|
It is recommended specifying comma-separated list of field names, which uniquely identify every log stream collected by Fluentbit, in the `_stream_fields` parameter.
|
||
|
|
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details.
|
||
|
|
|
||
|
|
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress` option.
|
||
|
|
This usually allows saving network bandwidth and costs by up to 5 times:
|
||
|
|
|
||
|
|
```conf
|
||
|
|
[Output]
|
||
|
|
Name http
|
||
|
|
Match *
|
||
|
|
host localhost
|
||
|
|
port 9428
|
||
|
|
uri /insert/jsonline/?_stream_fields=stream&_msg_field=log&_time_field=date
|
||
|
|
format json_lines
|
||
|
|
json_date_format iso8601
|
||
|
|
compress gzip
|
||
|
|
```
|
||
|
|
|
||
|
|
By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#multitenancy).
|
||
|
|
If you need storing logs in other tenant, then specify the needed tenant via `headers` at `output.elasticsearch` section.
|
||
|
|
For example, the following `fluentbit.conf` config instructs Filebeat to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
||
|
|
|
||
|
|
```conf
|
||
|
|
[Output]
|
||
|
|
Name http
|
||
|
|
Match *
|
||
|
|
host localhost
|
||
|
|
port 9428
|
||
|
|
uri /insert/jsonline/?_stream_fields=stream&_msg_field=log&_time_field=date
|
||
|
|
format json_lines
|
||
|
|
json_date_format iso8601
|
||
|
|
header AccountID 12
|
||
|
|
header ProjectID 23
|
||
|
|
```
|
||
|
|
|
||
|
|
More info about output tuning you can find in [these docs](https://docs.fluentbit.io/manual/pipeline/outputs/http).
|
||
|
|
|
||
|
|
[Here is a demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker)
|
||
|
|
for running Fluentbit with VictoriaLogs with docker-compose and collecting logs from docker-containers to VictoriaLogs.
|
||
|
|
|
||
|
|
The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||
|
|
|
||
|
|
See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs.
|