mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
docs/VictoriaLogs/data-ingestion: small fixes
This commit is contained in:
Aliaksandr Valialkin
parent
ee720d1bbc
commit
7206f91201
6 changed files with 96 additions and 85 deletions
|
|
@ -1,9 +1,5 @@
|
|||
# Filebeat setup
|
||||
|
||||
[Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) log collector supports
|
||||
[Elasticsearch output](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) compatible with
|
||||
VictoriaMetrics [ingestion format](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#elasticsearch-bulk-api).
|
||||
|
||||
Specify [`output.elasicsearch`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) section in the `filebeat.yml`
|
||||
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/):
|
||||
|
||||
|
|
@ -92,12 +88,9 @@ output.elasticsearch:
|
|||
_stream_fields: "host.name,log.file.path"
|
||||
```
|
||||
|
||||
More info about output parameters you can find in [these docs](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html).
|
||||
|
||||
[Here is a demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/filebeat-docker) for
|
||||
running Filebeat with VictoriaLogs with docker-compose and collecting logs to VictoriaLogs.
|
||||
|
||||
The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
|
||||
See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs.
|
||||
See also:
|
||||
|
||||
- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting).
|
||||
- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
- [Filebeat `output.elasticsearch` docs](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html).
|
||||
- [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/filebeat-docker).
|
||||
|
|
|
|||
|
|
@ -1,10 +1,7 @@
|
|||
## Fluentbit setup
|
||||
|
||||
[Fluentbit](https://docs.fluentbit.io/manual) log collector supports [HTTP output](https://docs.fluentbit.io/manual/pipeline/outputs/http) compatible with
|
||||
VictoriaMetrics [JSON stream API](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#json-stream-api).
|
||||
|
||||
Specify [`output`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) section with `Name http` in the `fluentbit.conf`
|
||||
for sending the collected logs to VictoriaLogs:
|
||||
Specify [http output](https://docs.fluentbit.io/manual/pipeline/outputs/http) section in the `fluentbit.conf`
|
||||
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/):
|
||||
|
||||
```conf
|
||||
[Output]
|
||||
|
|
@ -17,18 +14,42 @@ for sending the collected logs to VictoriaLogs:
|
|||
json_date_format iso8601
|
||||
```
|
||||
|
||||
Substitute the address (`localhost`) and port (`9428`) inside `Output` section with the real TCP address of VictoriaLogs.
|
||||
Substitute the host (`localhost`) and port (`9428`) with the real TCP address of VictoriaLogs.
|
||||
|
||||
The `_msg_field` parameter must contain the field name with the log message generated by Fluentbit. This is usually `message` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the query args specified in the `uri`.
|
||||
|
||||
The `_time_field` parameter must contain the field name with the log timestamp generated by Fluentbit. This is usually `@timestamp` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) for details.
|
||||
It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) in the `uri`
|
||||
and inspecting VictoriaLogs logs then:
|
||||
|
||||
It is recommended specifying comma-separated list of field names, which uniquely identify every log stream collected by Fluentbit, in the `_stream_fields` parameter.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details.
|
||||
```conf
|
||||
[Output]
|
||||
Name http
|
||||
Match *
|
||||
host localhost
|
||||
port 9428
|
||||
uri /insert/jsonline/?_stream_fields=stream&_msg_field=log&_time_field=date&debug=1
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
```
|
||||
|
||||
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress` option.
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped
|
||||
during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters).
|
||||
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
|
||||
|
||||
```conf
|
||||
[Output]
|
||||
Name http
|
||||
Match *
|
||||
host localhost
|
||||
port 9428
|
||||
uri /insert/jsonline/?_stream_fields=stream&_msg_field=log&_time_field=date&ignore_fields=log.offset,event.original
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
```
|
||||
|
||||
If the Fluentbit sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option.
|
||||
This usually allows saving network bandwidth and costs by up to 5 times:
|
||||
|
||||
```conf
|
||||
|
|
@ -44,8 +65,8 @@ This usually allows saving network bandwidth and costs by up to 5 times:
|
|||
```
|
||||
|
||||
By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#multitenancy).
|
||||
If you need storing logs in other tenant, then specify the needed tenant via `headers` at `output.elasticsearch` section.
|
||||
For example, the following `fluentbit.conf` config instructs Filebeat to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
||||
If you need storing logs in other tenant, then specify the needed tenant via `header` options.
|
||||
For example, the following `fluentbit.conf` config instructs Fluentbit to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
||||
|
||||
```conf
|
||||
[Output]
|
||||
|
|
@ -60,11 +81,9 @@ For example, the following `fluentbit.conf` config instructs Filebeat to store t
|
|||
header ProjectID 23
|
||||
```
|
||||
|
||||
More info about output tuning you can find in [these docs](https://docs.fluentbit.io/manual/pipeline/outputs/http).
|
||||
See also:
|
||||
|
||||
[Here is a demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker)
|
||||
for running Fluentbit with VictoriaLogs with docker-compose and collecting logs from docker-containers to VictoriaLogs.
|
||||
|
||||
The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
|
||||
See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs.
|
||||
- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting).
|
||||
- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
- [Fluentbit HTTP output config docs](https://docs.fluentbit.io/manual/pipeline/outputs/http).
|
||||
- [Docker-compose demo for Fluentbit integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker).
|
||||
|
|
|
|||
|
|
@ -1,10 +1,5 @@
|
|||
# Logstash setup
|
||||
|
||||
[Logstash](https://www.elastic.co/guide/en/logstash/8.8/introduction.html) log collector supports
|
||||
[Opensearch output plugin](https://github.com/opensearch-project/logstash-output-opensearch) compatible with
|
||||
[Elasticsearch bulk API](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#elasticsearch-bulk-api)
|
||||
in VictoriaMetrics.
|
||||
|
||||
Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) section in the `logstash.conf` file
|
||||
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/):
|
||||
|
||||
|
|
@ -100,12 +95,9 @@ output {
|
|||
}
|
||||
```
|
||||
|
||||
More info about output tuning you can find in [these docs](https://github.com/opensearch-project/logstash-output-opensearch/blob/main/README.md).
|
||||
See also:
|
||||
|
||||
[Here is a demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash)
|
||||
for running Logstash with VictoriaLogs with docker-compose and collecting logs to VictoriaLogs
|
||||
(via [Elasticsearch bulk API](https://docs.victoriametrics.com/VictoriaLogs/daat-ingestion/#elasticsearch-bulk-api)).
|
||||
|
||||
The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
|
||||
See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs.
|
||||
- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting).
|
||||
- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
- [Logstash `output.elasticsearch` docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html).
|
||||
- [Docker-compose demo for Logstash integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash).
|
||||
|
|
|
|||
|
|
@ -7,11 +7,13 @@
|
|||
- Logstash. See [how to setup Logstash for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html).
|
||||
- Vector. See [how to setup Vector for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html).
|
||||
|
||||
See also [Log collectors and data ingestion formats](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#log-collectors-and-data-ingestion-formats) in VictoriaMetrics.
|
||||
|
||||
The ingested logs can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
|
||||
See also [data ingestion troubleshooting](#troubleshooting) docs.
|
||||
See also:
|
||||
|
||||
- [Log collectors and data ingestion formats](#log-collectors-and-data-ingestion-formats).
|
||||
- [Data ingestion troubleshooting](#troubleshooting).
|
||||
|
||||
|
||||
## HTTP APIs
|
||||
|
||||
|
|
@ -122,11 +124,11 @@ VictoriaLogs exposes various [metrics](https://docs.victoriametrics.com/Victoria
|
|||
|
||||
## Log collectors and data ingestion formats
|
||||
|
||||
Here is the list of supported collectors and their ingestion formats supported by VictoriaLogs:
|
||||
Here is the list of log collectors and their ingestion formats supported by VictoriaLogs:
|
||||
|
||||
| Collector | Elasticsearch | JSON Stream |
|
||||
| How to setup the collector | Format: Elasticsearch | Format: JSON Stream |
|
||||
|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------------------------------------------------------|
|
||||
| [filebeat](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html) | [Yes](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) | No |
|
||||
| [fluentbit](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html) | No | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/http) |
|
||||
| [logstash](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html) | [Yes](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) | No |
|
||||
| [vector](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html) | [Yes](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) | No |
|
||||
| [Filebeat](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html) | [Yes](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) | No |
|
||||
| [Fluentbit](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html) | No | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/http) |
|
||||
| [Logstash](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html) | [Yes](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) | No |
|
||||
| [Vector](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html) | [Yes](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) | No |
|
||||
|
|
|
|||
|
|
@ -1,11 +1,7 @@
|
|||
# Vector setup
|
||||
|
||||
[Vector](http://vector.dev) log collector supports
|
||||
[Elasticsearch sink](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) compatible with
|
||||
[VictoriaMetrics Elasticsearch bulk API](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#elasticsearch-bulk-api).
|
||||
|
||||
Specify [`sinks.vlogs`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) with `type=elasticsearch` section in the `vector.toml`
|
||||
for sending the collected logs to VictoriaLogs:
|
||||
Specify [Elasticsearch sink type](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) in the `vector.toml`
|
||||
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/):
|
||||
|
||||
```toml
|
||||
[sinks.vlogs]
|
||||
|
|
@ -26,17 +22,32 @@ Substitute the `localhost:9428` address inside `endpoints` section with the real
|
|||
|
||||
Replace `your_input` with the name of the `inputs` section, which collects logs. See [these docs](https://vector.dev/docs/reference/configuration/sources/) for details.
|
||||
|
||||
The `_msg_field` parameter must contain the field name with the log message generated by Vector. This is usually `message` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field) for details.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on parameters specified
|
||||
in the `[sinks.vlogs.query]` section.
|
||||
|
||||
The `_time_field` parameter must contain the field name with the log timestamp generated by Vector. This is usually `@timestamp` field.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#time-field) for details.
|
||||
It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
||||
and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
||||
This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters)
|
||||
in the `[sinks.vlogs.query]` section and inspecting VictoriaLogs logs then:
|
||||
|
||||
It is recommended specifying comma-separated list of field names, which uniquely identify every log stream collected by Vector, in the `_stream_fields` parameter.
|
||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details.
|
||||
```toml
|
||||
[sinks.vlogs]
|
||||
inputs = [ "your_input" ]
|
||||
type = "elasticsearch"
|
||||
endpoints = [ "http://localhost:9428/insert/elasticsearch/" ]
|
||||
mode = "bulk"
|
||||
api_version = "v8"
|
||||
healthcheck.enabled = false
|
||||
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) aren't needed,
|
||||
then VictoriaLogs can be instructed to ignore them during data ingestion - just pass `ignore_fields` parameter with comma-separated list of fields to ignore.
|
||||
[sinks.vlogs.query]
|
||||
_msg_field = "message"
|
||||
_time_field = "timestamp"
|
||||
_stream_fields = "host,container_name"
|
||||
debug = "1"
|
||||
```
|
||||
|
||||
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped
|
||||
during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters).
|
||||
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
|
||||
|
||||
```toml
|
||||
|
|
@ -55,9 +66,6 @@ For example, the following config instructs VictoriaLogs to ignore `log.offset`
|
|||
ignore_fields = "log.offset,event.original"
|
||||
```
|
||||
|
||||
More details about `_msg_field`, `_time_field`, `_stream_fields` and `ignore_fields` are
|
||||
available [here](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters).
|
||||
|
||||
When Vector ingests logs into VictoriaLogs at a high rate, then it may be needed to tune `batch.max_events` option.
|
||||
For example, the following config is optimized for higher than usual ingestion rate:
|
||||
|
||||
|
|
@ -79,7 +87,7 @@ For example, the following config is optimized for higher than usual ingestion r
|
|||
max_events = 1000
|
||||
```
|
||||
|
||||
If the Vector sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compression` option.
|
||||
If the Vector sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compression = "gzip"` option.
|
||||
This usually allows saving network bandwidth and costs by up to 5 times:
|
||||
|
||||
```toml
|
||||
|
|
@ -99,8 +107,8 @@ This usually allows saving network bandwidth and costs by up to 5 times:
|
|||
```
|
||||
|
||||
By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#multitenancy).
|
||||
If you need storing logs in other tenant, then specify the needed tenant via `custom_headers` at `output.elasticsearch` section.
|
||||
For example, the following `vector.toml` config instructs Logstash to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
||||
If you need storing logs in other tenant, then specify the needed tenant via `[sinks.vlogq.request.headers]` section.
|
||||
For example, the following `vector.toml` config instructs Vector to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
||||
|
||||
```toml
|
||||
[sinks.vlogs]
|
||||
|
|
@ -121,12 +129,9 @@ For example, the following `vector.toml` config instructs Logstash to store the
|
|||
ProjectID = "34"
|
||||
```
|
||||
|
||||
More info about output tuning you can find in [these docs](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/).
|
||||
See also:
|
||||
|
||||
[Here is a demo](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-docker)
|
||||
for running Vector with VictoriaLogs with docker-compose and collecting logs from docker-containers
|
||||
to VictoriaLogs (via [Elasticsearch API](https://docs.victoriametrics.com/VictoriaLogs/ingestion/#elasticsearch-bulk-api)).
|
||||
|
||||
The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
|
||||
See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs.
|
||||
- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting).
|
||||
- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
||||
- [Elasticsearch output docs for Vector.dev](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/).
|
||||
- [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-docker).
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ via the following ways:
|
|||
|
||||
## HTTP API
|
||||
|
||||
[VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can be queried at the `/select/logsql/query` HTTP endpoint.
|
||||
VictoriaLogs can be queried at the `/select/logsql/query` HTTP endpoint.
|
||||
The [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) query must be passed via `query` argument.
|
||||
For example, the following query returns all the log entries with the `error` word:
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue