mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-01 14:47:38 +00:00
3b79f98459
This reverts commit 3d7a77bf82.
Reason for revert: relative links do not work properly at GitHub code
and at GitHub wiki. For example, the following page contains broken links
before reverting this commit:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/VictoriaLogs/CHANGELOG.md
It is always better to use absolute links thank relative links, since the page contents
can be copy-n-pasted to other pages, which are located in vastly different directories,
and all the links will remain working.
115 lines
4.1 KiB
Markdown
115 lines
4.1 KiB
Markdown
---
|
|
weight: 3
|
|
title: Logstash setup
|
|
disableToc: true
|
|
menu:
|
|
docs:
|
|
parent: "victorialogs-data-ingestion"
|
|
weight: 3
|
|
aliases:
|
|
- /VictoriaLogs/data-ingestion/Logstash.html
|
|
---
|
|
|
|
# Logstash setup
|
|
|
|
Specify [`output.elasticsearch`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) section in the `logstash.conf` file
|
|
for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/):
|
|
|
|
```conf
|
|
output {
|
|
elasticsearch {
|
|
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
|
parameters => {
|
|
"_msg_field" => "message"
|
|
"_time_field" => "@timestamp"
|
|
"_stream_fields" => "host.name,process.name"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Substitute `localhost:9428` address inside `hosts` with the real TCP address of VictoriaLogs.
|
|
|
|
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the `parameters` section.
|
|
|
|
It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model)
|
|
and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields).
|
|
This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters)
|
|
and inspecting VictoriaLogs logs then:
|
|
|
|
```conf
|
|
output {
|
|
elasticsearch {
|
|
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
|
parameters => {
|
|
"_msg_field" => "message"
|
|
"_time_field" => "@timestamp"
|
|
"_stream_fields" => "host.name,process.name"
|
|
"debug" => "1"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped
|
|
during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters).
|
|
For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs:
|
|
|
|
```conf
|
|
output {
|
|
elasticsearch {
|
|
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
|
parameters => {
|
|
"_msg_field" => "message"
|
|
"_time_field" => "@timestamp"
|
|
"_stream_fields" => "host.hostname,process.name"
|
|
"ignore_fields" => "log.offset,event.original"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
If the Logstash sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `http_compression: true` option.
|
|
This usually allows saving network bandwidth and costs by up to 5 times:
|
|
|
|
```conf
|
|
output {
|
|
elasticsearch {
|
|
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
|
parameters => {
|
|
"_msg_field" => "message"
|
|
"_time_field" => "@timestamp"
|
|
"_stream_fields" => "host.hostname,process.name"
|
|
}
|
|
http_compression => true
|
|
}
|
|
}
|
|
```
|
|
|
|
By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy).
|
|
If you need storing logs in other tenant, then specify the needed tenant via `custom_headers` at `output.elasticsearch` section.
|
|
For example, the following `logstash.conf` config instructs Logstash to store the data to `(AccountID=12, ProjectID=34)` tenant:
|
|
|
|
```conf
|
|
output {
|
|
elasticsearch {
|
|
hosts => ["http://localhost:9428/insert/elasticsearch/"]
|
|
custom_headers => {
|
|
"AccountID" => "1"
|
|
"ProjectID" => "2"
|
|
}
|
|
parameters => {
|
|
"_msg_field" => "message"
|
|
"_time_field" => "@timestamp"
|
|
"_stream_fields" => "host.hostname,process.name"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
See also:
|
|
|
|
- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting).
|
|
- [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/).
|
|
- [Logstash `output.elasticsearch` docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html).
|
|
- [Docker-compose demo for Logstash integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash).
|