VictoriaMetrics/deployment/docker/victorialogs/logstash
2024-09-08 21:23:16 +02:00
..
compose.yml deployment: update VictoriaLogs from v0.28.0-victorialogs to v0.29.0-victorialogs 2024-09-08 21:23:16 +02:00
Dockerfile app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers 2024-09-03 17:43:26 +02:00
logstash.yml Add docker compose examples: filebeat(docker, syslog), fluentbit(docker), logstash, vector(docker) 2023-06-21 03:59:31 -07:00
pipeline.conf app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers 2024-09-03 17:43:26 +02:00
README.md lib/logstorage: work-in-progress 2024-05-25 00:30:58 +02:00

Docker compose Logstash integration with VictoriaLogs for syslog

It is required to use OpenSearch plugin for output configuration. Plugin can be installed by using the following command:

bin/logstash-plugin install logstash-output-opensearch

OpenSearch plugin is required because elasticsearch output plugin performs various checks for Elasticsearch version and license which are not applicable for VictoriaLogs.

To spin-up environment run the following command:

docker compose up -d 

To shut down the docker-compose environment run the following command:

docker compose down
docker compose rm -f

The docker compose file contains the following components:

  • logstash - logstash is configured to accept syslog on 5140 port, you can find configuration in the pipeline.conf. It writes data in VictoriaLogs
  • VictoriaLogs - the log database, it accepts the data from logstash by elastic protocol

Querying the data

  • vmui - a web UI is accessible by http://localhost:9428/select/vmui
  • for querying the data via command-line please check these docs

Here is an example of logstash configuration(pipeline.conf):

input {
  syslog {
    port => 5140
  }
}
output {
  opensearch {
    hosts => ["http://victorialogs:9428/insert/elasticsearch"]
    custom_headers => {
        "AccountID" => "0"
        "ProjectID" => "0"
    }
    parameters => {
        "_stream_fields" => "host.ip,process.name"
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
    }
  }
}

Please, note that _stream_fields parameter must follow recommended best practices to achieve better performance.