mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00

History

Andrii Chubatiuk d5fe4566e5 app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers * Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution. * This commit adds the following changes: * Adds fallback to to headers params, if query param is empty for: _msg_field -> VL-Msg-Field _stream_fields -> VL-Stream-Fields _ignore_fields -> VL-Ignore-Fields _time_field -> VL-Time-Field * removes deprecations from victorialogs compose files, added more output format examples for logstash, telegraf, fluent-bit related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310		2024-09-03 20:24:00 +02:00
..
compose.yml	app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers	2024-09-03 20:24:00 +02:00
Dockerfile	app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers	2024-09-03 20:24:00 +02:00
logstash.yml	Add docker compose examples: filebeat(docker, syslog), fluentbit(docker), logstash, vector(docker)	2023-07-06 21:25:31 -07:00
pipeline.conf	app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers	2024-09-03 20:24:00 +02:00
README.md	lib/logstorage: work-in-progress	2024-05-25 00:31:55 +02:00

README.md

Docker compose Logstash integration with VictoriaLogs for syslog

It is required to use OpenSearch plugin for output configuration. Plugin can be installed by using the following command:

bin/logstash-plugin install logstash-output-opensearch

OpenSearch plugin is required because elasticsearch output plugin performs various checks for Elasticsearch version and license which are not applicable for VictoriaLogs.

To spin-up environment run the following command:

docker compose up -d

To shut down the docker-compose environment run the following command:

docker compose down
docker compose rm -f

The docker compose file contains the following components:

logstash - logstash is configured to accept syslog on 5140 port, you can find configuration in the pipeline.conf. It writes data in VictoriaLogs
VictoriaLogs - the log database, it accepts the data from logstash by elastic protocol

Querying the data

vmui - a web UI is accessible by http://localhost:9428/select/vmui
for querying the data via command-line please check these docs

Here is an example of logstash configuration(pipeline.conf):

input {
  syslog {
    port => 5140
  }
}
output {
  opensearch {
    hosts => ["http://victorialogs:9428/insert/elasticsearch"]
    custom_headers => {
        "AccountID" => "0"
        "ProjectID" => "0"
    }
    parameters => {
        "_stream_fields" => "host.ip,process.name"
        "_msg_field" => "message"
        "_time_field" => "@timestamp"
    }
  }
}

Please, note that _stream_fields parameter must follow recommended best practices to achieve better performance.