github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/deployment/docker/victorialogs/vector/README.md
Andrii Chubatiuk d5fe4566e5
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers 
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 20:24:00 +02:00

1.9 KiB
Raw Blame History

Docker compose Vector integration with VictoriaLogs for docker

The folder contains the example of integration of vector with Victorialogs

To spin-up environment run the following command:

docker compose up -d

To shut down the docker-compose environment run the following command:

docker compose down
docker compose rm -f

The docker compose file contains the following components:

  • vector - vector is configured to collect logs from the docker, you can find configuration in the vector.yaml. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics.
  • VictoriaLogs - the log database, it accepts the data from vector by elastic protocol
  • VictoriaMetrics - collects metrics from VictoriaLogs and VictoriaMetrics

Querying the data

  • vmui - a web UI is accessible by http://localhost:9428/select/vmui
  • for querying the data via command-line please check these docs

the example of vector configuration(vector.yaml)

sources:
  docker:
    type: docker_logs
transforms:
  msg_parser:
    type: remap
    inputs:
      - docker
    source: |
      .log = parse_json!(.message)
      del(.message)
sinks:
  vlogs_es:
    type: elasticsearch
    inputs:
      - msg_parser
    endpoints:
      - http://victorialogs:9428/insert/elasticsearch/
    mode: bulk
    api_version: v8
    compression: gzip
    healthcheck.enabled: false
    query:
      _msg_field: log.msg
      _time_field: timestamp
      _stream_fields: source_type,host,container_name
    request:
      headers:
        AccountID: "0"
        ProjectID: "0"

Please, note that _stream_fields parameter must follow recommended best practices to achieve better performance.