github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/docs/VictoriaLogs/data-ingestion/Telegraf.md
Andrii Chubatiuk d5fe4566e5
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers 
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 20:24:00 +02:00

3.9 KiB
Raw Blame History

weight title disableToc menu aliases
5 Telegraf setup true
docs
parent weight
victorialogs-data-ingestion 5
/VictoriaLogs/data-ingestion/Telegraf.html

Telegraf setup

VictoriaLogs supports given below Telegraf outputs:

Elasticsearch

Specify Elasticsearch output in the telegraf.toml for sending the collected logs to VictoriaLogs:

[[outputs.elasticsearch]]
  urls = ["http://localhost:9428/insert/elasticsearch"]
  timeout = "1m"
  flush_interval = "30s"
  enable_sniffer = false
  health_check_interval = "0s"
  index_name = "device_log-%Y.%m.%d"
  manage_template = false
  template_name = "telegraf"
  overwrite_template = false
  namepass = ["tail"]
  [outputs.elasticsearch.headers]
    "VL-Msg-Field" = "tail.value"
    "VL-Time-Field" = "@timestamp"
    "VL-Stream-Fields" = "tag.log_source,tag.metric_type"

[[inputs.tail]]
  files = ["/tmp/telegraf.log"]
  from_beginning = false
  interval = "10s"
  pipe = false
  watch_method = "inotify"
  data_format = "value"
  data_type = "string"
  character_encoding = "utf-8"
  [inputs.tail.tags]
     metric_type = "logs"
     log_source = "telegraf"

Loki

Specify Loki output in the telegraf.toml for sending the collected logs to VictoriaLogs:

[[outputs.loki]]
  domain = "http://localhost:9428"
  endpoint = "/insert/loki/api/v1/push&_msg_field=tail.value&_time_field=@timefield&_stream_fields=log_source,metric_type"
  namepass = ["tail"]
  gzip_request = true
  sanitize_label_names = true

[[inputs.tail]]
  files = ["/tmp/telegraf.log"]
  from_beginning = false
  interval = "10s"
  pipe = false
  watch_method = "inotify"
  data_format = "value"
  data_type = "string"
  character_encoding = "utf-8"
  [inputs.tail.tags]
     metric_type = "logs"
     log_source = "telegraf"

HTTP

Specify HTTP output in the telegraf.toml with batch mode disabled for sending the collected logs to VictoriaLogs:

[[inputs.tail]]
  files = ["/tmp/telegraf.log"]
  from_beginning = false
  interval = "10s"
  pipe = false
  watch_method = "inotify"
  data_format = "value"
  data_type = "string"
  character_encoding = "utf-8"
  [inputs.tail.tags]
     metric_type = "logs"
     log_source = "telegraf"

[[outputs.http]]
  url = "http://localhost:9428/insert/jsonline?_msg_field=fields.message&_time_field=timestamp,_stream_fields=tags.log_source,tags.metric_type"
  data_format = "json"
  namepass = ["docker_log"]
  use_batch_format = false

Substitute the localhost:9428 address inside endpoints section with the real TCP address of VictoriaLogs.

See these docs for details on headers specified in the [[output.elasticsearch]] section.

It is recommended verifying whether the initial setup generates the needed log fields and uses the correct stream fields.

See also: