VictoriaMetrics/deployment/docker/victorialogs/telegraf/README.md

# Docker compose Telegraf integration with VictoriaLogs for docker

The folder contains the examples of integration of [telegraf](https://www.influxdata.com/time-series-platform/telegraf/) with VictoriaLogs using:

* [elasticsearch](./elasticsearch)
* [loki](./loki)
* [jsonline](./jsonline)
* [syslog](./syslog)

To spin-up environment `cd` to any of listed above directories run the following command:
```
docker compose up -d 
```

To shut down the docker-compose environment run the following command:
```
docker compose down
docker compose rm -f
```

The docker compose file contains the following components:

* telegraf - telegraf is configured to collect logs from the `docker`, you can find configuration in the `telegraf.conf`. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics.
* VictoriaLogs - the log database, it accepts the data from `telegraf` by elastic protocol
* VictoriaMetrics - collects metrics from `VictoriaLogs` and `VictoriaMetrics`

Querying the data

* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:9428/select/vmui`
* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)

Telegraf configuration example can be found below:
* [elasticsearch](./elasticsearch/telegraf.conf)
* [loki](./loki/telegraf.conf)
* [jsonline](./jsonline/telegraf.conf)
* [syslog](./syslog/telegraf.conf)

Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers * Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution. * This commit adds the following changes: * Adds fallback to to headers params, if query param is empty for: _msg_field -> VL-Msg-Field _stream_fields -> VL-Stream-Fields _ignore_fields -> VL-Ignore-Fields _time_field -> VL-Time-Field * removes deprecations from victorialogs compose files, added more output format examples for logstash, telegraf, fluent-bit related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310 2024-09-03 15:43:26 +00:00			`# Docker compose Telegraf integration with VictoriaLogs for docker`

deployment: restructure victorialogs examples (#6971) ### Describe Your Changes - Use common compose.yaml file for all victorialogs setups to set version in a single place and override it on demand for each agent and protocol - Replaced multiple victorialogs instances in HA setup with single setup with `deploy.replica` parameter set - Added fluentd setup ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit e75ae1b2740d1692f06907ec9aa38e1972e439b6) 2024-09-25 16:33:26 +00:00			`The folder contains the examples of integration of [telegraf](https://www.influxdata.com/time-series-platform/telegraf/) with VictoriaLogs using:`
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers * Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution. * This commit adds the following changes: * Adds fallback to to headers params, if query param is empty for: _msg_field -> VL-Msg-Field _stream_fields -> VL-Stream-Fields _ignore_fields -> VL-Ignore-Fields _time_field -> VL-Time-Field * removes deprecations from victorialogs compose files, added more output format examples for logstash, telegraf, fluent-bit related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310 2024-09-03 15:43:26 +00:00
deployment: restructure victorialogs examples (#6971) ### Describe Your Changes - Use common compose.yaml file for all victorialogs setups to set version in a single place and override it on demand for each agent and protocol - Replaced multiple victorialogs instances in HA setup with single setup with `deploy.replica` parameter set - Added fluentd setup ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit e75ae1b2740d1692f06907ec9aa38e1972e439b6) 2024-09-25 16:33:26 +00:00			`* [elasticsearch](./elasticsearch)`
			`* [loki](./loki)`
			`* [jsonline](./jsonline)`
			`* [syslog](./syslog)`

			To spin-up environment `cd` to any of listed above directories run the following command:
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers * Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution. * This commit adds the following changes: * Adds fallback to to headers params, if query param is empty for: _msg_field -> VL-Msg-Field _stream_fields -> VL-Stream-Fields _ignore_fields -> VL-Ignore-Fields _time_field -> VL-Time-Field * removes deprecations from victorialogs compose files, added more output format examples for logstash, telegraf, fluent-bit related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310 2024-09-03 15:43:26 +00:00			```
			`docker compose up -d`
			```

			`To shut down the docker-compose environment run the following command:`
			```
			`docker compose down`
			`docker compose rm -f`
			```

			`The docker compose file contains the following components:`

			* telegraf - telegraf is configured to collect logs from the `docker`, you can find configuration in the `telegraf.conf`. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics.
			* VictoriaLogs - the log database, it accepts the data from `telegraf` by elastic protocol
			* VictoriaMetrics - collects metrics from `VictoriaLogs` and `VictoriaMetrics`

			`Querying the data`

			* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:9428/select/vmui`
			`* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)`
deployment: restructure victorialogs examples (#6971) ### Describe Your Changes - Use common compose.yaml file for all victorialogs setups to set version in a single place and override it on demand for each agent and protocol - Replaced multiple victorialogs instances in HA setup with single setup with `deploy.replica` parameter set - Added fluentd setup ### Checklist The following checks are mandatory: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). (cherry picked from commit e75ae1b2740d1692f06907ec9aa38e1972e439b6) 2024-09-25 16:33:26 +00:00
			`Telegraf configuration example can be found below:`
			`* [elasticsearch](./elasticsearch/telegraf.conf)`
			`* [loki](./loki/telegraf.conf)`
			`* [jsonline](./jsonline/telegraf.conf)`
			`* [syslog](./syslog/telegraf.conf)`

			Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.