diff --git a/deployment/docker/victorialogs/filebeat/README.md b/deployment/docker/victorialogs/filebeat/README.md index 40bb451c0..62d652e01 100644 --- a/deployment/docker/victorialogs/filebeat/README.md +++ b/deployment/docker/victorialogs/filebeat/README.md @@ -1,10 +1,10 @@ -# Docker compose Filebeat integration with VictoriaLogs using listed below protocols: +# Docker compose Filebeat integration with VictoriaLogs + +The folder contains examples of [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) integration with VictoriaLogs using protocols: * [syslog](./syslog) * [elasticsearch](./elasticsearch) -The folder contains the example of integration of [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) with Victorialogs - To spin-up environment `cd` to any of listed above directories run the following command: ``` docker compose up -d @@ -18,9 +18,9 @@ docker compose rm -f The docker compose file contains the following components: -* filebeat - fileabeat is configured to collect logs from the `docker`, you can find configuration in the `filebeat.yml`. It writes data in VictoriaLogs -* VictoriaLogs - the log database, it accepts the data from `filebeat` by elastic protocol -* VictoriaMetrics - collects metrics from `filebeat` via `filebeat-exporter`, `VictoriaLogs` and `VictoriaMetrics` +* filebeat - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `filebeat` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `filebeat` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/filebeat/compose.yml b/deployment/docker/victorialogs/filebeat/compose.yml index 308ff363b..0b3b38359 100644 --- a/deployment/docker/victorialogs/filebeat/compose.yml +++ b/deployment/docker/victorialogs/filebeat/compose.yml @@ -1,8 +1,8 @@ include: - ../compose.yml services: - filebeat-victorialogs: - image: docker.elastic.co/beats/filebeat:8.15.0 + filebeat: + image: docker.elastic.co/beats/filebeat:8.15.1 restart: on-failure volumes: - type: bind diff --git a/deployment/docker/victorialogs/fluentbit/README.md b/deployment/docker/victorialogs/fluentbit/README.md index 5a8f51709..c6727c206 100644 --- a/deployment/docker/victorialogs/fluentbit/README.md +++ b/deployment/docker/victorialogs/fluentbit/README.md @@ -1,11 +1,11 @@ -# Docker compose Fluentbit integration with VictoriaLogs using given below protocols: +# Docker compose FluentBit integration with VictoriaLogs + +The folder contains examples of [FluentBit](https://docs.fluentbit.io/manual) integration with VictoriaLogs using protocols: * [loki](./loki) * [jsonline single node](./jsonline) * [jsonline HA setup](./jsonline-ha) -The folder contains the example of integration of [fluentbit](https://docs.fluentbit.io/manual) with Victorialogs - To spin-up environment `cd` to any of listed above directories run the following command: ``` docker compose up -d @@ -19,8 +19,9 @@ docker compose rm -f The docker compose file contains the following components: -* fluentbit - fluentbit is configured to collect logs from the `docker`, you can find configuration in the `fluent-bit.conf`. It writes data in VictoriaLogs -* VictoriaLogs - the log database, it accepts the data from `fluentbit` by json line protocol +* fluentbit - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `fluentbit` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `fluentbit` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/fluentbit/jsonline-ha/fluent-bit.conf b/deployment/docker/victorialogs/fluentbit/jsonline-ha/fluent-bit.conf index fb439860c..aaca83619 100644 --- a/deployment/docker/victorialogs/fluentbit/jsonline-ha/fluent-bit.conf +++ b/deployment/docker/victorialogs/fluentbit/jsonline-ha/fluent-bit.conf @@ -13,11 +13,23 @@ Parser syslog-rfc3164 Mode tcp +[INPUT] + name fluentbit_metrics + tag internal_metrics + scrape_interval 2 + [SERVICE] Flush 1 Parsers_File parsers.conf -[Output] +[OUTPUT] + Name prometheus_remote_write + Match internal_metrics + Host victoriametrics + Port 8428 + Uri /api/v1/write + +[OUTPUT] Name http Match * host victorialogs @@ -29,7 +41,7 @@ header AccountID 0 header ProjectID 0 -[Output] +[OUTPUT] Name http Match * host victorialogs-2 diff --git a/deployment/docker/victorialogs/fluentbit/jsonline/fluent-bit.conf b/deployment/docker/victorialogs/fluentbit/jsonline/fluent-bit.conf index 42d349338..bacbc228f 100644 --- a/deployment/docker/victorialogs/fluentbit/jsonline/fluent-bit.conf +++ b/deployment/docker/victorialogs/fluentbit/jsonline/fluent-bit.conf @@ -13,11 +13,23 @@ Parser syslog-rfc3164 Mode tcp +[INPUT] + name fluentbit_metrics + tag internal_metrics + scrape_interval 2 + [SERVICE] Flush 1 Parsers_File parsers.conf -[Output] +[OUTPUT] + Name prometheus_remote_write + Match internal_metrics + Host victoriametrics + Port 8428 + Uri /api/v1/write + +[OUTPUT] Name http Match * host victorialogs diff --git a/deployment/docker/victorialogs/fluentbit/loki/fluent-bit.conf b/deployment/docker/victorialogs/fluentbit/loki/fluent-bit.conf index 6c7906d0d..17bd57ae3 100644 --- a/deployment/docker/victorialogs/fluentbit/loki/fluent-bit.conf +++ b/deployment/docker/victorialogs/fluentbit/loki/fluent-bit.conf @@ -13,10 +13,22 @@ Parser syslog-rfc3164 Mode tcp +[INPUT] + name fluentbit_metrics + tag internal_metrics + scrape_interval 2 + [SERVICE] Flush 1 Parsers_File parsers.conf +[OUTPUT] + Name prometheus_remote_write + Match internal_metrics + Host victoriametrics + Port 8428 + Uri /api/v1/write + [OUTPUT] name loki match * diff --git a/deployment/docker/victorialogs/fluentd/Dockerfile b/deployment/docker/victorialogs/fluentd/Dockerfile index b3bba2e0f..dad82dfbc 100644 --- a/deployment/docker/victorialogs/fluentd/Dockerfile +++ b/deployment/docker/victorialogs/fluentd/Dockerfile @@ -4,5 +4,6 @@ RUN \ gem install \ fluent-plugin-datadog \ fluent-plugin-grafana-loki \ - fluent-plugin-elasticsearch + fluent-plugin-elasticsearch \ + fluent-plugin-remote_syslog USER fluent diff --git a/deployment/docker/victorialogs/fluentd/README.md b/deployment/docker/victorialogs/fluentd/README.md index 0d7e42298..2aa9ea341 100644 --- a/deployment/docker/victorialogs/fluentd/README.md +++ b/deployment/docker/victorialogs/fluentd/README.md @@ -1,10 +1,12 @@ -# Docker compose Fluentd integration with VictoriaLogs using given below protocols: +# Docker compose Fluentd integration with VictoriaLogs + +The folder contains examples of [Fluentd](https://www.fluentd.org/) integration with VictoriaLogs using protocols: * [loki](./loki) * [jsonline](./jsonline) * [elasticsearch](./elasticsearch) -The folder contains the example of integration of [fluentd](https://www.fluentd.org/) with Victorialogs +All required plugins, that should be installed in order to support protocols listed above can be found in a [Dockerfile](./Dockerfile) To spin-up environment `cd` to any of listed above directories run the following command: ``` @@ -19,8 +21,9 @@ docker compose rm -f The docker compose file contains the following components: -* fluentd - fluentd is configured to collect logs from the `docker`, you can find configuration in the `fluent-bit.conf`. It writes data in VictoriaLogs -* VictoriaLogs - the log database, it accepts the data from `fluentd` by json line protocol +* fluentd - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `fluentd` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `fluentd` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/fluentd/syslog/compose.yml b/deployment/docker/victorialogs/fluentd/syslog/compose.yml new file mode 100644 index 000000000..e9e4e82da --- /dev/null +++ b/deployment/docker/victorialogs/fluentd/syslog/compose.yml @@ -0,0 +1,3 @@ +include: + - ../compose.yml +name: fluentbit-syslog diff --git a/deployment/docker/victorialogs/fluentd/syslog/fluent.conf b/deployment/docker/victorialogs/fluentd/syslog/fluent.conf new file mode 100644 index 000000000..b996f74d5 --- /dev/null +++ b/deployment/docker/victorialogs/fluentd/syslog/fluent.conf @@ -0,0 +1,19 @@ + + @type tail + format none + tag docker.testlog + path /var/lib/docker/containers/**/*.log + + + + @type remote_syslog + host victorialogs + port 8094 + severity debug + program fluentd + protocol tcp + + @type single_value + message_key message + + diff --git a/deployment/docker/victorialogs/logstash/README.md b/deployment/docker/victorialogs/logstash/README.md index de39545fa..746450867 100644 --- a/deployment/docker/victorialogs/logstash/README.md +++ b/deployment/docker/victorialogs/logstash/README.md @@ -1,16 +1,13 @@ -# Docker compose Logstash integration with VictoriaLogs for given below protocols: +# Docker compose Logstash integration with VictoriaLogs + +The folder contains examples of [Logstash](https://www.elastic.co/logstash) integration with VictoriaLogs using protocols: * [loki](./loki) * [jsonline single node](./jsonline) * [jsonline HA setup](./jsonline-ha) * [elasticsearch](./elasticsearch) -It is required to use [OpenSearch plugin](https://github.com/opensearch-project/logstash-output-opensearch) for output configuration. -Plugin can be installed by using the following command: -``` -bin/logstash-plugin install logstash-output-opensearch -``` -OpenSearch plugin is required because elasticsearch output plugin performs various checks for Elasticsearch version and license which are not applicable for VictoriaLogs. +All required plugins, that should be installed in order to support protocols listed above can be found in a [Dockerfile](./Dockerfile) To spin-up environment `cd` to any of listed above directories run the following command: ``` @@ -25,8 +22,9 @@ docker compose rm -f The docker compose file contains the following components: -* logstash - logstash is configured to accept `syslog` on `5140` port, you can find configuration in the `pipeline.conf`. It writes data in VictoriaLogs -* VictoriaLogs - the log database, it accepts the data from `logstash` by elastic protocol +* logstash - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `logstash` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `logstash` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/opentelemetry-collector/README.md b/deployment/docker/victorialogs/opentelemetry-collector/README.md index 6dab25daf..1e10ea199 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/README.md +++ b/deployment/docker/victorialogs/opentelemetry-collector/README.md @@ -1,4 +1,6 @@ -# Docker compose OpenTelemetry integration with VictoriaLogs using protocols: +# Docker compose OpenTelemetry collector integration with VictoriaLogs + +The folder contains examples of [OpenTelemetry collector](https://opentelemetry.io/docs/collector/) integration with VictoriaLogs using protocols: * [loki](./loki) * [otlp](./otlp) @@ -6,8 +8,6 @@ * [elasticsearch single node](./elasticsearch) * [elasticsearch HA mode](./elasticsearch-ha/) -The folder contains the example of integration of [OpenTelemetry collector](https://opentelemetry.io/docs/collector/) with Victorialogs - To spin-up environment `cd` to any of listed above directories run the following command: ``` docker compose up -d @@ -21,9 +21,9 @@ docker compose rm -f The docker compose file contains the following components: -* collector - vector is configured to collect logs from the `docker`, you can find configuration in the `config.yaml`. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics. -* VictoriaLogs - the log database, it accepts the data from `collector` by elastic protocol -* VictoriaMetrics - collects metrics from `VictoriaLogs` and `VictoriaMetrics` +* collector - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `collector` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `collector` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch-ha/config.yaml b/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch-ha/config.yaml index 46b6073a0..850e6f4e4 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch-ha/config.yaml +++ b/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch-ha/config.yaml @@ -9,6 +9,15 @@ receivers: resource: region: us-east-1 service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: http/protobuf + endpoint: http://victoriametrics:8428/opentelemetry/api/v1/push pipelines: logs: receivers: [filelog] diff --git a/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch/config.yaml b/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch/config.yaml index 072525175..76f56f9d8 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch/config.yaml +++ b/deployment/docker/victorialogs/opentelemetry-collector/elasticsearch/config.yaml @@ -8,6 +8,15 @@ receivers: resource: region: us-east-1 service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: http/protobuf + endpoint: http://victoriametrics:8428/opentelemetry/api/v1/push pipelines: logs: receivers: [filelog] diff --git a/deployment/docker/victorialogs/opentelemetry-collector/loki/config.yaml b/deployment/docker/victorialogs/opentelemetry-collector/loki/config.yaml index 1891bd531..920ab71cb 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/loki/config.yaml +++ b/deployment/docker/victorialogs/opentelemetry-collector/loki/config.yaml @@ -7,6 +7,15 @@ receivers: resource: region: us-east-1 service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: http/protobuf + endpoint: http://victoriametrics:8428/opentelemetry/api/v1/push pipelines: logs: receivers: [filelog] diff --git a/deployment/docker/victorialogs/opentelemetry-collector/otlp/config.yaml b/deployment/docker/victorialogs/opentelemetry-collector/otlp/config.yaml index b06ac851d..059e5d686 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/otlp/config.yaml +++ b/deployment/docker/victorialogs/opentelemetry-collector/otlp/config.yaml @@ -9,6 +9,15 @@ receivers: resource: region: us-east-1 service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: http/protobuf + endpoint: http://victoriametrics:8428/opentelemetry/api/v1/push pipelines: logs: receivers: [filelog] diff --git a/deployment/docker/victorialogs/opentelemetry-collector/syslog/config.yaml b/deployment/docker/victorialogs/opentelemetry-collector/syslog/config.yaml index d556ee111..ba4711e72 100644 --- a/deployment/docker/victorialogs/opentelemetry-collector/syslog/config.yaml +++ b/deployment/docker/victorialogs/opentelemetry-collector/syslog/config.yaml @@ -17,6 +17,15 @@ receivers: filelog: include: [/tmp/logs/*.log] service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: http/protobuf + endpoint: http://victoriametrics:8428/opentelemetry/api/v1/push pipelines: logs: receivers: [filelog] diff --git a/deployment/docker/victorialogs/promtail/README.md b/deployment/docker/victorialogs/promtail/README.md new file mode 100644 index 000000000..f48751fb8 --- /dev/null +++ b/deployment/docker/victorialogs/promtail/README.md @@ -0,0 +1,32 @@ +# Docker compose Promtail integration with VictoriaLogs + +The folder contains the example of integration of [Promtail agent](https://grafana.com/docs/loki/latest/send-data/promtail/) with VictoriaLogs using protocols: + +* [loki](./loki) + +To spin-up environment `cd` to any of listed above directories run the following command: +``` +docker compose up -d +``` + +To shut down the docker-compose environment run the following command: +``` +docker compose down +docker compose rm -f +``` + +The docker compose file contains the following components: + +* promtail - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `promtail` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `promtail` for observability purposes + +Querying the data + +* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:9428/select/vmui` +* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line) + +Promtail agent configuration example can be found below: +* [loki](./loki/config.yml) + +Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance. diff --git a/deployment/docker/victorialogs/telegraf/README.md b/deployment/docker/victorialogs/telegraf/README.md index 6c08ef8a3..69099ef96 100644 --- a/deployment/docker/victorialogs/telegraf/README.md +++ b/deployment/docker/victorialogs/telegraf/README.md @@ -1,6 +1,6 @@ -# Docker compose Telegraf integration with VictoriaLogs for docker +# Docker compose Telegraf integration with VictoriaLogs -The folder contains the examples of integration of [telegraf](https://www.influxdata.com/time-series-platform/telegraf/) with VictoriaLogs using: +The folder contains examples of [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) integration with VictoriaLogs using protocols: * [elasticsearch](./elasticsearch) * [loki](./loki) @@ -20,9 +20,9 @@ docker compose rm -f The docker compose file contains the following components: -* telegraf - telegraf is configured to collect logs from the `docker`, you can find configuration in the `telegraf.conf`. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics. -* VictoriaLogs - the log database, it accepts the data from `telegraf` by elastic protocol -* VictoriaMetrics - collects metrics from `VictoriaLogs` and `VictoriaMetrics` +* telegraf - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `telegraf` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `telegraf` for observability purposes Querying the data diff --git a/deployment/docker/victorialogs/vector/README.md b/deployment/docker/victorialogs/vector/README.md index 502298222..5169dc328 100644 --- a/deployment/docker/victorialogs/vector/README.md +++ b/deployment/docker/victorialogs/vector/README.md @@ -1,12 +1,12 @@ -# Docker compose Vector integration with VictoriaLogs using given below protocols: +# Docker compose Vector integration with VictoriaLogs + +The folder contains examples of [Vector](https://vector.dev/docs/) integration with VictoriaLogs using protocols: * [elasticsearch](./elasticsearch) * [loki](./loki) * [jsonline single node](./jsonline) * [jsonline HA setup](./jsonline-ha) -The folder contains the example of integration of [vector](https://vector.dev/docs/) with Victorialogs - To spin-up environment `cd` to any of listed above directories run the following command: ``` docker compose up -d @@ -20,9 +20,9 @@ docker compose rm -f The docker compose file contains the following components: -* vector - vector is configured to collect logs from the `docker`, you can find configuration in the `vector.yaml`. It writes data in VictoriaLogs. It pushes metrics to VictoriaMetrics. -* VictoriaLogs - the log database, it accepts the data from `vector` by DataDog protocol -* VictoriaMetrics - collects metrics from `VictoriaLogs` and `VictoriaMetrics` +* vector - logs collection agent configured to collect and write data to `victorialogs` +* victorialogs - logs database, receives data from `vector` agent +* victoriametrics - metrics database, which collects metrics from `victorialogs` and `vector` for observability purposes Querying the data diff --git a/docs/VictoriaLogs/Roadmap.md b/docs/VictoriaLogs/Roadmap.md index 4848d1aaa..3dbabcb95 100644 --- a/docs/VictoriaLogs/Roadmap.md +++ b/docs/VictoriaLogs/Roadmap.md @@ -22,7 +22,7 @@ The following functionality is planned in the future versions of VictoriaLogs: - Support for [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) from popular log collectors and formats: - [x] [OpenTelemetry for logs](https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/) - - [ ] Fluentd + - [x] [Fluentd](https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentd/) - [ ] [Journald](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618) (systemd) - [ ] [Datadog protocol for logs](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632) - [x] [Telegraf](https://docs.victoriametrics.com/victorialogs/data-ingestion/telegraf/) diff --git a/docs/VictoriaLogs/data-ingestion/Fluentbit.md b/docs/VictoriaLogs/data-ingestion/Fluentbit.md index 645734e18..d69b5c672 100644 --- a/docs/VictoriaLogs/data-ingestion/Fluentbit.md +++ b/docs/VictoriaLogs/data-ingestion/Fluentbit.md @@ -11,9 +11,6 @@ aliases: - /victorialogs/data-ingestion/fluentbit.html - /victorialogs/data-ingestion/Fluentbit.html --- - -# Fluentbit setup - VictoriaLogs supports given below Fluentbit outputs: - [Loki](#loki) - [HTTP JSON](#http) diff --git a/docs/VictoriaLogs/data-ingestion/Fluentd.md b/docs/VictoriaLogs/data-ingestion/Fluentd.md new file mode 100644 index 000000000..fcfc7b18f --- /dev/null +++ b/docs/VictoriaLogs/data-ingestion/Fluentd.md @@ -0,0 +1,109 @@ +--- +weight: 2 +title: Fluentd setup +disableToc: true +menu: + docs: + parent: "victorialogs-data-ingestion" + weight: 2 +aliases: + - /VictoriaLogs/data-ingestion/Fluentd.html + - /victorialogs/data-ingestion/fluentd.html + - /victorialogs/data-ingestion/Fluentd.html +--- +VictoriaLogs supports given below Fluentd outputs: +- [Loki](#loki) +- [HTTP JSON](#http) + +## Loki + +Specify [loki output](https://docs.fluentd.io/manual/pipeline/outputs/loki) section in the `fluentd.conf` +for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): + +```conf + + @type loki + url "http://localhost:9428/insert" + + flush_interval 10s + flush_at_shutdown true + + custom_headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + buffer_chunk_limit 1m + +``` + +## HTTP + +Specify [http output](https://docs.fluentd.io/manual/pipeline/outputs/http) section in the `fluentd.conf` +for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): + +```fluentd + + @type http + endpoint "http://localhost:9428/insert/jsonline" + headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + +``` + +Substitute the host (`localhost`) and port (`9428`) with the real TCP address of VictoriaLogs. + +See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on the query args specified in the `endpoint`. + +It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) +and uses the correct [stream fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) in the `endpoint` +and inspecting VictoriaLogs logs then: + +```fluentd + + @type http + endpoint "http://localhost:9428/insert/jsonline&debug=1" + headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + +``` + +If some [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) must be skipped +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). +For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: + +```fluentd + + @type http + endpoint "http://localhost:9428/insert/jsonline&ignore_fields=log.offset,event.original" + headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + +``` + +If the Fluentd sends logs to VictoriaLogs in another datacenter, then it may be useful enabling data compression via `compress gzip` option. +This usually allows saving network bandwidth and costs by up to 5 times: + +```fluentd + + @type http + endpoint "http://localhost:9428/insert/jsonline&ignore_fields=log.offset,event.original" + headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + compress gzip + +``` + +By default, the ingested logs are stored in the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/victorialogs/keyconcepts/#multitenancy). +If you need storing logs in other tenant, then specify the needed tenant via `header` options. +For example, the following `fluentd.conf` config instructs Fluentd to store the data to `(AccountID=12, ProjectID=34)` tenant: + +```fluentd + + @type http + endpoint "http://localhost:9428/insert/jsonline" + headers {"VL-Msg-Field": "log", "VL-Time-Field": "time", "VL-Stream-Fields": "path"} + header AccountID 12 + header ProjectID 23 + +``` + +See also: + +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting). +- [How to query VictoriaLogs](https://docs.victoriametrics.com/victorialogs/querying/). +- [Fluentd HTTP output config docs](https://docs.fluentd.org/output/http) +- [Docker-compose demo for Fluentd integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentd). diff --git a/docs/VictoriaLogs/data-ingestion/README.md b/docs/VictoriaLogs/data-ingestion/README.md index 523e05697..a548a195a 100644 --- a/docs/VictoriaLogs/data-ingestion/README.md +++ b/docs/VictoriaLogs/data-ingestion/README.md @@ -3,6 +3,7 @@ - Syslog, Rsyslog and Syslog-ng - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/). - Filebeat - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/filebeat/). - Fluentbit - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentbit/). +- Fluentd - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentd/). - Logstash - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/logstash/). - Vector - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/). - Promtail (aka Grafana Loki) - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/promtail/). @@ -286,3 +287,5 @@ Here is the list of log collectors and their ingestion formats supported by Vict | [Promtail](https://docs.victoriametrics.com/victorialogs/data-ingestion/promtail/) | No | No | [Yes](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#clients) | No | No | | [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) | [Yes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter) | No | [Yes](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter) | [Yes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/syslogexporter) | [Yes](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) | | [Telegraf](https://docs.victoriametrics.com/victorialogs/data-ingestion/telegraf/) | [Yes](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/elasticsearch) | [Yes](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/http) | [Yes](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/loki) | [Yes](https://github.com/influxdata/telegraf/blob/master/plugins/outputs/syslog) | Yes | +| [Fluentd](https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentd/) | [Yes](https://github.com/uken/fluent-plugin-elasticsearch) | [Yes](https://docs.fluentd.org/output/http) | [Yes](https://grafana.com/docs/loki/latest/send-data/fluentd/) | [Yes](https://github.com/fluent-plugins-nursery/fluent-plugin-remote_syslog) | No | + diff --git a/docs/VictoriaLogs/data-ingestion/Telegraf.md b/docs/VictoriaLogs/data-ingestion/Telegraf.md index a31b86d95..3cd1c737c 100644 --- a/docs/VictoriaLogs/data-ingestion/Telegraf.md +++ b/docs/VictoriaLogs/data-ingestion/Telegraf.md @@ -9,8 +9,6 @@ menu: aliases: - /VictoriaLogs/data-ingestion/Telegraf.html --- -# Telegraf setup - VictoriaLogs supports given below Telegraf outputs: - [Elasticsearch](#elasticsearch) - [Loki](#loki) diff --git a/docs/VictoriaLogs/data-ingestion/opentelemetry.md b/docs/VictoriaLogs/data-ingestion/opentelemetry.md index db342d9a5..16df5742a 100644 --- a/docs/VictoriaLogs/data-ingestion/opentelemetry.md +++ b/docs/VictoriaLogs/data-ingestion/opentelemetry.md @@ -9,8 +9,6 @@ menu: aliases: - /VictoriaLogs/data-ingestion/OpenTelemetry.html --- - - VictoriaLogs supports both client open-telemetry [SDK](https://opentelemetry.io/docs/languages/) and [collector](https://opentelemetry.io/docs/collector/). ## Client SDK