From bafdc9dd7822628f7a5f61919983ed3d747a03cc Mon Sep 17 00:00:00 2001 From: Zhu Jiekun Date: Mon, 22 Jul 2024 22:51:45 +0800 Subject: [PATCH] docs: [vmagent] Add docs for remote write protocol control in Kafka integration (#6682) ### Describe Your Changes Improve documentation to help: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6670 Currently, the documentation of Kafka Integration did not describe: - How to switch between different remote write protocols in producer side and consumer side. --- docs/vmagent.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/vmagent.md b/docs/vmagent.md index f0732d54c2..20851a1286 100644 --- a/docs/vmagent.md +++ b/docs/vmagent.md @@ -1437,6 +1437,9 @@ These formats can be configured with `-kafka.consumer.topic.defaultFormat` or `- * `graphite` - [Graphite plaintext format](https://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol). * `jsonline` - [JSON line format](https://docs.victoriametrics.com/#how-to-import-data-in-json-line-format). +For Kafka messages in the `promremotewrite` format, `vmagent` will automatically detect whether they are using [the Prometheus remote write protocol](https://prometheus.io/docs/specs/remote_write_spec/#protocol) +or [the VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol), and handle them accordingly. + Every Kafka message may contain multiple lines in `influx`, `prometheus`, `graphite` and `jsonline` format delimited by `\n`. `vmagent` consumes messages from Kafka topics specified by `-kafka.consumer.topic` command-line flag. Multiple topics can be specified @@ -1538,6 +1541,11 @@ These messages can be read later from Kafka by another `vmagent` - see [these do Additional Kafka options can be passed as query params to `-remoteWrite.url`. For instance, `kafka://localhost:9092/?topic=prom-rw&client.id=my-favorite-id` sets `client.id` Kafka option to `my-favorite-id`. The full list of Kafka options is available [here](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). +By default, `vmagent` sends compressed messages using Google's Snappy, as defined in [the Prometheus remote write protocol](https://prometheus.io/docs/specs/remote_write_spec/#protocol). +To switch to [the VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent/#victoriametrics-remote-write-protocol) and reduce network bandwidth, +simply set the `-remoteWrite.forceVMProto=true` flag. It is also possible to adjust the compression level for the VictoriaMetrics remote write protocol using the `-remoteWrite.vmProtoCompressLevel` +command-line flag. + #### Kafka broker authorization and authentication Two types of auth are supported: