github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

Vmagent kafka updates (#535)

Browse source 
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
This commit is contained in:
Nikolay 2023-03-15 20:33:10 +01:00 committed by Aliaksandr Valialkin
parent 55afae8641
commit 91cbb9063d
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
2 changed files with 3 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
deployment/docker/Makefile
View file

@ -162,12 +162,7 @@ package-via-docker-arm:
GOARCH=arm $(MAKE) package-via-docker-goarch-nocgo GOARCH=arm $(MAKE) package-via-docker-goarch-nocgo
package-via-docker-arm64: package-via-docker-arm64:
ifeq ($(APP_NAME),vmagent)
GOARCH=arm64 $(MAKE) package-via-docker-goarch-nocgo
else
$(MAKE) package-via-docker-goarch-arm64 $(MAKE) package-via-docker-goarch-arm64
endif
package-via-docker-ppc64le: package-via-docker-ppc64le:
GOARCH=ppc64le $(MAKE) package-via-docker-goarch-nocgo GOARCH=ppc64le $(MAKE) package-via-docker-goarch-nocgo

3
docs/CHANGELOG.md
View file

@ -16,6 +16,9 @@ The following tip changes can be tested by building VictoriaMetrics components f
## tip ## tip
* FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds `-maxLabelValueLen`. This should simplify debugging for this case. * FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds `-maxLabelValueLen`. This should simplify debugging for this case.
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [VictoriaMetrics remote write protocol](https://docs.victoriametrics.com/vmagent.html#victoriametrics-remote-write-protocol) when [sending / receiving data to / from Kafka](https://docs.victoriametrics.com/vmagent.html#kafka-integration). This protocol allows saving egress network bandwidth costs when sending data from `vmagent` to `Kafka` located in another datacenter or availability zone. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `--kafka.consumer.topic.concurrency` command-line flag. It controls the number of Kafka consumer workers to use by `vmagent`. It should eliminate the need to start multiple `vmagent` instances to improve data transfer rate. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [Kafka producer and consumer](https://docs.victoriametrics.com/vmagent.html#kafka-integration) on `arm64` machines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271).
## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1) ## [v1.89.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.89.1)