From c41a9b8d1707c65de12e91f5b18d764c25387ad8 Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Wed, 7 Aug 2024 16:49:43 +0200 Subject: [PATCH] lib/bytesutil: smooth buffer growth rate (#6761) Before, buffer growth was always x2 of its size, which could lead to excessive memory usage when processing big amount of data. For example, scraping a target with hundreds of MBs in response could result into hih memory spikes in vmagent because buffer has to double its size to fit the response. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759 The change smoothes out the growth rate, trading higher allocation rate for lower mem usage at certain conditions. --------- Signed-off-by: hagen1778 (cherry picked from commit f28f496a9df4ae867f8e12537a344720e42b5e8a) --- docs/CHANGELOG.md | 1 + lib/bytesutil/bytebuffer.go | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 833250b5d..b2e4bcf8e 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -32,6 +32,7 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/). * FEATURE: add `/influx/health` health-check handler for Influx endpoints. This is needed as some clients use the health endpoint to determine if the server is healthy and ready for data ingestion. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6653) for the details. * FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl/): add `--vm-backoff-retries`, `--vm-backoff-factor`, `--vm-backoff-min-duration` and `--vm-native-backoff-retries`, `--vm-native-backoff-factor`, `--vm-native-backoff-min-duration` command-line flags. These flags allow to change backoff policy config for import requests to VictoriaMetrics. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6622). * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): allow overriding the `sample_limit` option at [scrape_configs](https://docs.victoriametrics.com/sd_configs/#scrape_configs) when a label `__sample_limit__` is specified for target. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6665). Thanks to @zoglam for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6666). +* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce memory usage when scraping targets with big response body. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759). * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert): respect HTTP headers defined in [notifier configuration file](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file) for each request to notifiers. Previously, this param was ignored by mistake. * BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): correctly apply `-streamAggr.dropInputLabels` when global stream deduplication is enabled without `-streamAggr.config`. Previously, `-remoteWrite.streamAggr.dropInputLabels` was used instead. diff --git a/lib/bytesutil/bytebuffer.go b/lib/bytesutil/bytebuffer.go index 2060cae89..aceb6391e 100644 --- a/lib/bytesutil/bytebuffer.go +++ b/lib/bytesutil/bytebuffer.go @@ -65,7 +65,11 @@ func (bb *ByteBuffer) ReadFrom(r io.Reader) (int64, error) { offset := bLen for { if free := len(b) - offset; free < offset { - b = slicesutil.SetLength(b, 2*len(b)) + // grow slice by 30% similar to how Go does this + // https://go.googlesource.com/go/+/2dda92ff6f9f07eeb110ecbf0fc2d7a0ddd27f9d + // higher growth rates could consume excessive memory when reading big amounts of data. + n := 1.3 * float64(len(b)) + b = slicesutil.SetLength(b, int(n)) } n, err := r.Read(b[offset:]) offset += n