VictoriaMetrics/lib/filestream
Aliaksandr Valialkin 53849c95b7
lib/filestream: use smaller sizes for read buffers than for write buffers
The number of filestream readers is proportional to the number of parts to be merged,
while the number of filestream writers is proportional to the number of concurrent merges.
Usually around 4-16 parts are merged at once, so the number of active filestream readers is ~8x
bigger than the number of active filestream writers.

So it is a good idea to use smaller size of read buffers comparing to the size of write buffers.
Limit read buffer size by 64Kb, while write buffer size is limited by 128Kb.
This should reduce the overall memory usage when merging parts with big number of files.
This is the case for VictoriaLogs, which works with logs containing hundreds of fields (aka wide events).

(cherry picked from commit 659251beaa)
2025-02-19 13:30:03 +01:00
..
filestream.go lib/filestream: use smaller sizes for read buffers than for write buffers 2025-02-19 13:30:03 +01:00
filestream_darwin.go vlinsert: added opentelemetry logs support 2024-09-03 20:24:01 +02:00
filestream_freebsd.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:33:46 +03:00
filestream_linux.go all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:33:46 +03:00
filestream_openbsd.go lib/{fs,filestream}: small consistency-related updates after cc90a548b1 2020-09-29 00:43:20 +03:00
filestream_solaris.go fixes solaris build () 2021-06-04 11:56:06 +03:00
filestream_test.go lib/filestream: change Create() to MustCreate() 2023-04-14 15:14:24 -07:00
filestream_windows.go all: follow-up after 34634ec357 2023-03-25 12:00:48 -07:00