VictoriaMetrics/lib
Roman Khavronenko f28f496a9d
lib/bytesutil: smooth buffer growth rate (#6761)
Before, buffer growth was always x2 of its size, which could lead to
excessive memory usage when processing big amount of data.
For example, scraping a target with hundreds of MBs in response could
result into hih memory spikes in vmagent because buffer has to double
its size to fit the response. See
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759

The change smoothes out the growth rate, trading higher allocation rate
for lower mem usage at certain conditions.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-07 16:49:43 +02:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:20:40 +02:00
auth lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant 2023-08-30 14:17:55 +02:00
awsapi lib/awsapi: properly assume role with webIdentity token (#5495) 2023-12-20 19:05:39 +02:00
backup lib/backup/s3remote: fix typos (#6694) 2024-07-29 14:18:31 +02:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-06-19 22:34:20 -07:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil lib/bytesutil: smooth buffer growth rate (#6761) 2024-08-07 16:49:43 +02:00
cgroup lib/cgroup: remove SetGOGC() function 2024-02-05 12:11:08 +02:00
decimal lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:32:17 +02:00
encoding lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:26:53 +02:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 15:43:05 -07:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:13 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
filestream lib/filestream: do not measure read / write duration from / to in-memory buffers 2024-01-23 14:52:22 +02:00
flagutil lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag() 2024-07-16 01:00:45 +02:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:06 -08:00
fs fix typos in comments 2024-08-06 14:54:49 +02:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:29:05 +01:00
httpserver all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 13:52:13 +02:00
httputils lib/{httputils,netutil}: move httputils.GetStatDialFunc to netutil.NewStatDialFunc 2024-07-15 23:02:34 +02:00
influxutils app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749) 2024-08-05 09:34:54 +02:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:51:56 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 16:56:25 +02:00
logger all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
logstorage app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:17 +02:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:12:51 +01:00
mergeset lib/mergeset: fix typos in comments 2024-08-07 15:54:15 +02:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:22:15 +03:00
netutil lib/netutil: validate TLS cert and key files immediately (#6621) 2024-07-29 13:58:53 +02:00
persistentqueue app/vmagent/remotewrite: follow-up for 87fd400dfc 2024-07-13 02:25:19 +02:00
procutil all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:12:51 +01:00
promauth all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
prompb Revert "Exemplar support (#5982)" 2024-07-03 15:30:21 +02:00
prompbmarshal Revert "Exemplar support (#5982)" 2024-07-03 15:30:21 +02:00
promrelabel lib: consistently use regexp.Regexp.ReplaceAllLiteralString instead of regexp.Regexp.ReplaceAllString in places where the replacement cannot contain matching group placeholders 2024-07-17 12:41:54 +02:00
promscrape app/vmagent: read __sample_limit__ from labels (#6665) (#6666) 2024-08-07 09:36:14 +02:00
promutils all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
protoparser all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 13:52:13 +02:00
proxy all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-15 13:50:36 +02:00
querytracer lib: consistently use regexp.Regexp.ReplaceAllLiteralString instead of regexp.Regexp.ReplaceAllString in places where the replacement cannot contain matching group placeholders 2024-07-17 12:41:54 +02:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-03-30 06:43:48 +02:00
regexutil lib/logstorage: work-in-progress 2024-05-25 22:59:13 +02:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:32:17 +02:00
snapshot lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:40:50 +02:00
storage Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:32:09 +02:00
streamaggr lib/streamaggr: remove resetState arg from aggrState.flushState() 2024-08-07 11:39:14 +02:00
stringsutil all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 13:52:13 +02:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 2023-07-20 20:56:11 -07:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:20:10 +03:00
timeutil all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:40:32 +02:00
uint64set lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now 2024-07-15 23:59:20 +02:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:43:08 +02:00