Aliaksandr Valialkin bc7cf4950b lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode ... While fasthttp.Client uses less CPU and RAM when scraping targets with small responses (up to 10K metrics), it doesn't work well when scraping targets with big responses such as kube-state-metrics. In this case it could use big amounts of additional memory comparing to net/http.Client, since fasthttp.Client reads the full response in memory and then tries re-using the large buffer for further scrapes. Additionally, fasthttp.Client-based scraping had various issues with proxying, redirects and scrape timeouts like the following ones: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5425 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017 This should help reducing memory usage for the case when target returns big response and this response is scraped by fasthttp.Client at first before switching to stream parsing mode for subsequent scrapes. Now the switch to stream parsing mode is performed on the first scrape after reading the response body in memory and noticing that its size exceeds the value passed to -promscrape.minResponseSizeForStreamParse command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567 Overrides https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931		2024-01-30 18:39:10 +02:00
..
appmetrics	all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page	2023-12-19 03:20:40 +02:00
auth	lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant	2023-08-30 14:17:55 +02:00
awsapi	lib/awsapi: properly assume role with webIdentity token (#5495)	2023-12-20 19:05:39 +02:00
backup	vendor: run make vendor-update	2023-12-11 10:48:36 +02:00
blockcache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:40:32 +02:00
bloomfilter	all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time	2024-01-21 14:04:54 +02:00
bufferedwriter	app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils	2023-06-19 22:34:20 -07:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args	2023-11-11 12:30:08 +01:00
cgroup	lib/cgroup: add SetGOGC() function	2023-06-19 22:39:00 -07:00
decimal	lib/decimal: follow-up for e6bad5174f	2024-01-27 00:08:57 +01:00
encoding	lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint*	2024-01-23 22:50:29 +02:00
envflag	lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value	2023-08-17 13:26:53 +02:00
envtemplate	allowed using dashes and dots in environment variables names (#4009)	2023-03-24 15:43:05 -07:00
fastnum	Makefile: add build and test rules with enabled race detector. These rules have -race suffix	2020-03-05 12:03:38 +02:00
fasttime	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
filestream	lib/filestream: do not measure read / write duration from / to in-memory buffers	2024-01-23 14:52:22 +02:00
flagutil	add inserting comma inside value instruction to flag description (#5666)	2024-01-26 22:46:49 +01:00
formatutil	app/vmbackupmanager: add metrics for better observability (#488)	2022-12-20 14:18:06 -08:00
fs	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-21 22:03:38 +02:00
htmlcomponents	lib/htmlcomponents: use relative links for the top page and for favicon.ico	2023-11-13 20:29:05 +01:00
httpserver	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-21 22:03:38 +02:00
httputils	lib/httputils: handle step=undefined query arg as an empty value	2024-01-16 18:59:26 +02:00
influxutils	lib/flagutil: rename Array to ArrayString	2022-10-01 18:26:36 +03:00
ingestserver	lib/ingestserver: properly log the number of closed connections	2023-11-14 21:53:24 +01:00
leveledbytebufferpool	all: make fmt via the upcoming Go1.19	2022-07-11 19:22:15 +03:00
logger	lib/logger: increase default -loggerMaxArgLen command-line flag value from 500 to 1000	2023-11-14 19:52:27 +01:00
logjson	app/vlinsert/jsonline: code prettifying	2023-06-21 19:39:22 -07:00
logstorage	lib/logstorage: make sure that WaitGroup.Add isnt called after stopCh is closed and WaitGroup.Wait is called	2024-01-26 21:17:02 +01:00
lrucache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:40:32 +02:00
memory	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:12:51 +01:00
mergeset	lib/{mergeset,storage}: make background merge more responsive and scalable	2024-01-26 22:27:47 +01:00
metricsql	all: make fmt via the upcoming Go1.19	2022-07-11 19:22:15 +03:00
netutil	all: consistently use %w instead of %s in when error is passed to fmt.Errorf()	2023-10-25 21:24:03 +02:00
persistentqueue	app/vmagent: follow-up for 090cb2c9de	2023-11-25 12:09:44 +02:00
procutil	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:12:51 +01:00
promauth	lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode	2024-01-30 18:39:10 +02:00
prompb	lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto	2024-01-14 23:04:45 +02:00
prompbmarshal	lib/prompbmarshal: move WriteRequest proto definition to the correct place	2024-01-16 00:20:37 +02:00
promrelabel	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-21 22:03:38 +02:00
promscrape	lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode	2024-01-30 18:39:10 +02:00
promutils	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-21 05:11:05 +02:00
protoparser	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-21 05:11:05 +02:00
proxy	lib/promscrape: use the standard net/http.Client instead of fasthttp.Client for scraping targets in non-streaming mode	2024-01-30 18:39:10 +02:00
pushmetrics	lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop()	2024-01-15 13:50:36 +02:00
querytracer	lib/querytracer: add missing blank comment line after 3121d76bee	2023-11-15 16:10:43 +01:00
regexutil	lib/regexutil: properly handle alternate regexps surrounded by .+ or .*	2023-11-13 18:23:38 +01:00
snapshot	Makefile: update golangci-lint version from v1.54.2 to v1.55.1	2023-11-10 20:23:48 +01:00
storage	lib/storage: keep (date, metricID) entries only for the last two dates	2024-01-29 18:43:59 +01:00
streamaggr	lib/streamaggr: fix incorrect err message for min interval value	2024-01-29 09:53:05 +01:00
stringsutil	lib/stringsutil: add tests for LimitStringLen() function	2023-11-13 10:32:33 +01:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...	2023-07-20 20:56:11 -07:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:20:10 +03:00
timeutil	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:40:32 +02:00
uint64set	lib/uint64set: use repeatable randomizer in tests	2023-01-23 19:22:58 -08:00
workingsetcache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:40:32 +02:00
writeconcurrencylimiter	lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting vm_concurrent_insert_capacity and vm_concurrent_insert_current metrics	2023-02-07 11:08:17 -08:00