Aliaksandr Valialkin 12698b9136 lib/mergeset: really limit the number of in-memory parts to 15 ... It has been appeared that the registration of new time series slows down linearly with the number of indexdb parts, since VictoriaMetrics needs to check every indexdb part when it searches for TSID by newly ingested metric name. The number of in-memory parts grows when new time series are registered at high rate. The number of in-memory parts grows faster on systems with big number of CPU cores, because the mergeset maintains per-CPU buffers with newly added entries for the indexdb, and every such entry is transformed eventually into a separate in-memory part. The solution has been suggested in https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 by @misutoth - to limit the number of in-memory parts with buffered channel. This solution is implemented in this commit. Additionally, this commit merges per-CPU parts into a single part before adding it to the list of in-memory parts. This reduces CPU load when searching for TSID by newly ingested metric name. The https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212 recommends setting the limit on the number of in-memory parts to 100, but my internal testing shows that much lower limit 15 works with the same efficiency on a system with 16 CPU cores while reducing memory usage for `indexdb/dataBlocks` cache by up to 50%. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190		2024-01-24 03:41:19 +02:00
..
appmetrics	all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page	2023-12-19 03:26:02 +02:00
auth	lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant	2023-08-30 14:13:51 +02:00
awsapi	lib/awsapi: properly assume role with webIdentity token (#5495)	2023-12-20 19:07:04 +02:00
backup	vendor: run make vendor-update	2023-12-11 10:48:47 +02:00
blockcache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:39:16 +02:00
bloomfilter	all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time	2024-01-22 01:13:41 +02:00
bufferedwriter	app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils	2023-07-06 17:22:23 -07:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args	2023-11-13 09:43:49 +01:00
cgroup	lib/cgroup: add SetGOGC() function	2023-07-06 17:24:31 -07:00
consts	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage	2022-04-05 15:39:58 +03:00
decimal	lib/decimal: use consistent randomizer in tests	2023-01-23 19:24:05 -08:00
encoding	lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint*	2024-01-23 22:50:11 +02:00
envflag	lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value	2023-08-17 13:37:05 +02:00
envtemplate	allowed using dashes and dots in environment variables names (#4009)	2023-03-24 17:57:19 -07:00
fastnum	Makefile: add build and test rules with enabled race detector. These rules have -race suffix	2020-03-05 12:05:16 +02:00
fasttime	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
filestream	lib/filestream: do not measure read / write duration from / to in-memory buffers	2024-01-23 14:53:35 +02:00
flagutil	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
formatutil	app/vmbackupmanager: add metrics for better observability (#488)	2022-12-20 14:18:43 -08:00
fs	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
handshake	lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow	2024-01-23 18:39:28 +02:00
htmlcomponents	lib/htmlcomponents: use relative links for the top page and for favicon.ico	2023-11-13 20:28:17 +01:00
httpserver	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
httputils	lib/httputils: handle step=undefined query arg as an empty value	2024-01-17 00:13:04 +02:00
influxutils	lib/flagutil: rename Array to ArrayString	2022-10-01 18:28:19 +03:00
ingestserver	lib/ingestserver: properly log the number of closed connections	2023-11-14 21:53:10 +01:00
leveledbytebufferpool	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
logger	lib/logger: increase default -loggerMaxArgLen command-line flag value from 500 to 1000	2023-11-14 19:55:55 +01:00
logjson	app/vlinsert/jsonline: code prettifying	2023-07-06 21:35:55 -07:00
logstorage	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:39:16 +02:00
lrucache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:39:16 +02:00
memory	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:15:42 +01:00
mergeset	lib/mergeset: really limit the number of in-memory parts to 15	2024-01-24 03:41:19 +02:00
metricsql	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
netutil	app/vmselect: add support for vmstorage groups with independent -replicationFactor per group	2023-12-13 00:14:34 +02:00
persistentqueue	app/vmagent: follow-up for 090cb2c9de	2023-11-25 12:13:39 +02:00
procutil	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:15:42 +01:00
promauth	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
prompb	lib/prompbmarshal: switch to github.com/VictoriaMetrics/easyproto	2024-01-16 20:48:30 +02:00
prompbmarshal	lib/prompbmarshal: move WriteRequest proto definition to the correct place	2024-01-16 21:57:03 +02:00
promrelabel	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
promscrape	lib/promscrape: respect 0 value for series_limit param (#5663)	2024-01-23 13:09:36 +02:00
promutils	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-22 01:11:59 +02:00
protoparser	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-22 01:11:59 +02:00
proxy	lib/promauth: follow-up for e16d3f5639	2023-10-26 09:55:47 +02:00
pushmetrics	lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop()	2024-01-16 21:18:22 +02:00
querytracer	lib/querytracer: add missing blank comment line after 3121d76bee	2023-11-15 16:11:50 +01:00
regexutil	lib/regexutil: properly handle alternate regexps surrounded by .+ or .*	2023-11-13 18:25:57 +01:00
snapshot	Makefile: update golangci-lint version from v1.54.2 to v1.55.1	2023-11-02 21:42:35 +01:00
storage	lib/{storage,mergeset}: reduce the maxium compression level for the stored data	2024-01-23 17:47:40 +02:00
streamaggr	all: allow dynamically reading *AuthKey flag values from files and urls	2024-01-22 01:23:23 +02:00
stringsutil	lib/stringsutil: add tests for LimitStringLen() function	2023-11-13 10:33:07 +01:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...	2023-07-20 21:54:51 -07:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:30:10 +03:00
timeutil	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:39:16 +02:00
uint64set	lib/uint64: remove accidentally added test	2024-01-09 13:32:22 +01:00
vmselectapi	vmcluster: re-routing enhancement (#5293)	2023-11-14 01:00:42 +01:00
workingsetcache	all: add up to 10% random jitter to the interval between periodic tasks performed by various components	2024-01-22 18:39:16 +02:00
writeconcurrencylimiter	lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting vm_concurrent_insert_capacity and vm_concurrent_insert_current metrics	2023-02-07 11:08:39 -08:00