Artem Fetishev a722ddbdd9 lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267) ... This commit fixes the TestStorageRotateIndexDB flaky test reported at: #6977. Sample test failure: https://pastebin.com/bTSs8HP1 The test fails because one goroutine adds items to the indexDB table while another goroutine is closing that table. This may happen if indexDB rotation happens twice during one Storage.add() operation: - Storage.add() takes the current indexDB and adds index recods to it - First index db rotation makes the current index DB a previous one (still ok at this point) - Second index db rotation removes the indexDB that was current two rotations earlier. It does this by setting the mustDrop flag to true and decrementing the ref counter. The ref counter reaches zero which cases the underlying indexdb table to release its resources gracefully. Graceful release assumes that the table is not written anymore. But Storage.add() still adds items to it. The solution is to increment the indexDB ref counters while it is used inside add(). The unit test has been changed a little so that the test fails reliably. The idea is to make add() function invocation to last much longer, therefore the test inserts not just one record at a time but thouthands of them. To see the test fail, just replace the idbsLocked() func with: ```go unc (s *Storage) idbsLocked2() (*indexDB, *indexDB, func()) {       return s.idbCurr.Load(), s.idbNext.Load(), func() {} } ``` --------- Signed-off-by: Artem Fetishev <rtm@victoriametrics.com> (cherry picked from commit 6b9f57e5f7) Signed-off-by: hagen1778 <roman@victoriametrics.com>		2024-10-24 15:45:54 +02:00
..
appmetrics	all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page	2023-12-19 03:26:02 +02:00
auth	vmselect: add support of multi-tenant queries (#6346)	2024-10-01 16:37:18 +02:00
awsapi	lib/awsapi: properly assume role with webIdentity token (#5495)	2023-12-20 19:07:04 +02:00
backup	lib/backup/s3remote: add retryer configuration (#6747)	2024-08-07 16:59:23 +02:00
blockcache	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:23:26 +02:00
bloomfilter	lib: consistently use atomic.* types instead of atomic.* functions	2024-02-24 02:10:04 +02:00
bufferedwriter	app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils	2023-07-06 17:22:23 -07:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	lib/bytesutil: smooth buffer growth rate (#6761)	2024-08-07 16:59:23 +02:00
cgroup	lib/cgroup: round GOMAXPROCS to the lower integer value of cpuQuota	2024-09-23 16:11:59 +02:00
consts	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage	2022-04-05 15:39:58 +03:00
contextutil	lib/contextutil: make golanci-lint happy by substituing unused function arg name with _	2024-09-26 17:07:25 +02:00
decimal	lib/slicesutil: add helper functions for setting slice length and extending its capacity	2024-05-12 11:33:49 +02:00
encoding	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit	2024-05-14 01:30:25 +02:00
envflag	lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value	2023-08-17 13:37:05 +02:00
envtemplate	allowed using dashes and dots in environment variables names (#4009)	2023-03-24 17:57:19 -07:00
envutil	testing: allow disabling fsync to make tests run faster (#6871)	2024-08-30 11:18:21 +02:00
fastnum	lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader	2024-02-29 17:17:24 +02:00
fasttime	lib: consistently use atomic.* types instead of atomic.* functions	2024-02-24 02:10:04 +02:00
filestream	vlinsert: added opentelemetry logs support	2024-09-03 20:24:01 +02:00
flagutil	docs: follow-up after f0d1db81dc	2024-10-17 11:19:03 -03:00
formatutil	app/vmbackupmanager: add metrics for better observability (#488)	2022-12-20 14:18:43 -08:00
fs	fscore: rollback trailing space trim (#7106)	2024-09-29 14:48:36 +02:00
handshake	lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow	2024-01-23 18:39:28 +02:00
htmlcomponents	lib/htmlcomponents: use relative links for the top page and for favicon.ico	2023-11-13 20:28:17 +01:00
httpserver	app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert	2024-09-06 23:00:58 +02:00
httputils	app/vlinsert: support _time field without timezone information during data ingestion	2024-09-26 12:50:14 +02:00
influxutils	app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749)	2024-08-05 09:45:32 +02:00
ingestserver	Revert c6c5a5a186 and b2765c45d0	2024-07-03 23:57:49 +02:00
leveledbytebufferpool	lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes	2024-06-13 17:02:05 +02:00
logger	app/vlogscli: add interactive command-line tool for querying VictoriaLogs	2024-10-01 12:24:53 +02:00
logstorage	lib/logstorage: use simpler in-memory cache instead of workingsetcache for caching recently ingested _stream values and recently queried set of streams	2024-10-18 11:42:16 +02:00
lrucache	all: consistently use 'any' instead of 'interface{}'	2024-07-10 00:23:26 +02:00
memory	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:15:42 +01:00
mergeset	lib/mergeset: fix typos in comments	2024-08-07 16:59:22 +02:00
metricsql	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
netutil	lib/promscrape: fixes proxy autorization (#6783)	2024-08-19 22:50:39 +02:00
persistentqueue	app/vmagent/remotewrite: follow-up for 87fd400dfc	2024-07-13 02:30:10 +02:00
procutil	all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20	2023-11-13 19:15:42 +01:00
promauth	vmagent: add support of HTTP2 client for Kubernetes SD (#7114)	2024-10-08 10:37:48 +02:00
prompb	Revert "Exemplar support (#5982)"	2024-07-03 16:09:18 +02:00
prompbmarshal	Revert "Exemplar support (#5982)"	2024-07-03 16:09:18 +02:00
promrelabel	lib/promrelabel: follow-up for 8958cecad6	2024-08-27 15:44:07 +02:00
promscrape	lib/promscrape: fixed reload on max_scrape_size change (#7282)	2024-10-18 11:42:47 +02:00
promutils	stream aggregation: fix possible duplicated aggregation results (#7118)	2024-09-30 14:30:34 +02:00
protoparser	lib/protoparser/influx: enable batch processing by default (#7165)	2024-10-15 11:51:48 +02:00
proxy	lib/promscrape: fixes proxy autorization (#6783)	2024-08-19 22:50:39 +02:00
pushmetrics	lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop()	2024-01-16 21:18:22 +02:00
querytracer	make go vet happy	2024-08-19 21:41:44 +02:00
ratelimiter	app/vmagent: properly shutdown when -maxIngestionRate limit is reached	2024-04-03 02:41:11 +03:00
regexutil	lib/logstorage: work-in-progress	2024-05-25 22:59:21 +02:00
slicesutil	lib/slicesutil: add helper functions for setting slice length and extending its capacity	2024-05-12 11:33:49 +02:00
snapshot	lib/httputils: parse URL before creating HTTP transport (#6820)	2024-08-16 11:34:49 +02:00
storage	lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267)	2024-10-24 15:45:54 +02:00
streamaggr	tests: fix slice init length (#6897)	2024-08-30 11:18:21 +02:00
stringsutil	all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter	2024-07-17 14:01:37 +02:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	vmselect: add support of multi-tenant queries (#6346)	2024-10-01 16:37:18 +02:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:30:10 +03:00
timeutil	vmselect: add support of multi-tenant queries (#6346)	2024-10-01 16:37:18 +02:00
uint64set	lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now	2024-07-16 00:00:46 +02:00
vmselectapi	lib: consistently use atomic.* types instead of atomic.* functions	2024-02-24 02:10:04 +02:00
workingsetcache	lib: consistently use atomic.* types instead of atomic.* functions	2024-02-24 02:10:04 +02:00
writeconcurrencylimiter	app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c	2024-03-06 13:57:53 +02:00