Nikolay 30b32583f4 lib/storage: pre-create timeseries before indexDB rotation (#4652) ... * lib/storage: pre-create timeseries before indexDB rotation during an hour before indexDB rotation start creating records at the next indexDB it must improve performance during switch for the next indexDB and remove ingestion issues. Since there is no need for creation new index records for timeseries already ingested into current indexDB https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 * lib/storage: further work on indexdb rotation optimization - Document the change at docs/CHAGNELOG.md - Move back various caches from indexDB to Storage. This makes the change less intrusive. The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID) entries for both the current and the next indexDB. - Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function. This improves code readability and maintainability a bit. - Rewrite and simplify the code responsible for calculating the next retention timestamp. Add various tests for corner cases of this code. - Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called. It is OK to add indexdb entries on demand in this function. This simplifies the code. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 * docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563 --------- Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>		2023-07-22 15:23:14 -07:00
..
appmetrics	all: replace atomic.Value with atomic.Pointer[T]	2023-07-19 17:48:26 -07:00
auth	app/vminsert: allows parsing tenant id from labels (#3009)	2022-09-30 17:28:35 +03:00
awsapi	.golangci.yml: properly enable revive linter and fix all the warnings it detects	2023-02-26 12:19:58 -08:00
backup	fix removing storage data dir before restoring from backup (#598)	2023-07-06 22:32:12 -07:00
blockcache	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package	2022-06-21 20:24:28 +03:00
bloomfilter	all: replace atomic.Value with atomic.Pointer[T]	2023-07-19 17:48:26 -07:00
bufferedwriter	app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils	2023-07-06 17:22:23 -07:00
buildinfo	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
bytesutil	docs: follow-up after aec4b5db81 (#4638)	2023-07-19 14:48:17 -07:00
cgroup	lib/cgroup: add SetGOGC() function	2023-07-06 17:24:31 -07:00
consts	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage	2022-04-05 15:39:58 +03:00
decimal	lib/decimal: use consistent randomizer in tests	2023-01-23 19:24:05 -08:00
encoding	lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...	2023-07-20 21:54:51 -07:00
envflag	docs: follow-up after aec4b5db81 (#4638)	2023-07-19 14:48:17 -07:00
envtemplate	allowed using dashes and dots in environment variables names (#4009)	2023-03-24 17:57:19 -07:00
fastnum	Makefile: add build and test rules with enabled race detector. These rules have -race suffix	2020-03-05 12:05:16 +02:00
fasttime	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
filestream	lib/filestream: change Create() to MustCreate()	2023-04-14 15:14:24 -07:00
flagutil	lib/flagutil: ArrayString: support commas inside quoted strings and inside [], {} and () braces	2023-03-28 21:25:07 -07:00
formatutil	app/vmbackupmanager: add metrics for better observability (#488)	2022-12-20 14:18:43 -08:00
fs	lib/fs: add ReaderAt.Path() function	2023-07-06 17:25:19 -07:00
handshake	lib/handshake: do not pollute logs with cannot read hello messages on TCP health checks	2023-05-18 10:37:59 -07:00
htmlcomponents	app/vmselect: remove dependency on lib/promscrape from app/vmselect	2023-01-03 23:27:36 -08:00
httpserver	docs: make httpAuth.* flags description less ambiguous (#4588)	2023-07-09 12:36:14 -07:00
httputils	lib/httputils: fix test after b49d04b3dc	2023-07-06 22:21:43 -07:00
influxutils	lib/flagutil: rename Array to ArrayString	2022-10-01 18:28:19 +03:00
ingestserver	lib/netutil: init implimentation of proxy protocol (#3687)	2023-01-26 23:25:22 -08:00
leveledbytebufferpool	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
logger	add error handler for parsing prometheus text format to vmagent and v… (#3693)	2023-01-23 22:36:23 -08:00
logjson	app/vlinsert/jsonline: code prettifying	2023-07-06 21:35:55 -07:00
logstorage	app/vlinsert/loki: follow-up after 09df5b66fd	2023-07-20 21:52:11 -07:00
lrucache	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package	2022-06-21 20:24:28 +03:00
memory	docs: follow-up after aec4b5db81 (#4638)	2023-07-19 14:48:17 -07:00
mergeset	lib/mergeset: skip common prefix in binarySearchKey() function	2023-07-13 22:05:14 -07:00
metricsql	all: make fmt via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
netutil	docs: follow-up after aec4b5db81 (#4638)	2023-07-19 14:48:17 -07:00
persistentqueue	app/vmagent,lib/persistentqueue: show warning message if --remoteWrite.maxDiskUsagePerURL flag lower than 500MB (#4196)	2023-05-08 15:45:21 -07:00
procutil	lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal	2022-10-20 21:58:49 +03:00
promauth	lib/promscrape: disable support for service discovery and metrics scrape via http2	2023-07-06 16:04:31 -07:00
prompb	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 22:53:50 +02:00
prompbmarshal	all: use %w instead of %s for wrapping errors in fmt.Errorf	2020-06-30 23:33:46 +03:00
promrelabel	all: add support for or filters in series selectors	2023-07-15 23:56:18 -07:00
promscrape	lib/promscrape: follow-up after 6aa50ca954	2023-07-20 21:54:09 -07:00
promutils	vmalert: init unit test (#4596)	2023-07-20 21:19:45 -07:00
protoparser	lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString()	2023-05-12 17:02:06 -07:00
proxy	lib/promauth: add ability to send additional http headers in requests to scrape targets	2022-06-22 20:40:50 +03:00
pushmetrics	fixed typos in documentation and commandline flags descriptions (#4275)	2023-05-10 02:22:06 -07:00
querytracer	lib/querytracer: fix remaining tests after 49ebc48809	2022-12-08 18:18:50 -08:00
regexutil	app,lib: fix typos in comments (#3804)	2023-02-13 09:32:35 -08:00
snapshot	app/vmbackup: prevent password leaks (#3672)	2023-01-18 11:40:52 -08:00
storage	lib/storage: pre-create timeseries before indexDB rotation (#4652)	2023-07-22 15:23:14 -07:00
streamaggr	lib/streamaggr: follow up for 70773f53d7	2023-07-20 21:56:36 -07:00
syncwg	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00
tenantmetrics	lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...	2023-07-20 21:54:51 -07:00
timerpool	lib/timerpool: use timer pool in concurrency limiters	2019-05-28 17:30:10 +03:00
uint64set	lib/uint64set: use repeatable randomizer in tests	2023-01-23 19:24:05 -08:00
vmselectapi	lib/vmselectapi: move the code for checking the expected client errors into a isExpectedError() function	2023-07-06 16:37:59 -07:00
workingsetcache	all: replace atomic.Value with atomic.Pointer[T]	2023-07-19 17:48:26 -07:00
writeconcurrencylimiter	lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting vm_concurrent_insert_capacity and vm_concurrent_insert_current metrics	2023-02-07 11:08:39 -08:00