VictoriaMetrics/app
Nikolay 30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652)
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
..
victoria-logs app/vlselect: handle vmui at /select/vmui path instead of /vmui 2023-07-06 21:36:28 -07:00
vlinsert app/vlinsert/loki: fix build for architectures where int is 32-bit 2023-07-20 21:55:28 -07:00
vlselect app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI 2023-07-20 21:53:51 -07:00
vlstorage docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vmagent lib/promscrape: follow-up after 6aa50ca954 2023-07-20 21:54:09 -07:00
vmalert app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687) 2023-07-22 14:18:52 -07:00
vmauth docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vmbackup docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vmbackupmanager docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vmctl docs: mention the simplest way to migrate data in vmctl docs 2023-07-20 21:12:18 -07:00
vmgateway docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vminsert all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
vmrestore docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
vmselect app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI 2023-07-20 21:53:51 -07:00
vmstorage lib/storage: pre-create timeseries before indexDB rotation (#4652) 2023-07-22 15:23:14 -07:00
vmui fix: change getting serverUrl for vmui-logs (#4604) (#4677) 2023-07-20 21:53:30 -07:00