VictoriaMetrics/app
Zakhar Bessarab 837d0d136d
lib/mergeset: add sparse indexdb cache (#7269)
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182

- add a separate index cache for searches which might read through large
amounts of random entries. Primary use-case for this is retention and
downsampling filters, when applying filters background merge needs to
fetch large amount of random entries which pollutes an index cache.
Using different caches allows to reduce effect on memory usage and cache
efficiency of the main cache while still having high cache hit rate. A
separate cache size is 5% of allowed memory.

- reduce size of indexdb/dataBlocks cache in order to free memory for
new sparse cache. Reduced size by 5% and moved this to a separate cache.

- add a separate metricName search which does not cache metric names -
this is needed in order to allow disabling metric name caching when
applying downsampling/retention filters. Applying filters during
background merge accesses random entries, this fills up cache and does
not provide an actual improvement due to random access nature.


Merge performance and memory usage stats before and after the change:

- before

![image](https://github.com/user-attachments/assets/485fffbb-c225-47ae-b5c5-bc8a7c57b36e)


- after

![image](https://github.com/user-attachments/assets/f4ba3440-7c1c-4ec1-bc54-4d2ab431eef5)

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-24 15:21:17 +02:00
..
victoria-logs app/vlstorage: add support for forced merge via /internal/force_merge HTTP endpoint 2024-10-13 22:20:31 +02:00
victoria-metrics app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:12:17 +02:00
vlinsert app/vlselect: do not show empty fields in query results 2024-10-14 23:43:58 +02:00
vlogscli app/vlogscli: add -accountID and -projectID command-line flags for querying the given tenants 2024-10-09 12:56:49 +02:00
vlogsgenerator all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 13:52:13 +02:00
vlselect app/vlselect/vmui: run make vmui-logs-update after 423df09d7d 2024-10-18 02:33:41 +02:00
vlstorage lib/flagutil: rename Duration to RetentionDuration (#7284) 2024-10-17 13:47:48 +02:00
vmagent vmagent: fix type of command-line flag -streamAggr.dedupInterval (#7081) 2024-10-17 13:27:59 +02:00
vmalert app/vmalert: update -remoteWrite.concurrency and -remoteWrite.flushInterval (#7272) 2024-10-22 14:43:55 +02:00
vmalert-tool vmalert-tool: add more syntax checks for input_series and exp_samples (#7263) 2024-10-17 11:00:34 +02:00
vmauth app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:12:17 +02:00
vmbackup app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:12:17 +02:00
vmbackupmanager all: replace old https://docs.victoriametrics.com/vmbackupmanager.html url with the new one - https://docs.victoriametrics.com/vmbackupmanager/ 2024-04-18 02:03:58 +02:00
vmctl app/vmctl: fix match expression for vm-native protocol with --vm-native-disable-per-metric-migration flag enabled (#7310) 2024-10-24 14:57:58 +02:00
vmgateway all: replace old https://docs.victoriametrics.com/vmgateway.html url with the new one - https://docs.victoriametrics.com/vmgateway/ 2024-04-18 02:08:18 +02:00
vminsert lib/protoparser/influx: enable batch processing by default (#7165) 2024-10-15 11:48:40 +02:00
vmrestore app/*/multiarch: return back empty value for TARGETARCH 2024-09-06 18:12:17 +02:00
vmselect app/vmui: add missing assets after a710d43a20 2024-10-18 19:52:25 +02:00
vmstorage lib/mergeset: add sparse indexdb cache (#7269) 2024-10-24 15:21:17 +02:00
vmui vmui/logs: add ability to hide hits chart (#7206) 2024-10-18 02:30:56 +02:00