VictoriaMetrics/app/vmselect/netstorage
Artem Fetishev ba0d7dc2fc
Allow disabling per-day index (#6976)
### Describe Your Changes

Allow disabling the per-day index using the `-disablePerDayIndex` flag.
This should significantly improve the ingestion rate and decrease the
disk space usage for the use cases that assume small or no churn rate.
See the docs added to `docs/README.md` for details.

Both improvements are due to no data written to the per-day index.
Benchmark results:

```shell
rm -Rf ./lib/storage/Benchmark*; go test ./lib/storage -run=NONE -bench=BenchmarkStorageInsertWithAndWithoutPerDayIndex --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/perDayIndexes-12                 1        3850268120 ns/op                39.56 data-MiB          28.20 indexdb-MiB           259722 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/noPerDayIndexes-12               1        2916865725 ns/op                39.57 data-MiB          25.73 indexdb-MiB           342834 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/perDayIndexes-12                   1        2218073474 ns/op                 9.772 data-MiB         13.73 indexdb-MiB           450842 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/noPerDayIndexes-12                 1        1295140898 ns/op                 9.771 data-MiB          0.3566 indexdb-MiB         772119 rows/s
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  11.421s
```

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-02-14 12:35:51 +01:00
..
netstorage.go Allow disabling per-day index (#6976) 2025-02-14 12:35:51 +01:00
netstorage_test.go app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks() 2023-01-09 21:08:51 -08:00
netstorage_timing_test.go app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks() 2023-01-09 21:08:51 -08:00
tmp_blocks_file.go app/vmselect:add command-line flag -search.inmemoryBufSizeBytes (#6869) 2024-08-26 14:48:53 +02:00