VictoriaMetrics/lib/mergeset
Aliaksandr Valialkin 7d5c64eb7a
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:28:09 -08:00
..
block_header.go lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() 2022-11-16 21:22:51 +02:00
block_stream_reader.go lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() 2022-11-16 21:22:51 +02:00
block_stream_reader_test.go lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation 2021-02-21 22:09:10 +02:00
block_stream_writer.go lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 2022-12-03 22:47:06 -08:00
encoding.go lib/mergeset: reduce the time needed for the slowest tests 2022-12-03 22:26:46 -08:00
encoding_test.go lib/mergeset: reduce the time needed for the slowest tests 2022-12-03 22:26:46 -08:00
encoding_timing_test.go benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length 2022-07-27 22:19:26 +03:00
inmemory_part.go all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk 2022-12-05 15:28:09 -08:00
merge.go lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge 2022-07-28 00:05:43 +03:00
merge_test.go lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 2022-12-03 22:47:06 -08:00
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:04 +03:00
part.go lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf 2022-11-16 12:10:15 +02:00
part_header.go lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
part_search.go lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf 2022-11-16 12:10:15 +02:00
part_search_test.go lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 2022-12-03 22:47:06 -08:00
table.go all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk 2022-12-05 15:28:09 -08:00
table_search.go optimized code (#2103) 2022-01-28 12:10:47 +02:00
table_search_test.go lib/mergeset: panic when too long item is passed to Table.AddItems() 2022-12-03 23:37:20 -08:00
table_search_timing_test.go lib/storage: stop background merge when storage enters read-only mode 2022-06-01 14:22:12 +03:00
table_test.go all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk 2022-12-05 15:28:09 -08:00