VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin da9ef90277
lib/logstorage: assist merging in-memory parts at data ingestion path if their number starts exceeding maxInmemoryPartsPerPartition
This is a follow-up for 9310e9f584 , which removed data ingestion pacing.
This can result in uncontrolled growth of in-memory parts under high data ingestion rate,
which, in turn, can result in unbounded RAM usage, OOM crashes and slow query performance.

While at it, consistently reset isInMerge field for parts passed to mergeParts() before returning from this function.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4828
2023-10-02 08:24:58 +02:00
..
arena.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
block_data.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_data_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_search.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:16:42 +02:00
block_stream_merger.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
block_stream_reader.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_stream_writer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
consts.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
datadb.go lib/logstorage: assist merging in-memory parts at data ingestion path if their number starts exceeding maxInmemoryPartsPerPartition 2023-10-02 08:24:58 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filters.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:16:42 +02:00
filters_test.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go app/vlstorage: expose vl_data_size_bytes metric at /metrics page for tracking the on-disk data size (both indexdb and the data itself) 2023-07-31 07:56:53 -07:00
indexdb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
log_rows.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
parser.go lib/logstorage: properly encode "offset" search word just after _time filter 2023-07-18 16:00:06 -07:00
parser_test.go lib/logstorage: go fmt after a8000b74c5 2023-07-18 16:04:51 -07:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
partition_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
rows.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
rows_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
storage_search.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage_search_test.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
storage_test.go lib/logstorage: switch to read-only mode when running out of disk space (#4945) 2023-09-29 11:55:38 +02:00
stream_filter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_tags.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tenant_id.go app/vlinsert/loki: follow-up after 09df5b66fd 2023-07-20 16:48:21 -07:00
tenant_id_test.go app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 10:10:55 +02:00
tokenizer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder_test.go lib/logstorage: fix TestValuesEncoder() on 32-bit architectures 2023-07-13 11:27:13 -07:00
values_encoder_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00