VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin 7b33a27874
lib/logstorage: follow-up for 8a23d08c21
- Compare the actual free disk space to the value provided via -storage.minFreeDiskSpaceBytes
  directly inside the Storage.IsReadOnly(). This should work fast in most cases.
  This simplifies the logic at lib/storage.

- Do not take into account -storage.minFreeDiskSpaceBytes during background merges, since
  it results in uncontrolled growth of small parts when the free disk space approaches -storage.minFreeDiskSpaceBytes.
  The background merge logic uses another mechanism for determining whether there is enough
  disk space for the merge - it reserves the needed disk space before the merge
  and releases it after the merge. This prevents from out of disk space errors during background merge.

- Properly handle corner cases for flushing in-memory data to disk when the storage
  enters read-only mode. This is better than losing the in-memory data.

- Return back Storage.MustAddRows() instead of Storage.AddRows(),
  since the only case when AddRows() can return error is when the storage is in read-only mode.
  This case must be handled by the caller by calling Storage.IsReadOnly()
  before adding rows to the storage.
  This simplifies the code a bit, since the caller of Storage.MustAddRows() shouldn't handle
  errors returned by Storage.AddRows().

- Properly store parsed logs to Storage if parts of the request contain invalid log lines.
  Previously the parsed logs could be lost in this case.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4945
2023-10-02 16:52:23 +02:00
..
arena.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
block_data.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_data_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_search.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:16:42 +02:00
block_stream_merger.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
block_stream_reader.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_stream_writer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
consts.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
datadb.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filters.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
filters_test.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go app/vlstorage: expose vl_data_size_bytes metric at /metrics page for tracking the on-disk data size (both indexdb and the data itself) 2023-07-31 07:56:53 -07:00
indexdb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
log_rows.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
parser.go lib/logstorage: properly encode "offset" search word just after _time filter 2023-07-18 16:00:06 -07:00
parser_test.go lib/logstorage: go fmt after a8000b74c5 2023-07-18 16:04:51 -07:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
partition_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
rows.go lib/logstorage: prevent from panic during background merge (#4969) 2023-09-29 11:58:20 +02:00
rows_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
storage_search.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
storage_search_test.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
storage_test.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
stream_filter.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_id_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
stream_tags.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tenant_id.go app/vlinsert/loki: follow-up after 09df5b66fd 2023-07-20 16:48:21 -07:00
tenant_id_test.go app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 10:10:55 +02:00
tokenizer.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder_test.go lib/logstorage: fix TestValuesEncoder() on 32-bit architectures 2023-07-13 11:27:13 -07:00
values_encoder_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00