VictoriaMetrics/lib/persistentqueue
Nikolay 090cb2c9de
app/vmagent: allow to disabled on-disk persistence (#5088)
* app/vmagent: allow to disabled on-disk queue
Previously, it wasn't possible to build data processing pipeline with a
chain of vmagents. In case when remoteWrite for the last vmagent in the
chain wasn't accessible, it persisted data only when it has enough disk
capacity. If disk queue is full, it started to silently drop ingested
metrics.

New flags allows to disable on-disk persistent and immediatly return an
error if remoteWrite is not accessible anymore. It blocks any writes and
notify client, that data ingestion isn't possible.

Main use case for this feature - use external queue such as kafka for
data persistence.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2110

* adds test, updates readme

* apply review suggestions

* update docs for vmagent

* makes linter happy

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-11-24 13:42:11 +01:00
..
fastqueue.go app/vmagent: allow to disabled on-disk persistence (#5088) 2023-11-24 13:42:11 +01:00
fastqueue_test.go app/vmagent: allow to disabled on-disk persistence (#5088) 2023-11-24 13:42:11 +01:00
fastqueue_timing_test.go app/vmagent: allow to disabled on-disk persistence (#5088) 2023-11-24 13:42:11 +01:00
filenames.go lib/{fs,persistentqueue}: use filepath.Join() instead of concatenating path parts with / 2023-04-13 20:13:45 -07:00
persistentqueue.go lib/persistentqueue: properly re-create flock.lock file inside directory if persistent queue is broken. 2023-10-31 18:38:32 +01:00
persistentqueue_test.go app/vmagent,lib/persistentqueue: show warning message if --remoteWrite.maxDiskUsagePerURL flag lower than 500MB (#4196) 2023-04-26 13:23:01 +03:00
persistentqueue_timing_test.go lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error 2021-04-05 19:26:11 +03:00