Aliaksandr Valialkin
f14412321b
lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value
2021-04-27 00:26:32 +03:00
Aliaksandr Valialkin
4d6a3aba4d
lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030
2021-04-05 19:26:51 +03:00
Aliaksandr Valialkin
d98a2f217b
lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964
2020-12-14 19:25:53 +02:00
Aliaksandr Valialkin
90d2549428
lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue
2020-09-23 02:17:53 +03:00
Aliaksandr Valialkin
df547bf345
lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush
2020-09-19 12:51:46 +03:00
Aliaksandr Valialkin
adb4d3b75c
lib/persistentqueue: flush data to disk every second
...
Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-18 13:06:03 +03:00
Aliaksandr Valialkin
eee6f1e56d
lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset)
...
The strategy is:
- Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown.
- Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data.
- Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown.
- Skipping of broken chunk file if it cannot be read.
- Fsyncing finalized chunk files.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-16 18:13:24 +03:00
Roman Khavronenko
605711bde5
lib/persistentqueue: add vm_persistentqueue_bytes_pending metric ( #619 )
...
Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount
of bytes in persistentqueue flushed on disk as a difference between write and read
offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes`
except of accounting for bytes in-memory.
2020-07-13 21:54:54 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
dc16cdd1ca
lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484
2020-05-16 09:32:30 +03:00
Aliaksandr Valialkin
f01d1bf4a8
app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
2020-03-03 19:49:20 +02:00
Aliaksandr Valialkin
c2e602286c
lib/persistentqueue: reset chunk file when the persistent queue is empty
2020-02-28 20:06:59 +02:00
Aliaksandr Valialkin
7ee7614e90
app/vmagent: initial implementation for vmagent
2020-02-23 17:31:54 +02:00
