Aliaksandr Valialkin
5a36e241f4
lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase
...
This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981
2021-12-20 17:25:11 +02:00
Aliaksandr Valialkin
b688960db0
lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent
2021-11-17 16:41:35 +02:00
Aliaksandr Valialkin
4401464c22
all: add support for Prometheus staleness markers
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin
60947fb2d5
lib/persistentqueue: eliminate possible data race when obtaining vm_persistentqueue_bytes_pending metric value
2021-04-27 00:25:52 +03:00
Aliaksandr Valialkin
95dbebf512
lib/persistentqueue: delete corrupted persistent queue instead of throwing a fatal error
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1030
2021-04-05 19:26:11 +03:00
Aliaksandr Valialkin
d006b41eff
lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964
2020-12-14 19:25:25 +02:00
Aliaksandr Valialkin
c5ef0e6327
lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue
2020-09-23 02:17:47 +03:00
Aliaksandr Valialkin
f00e0e0103
lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush
2020-09-19 12:51:41 +03:00
Aliaksandr Valialkin
70c721c01b
lib/persistentqueue: flush data to disk every second
...
Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-18 13:05:40 +03:00
Aliaksandr Valialkin
d455764a6f
lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset)
...
The strategy is:
- Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown.
- Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data.
- Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown.
- Skipping of broken chunk file if it cannot be read.
- Fsyncing finalized chunk files.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-16 18:13:44 +03:00
Roman Khavronenko
19dd121968
lib/persistentqueue: add vm_persistentqueue_bytes_pending metric ( #619 )
...
Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount
of bytes in persistentqueue flushed on disk as a difference between write and read
offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes`
except of accounting for bytes in-memory.
2020-07-13 21:54:09 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
4e850cd6a7
lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484
2020-05-16 09:31:46 +03:00
Aliaksandr Valialkin
76036c1897
app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin
e22fdc1073
lib/persistentqueue: reset chunk file when the persistent queue is empty
2020-02-28 20:05:53 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00
