From be313763496dbe56ea047243dc642bf6140240c4 Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Sat, 1 Apr 2023 08:50:27 +0200 Subject: [PATCH] lib/storage: check for free disk space before opening tables (#4035) * lib/storage: check for free disk space before opening tables We check for free disk space before call to `openTable`, so `Storage` can be set to ReadOnly before mergeWorkers start. Before the change, there was a chance that merges will start even if Storage has to start in ReadOnly mode because of `-storage.minFreeDiskSpaceBytes` limit. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023 Signed-off-by: hagen1778 * lib/storage: chore Signed-off-by: hagen1778 * Update lib/storage/storage.go --------- Signed-off-by: hagen1778 Co-authored-by: Aliaksandr Valialkin --- docs/CHANGELOG.md | 1 + lib/storage/storage.go | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f30455603..5e06bda97 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -18,6 +18,7 @@ The following tip changes can be tested by building VictoriaMetrics components f * SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See [alpine 3.17.3 release notes](https://alpinelinux.org/posts/Alpine-3.17.3-released.html). * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly convert [VictoriaMetrics historgram buckets](https://valyala.medium.com/improving-histogram-usability-for-prometheus-and-grafana-bc7e5df0e350) to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021). +* BUGFIX: prevent unexpected merges on start-up when `-storage.minFreeDiskSpaceBytes` is set. See [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023). * BUGFIX: properly support comma-separated filters inside [retention filters](https://docs.victoriametrics.com/#retention-filters). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915). * BUGFIX: verify response code when fetching configuration files via HTTP. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4034). diff --git a/lib/storage/storage.go b/lib/storage/storage.go index c74bec43e..72f50c50f 100644 --- a/lib/storage/storage.go +++ b/lib/storage/storage.go @@ -261,6 +261,10 @@ func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySer s.setDeletedMetricIDs(dmisCurr) s.updateDeletedMetricIDs(dmisPrev) + // check for free disk space before opening the table + // to prevent unexpected part merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023 + s.startFreeDiskSpaceWatcher() + // Load data tablePath := path + "/data" tb, err := openTable(tablePath, s) @@ -273,7 +277,6 @@ func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySer s.startCurrHourMetricIDsUpdater() s.startNextDayMetricIDsUpdater() s.startRetentionWatcher() - s.startFreeDiskSpaceWatcher() return s, nil }