From 59a570f5b9f4332e4326c1016e4826966775c5d7 Mon Sep 17 00:00:00 2001 From: Nikolay Date: Thu, 26 Jan 2023 17:05:20 +0100 Subject: [PATCH] lib/storage: properly release parts inMerge lock (#3711) if storage doesn't have enough disk space, finalDedupWatcher holds inMerge lock for all parts and never release it until storage restart --- docs/CHANGELOG.md | 1 + lib/storage/partition.go | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 1bc7c31d9..9b78e5032 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -15,6 +15,7 @@ The following tip changes can be tested by building VictoriaMetrics components f ## v1.79.x long-time support release (LTS) +* BUGFIX: release `inMerge` parts lock properly at `finalDedupWatcher`. It prevented partition merges until storage restart if storage didn't have enough disk space for final deduplication and down-sampling. * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): update API version for [ec2_sd_configs](https://docs.victoriametrics.com/sd_configs.html#ec2_sd_configs) to fix [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3700) with missing `__meta_ec2_availability_zone_id` attribute. * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs.html#dockerswarm_sd_configs): apply `filters` only to objects of the specified `role`. Previously filters were applied to all the objects, which could cause errors when different types of objects were used with filters that were not compatible with them. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579). * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): suppress all the scrape errors when `-promscrape.suppressScrapeErrors` is enabled. Previously some scrape errors were logged even if `-promscrape.suppressScrapeErrors` flag was set. diff --git a/lib/storage/partition.go b/lib/storage/partition.go index 768947c6f..de7fb9c86 100644 --- a/lib/storage/partition.go +++ b/lib/storage/partition.go @@ -837,6 +837,7 @@ func (pt *partition) ForceMergeAllParts() error { if newPartSize > maxOutBytes { freeSpaceNeededBytes := newPartSize - maxOutBytes forceMergeLogger.Warnf("cannot initiate force merge for the partition %s; additional space needed: %d bytes", pt.name, freeSpaceNeededBytes) + pt.releasePartsToMerge(pws) return nil }