github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673)

Browse source 
Workers count for merges affects the max part size during merges. Such behaviour
protects storage from running out of disk space for scenario when all workers
are merging parts with the max size.

This works very well for most cases. But for systems where high number of CPUs
is allocated for vmstorage components this could significantly impact the max
part size and result in more unmerged parts than expected.

While checking multiple production highly loaded setups it was discovered that
`max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2,
and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4.
The change in this commit limits the max value for concurrency accordingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
This commit is contained in:
Roman Khavronenko 2022-06-07 13:55:09 +02:00 committed by GitHub
parent 194258c7b4
commit 1ee1e986da
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 10 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
lib/storage/partition.go
View file

@ -869,10 +869,18 @@ func hasActiveMerges(pws []*partWrapper) bool {
}
var (
bigMergeWorkersCount = (cgroup.AvailableCPUs() + 1) / 2
smallMergeWorkersCount = (cgroup.AvailableCPUs() + 1) / 2
bigMergeWorkersCount = getDefaultMergeConcurrency(4)
smallMergeWorkersCount = getDefaultMergeConcurrency(8)
)
func getDefaultMergeConcurrency(max int) int {
v := (cgroup.AvailableCPUs() + 1) / 2
if v > max {
v = max
}
return v
}
// SetBigMergeWorkersCount sets the maximum number of concurrent mergers for big blocks.
//
// The function must be called before opening or creating any storage.