From 01520d3e5df05bb51a3e9dc93f0eaf8193ca41be Mon Sep 17 00:00:00 2001 From: Roman Khavronenko Date: Sun, 7 May 2023 22:18:56 +0200 Subject: [PATCH] alerts: update TooHighMemoryUsage threshold (#4256) It appears that 90% usage for anonymous mem usage is already concerning. So we lowering the threshold to 80%. Signed-off-by: hagen1778 --- deployment/docker/alerts-health.yml | 4 ++-- docs/CHANGELOG.md | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/deployment/docker/alerts-health.yml b/deployment/docker/alerts-health.yml index 63f66c8c2..0a58cc853 100644 --- a/deployment/docker/alerts-health.yml +++ b/deployment/docker/alerts-health.yml @@ -34,12 +34,12 @@ groups: Consider to increase the limit as fast as possible." - alert: TooHighMemoryUsage - expr: (process_resident_memory_anon_bytes / vm_available_memory_bytes) > 0.9 + expr: (process_resident_memory_anon_bytes / vm_available_memory_bytes) > 0.8 for: 5m labels: severity: critical annotations: - summary: "It is more than 90% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m" + summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m" description: "Too high memory usage may result into multiple issues such as OOMs or degraded performance. Consider to either increase available memory or decrease the load on the process." diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 3c40433ce..36ccdaa45 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -55,6 +55,7 @@ The following tip changes can be tested by building VictoriaMetrics components f * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue. * BUGFIX: fix indexdb rotation getting in infinite loop when using `retentionTimezoneOffset` and local timezone is not UTC. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207) for details. Thanks to @faceair for the fix. * BUGFIX: max value for `memory.allowedPercent` changed from 200 to 100. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4171). +* BUGFIX: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): update threshold for TooHighMemoryUsage from 90% to 80%. ## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0)