From 939c8b83722663db068230ee795f45babcaa56ba Mon Sep 17 00:00:00 2001 From: Haleygo Date: Thu, 20 Jul 2023 21:07:10 +0800 Subject: [PATCH] vmalert: init unit test (#4596) vmalert: support unit tests See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945 --------- Signed-off-by: hagen1778 Co-authored-by: hagen1778 --- app/vmalert/README.md | 244 +++++++++++++ app/vmalert/datasource/datasource.go | 70 ++++ app/vmalert/group.go | 4 +- app/vmalert/main.go | 2 +- app/vmalert/manager.go | 2 +- app/vmalert/manager_test.go | 2 +- app/vmalert/remotewrite/client.go | 320 ++++++++++++++++++ .../{remotewrite_test.go => client_test.go} | 0 app/vmalert/remotewrite/debug_client.go | 97 ++++++ app/vmalert/remotewrite/debug_client_test.go | 50 +++ app/vmalert/remotewrite/remotewrite.go | 319 +---------------- app/vmalert/replay.go | 6 +- docs/CHANGELOG.md | 1 + docs/vmalert.md | 231 +++++++++++++ go.mod | 6 +- lib/promutils/duration.go | 8 + 16 files changed, 1039 insertions(+), 323 deletions(-) create mode 100644 app/vmalert/remotewrite/client.go rename app/vmalert/remotewrite/{remotewrite_test.go => client_test.go} (100%) create mode 100644 app/vmalert/remotewrite/debug_client.go create mode 100644 app/vmalert/remotewrite/debug_client_test.go diff --git a/app/vmalert/README.md b/app/vmalert/README.md index b0b3bdebdf..009a8ded69 100644 --- a/app/vmalert/README.md +++ b/app/vmalert/README.md @@ -732,6 +732,245 @@ See full description for these flags in `./vmalert -help`. * `query` template function is disabled for performance reasons (might be changed in future); * `limit` group's param has no effect during replay (might be changed in future); +## Unit Testing for Rules + +You can use `vmalert` to run unit tests for alerting and recording rules. +In unit test mode vmalert performs the following actions: +* sets up an isolated VictoriaMetrics instance; +* simulates the periodic ingestion of time series; +* queries the ingested data for recording and alerting rules evaluation; +* tests whether the firing alerts or resulting recording rules match the expected results. + +See how to run vmalert in unit test mode below: +``` +# Run vmalert with one or multiple test files via -unittestFile cmd-line flag +./vmalert -unittestFile=test1.yaml -unittestFile=test2.yaml +``` + +vmalert is compatible with [Prometheus config format for tests](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/#test-file-format) +except `promql_expr_test` field. Use `metricsql_expr_test` field name instead. The name is different because vmalert +validates and executes [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) expressions, +which aren't always backward compatible with [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/). + +### Test file format + +The configuration format for files specified in `-unittestFile` cmd-line flag is the following: +``` +# Path to the files or http url containing [rule groups](https://docs.victoriametrics.com/vmalert.html#groups) configuration. +# Enterprise version of vmalert supports S3 and GCS paths to rules. +rule_files: + [ - ] + +# The evaluation interval for rules specified in `rule_files` +[ evaluation_interval: | default = 1m ] + +# Groups listed below will be evaluated by order. +# Not All the groups need not be mentioned, if not, they will be evaluated by define order in rule_files. +group_eval_order: + [ - ] + +# The list of unit test files to be checked during evaluation. +tests: + [ - ] +``` + +#### `` + +``` +# Interval between samples for input series +interval: +# Time series to persist into the database according to configured before running tests. +input_series: + [ - ] + +# Name of the test group, optional +[ name: ] + +# Unit tests for alerting rules +alert_rule_test: + [ - ] + +# Unit tests for Metricsql expressions. +metricsql_expr_test: + [ - ] + +# External labels accessible for templating. +external_labels: + [ : ... ] + +``` + +#### `` + +``` +# series in the following format '{