Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

2024-11-21 14:44:00 +00:00 · 2021-12-02 15:03:46 +02:00 · 2021-12-02 15:03:46 +02:00 · b94e986710
commit b94e986710
parent a29565d1bd 7f2f26b25f
379 changed files with 30945 additions and 9332 deletions
--- a/.github/workflows/check-licenses.yml
+++ b/.github/workflows/check-licenses.yml
@ -14,7 +14,7 @@ jobs:
      - name: Setup Go
        uses: actions/setup-go@main
        with:
-          go-version: 1.16
+          go-version: 1.17
        id: go
      - name: Code checkout
        uses: actions/checkout@master
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@ -16,7 +16,7 @@ jobs:
      - name: Setup Go
        uses: actions/setup-go@main
        with:
-          go-version: 1.16
+          go-version: 1.17
        id: go
      - name: Code checkout
        uses: actions/checkout@master
--- a/README.md
+++ b/README.md
@ -765,33 +765,10 @@ when new data is ingested into it.

 VictoriaMetrics provides the following handlers for exporting data:

-* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export.
-  See [these docs](#how-to-export-data-in-native-format) for details.
 * `/api/v1/export` for exporing data in JSON line format. See [these docs](#how-to-export-data-in-json-line-format) for details.
 * `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details.
-
-
-### How to export data in native format
-
-Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
-where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
-for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series.
-
-On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter:
-
-```bash
-# count unique timeseries in database
-wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
-
-# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command
-```
-
-Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either
-unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values.
-
-The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
-The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X
-can fail to be imported into VictoriaMetrics release Y.
+* `/api/v1/export/native` for exporting data in native binary format. This is the most efficient format for data export.
+  See [these docs](#how-to-export-data-in-native-format) for details.


 ### How to export data in JSON line format
@ -812,7 +789,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val

 Optional `max_rows_per_line` arg may be added to the request for limiting the maximum number of rows exported per each JSON line.
 Optional `reduce_mem_usage=1` arg may be added to the request for reducing memory usage when exporting big number of time series.
-In this case the output may contain multiple lines with distinct samples for the same time series.
+In this case the output may contain multiple lines with samples for the same time series.

 Pass `Accept-Encoding: gzip` HTTP header in the request to `/api/v1/export` in order to reduce network bandwidth during exporing big amounts
 of time series data. This enables gzip compression for the exported data. Example for exporting gzipped data:
@ -825,6 +802,9 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search

 Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format).

+The [deduplication](#deduplication) is applied to the data exported via `/api/v1/export` by default. The deduplication
+isn't applied if `reduce_mem_usage=1` query arg is passed to the request.
+

 ### How to export CSV data

@ -849,6 +829,33 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val

 The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data).

+The [deduplication](#deduplication) isn't applied for the data exported in CSV. It is expected that the de-duplication is performed during data import.
+
+
+### How to export data in native format
+
+Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
+where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
+for metrics to export. Use `{__name__=~".*"}` selector for fetching all the time series.
+
+On large databases you may experience problems with limit on unique timeseries (default value is 300000). In this case you need to adjust `-search.maxUniqueTimeseries` parameter:
+
+```bash
+# count unique timeseries in database
+wget -O- -q 'http://your_victoriametrics_instance:8428/api/v1/series/count' | jq '.data[0]'
+
+# relaunch victoriametrics with search.maxUniqueTimeseries more than value from previous command
+```
+
+Optional `start` and `end` args may be added to the request in order to limit the time frame for the exported data. These args may contain either
+unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) values.
+
+The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
+The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release X
+can fail to be imported into VictoriaMetrics release Y.
+
+The [deduplication](#deduplication) isn't applied for the data exported in native format. It is expected that the de-duplication is performed during data import.
+

 ## How to import time series data

@ -868,36 +875,6 @@ Time series data can be imported via any supported ingestion protocol:
 * `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.


-### How to import data in native format
-
-The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file.
-
-If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in.
-
-```bash
-# Export the data from <source-victoriametrics>:
-curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin
-
-# Import the data to <destination-victoriametrics>:
-curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin
-```
-
-Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/native` for importing gzipped data:
-
-```bash
-# Export gzipped data from <source-victoriametrics>:
-curl -H 'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin.gz
-
-# Import gzipped data to <destination-victoriametrics>:
-curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin.gz
-```
-
-Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args.
-For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series.
-
-Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
-
-
 ### How to import data in JSON line format

 Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
@ -928,6 +905,26 @@ Note that it could be required to flush response cache after importing historica
 VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage. This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines. The solution is to split too long JSON lines into smaller lines. It is OK if samples for a single time series are split among multiple JSON lines.


+### How to import data in native format
+
+The specification of VictoriaMetrics' native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file.
+
+If you have a native format file obtained via [/api/v1/export/native](#how-to-export-data-in-native-format) however this is the most efficient protocol for importing data in.
+
+```bash
+# Export the data from <source-victoriametrics>:
+curl http://source-victoriametrics:8428/api/v1/export/native -d 'match={__name__!=""}' > exported_data.bin
+
+# Import the data to <destination-victoriametrics>:
+curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin
+```
+
+Extra labels may be added to all the imported time series by passing `extra_label=name=value` query args.
+For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"` label to all the imported time series.
+
+Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
+
+
 ### How to import CSV data

 Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
@ -1003,6 +1000,13 @@ It should return something like the following:
 {"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]}
 ```

+Pass `Content-Encoding: gzip` HTTP request header to `/api/v1/import/prometheus` for importing gzipped data:
+
+```bash
+# Import gzipped data to <destination-victoriametrics>:
+curl -X POST -H 'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz
+```
+
 Extra labels may be added to all the imported metrics by passing `extra_label=name=value` query args.
 For example, `/api/v1/import/prometheus?extra_label=foo=bar` would add `{foo="bar"}` label to all the imported metrics.

@ -1144,7 +1148,7 @@ Older data is eventually deleted during [background merge](https://medium.com/@v

 ## Multiple retentions

-Just start multiple VictoriaMetrics instances with distinct values for the following flags:
+A single instance of VictoriaMetrics supports only a single retention, which can be configured via `-retentionPeriod` command-line flag. If you need multiple retentions, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:

 * `-retentionPeriod`
 * `-storageDataPath`, so the data for each retention period is saved in a separate directory
@ -1153,6 +1157,7 @@ Just start multiple VictoriaMetrics instances with distinct values for the follo
 Then set up [vmauth](https://docs.victoriametrics.com/vmauth.html) in front of VictoriaMetrics instances,
 so it could route requests from particular user to VictoriaMetrics with the desired retention.
 The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
+See [these docs](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html) for multi-retention setup details.


 ## Downsampling
@ -1248,8 +1253,9 @@ or Prometheus by adding the corresponding scrape config to it.
 Alternatively they can be self-scraped by setting `-selfScrapeInterval` command-line flag to duration greater than 0.
 For example, `-selfScrapeInterval=10s` would enable self-scraping of `/metrics` page with 10 seconds interval.

-There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176).
-There is also an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831).
+There are officials Grafana dashboards for [single-node VictoriaMetrics](https://grafana.com/dashboards/10229) and [clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11176). There is also an [alternative dashboard for clustered VictoriaMetrics](https://grafana.com/grafana/dashboards/11831).
+
+Graphs on these dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.

 It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts.yml).

@ -1393,6 +1399,7 @@ See [vmctl docs](https://docs.victoriametrics.com/vmctl.html) for more details.
 ## Backfilling

 VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data).
+See [how to backfill data with recording rules in vmalert](https://docs.victoriametrics.com/vmalert.html#rules-backfilling).
 Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data.

 It is recommended disabling query cache with `-search.disableCache` command-line flag when writing
--- a/app/victoria-metrics/main.go
+++ b/app/victoria-metrics/main.go
@ -96,15 +96,15 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, "See docs at <a href='https://docs.victoriametrics.com/'>https://docs.victoriametrics.com/</a></br>")
 		fmt.Fprintf(w, "Useful endpoints:</br>")
 		httpserver.WriteAPIHelp(w, [][2]string{
-			{"/vmui", "Web UI"},
-			{"/targets", "discovered targets list"},
-			{"/api/v1/targets", "advanced information about discovered targets in JSON format"},
-			{"/config", "-promscrape.config contents"},
-			{"/metrics", "available service metrics"},
-			{"/flags", "command-line flags"},
-			{"/api/v1/status/tsdb", "tsdb status page"},
-			{"/api/v1/status/top_queries", "top queries"},
-			{"/api/v1/status/active_queries", "active queries"},
+			{"vmui", "Web UI"},
+			{"targets", "discovered targets list"},
+			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
+			{"config", "-promscrape.config contents"},
+			{"metrics", "available service metrics"},
+			{"flags", "command-line flags"},
+			{"api/v1/status/tsdb", "tsdb status page"},
+			{"api/v1/status/top_queries", "top queries"},
+			{"api/v1/status/active_queries", "active queries"},
 		})
 		for _, p := range customAPIPathList {
 			p, doc := p[0], p[1]
--- a/app/vmagent/README.md
+++ b/app/vmagent/README.md
@ -428,7 +428,7 @@ These limits are approximate, so `vmagent` can underflow/overflow the limit by a

 `vmagent` exports various metrics in Prometheus exposition format at `http://vmagent-host:8429/metrics` page. We recommend setting up regular scraping of this page
 either through `vmagent` itself or by Prometheus so that the exported metrics may be analyzed later.
-Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/12683) for `vmagent` state overview.
+Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/12683) for `vmagent` state overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
 If you have suggestions for improvements or have found a bug - please open an issue on github or add a review to the dashboard.

 `vmagent` also exports the status for various targets at the following handlers:
--- a/app/vmagent/main.go
+++ b/app/vmagent/main.go
@ -158,12 +158,12 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, "See docs at <a href='https://docs.victoriametrics.com/vmagent.html'>https://docs.victoriametrics.com/vmagent.html</a></br>")
 		fmt.Fprintf(w, "Useful endpoints:</br>")
 		httpserver.WriteAPIHelp(w, [][2]string{
-			{"/targets", "discovered targets list"},
-			{"/api/v1/targets", "advanced information about discovered targets in JSON format"},
-			{"/config", "-promscrape.config contents"},
-			{"/metrics", "available service metrics"},
-			{"/flags", "command-line flags"},
-			{"/-/reload", "reload configuration"},
+			{"targets", "discovered targets list"},
+			{"api/v1/targets", "advanced information about discovered targets in JSON format"},
+			{"config", "-promscrape.config contents"},
+			{"metrics", "available service metrics"},
+			{"flags", "command-line flags"},
+			{"-/reload", "reload configuration"},
 		})
 		return true
 	}
@ -236,26 +236,26 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 			return true
 		}
 		// See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(202)
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "/datadog/api/v1/validate":
 		datadogValidateRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/authentication/#validate-api-key
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{"valid":true}`)
 		return true
 	case "/datadog/api/v1/check_run":
 		datadogCheckRunRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/service-checks/#submit-a-service-check
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(202)
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "/datadog/intake/":
 		datadogIntakeRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{}`)
 		return true
 	case "/targets":
@ -277,7 +277,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		state := r.FormValue("state")
 		promscrape.WriteAPIV1Targets(w, state)
 		return true
@ -391,19 +391,19 @@ func processMultitenantRequest(w http.ResponseWriter, r *http.Request, path stri
 	case "datadog/api/v1/validate":
 		datadogValidateRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/authentication/#validate-api-key
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{"valid":true}`)
 		return true
 	case "datadog/api/v1/check_run":
 		datadogCheckRunRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/service-checks/#submit-a-service-check
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(202)
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "datadog/intake/":
 		datadogIntakeRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{}`)
 		return true
 	default:
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@ -2,7 +2,11 @@

 `vmalert` executes a list of the given [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
 or [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
-rules against configured address. It is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
+rules against configured `-datasource.url`. For sending alerting notifications
+vmalert relies on [Alertmanager]((https://github.com/prometheus/alertmanager)) configured via `-notifier.url` flag.
+Recording rules results are persisted via [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
+protocol and require `-remoteWrite.url` to be configured.
+Vmalert is heavily inspired by [Prometheus](https://prometheus.io/docs/alerting/latest/overview/)
 implementation and aims to be compatible with its syntax.

 ## Features
@ -18,12 +22,12 @@ implementation and aims to be compatible with its syntax.
 * Lightweight without extra dependencies.

 ## Limitations
-* `vmalert` execute queries against remote datasource which has reliability risks because of network.
-It is recommended to configure alerts thresholds and rules expressions with understanding that network request
-may fail;
+* `vmalert` execute queries against remote datasource which has reliability risks because of the network.
+It is recommended to configure alerts thresholds and rules expressions with the understanding that network 
+requests may fail;
 * by default, rules execution is sequential within one group, but persistence of execution results to remote
 storage is asynchronous. Hence, user shouldn't rely on chaining of recording rules when result of previous
-recording rule is reused in next one;
+recording rule is reused in the next one;

 ## QuickStart

@ -33,28 +37,35 @@ git clone https://github.com/VictoriaMetrics/VictoriaMetrics
 cd VictoriaMetrics
 make vmalert
 ```
-The build binary will be placed to `VictoriaMetrics/bin` folder.
+The build binary will be placed in `VictoriaMetrics/bin` folder.

 To start using `vmalert` you will need the following things:
 * list of rules - PromQL/MetricsQL expressions to execute;
-* datasource address - reachable VictoriaMetrics instance for rules execution;
-* notifier address - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing,
-aggregating alerts and sending notifications.
+* datasource address - reachable MetricsQL endpoint to run queries against;
+* notifier address [optional] - reachable [Alert Manager](https://github.com/prometheus/alertmanager) instance for processing,
+aggregating alerts, and sending notifications.
 * remote write address [optional] - [remote write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations)
-compatible storage address for storing recording rules results and alerts state in for of timeseries.
+  compatible storage to persist rules and alerts state info;
+* remote read address [optional] - MetricsQL compatible datasource to restore alerts state from.

 Then configure `vmalert` accordingly:
 ```
 ./bin/vmalert -rule=alert.rules \            # Path to the file with rules configuration. Supports wildcard
    -datasource.url=http://localhost:8428 \  # PromQL compatible datasource
-    -notifier.url=http://localhost:9093 \    # AlertManager URL
+    -notifier.url=http://localhost:9093 \    # AlertManager URL (required if alerting rules are used)
    -notifier.url=http://127.0.0.1:9093 \    # AlertManager replica URL
-    -remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules
+    -remoteWrite.url=http://localhost:8428 \ # Remote write compatible storage to persist rules and alerts state info (required if recording rules are used)
    -remoteRead.url=http://localhost:8428 \  # MetricsQL compatible datasource to restore alerts state from
    -external.label=cluster=east-1 \         # External label to be applied for each rule
    -external.label=replica=a                # Multiple external labels may be set
 ```

+Note there's a separate `remoteRead.url` to allow writing results of
+alerting/recording rules into a different storage than the initial data that's
+queried. This allows using `vmalert` to aggregate data from a short-term,
+high-frequency, high-cardinality storage into a long-term storage with
+decreased cardinality and a bigger interval between samples.
+
 See the full list of configuration flags in [configuration](#configuration) section.

 If you run multiple `vmalert` services for the same datastore or AlertManager - do not forget
@ -88,12 +99,24 @@ name: <string>
 # By default "prometheus" type is used.
 [ type: <string> ]

-# Optional list of label filters applied to every rule's
-# request withing a group. Is compatible only with VM datasource.
-# See more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements
+# Warning: DEPRECATED
+# Please use `params` instead:
+#   params:
+#     extra_label: ["job=nodeexporter", "env=prod"]
 extra_filter_labels:
  [ <labelname>: <labelvalue> ... ]

+# Optional list of HTTP URL parameters
+# applied for all rules requests within a group
+# For example:
+#  params:
+#    nocache: ["1"]                # disable caching for vmselect
+#    denyPartialResponse: ["true"] # fail if one or more vmstorage nodes returned an error
+#    extra_label: ["env=dev"]      # apply additional label filter "env=dev" for all requests 
+# see more details at https://docs.victoriametrics.com#prometheus-querying-api-enhancements
+params:
+  [ <string>: [<string>, ...]]
+
 # Optional list of labels added to every rule within a group.
 # It has priority over the external labels.
 # Labels are commonly used for adding environment
@ -113,14 +136,14 @@ expression and then act according to the Rule type.

 There are two types of Rules:
 * [alerting](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) -
-Alerting rules allow to define alert conditions via `expr` field and to send notifications to
+Alerting rules allow defining alert conditions via `expr` field and to send notifications to
 [Alertmanager](https://github.com/prometheus/alertmanager) if execution result is not empty.
 * [recording](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) -
-Recording rules allow to define `expr` which result will be then backfilled to configured
+Recording rules allow defining `expr` which result will be then backfilled to configured
 `-remoteWrite.url`. Recording rules are used to precompute frequently needed or computationally
 expensive expressions and save their result as a new set of time series.

-`vmalert` forbids defining duplicates - rules with the same combination of name, expression and labels
+`vmalert` forbids defining duplicates - rules with the same combination of name, expression, and labels
 within one group.

 #### Alerting rules
@ -136,7 +159,7 @@ alert: <string>
 expr: <string>

 # Alerts are considered firing once they have been returned for this long.
-# Alerts which have not yet fired for long enough are considered pending.
+# Alerts which have not yet been fired for long enough are considered pending.
 # If param is omitted or set to 0 then alerts will be immediately considered
 # as firing once they return.
 [ for: <duration> | default = 0s ]
@ -181,19 +204,19 @@ For recording rules to work `-remoteWrite.url` must be specified.
 the process alerts state will be lost. To avoid this situation, `vmalert` should be configured via the following flags:
 * `-remoteWrite.url` - URL to VictoriaMetrics (Single) or vminsert (Cluster). `vmalert` will persist alerts state
 into the configured address in the form of time series named `ALERTS` and `ALERTS_FOR_STATE` via remote-write protocol.
-These are regular time series and may be queried from VM just as any other time series.
+These are regular time series and maybe queried from VM just as any other time series.
 The state is stored to the configured address on every rule evaluation.
 * `-remoteRead.url` - URL to VictoriaMetrics (Single) or vmselect (Cluster). `vmalert` will try to restore alerts state
 from configured address by querying time series with name `ALERTS_FOR_STATE`.

-Both flags are required for proper state restoring. Restore process may fail if time series are missing
+Both flags are required for proper state restoration. Restore process may fail if time series are missing
 in configured `-remoteRead.url`, weren't updated in the last `1h` (controlled by `-remoteRead.lookback`)
 or received state doesn't match current `vmalert` rules configuration.


 ### Multitenancy

-The following are the approaches for alerting and recording rules across
+There are the following approaches exist for alerting and recording rules across
 [multiple tenants](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy):

 * To run a separate `vmalert` instance per each tenant.
@ -233,6 +256,110 @@ The enterprise version of vmalert is available in `vmutils-*-enterprise.tar.gz`
 at [release page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) and in `*-enterprise`
 tags at [Docker Hub](https://hub.docker.com/r/victoriametrics/vmalert/tags).

+### Topology examples
+
+The following sections are showing how `vmalert` may be used and configured 
+for different scenarios. 
+
+Please note, not all flags in examples are required: 
+* `-remoteWrite.url` and `-remoteRead.url` are optional and are needed only if
+you have recording rules or want to store [alerts state](#alerts-state-on-restarts) on `vmalert` restarts;
+* `-notifier.url` is optional and is needed only if you have alerting rules.
+
+#### Single-node VictoriaMetrics
+
+The simplest configuration where one single-node VM server is used for
+rules execution, storing recording rules results and alerts state.
+
+`vmalert` configuration flags:
+```
+./bin/vmalert -rule=rules.yml  \                    # Path to the file with rules configuration. Supports wildcard
+    -datasource.url=http://victoriametrics:8428 \   # VM-single addr for executing rules expressions
+    -remoteWrite.url=http://victoriametrics:8428 \  # VM-single addr to persist alerts state and recording rules results
+    -remoteRead.url=http://victoriametrics:8428 \   # VM-single addr for restoring alerts state after restart
+    -notifier.url=http://alertmanager:9093          # AlertManager addr to send alerts when they trigger
+```
+
+<img alt="vmalert single" width="500" src="vmalert_single.png">
+
+
+#### Cluster VictoriaMetrics
+
+In [cluster mode](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html)
+VictoriaMetrics has separate components for writing and reading path:
+`vminsert` and `vmselect` components respectively. `vmselect` is used for executing rules expressions
+and `vminsert` is used to persist recording rules results and alerts state.
+Cluster mode could have multiple `vminsert` and `vmselect` components. 
+
+`vmalert` configuration flags:
+```
+./bin/vmalert -rule=rules.yml  \                                # Path to the file with rules configuration. Supports wildcard
+    -datasource.url=http://vmselect:8481/select/0/prometheus    # vmselect addr for executing rules expressions
+    -remoteWrite.url=http://vminsert:8480/insert/0/prometheuss  # vminsert addr to persist alerts state and recording rules results
+    -remoteRead.url=http://vmselect:8481/select/0/prometheus    # vmselect addr for restoring alerts state after restart
+    -notifier.url=http://alertmanager:9093                      # AlertManager addr to send alerts when they trigger
+```
+
+<img alt="vmalert cluster" src="vmalert_cluster.png">
+
+In case when you want to spread the load on these components - add balancers before them and configure
+`vmalert` with balancer's addresses. Please, see more about VM's cluster architecture
+[here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview).
+
+#### HA vmalert
+
+For HA user can run multiple identically configured `vmalert` instances.
+It means all of them will execute the same rules, write state and results to
+the same destinations, and send alert notifications to multiple configured
+Alertmanagers.
+
+`vmalert` configuration flags:
+```
+./bin/vmalert -rule=rules.yml \                   # Path to the file with rules configuration. Supports wildcard
+    -datasource.url=http://victoriametrics:8428 \   # VM-single addr for executing rules expressions
+    -remoteWrite.url=http://victoriametrics:8428 \  # VM-single addr to persist alerts state and recording rules results
+    -remoteRead.url=http://victoriametrics:8428 \   # VM-single addr for restoring alerts state after restart
+    -notifier.url=http://alertmanager1:9093 \       # Multiple AlertManager addresses to send alerts when they trigger
+    -notifier.url=http://alertmanagerN:9093         # The same alert will be sent to all configured notifiers
+```
+
+<img alt="vmalert ha" width="800px" src="vmalert_ha.png">
+
+To avoid recording rules results and alerts state duplication in VictoriaMetrics server
+don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).
+
+Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
+all `vmalert`s are having the same config.
+
+Don't forget to configure [cluster mode](https://prometheus.io/docs/alerting/latest/alertmanager/)
+for Alertmanagers for better reliability.
+
+This example uses single-node VM server for the sake of simplicity. 
+Check how to replace it with [cluster VictoriaMetrics](#cluster-victoriametrics) if needed. 
+
+
+#### Downsampling and aggregation via vmalert
+
+Example shows how to build a topology where `vmalert` will process data from one cluster
+and write results into another. Such clusters may be called as "hot" (low retention,
+high-speed disks, used for operative monitoring) and "cold" (long term retention,
+slower/cheaper disks, low resolution data). With help of `vmalert`, user can setup
+recording rules to process raw data from "hot" cluster (by applying additional transformations
+or reducing resolution) and push results to "cold" cluster.
+
+`vmalert` configuration flags:
+```
+./bin/vmalert -rule=downsampling-rules.yml \                                        # Path to the file with rules configuration. Supports wildcard
+    -datasource.url=http://raw-cluster-vmselect:8481/select/0/prometheus            # vmselect addr for executing recordi ng rules expressions
+    -remoteWrite.url=http://aggregated-cluster-vminsert:8480/insert/0/prometheuss   # vminsert addr to persist recording rules results
+```
+
+<img alt="vmalert multi cluster" src="vmalert_multicluster.png">
+
+Please note, [replay](#rules-backfilling) feature may be used for transforming historical data.
+
+Flags `-remoteRead.url` and `-notifier.url` are omitted since we assume only recording rules are used.
+

 ### Web

@ -252,7 +379,7 @@ vmalert sends requests to `<-datasource.url>/render?format=json` during evaluati
 if the corresponding group or rule contains `type: "graphite"` config option. It is expected that the `<-datasource.url>/render`
 implements [Graphite Render API](https://graphite.readthedocs.io/en/stable/render_api.html) for `format=json`.
 When using vmalert with both `graphite` and `prometheus` rules configured against cluster version of VM do not forget
-to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on query type.
+to set `-datasource.appendTypePrefix` flag to `true`, so vmalert can adjust URL prefix automatically based on the query type.

 ## Rules backfilling

@ -311,11 +438,11 @@ to prevent cache pollution and unwanted time range boundaries adjustment during

 #### Recording rules

-Result of recording rules `replay` should match with results of normal rules evaluation.
+The result of recording rules `replay` should match with results of normal rules evaluation.

 #### Alerting rules

-Result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
+The result of alerting rules `replay` is time series reflecting [alert's state](#alerts-state-on-restarts).
 To see if `replayed` alert has fired in the past use the following PromQL/MetricsQL expression:
 ```
 ALERTS{alertname="your_alertname", alertstate="firing"}
@ -328,7 +455,7 @@ There are following non-required `replay` flags:

 * `-replay.maxDatapointsPerQuery` - the max number of data points expected to receive in one request.
 In two words, it affects the max time range for every `/query_range` request. The higher the value,
-the less requests will be issued during `replay`.
+the fewer requests will be issued during `replay`.
 * `-replay.ruleRetryAttempts` - when datasource fails to respond vmalert will make this number of retries
 per rule before giving up.
 * `-replay.rulesDelay` - delay between sequential rules execution. Important in cases if there are chaining
@ -350,13 +477,15 @@ See full description for these flags in `./vmalert --help`.
 We recommend setting up regular scraping of this page either through `vmagent` or by Prometheus so that the exported
 metrics may be analyzed later.

-Use official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview.
+Use the official [Grafana dashboard](https://grafana.com/grafana/dashboards/14950) for `vmalert` overview. Graphs on this dashboard contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.
 If you have suggestions for improvements or have found a bug - please open an issue on github or add
 a review to the dashboard.


 ## Configuration

+### Flags
+
 Pass `-help` to `vmalert` in order to see the full list of supported
 command-line flags with their descriptions.

@ -454,7 +583,7 @@ The shortlist of configuration flags is the following:
  -memory.allowedPercent float
    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from OS page cache which will result in higher disk IO usage (default 60)
  -metricsAuthKey string
-    	Auth key for /metrics. It overrides httpAuth settings
+    	Auth key for /metrics. It must be passed via authKey query arg. It overrides httpAuth.* settings
  -notifier.basicAuth.password array
    	Optional basic auth password for -notifier.url
    	Supports an array of values separated by comma or specified via multiple flags.
@ -477,10 +606,10 @@ The shortlist of configuration flags is the following:
    	Optional TLS server name to use for connections to -notifier.url. By default the server name from -notifier.url is used
    	Supports an array of values separated by comma or specified via multiple flags.
  -notifier.url array
-    	Prometheus alertmanager URL. Required parameter. e.g. http://127.0.0.1:9093
+    	Prometheus alertmanager URL, e.g. http://127.0.0.1:9093
    	Supports an array of values separated by comma or specified via multiple flags.
  -pprofAuthKey string
-    	Auth key for /debug/pprof. It overrides httpAuth settings
+    	Auth key for /debug/pprof. It must be passed via authKey query arg. It overrides httpAuth.* settings
  -remoteRead.basicAuth.password string
    	Optional basic auth password for -remoteRead.url
  -remoteRead.basicAuth.passwordFile string
@ -578,12 +707,32 @@ The shortlist of configuration flags is the following:
    	Show VictoriaMetrics version
 ```

+### Hot config reload
 `vmalert` supports "hot" config reload via the following methods:
 * send SIGHUP signal to `vmalert` process;
 * send GET request to `/-/reload` endpoint;
 * configure `-rule.configCheckInterval` flag for periodic reload
 on config change.

+### URL params
+
+To set additional URL params for `datasource.url`, `remoteWrite.url` or `remoteRead.url`
+just add them in address: `-datasource.url=http://localhost:8428?nocache=1`.
+
+To set additional URL params for specific [group of rules](#Groups) modify
+the `params` group:
+```yaml
+groups:
+  - name: TestGroup
+    params:
+      denyPartialResponse: ["true"]
+      extra_label: ["env=dev"]
+```
+Please note, `params` are used only for executing rules expressions (requests to `datasource.url`).
+If there would be a conflict between URL params set in `datasource.url` flag and params in group definition
+the latter will have higher priority.
+
+
 ## Contributing

 `vmalert` is mostly designed and built by VictoriaMetrics community.
@ -599,7 +748,7 @@ It is recommended using

 ### Development build

-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.16.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.17.
 2. Run `make vmalert` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
   It builds `vmalert` binary and puts it into the `bin` folder.

@ -616,7 +765,7 @@ ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://b

 ### Development ARM build

-1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.16.
+1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.17.
 2. Run `make vmalert-arm` or `make vmalert-arm64` from the root folder of [the repository](https://github.com/VictoriaMetrics/VictoriaMetrics).
   It builds `vmalert-arm` or `vmalert-arm64` binary respectively and puts it into the `bin` folder.

--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@ -71,7 +71,7 @@ func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule
 		q: qb.BuildWithParams(datasource.QuerierParams{
 			DataSourceType:     &group.Type,
 			EvaluationInterval: group.Interval,
-			ExtraLabels:        group.ExtraFilterLabels,
+			QueryParams:        group.Params,
 		}),
 		alerts:  make(map[uint64]*notifier.Alert),
 		metrics: &alertingRuleMetrics{},
--- a/app/vmalert/config/config.go
+++ b/app/vmalert/config/config.go
@ -5,17 +5,18 @@ import (
 	"fmt"
 	"hash/fnv"
 	"io/ioutil"
+	"net/url"
 	"path/filepath"
 	"sort"
 	"strings"

-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
+	"gopkg.in/yaml.v2"

+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envtemplate"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
-	"gopkg.in/yaml.v2"
 )

 // Group contains list of Rules grouped into
@ -30,6 +31,7 @@ type Group struct {
 	// ExtraFilterLabels is a list label filters applied to every rule
 	// request withing a group. Is compatible only with VM datasources.
 	// See https://docs.victoriametrics.com#prometheus-querying-api-enhancements
+	// DEPRECATED: use Params field instead
 	ExtraFilterLabels map[string]string `yaml:"extra_filter_labels"`
 	// Labels is a set of label value pairs, that will be added to every rule.
 	// It has priority over the external labels.
@ -37,11 +39,15 @@ type Group struct {
 	// Checksum stores the hash of yaml definition for this group.
 	// May be used to detect any changes like rules re-ordering etc.
 	Checksum string
+	// Optional HTTP URL parameters added to each rule request
+	Params url.Values `yaml:"params"`

 	// Catches all undefined fields and must be empty after parsing.
 	XXX map[string]interface{} `yaml:",inline"`
 }

+const extraLabelParam = "extra_label"
+
 // UnmarshalYAML implements the yaml.Unmarshaler interface.
 func (g *Group) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	type group Group
@ -57,6 +63,16 @@ func (g *Group) UnmarshalYAML(unmarshal func(interface{}) error) error {
 		g.Type.Set(datasource.NewPrometheusType())
 	}

+	// backward compatibility with deprecated `ExtraFilterLabels` param
+	if len(g.ExtraFilterLabels) > 0 {
+		if g.Params == nil {
+			g.Params = url.Values{}
+		}
+		for k, v := range g.ExtraFilterLabels {
+			g.Params.Add(extraLabelParam, fmt.Sprintf("%s=%s", k, v))
+		}
+	}
+
 	h := md5.New()
 	h.Write(b)
 	g.Checksum = fmt.Sprintf("%x", h.Sum(nil))
@ -178,6 +194,7 @@ func Parse(pathPatterns []string, validateAnnotations, validateExpressions bool)
 		fp = append(fp, matches...)
 	}
 	errGroup := new(utils.ErrGroup)
+	var isExtraFilterLabelsUsed bool
 	var groups []Group
 	for _, file := range fp {
 		uniqueGroups := map[string]struct{}{}
@ -197,6 +214,9 @@ func Parse(pathPatterns []string, validateAnnotations, validateExpressions bool)
 			}
 			uniqueGroups[g.Name] = struct{}{}
 			g.File = file
+			if len(g.ExtraFilterLabels) > 0 {
+				isExtraFilterLabelsUsed = true
+			}
 			groups = append(groups, g)
 		}
 	}
@ -206,6 +226,9 @@ func Parse(pathPatterns []string, validateAnnotations, validateExpressions bool)
 	if len(groups) < 1 {
 		logger.Warnf("no groups found in %s", strings.Join(pathPatterns, ";"))
 	}
+	if isExtraFilterLabelsUsed {
+		logger.Warnf("field `extra_filter_labels` is deprecated - use `params` instead")
+	}
 	return groups, nil
 }

--- a/app/vmalert/config/config_test.go
+++ b/app/vmalert/config/config_test.go
@ -7,10 +7,11 @@ import (
 	"testing"
 	"time"

+	"gopkg.in/yaml.v2"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
-	"gopkg.in/yaml.v2"
 )

 func TestMain(m *testing.M) {
@ -472,6 +473,85 @@ concurrency: 16
 rules:
  - alert: ExampleAlertWithFor
    expr: sum by(job) (up == 1)
+`)
+	})
+
+	t.Run("`params` change", func(t *testing.T) {
+		f(t, `
+name: TestGroup
+params:
+    nocache: ["1"]
+rules:
+  - alert: foo
+    expr: sum by(job) (up == 1)
+`, `
+name: TestGroup
+params:
+    nocache: ["0"]
+rules:
+  - alert: foo
+    expr: sum by(job) (up == 1)
 `)
 	})
 }
+
+func TestGroupParams(t *testing.T) {
+	f := func(t *testing.T, data string, expParams url.Values) {
+		t.Helper()
+		var g Group
+		if err := yaml.Unmarshal([]byte(data), &g); err != nil {
+			t.Fatalf("failed to unmarshal: %s", err)
+		}
+		got, exp := g.Params.Encode(), expParams.Encode()
+		if got != exp {
+			t.Fatalf("expected to have %q; got %q", exp, got)
+		}
+	}
+
+	t.Run("no params", func(t *testing.T) {
+		f(t, `
+name: TestGroup
+rules:
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+`, url.Values{})
+	})
+
+	t.Run("params", func(t *testing.T) {
+		f(t, `
+name: TestGroup
+params:
+  nocache: ["1"]
+  denyPartialResponse: ["true"]
+rules:
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+`, url.Values{"nocache": {"1"}, "denyPartialResponse": {"true"}})
+	})
+
+	t.Run("extra labels", func(t *testing.T) {
+		f(t, `
+name: TestGroup
+extra_filter_labels:
+  job: victoriametrics
+  env: prod
+rules:
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+`, url.Values{extraLabelParam: {"job=victoriametrics", "env=prod"}})
+	})
+
+	t.Run("extra labels and params", func(t *testing.T) {
+		f(t, `
+name: TestGroup
+extra_filter_labels:
+  job: victoriametrics
+params:
+  nocache: ["1"]
+  extra_label: ["env=prod"]
+rules:
+  - alert: ExampleAlertAlwaysFiring
+    expr: sum by(job) (up == 1)
+`, url.Values{"nocache": {"1"}, extraLabelParam: {"env=prod", "job=victoriametrics"}})
+	})
+}
--- a/app/vmalert/config/testdata/rules0-good.rules
+++ b/app/vmalert/config/testdata/rules0-good.rules
@ -1,5 +1,8 @@
 groups:
  - name: groupGorSingleAlert
+    params:
+      nocache: ["1"]
+      denyPartialResponse: ["true"]
    rules:
      - alert: VMRows
        for: 10s
--- a/app/vmalert/config/testdata/rules2-good.rules
+++ b/app/vmalert/config/testdata/rules2-good.rules
@ -2,8 +2,11 @@ groups:
  - name: TestGroup
    interval: 2s
    concurrency: 2
-    extra_filter_labels:
+    extra_filter_labels: # deprecated param, use `params` instead
        job: victoriametrics
+    params:
+      denyPartialResponse: ["true"]
+      extra_label: ["env=dev"]
    rules:
      - alert: Conns
        expr: sum(vm_tcplistener_conns) by(instance) > 1
--- a/app/vmalert/datasource/datasource.go
+++ b/app/vmalert/datasource/datasource.go
@ -2,6 +2,7 @@ package datasource

 import (
 	"context"
+	"net/url"
 	"time"
 )

@ -20,8 +21,7 @@ type QuerierBuilder interface {
 type QuerierParams struct {
 	DataSourceType     *Type
 	EvaluationInterval time.Duration
-	// see https://docs.victoriametrics.com/#prometheus-querying-api-enhancements
-	ExtraLabels map[string]string
+	QueryParams        url.Values
 }

 // Metric is the basic entity which should be return by datasource
--- a/app/vmalert/datasource/init.go
+++ b/app/vmalert/datasource/init.go
@ -4,6 +4,7 @@ import (
 	"flag"
 	"fmt"
 	"net/http"
+	"net/url"
 	"strings"

 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
@ -40,9 +41,9 @@ type Param struct {
 }

 // Init creates a Querier from provided flag values.
-// Provided extraParams will be added as GET params to
+// Provided extraParams will be added as GET params for
 // each request.
-func Init(extraParams []Param) (QuerierBuilder, error) {
+func Init(extraParams url.Values) (QuerierBuilder, error) {
 	if *addr == "" {
 		return nil, fmt.Errorf("datasource.url is empty")
 	}
@ -56,11 +57,11 @@ func Init(extraParams []Param) (QuerierBuilder, error) {
 		tr.MaxIdleConns = tr.MaxIdleConnsPerHost
 	}

+	if extraParams == nil {
+		extraParams = url.Values{}
+	}
 	if *roundDigits > 0 {
-		extraParams = append(extraParams, Param{
-			Key:   "round_digits",
-			Value: fmt.Sprintf("%d", *roundDigits),
-		})
+		extraParams.Set("round_digits", fmt.Sprintf("%d", *roundDigits))
 	}

 	authCfg, err := utils.AuthConfig(*basicAuthUsername, *basicAuthPassword, *basicAuthPasswordFile, *bearerToken, *bearerTokenFile)
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@ -5,6 +5,7 @@ import (
 	"fmt"
 	"io/ioutil"
 	"net/http"
+	"net/url"
 	"strings"
 	"time"

@ -22,8 +23,7 @@ type VMStorage struct {

 	dataSourceType     Type
 	evaluationInterval time.Duration
-	extraLabels        []string
-	extraParams        []Param
+	extraParams        url.Values
 	disablePathAppend  bool
 }

@ -47,9 +47,7 @@ func (s *VMStorage) ApplyParams(params QuerierParams) *VMStorage {
 		s.dataSourceType = *params.DataSourceType
 	}
 	s.evaluationInterval = params.EvaluationInterval
-	for k, v := range params.ExtraLabels {
-		s.extraLabels = append(s.extraLabels, fmt.Sprintf("%s=%s", k, v))
-	}
+	s.extraParams = params.QueryParams
 	return s
 }

@ -150,7 +148,7 @@ func (s *VMStorage) newRequestPOST() (*http.Request, error) {
 	if err != nil {
 		return nil, err
 	}
-	req.Header.Set("Content-Type", "application/json; charset=utf-8")
+	req.Header.Set("Content-Type", "application/json")
 	if s.authCfg != nil {
 		if auth := s.authCfg.GetAuthHeader(); auth != "" {
 			req.Header.Set("Authorization", auth)
--- a/app/vmalert/datasource/vm_graphite_api.go
+++ b/app/vmalert/datasource/vm_graphite_api.go
@ -54,6 +54,14 @@ func (s *VMStorage) setGraphiteReqParams(r *http.Request, query string, timestam
 	}
 	r.URL.Path += graphitePath
 	q := r.URL.Query()
+	for k, vs := range s.extraParams {
+		if q.Has(k) { // extraParams are prior to params in URL
+			q.Del(k)
+		}
+		for _, v := range vs {
+			q.Add(k, v)
+		}
+	}
 	q.Set("format", "json")
 	q.Set("target", query)
 	from := "-5min"
--- a/app/vmalert/datasource/vm_prom_api.go
+++ b/app/vmalert/datasource/vm_prom_api.go
@ -150,6 +150,14 @@ func (s *VMStorage) setPrometheusRangeReqParams(r *http.Request, query string, s

 func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string) {
 	q := r.URL.Query()
+	for k, vs := range s.extraParams {
+		if q.Has(k) { // extraParams are prior to params in URL
+			q.Del(k)
+		}
+		for _, v := range vs {
+			q.Add(k, v)
+		}
+	}
 	q.Set("query", query)
 	if s.evaluationInterval > 0 {
 		// set step as evaluationInterval by default
@ -159,11 +167,5 @@ func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string) {
 		// override step with user-specified value
 		q.Set("step", s.queryStep.String())
 	}
-	for _, l := range s.extraLabels {
-		q.Add("extra_label", l)
-	}
-	for _, p := range s.extraParams {
-		q.Add(p.Key, p.Value)
-	}
 	r.URL.RawQuery = q.Encode()
 }
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@ -5,6 +5,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/http/httptest"
+	"net/url"
 	"reflect"
 	"strconv"
 	"strings"
@ -440,10 +441,10 @@ func TestRequestParams(t *testing.T) {
 			},
 		},
 		{
-			"round digits",
+			"prometheus extra params",
 			false,
 			&VMStorage{
-				extraParams: []Param{{"round_digits", "10"}},
+				extraParams: url.Values{"round_digits": {"10"}},
 			},
 			func(t *testing.T, r *http.Request) {
 				exp := fmt.Sprintf("query=%s&round_digits=10&time=%d", query, timestamp.Unix())
@ -451,45 +452,32 @@ func TestRequestParams(t *testing.T) {
 			},
 		},
 		{
-			"extra labels",
-			false,
-			&VMStorage{
-				extraLabels: []string{
-					"env=prod",
-					"query=es=cape",
-				},
-			},
-			func(t *testing.T, r *http.Request) {
-				exp := fmt.Sprintf("extra_label=env%%3Dprod&extra_label=query%%3Des%%3Dcape&query=%s&time=%d", query, timestamp.Unix())
-				checkEqualString(t, exp, r.URL.RawQuery)
-			},
-		},
-		{
-			"extra labels range",
+			"prometheus extra params range",
 			true,
 			&VMStorage{
-				extraLabels: []string{
-					"env=prod",
-					"query=es=cape",
+				extraParams: url.Values{
+					"nocache":      {"1"},
+					"max_lookback": {"1h"},
 				},
 			},
 			func(t *testing.T, r *http.Request) {
-				exp := fmt.Sprintf("end=%d&extra_label=env%%3Dprod&extra_label=query%%3Des%%3Dcape&query=%s&start=%d",
+				exp := fmt.Sprintf("end=%d&max_lookback=1h&nocache=1&query=%s&start=%d",
 					timestamp.Unix(), query, timestamp.Unix())
 				checkEqualString(t, exp, r.URL.RawQuery)
 			},
 		},
 		{
-			"extra params",
+			"graphite extra params",
 			false,
 			&VMStorage{
-				extraParams: []Param{
-					{Key: "nocache", Value: "1"},
-					{Key: "max_lookback", Value: "1h"},
+				dataSourceType: NewGraphiteType(),
+				extraParams: url.Values{
+					"nocache":      {"1"},
+					"max_lookback": {"1h"},
 				},
 			},
 			func(t *testing.T, r *http.Request) {
-				exp := fmt.Sprintf("max_lookback=1h&nocache=1&query=%s&time=%d", query, timestamp.Unix())
+				exp := fmt.Sprintf("format=json&from=-5min&max_lookback=1h&nocache=1&target=%s&until=now", query)
 				checkEqualString(t, exp, r.URL.RawQuery)
 			},
 		},
@ -519,7 +507,7 @@ func TestRequestParams(t *testing.T) {
 func checkEqualString(t *testing.T, exp, got string) {
 	t.Helper()
 	if got != exp {
-		t.Errorf("expected to get %q; got %q", exp, got)
+		t.Errorf("expected to get: \n%q; \ngot: \n%q", exp, got)
 	}
 }

--- a/app/vmalert/group.go
+++ b/app/vmalert/group.go
@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"hash/fnv"
+	"net/url"
 	"sync"
 	"time"

@ -27,8 +28,8 @@ type Group struct {
 	Concurrency int
 	Checksum    string

-	ExtraFilterLabels map[string]string
 	Labels map[string]string
+	Params url.Values

 	doneCh     chan struct{}
 	finishedCh chan struct{}
@ -77,7 +78,7 @@ func newGroup(cfg config.Group, qb datasource.QuerierBuilder, defaultInterval ti
 		Interval:    cfg.Interval.Duration(),
 		Concurrency: cfg.Concurrency,
 		Checksum:    cfg.Checksum,
-		ExtraFilterLabels: cfg.ExtraFilterLabels,
+		Params:      cfg.Params,
 		Labels:      cfg.Labels,

 		doneCh:     make(chan struct{}),
@ -198,7 +199,7 @@ func (g *Group) updateWith(newGroup *Group) error {
 	// group.Start function
 	g.Type = newGroup.Type
 	g.Concurrency = newGroup.Concurrency
-	g.ExtraFilterLabels = newGroup.ExtraFilterLabels
+	g.Params = newGroup.Params
 	g.Labels = newGroup.Labels
 	g.Checksum = newGroup.Checksum
 	g.Rules = newRules
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@ -104,8 +104,7 @@ func main() {
 		}
 		// prevent queries from caching and boundaries aligning
 		// when querying VictoriaMetrics datasource.
-		noCache := datasource.Param{Key: "nocache", Value: "1"}
-		q, err := datasource.Init([]datasource.Param{noCache})
+		q, err := datasource.Init(url.Values{"nocache": {"1"}})
 		if err != nil {
 			logger.Fatalf("failed to init datasource: %s", err)
 		}
@ -284,13 +283,13 @@ func configReload(ctx context.Context, m *manager, groupsCfg []config.Group, sig
 			// config didn't change - skip it
 			continue
 		}
-		groupsCfg = newGroupsCfg
-		if err := m.update(ctx, groupsCfg, false); err != nil {
+		if err := m.update(ctx, newGroupsCfg, false); err != nil {
 			configReloadErrors.Inc()
 			configSuccess.Set(0)
 			logger.Errorf("error while reloading rules: %s", err)
 			continue
 		}
+		groupsCfg = newGroupsCfg
 		configSuccess.Set(1)
 		configTimestamp.Set(fasttime.UnixTimestamp())
 		logger.Infof("Rules reloaded successfully from %q", *rulePath)
--- a/app/vmalert/main_test.go
+++ b/app/vmalert/main_test.go
@ -10,6 +10,7 @@ import (
 	"time"

 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/remotewrite"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 )

@ -99,6 +100,8 @@ groups:
 		querierBuilder: &fakeQuerier{},
 		groups:         make(map[uint64]*Group),
 		labels:         map[string]string{},
+		notifiers:      []notifier.Notifier{&fakeNotifier{}},
+		rw:             &remotewrite.Client{},
 	}

 	syncCh := make(chan struct{})
--- a/app/vmalert/manager.go
+++ b/app/vmalert/manager.go
@ -3,6 +3,8 @@ package main
 import (
 	"context"
 	"fmt"
+	"net/url"
+	"sort"
 	"sync"

 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
@ -85,12 +87,31 @@ func (m *manager) startGroup(ctx context.Context, group *Group, restore bool) er
 }

 func (m *manager) update(ctx context.Context, groupsCfg []config.Group, restore bool) error {
+	var rrPresent, arPresent bool
 	groupsRegistry := make(map[uint64]*Group)
 	for _, cfg := range groupsCfg {
+		for _, r := range cfg.Rules {
+			if rrPresent && arPresent {
+				continue
+			}
+			if r.Record != "" {
+				rrPresent = true
+			}
+			if r.Alert != "" {
+				arPresent = true
+			}
+		}
 		ng := newGroup(cfg, m.querierBuilder, *evaluationInterval, m.labels)
 		groupsRegistry[ng.ID()] = ng
 	}

+	if rrPresent && m.rw == nil {
+		return fmt.Errorf("config contains recording rules but `-remoteWrite.url` isn't set")
+	}
+	if arPresent && m.notifiers == nil {
+		return fmt.Errorf("config contains alerting rules but `-notifier.url` isn't set")
+	}
+
 	type updateItem struct {
 		old *Group
 		new *Group
@ -147,7 +168,7 @@ func (g *Group) toAPI() APIGroup {
 		File:        g.File,
 		Interval:    g.Interval.String(),
 		Concurrency: g.Concurrency,
-		ExtraFilterLabels: g.ExtraFilterLabels,
+		Params:      urlValuesToStrings(g.Params),
 		Labels:      g.Labels,
 	}
 	for _, r := range g.Rules {
@ -160,3 +181,24 @@ func (g *Group) toAPI() APIGroup {
 	}
 	return ag
 }
+
+func urlValuesToStrings(values url.Values) []string {
+	if len(values) < 1 {
+		return nil
+	}
+
+	keys := make([]string, 0, len(values))
+	for k := range values {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	var res []string
+	for _, k := range keys {
+		params := values[k]
+		for _, v := range params {
+			res = append(res, fmt.Sprintf("%s=%s", k, v))
+		}
+	}
+	return res
+}
--- a/app/vmalert/manager_test.go
+++ b/app/vmalert/manager_test.go
@ -5,6 +5,7 @@ import (
 	"math/rand"
 	"net/url"
 	"os"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@ -12,6 +13,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/remotewrite"
 )

 func TestMain(m *testing.M) {
@ -218,7 +220,11 @@ func TestManagerUpdate(t *testing.T) {
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
 			ctx, cancel := context.WithCancel(context.TODO())
-			m := &manager{groups: make(map[uint64]*Group), querierBuilder: &fakeQuerier{}}
+			m := &manager{
+				groups:         make(map[uint64]*Group),
+				querierBuilder: &fakeQuerier{},
+				notifiers:      []notifier.Notifier{&fakeNotifier{}},
+			}

 			cfgInit := loadCfg(t, []string{tc.initPath}, true, true)
 			if err := m.update(ctx, cfgInit, false); err != nil {
@ -247,6 +253,78 @@ func TestManagerUpdate(t *testing.T) {
 	}
 }

+func TestManagerUpdateNegative(t *testing.T) {
+	testCases := []struct {
+		notifiers []notifier.Notifier
+		rw        *remotewrite.Client
+		cfg       config.Group
+		expErr    string
+	}{
+		{
+			nil,
+			nil,
+			config.Group{Name: "Recording rule only",
+				Rules: []config.Rule{
+					{Record: "record", Expr: "max(up)"},
+				},
+			},
+			"contains recording rules",
+		},
+		{
+			nil,
+			nil,
+			config.Group{Name: "Alerting rule only",
+				Rules: []config.Rule{
+					{Alert: "alert", Expr: "up > 0"},
+				},
+			},
+			"contains alerting rules",
+		},
+		{
+			[]notifier.Notifier{&fakeNotifier{}},
+			nil,
+			config.Group{Name: "Recording and alerting rules",
+				Rules: []config.Rule{
+					{Alert: "alert1", Expr: "up > 0"},
+					{Alert: "alert2", Expr: "up > 0"},
+					{Record: "record", Expr: "max(up)"},
+				},
+			},
+			"contains recording rules",
+		},
+		{
+			nil,
+			&remotewrite.Client{},
+			config.Group{Name: "Recording and alerting rules",
+				Rules: []config.Rule{
+					{Record: "record1", Expr: "max(up)"},
+					{Record: "record2", Expr: "max(up)"},
+					{Alert: "alert", Expr: "up > 0"},
+				},
+			},
+			"contains alerting rules",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.cfg.Name, func(t *testing.T) {
+			m := &manager{
+				groups:         make(map[uint64]*Group),
+				querierBuilder: &fakeQuerier{},
+				notifiers:      tc.notifiers,
+				rw:             tc.rw,
+			}
+			err := m.update(context.Background(), []config.Group{tc.cfg}, false)
+			if err == nil {
+				t.Fatalf("expected to get error; got nil")
+			}
+			if !strings.Contains(err.Error(), tc.expErr) {
+				t.Fatalf("expected err to contain %q; got %q", tc.expErr, err)
+			}
+		})
+	}
+}
+
 func loadCfg(t *testing.T, path []string, validateAnnotations, validateExpressions bool) []config.Group {
 	t.Helper()
 	cfg, err := config.Parse(path, validateAnnotations, validateExpressions)
--- a/app/vmalert/notifier/alertmanager.go
+++ b/app/vmalert/notifier/alertmanager.go
@ -32,7 +32,7 @@ func (am *AlertManager) Send(ctx context.Context, alerts []Alert) error {
 	if err != nil {
 		return err
 	}
-	req.Header.Set("Content-Type", "application/json; charset=utf-8")
+	req.Header.Set("Content-Type", "application/json")
 	req = req.WithContext(ctx)
 	if am.basicAuthPass != "" {
 		req.SetBasicAuth(am.basicAuthUser, am.basicAuthPass)
--- a/app/vmalert/notifier/init.go
+++ b/app/vmalert/notifier/init.go
@ -9,7 +9,7 @@ import (
 )

 var (
-	addrs             = flagutil.NewArray("notifier.url", "Prometheus alertmanager URL. Required parameter. e.g. http://127.0.0.1:9093")
+	addrs             = flagutil.NewArray("notifier.url", "Prometheus alertmanager URL, e.g. http://127.0.0.1:9093")
 	basicAuthUsername = flagutil.NewArray("notifier.basicAuth.username", "Optional basic auth username for -notifier.url")
 	basicAuthPassword = flagutil.NewArray("notifier.basicAuth.password", "Optional basic auth password for -notifier.url")

@ -24,10 +24,6 @@ var (

 // Init creates a Notifier object based on provided flags.
 func Init(gen AlertURLGenerator) ([]Notifier, error) {
-	if len(*addrs) == 0 {
-		return nil, fmt.Errorf("at least one `-notifier.url` must be set")
-	}
-
 	var notifiers []Notifier
 	for i, addr := range *addrs {
 		cert, key := tlsCertFile.GetOptionalArg(i), tlsKeyFile.GetOptionalArg(i)
--- a/app/vmalert/recording.go
+++ b/app/vmalert/recording.go
@ -70,7 +70,7 @@ func newRecordingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rul
 		q: qb.BuildWithParams(datasource.QuerierParams{
 			DataSourceType:     &group.Type,
 			EvaluationInterval: group.Interval,
-			ExtraLabels:        group.ExtraFilterLabels,
+			QueryParams:        group.Params,
 		}),
 	}

--- a/app/vmalert/vmalert_cluster.excalidraw
+++ b/app/vmalert/vmalert_cluster.excalidraw
@ -0,0 +1,525 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 794,
+      "versionNonce": 1855937036,
+      "isDeleted": false,
+      "id": "VgBUzo0blGR-Ijd2mQEEf",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 422.3502197265625,
+      "y": 215.55953979492188,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 123.7601318359375,
+      "height": 72.13211059570312,
+      "seed": 1194011660,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "miEbzHxOPXe4PEYvXiJp5",
+        "rcmiQfIWtfbTTlwxqr1sl",
+        "P-dpWlSTtnsux-zr5oqgF",
+        "oAToSPttH7aWoD_AqXGFX",
+        "Bpy5by47XGKB4yS99ZkuA",
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348083348
+    },
+    {
+      "type": "text",
+      "version": 659,
+      "versionNonce": 247957684,
+      "isDeleted": false,
+      "id": "e9TDm09y-GhPm84XWt0Jv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 443.89678955078125,
+      "y": 236.64378356933594,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 82,
+      "height": 24,
+      "seed": 327273100,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347948032,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmalert",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "middle"
+    },
+    {
+      "type": "rectangle",
+      "version": 1670,
+      "versionNonce": 2021681972,
+      "isDeleted": false,
+      "id": "dd52BjHfPMPRji9Tws7U-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 750.3317260742188,
+      "y": 226.5509033203125,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 1779959692,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "Bpy5by47XGKB4yS99ZkuA"
+      ],
+      "updated": 1638348054411
+    },
+    {
+      "type": "text",
+      "version": 1311,
+      "versionNonce": 1283453068,
+      "isDeleted": false,
+      "id": "9TEzv0sVCHAkc46ou0oNF",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 759.2862243652344,
+      "y": 238.68240356445312,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 152,
+      "height": 24,
+      "seed": 1617178804,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348054411,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vminsert:8480",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 897,
+      "versionNonce": 1983434892,
+      "isDeleted": false,
+      "id": "Sa4OBd1ZjD6itohm7Ll8z",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 61.744873046875,
+      "y": 224.9600830078125,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 126267060,
+      "groupIds": [
+        "ek-pq3umtz1yN-J_-preq"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh"
+      ],
+      "updated": 1638348077724
+    },
+    {
+      "type": "text",
+      "version": 719,
+      "versionNonce": 457402292,
+      "isDeleted": false,
+      "id": "we766A079lfGYu2_aC4Pl",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 70.69937133789062,
+      "y": 237.33523559570312,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 152,
+      "height": 24,
+      "seed": 478660236,
+      "groupIds": [
+        "ek-pq3umtz1yN-J_-preq"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348077725,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmselect:8481",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 1098,
+      "versionNonce": 1480603788,
+      "isDeleted": false,
+      "id": "8-XFSbd6Zw96EUSJbJXZv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 371.7434387207031,
+      "y": 398.50787353515625,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 240.10644531249997,
+      "height": 44.74725341796875,
+      "seed": 99322124,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348083348
+    },
+    {
+      "type": "text",
+      "version": 864,
+      "versionNonce": 1115813900,
+      "isDeleted": false,
+      "id": "GUs816aggGqUSdoEsSmea",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 393.73809814453125,
+      "y": 410.5976257324219,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 199,
+      "height": 24,
+      "seed": 1194745268,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348009180,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "alertmanager:9093",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "arrow",
+      "version": 2405,
+      "versionNonce": 959767732,
+      "isDeleted": false,
+      "id": "Bpy5by47XGKB4yS99ZkuA",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 556.6860961914062,
+      "y": 252.2582773408825,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 184.9195556640625,
+      "height": 1.6022679018915937,
+      "seed": 357577356,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638348054411,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": 0.0344528515859526,
+        "gap": 10.57574462890625
+      },
+      "endBinding": {
+        "elementId": "dd52BjHfPMPRji9Tws7U-",
+        "focus": -0.039393828258510157,
+        "gap": 8.72607421875
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          184.9195556640625,
+          -1.6022679018915937
+        ]
+      ]
+    },
+    {
+      "type": "arrow",
+      "version": 1173,
+      "versionNonce": 1248255756,
+      "isDeleted": false,
+      "id": "wRO0q9xKPHc8e8XPPsQWh",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 406.0439244722469,
+      "y": 246.80533728741074,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 157.86774649373126,
+      "height": 0.1417938392881979,
+      "seed": 656189364,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638348077725,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": 0.13736472619498497,
+        "gap": 16.306295254315614
+      },
+      "endBinding": {
+        "elementId": "Sa4OBd1ZjD6itohm7Ll8z",
+        "focus": -0.013200835330936087,
+        "gap": 14.437713623046875
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -157.86774649373126,
+          0.1417938392881979
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 557,
+      "versionNonce": 995289780,
+      "isDeleted": false,
+      "id": "RbVSa4PnOgAMtzoKb-DhW",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 552.4987182617188,
+      "y": 212.27996826171875,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 188,
+      "height": 76,
+      "seed": 1989838604,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348059525,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "persist alerts state\n\n\nand recording rules",
+      "baseline": 72,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "text",
+      "version": 803,
+      "versionNonce": 1576507444,
+      "isDeleted": false,
+      "id": "ia2QzZNl_tuvfY3ymLjyJ",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 290.34130859375,
+      "y": 210.56927490234375,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 122,
+      "height": 19,
+      "seed": 157304972,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh"
+      ],
+      "updated": 1638347948032,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "execute rules",
+      "baseline": 15,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "arrow",
+      "version": 1471,
+      "versionNonce": 1361321140,
+      "isDeleted": false,
+      "id": "sxEhnxlbT7ldlSsmHDUHp",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 484.18669893674246,
+      "y": 302.3424013553929,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 1.0484739253853945,
+      "height": 84.72775855671654,
+      "seed": 1818348300,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638348083348,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": 0.010768924644894236,
+        "gap": 14.650750964767894
+      },
+      "endBinding": {
+        "elementId": "8-XFSbd6Zw96EUSJbJXZv",
+        "focus": -0.051051952959743775,
+        "gap": 11.437713623046818
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          1.0484739253853945,
+          84.72775855671654
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 576,
+      "versionNonce": 2112088460,
+      "isDeleted": false,
+      "id": "E9Run6wCm2chQ6JHrmc_y",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 530.5612182617188,
+      "y": 318.60687255859375,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 122,
+      "height": 38,
+      "seed": 1836541708,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348023735,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "send alert \nnotifications",
+      "baseline": 34,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "text",
+      "version": 480,
+      "versionNonce": 1835119500,
+      "isDeleted": false,
+      "id": "ff5OkfgmkKLifS13_TFj3",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 291.37474060058594,
+      "y": 261.4861297607422,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 122,
+      "height": 19,
+      "seed": 264004620,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh"
+      ],
+      "updated": 1638347948032,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "restore state",
+      "baseline": 15,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}
--- a/app/vmalert/vmalert_cluster.png
+++ b/app/vmalert/vmalert_cluster.png
--- a/app/vmalert/vmalert_ha.excalidraw
+++ b/app/vmalert/vmalert_ha.excalidraw
@ -0,0 +1,911 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 906,
+      "versionNonce": 448716468,
+      "isDeleted": false,
+      "id": "VgBUzo0blGR-Ijd2mQEEf",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 302.2676696777344,
+      "y": 275.59356689453125,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 123.7601318359375,
+      "height": 72.13211059570312,
+      "seed": 1194011660,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "miEbzHxOPXe4PEYvXiJp5",
+        "rcmiQfIWtfbTTlwxqr1sl",
+        "P-dpWlSTtnsux-zr5oqgF",
+        "oAToSPttH7aWoD_AqXGFX",
+        "Bpy5by47XGKB4yS99ZkuA",
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "sxEhnxlbT7ldlSsmHDUHp",
+        "m9_BptFOFxbV2sS_xJDu2",
+        "fsGFp4NW4JlrCdF0HR3uA",
+        "OTKoeHmKtqCxFArDbY-sP"
+      ],
+      "updated": 1638355221749
+    },
+    {
+      "type": "text",
+      "version": 762,
+      "versionNonce": 223660724,
+      "isDeleted": false,
+      "id": "e9TDm09y-GhPm84XWt0Jv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 318.0538024902344,
+      "y": 296.6778106689453,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 327273100,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "m9_BptFOFxbV2sS_xJDu2"
+      ],
+      "updated": 1638355102070,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmalert1",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 2067,
+      "versionNonce": 817347852,
+      "isDeleted": false,
+      "id": "dd52BjHfPMPRji9Tws7U-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 329.6426086425781,
+      "y": 90.3275146484375,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 269.336669921875,
+      "height": 44.74725341796875,
+      "seed": 1779959692,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "Bpy5by47XGKB4yS99ZkuA",
+        "m9_BptFOFxbV2sS_xJDu2",
+        "S_dOHQrhGmu8SFJzobJK7"
+      ],
+      "updated": 1638355058507
+    },
+    {
+      "type": "text",
+      "version": 1717,
+      "versionNonce": 2040797452,
+      "isDeleted": false,
+      "id": "9TEzv0sVCHAkc46ou0oNF",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 349.01177978515625,
+      "y": 102.45901489257812,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 234,
+      "height": 24,
+      "seed": 1617178804,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "m9_BptFOFxbV2sS_xJDu2",
+        "S_dOHQrhGmu8SFJzobJK7"
+      ],
+      "updated": 1638355040938,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "victoriametrics:8428",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 1553,
+      "versionNonce": 39381044,
+      "isDeleted": false,
+      "id": "8-XFSbd6Zw96EUSJbJXZv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 173.67257690429688,
+      "y": 470.5100402832031,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 240.10644531249997,
+      "height": 44.74725341796875,
+      "seed": 99322124,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp",
+        "OTKoeHmKtqCxFArDbY-sP",
+        "XhPgLRBk-YhWAFcSQi9TJ",
+        "D6fkQH1E_MFbCuL697ArO"
+      ],
+      "updated": 1638355221749
+    },
+    {
+      "type": "text",
+      "version": 1309,
+      "versionNonce": 1112872844,
+      "isDeleted": false,
+      "id": "GUs816aggGqUSdoEsSmea",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 189.667236328125,
+      "y": 482.59979248046875,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 211,
+      "height": 24,
+      "seed": 1194745268,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355040938,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "alertmanager1:9093",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "text",
+      "version": 835,
+      "versionNonce": 2036483596,
+      "isDeleted": false,
+      "id": "RbVSa4PnOgAMtzoKb-DhW",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 525.83837890625,
+      "y": 147.33470153808594,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 188,
+      "height": 95,
+      "seed": 1989838604,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355040939,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n",
+      "baseline": 91,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "text",
+      "version": 777,
+      "versionNonce": 990468492,
+      "isDeleted": false,
+      "id": "E9Run6wCm2chQ6JHrmc_y",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 617.0690307617188,
+      "y": 376.9822692871094,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 122,
+      "height": 38,
+      "seed": 1836541708,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638355227582,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "send alert \nnotifications",
+      "baseline": 34,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 1112,
+      "versionNonce": 999136652,
+      "isDeleted": false,
+      "id": "mIu-d0lmShCxzMLD5iA_p",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 492.29505920410156,
+      "y": 272.3052215576172,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 123.7601318359375,
+      "height": 72.13211059570312,
+      "seed": 756416780,
+      "groupIds": [
+        "jbBot-UNdMoWy2jPXC1u5"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "miEbzHxOPXe4PEYvXiJp5",
+        "rcmiQfIWtfbTTlwxqr1sl",
+        "P-dpWlSTtnsux-zr5oqgF",
+        "oAToSPttH7aWoD_AqXGFX",
+        "Bpy5by47XGKB4yS99ZkuA",
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "sxEhnxlbT7ldlSsmHDUHp",
+        "S_dOHQrhGmu8SFJzobJK7",
+        "XhPgLRBk-YhWAFcSQi9TJ",
+        "Ar-hcDLlzVSoTPs2MaywO"
+      ],
+      "updated": 1638355153683
+    },
+    {
+      "type": "text",
+      "version": 970,
+      "versionNonce": 430760500,
+      "isDeleted": false,
+      "id": "ZqIR6SaLNDQl8s0zZbdnE",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 508.08119201660156,
+      "y": 293.38946533203125,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 1477404084,
+      "groupIds": [
+        "jbBot-UNdMoWy2jPXC1u5"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355105020,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmalertN",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "id": "qZFezRGU4Chwxgvb2451t",
+      "type": "line",
+      "x": 449.93653869628906,
+      "y": 306.30010986328125,
+      "width": 19.48321533203125,
+      "height": 0.3865966796875,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 1833426828,
+      "version": 69,
+      "versionNonce": 871948300,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355040939,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          19.48321533203125,
+          0.3865966796875
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": null,
+      "endBinding": null,
+      "startArrowhead": null,
+      "endArrowhead": null
+    },
+    {
+      "type": "rectangle",
+      "version": 1649,
+      "versionNonce": 93981708,
+      "isDeleted": false,
+      "id": "gNHiZJKo0ap69ALDobtZ-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 518.5596466064453,
+      "y": 471.4539489746094,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 240.10644531249997,
+      "height": 44.74725341796875,
+      "seed": 454422412,
+      "groupIds": [
+        "fEAIeQ0DxLnI_rPlKPZqW"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp",
+        "fsGFp4NW4JlrCdF0HR3uA",
+        "Ar-hcDLlzVSoTPs2MaywO",
+        "D6fkQH1E_MFbCuL697ArO"
+      ],
+      "updated": 1638355153683
+    },
+    {
+      "type": "text",
+      "version": 1412,
+      "versionNonce": 75038348,
+      "isDeleted": false,
+      "id": "O-zgjZBvt4RI1PrkBNQnb",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 534.3148040771484,
+      "y": 483.543701171875,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 211,
+      "height": 24,
+      "seed": 1026268980,
+      "groupIds": [
+        "fEAIeQ0DxLnI_rPlKPZqW"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355040939,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "alertmanagerN:9093",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "id": "m9_BptFOFxbV2sS_xJDu2",
+      "type": "arrow",
+      "x": 378.57185562792466,
+      "y": 262.1596984863281,
+      "width": 79.96146539019026,
+      "height": 111.53411865234375,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 141224884,
+      "version": 521,
+      "versionNonce": 283254540,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355102070,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          79.96146539019026,
+          -111.53411865234375
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": -0.23996018586441184,
+        "gap": 13.433868408203125
+      },
+      "endBinding": {
+        "elementId": "dd52BjHfPMPRji9Tws7U-",
+        "focus": -0.14207100087207922,
+        "gap": 15.550811767578125
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "S_dOHQrhGmu8SFJzobJK7",
+      "type": "arrow",
+      "x": 558.1990515577129,
+      "y": 263.4271240234375,
+      "width": 88.18940317574186,
+      "height": 114.15780639648438,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 1004789812,
+      "version": 314,
+      "versionNonce": 1259171724,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355105019,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -88.18940317574186,
+          -114.15780639648438
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "mIu-d0lmShCxzMLD5iA_p",
+        "focus": 0.4315094049079832,
+        "gap": 8.878097534179688
+      },
+      "endBinding": {
+        "elementId": "dd52BjHfPMPRji9Tws7U-",
+        "focus": 0.14840834302306677,
+        "gap": 14.194549560546875
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "fsGFp4NW4JlrCdF0HR3uA",
+      "type": "arrow",
+      "x": 356.6613630516658,
+      "y": 354.95416259765625,
+      "width": 278.74351860741064,
+      "height": 106.90020751953125,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 712384692,
+      "version": 334,
+      "versionNonce": 266493324,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355102070,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          278.74351860741064,
+          106.90020751953125
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": 0.7721210277890177,
+        "gap": 7.228485107421875
+      },
+      "endBinding": {
+        "elementId": "gNHiZJKo0ap69ALDobtZ-",
+        "focus": 0.4493598001028791,
+        "gap": 9.599578857421875
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "OTKoeHmKtqCxFArDbY-sP",
+      "type": "arrow",
+      "x": 361.02563221226757,
+      "y": 356.2252197265625,
+      "width": 95.3594006000182,
+      "height": 105.52130126953125,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 1992252596,
+      "version": 466,
+      "versionNonce": 472553100,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355221749,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -95.3594006000182,
+          105.52130126953125
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": -0.39325294425055324,
+        "gap": 8.499542236328125
+      },
+      "endBinding": {
+        "elementId": "8-XFSbd6Zw96EUSJbJXZv",
+        "focus": -0.400636314282374,
+        "gap": 8.763519287109375
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "XhPgLRBk-YhWAFcSQi9TJ",
+      "type": "arrow",
+      "x": 561.7189961327852,
+      "y": 349.238037109375,
+      "width": 266.271510370216,
+      "height": 112.383544921875,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 588869260,
+      "version": 453,
+      "versionNonce": 584723212,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355150131,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -266.271510370216,
+          112.383544921875
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "mIu-d0lmShCxzMLD5iA_p",
+        "focus": -0.7084007026732048,
+        "gap": 4.8007049560546875
+      },
+      "endBinding": {
+        "elementId": "8-XFSbd6Zw96EUSJbJXZv",
+        "focus": -0.4180430111580123,
+        "gap": 8.888458251953125
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "Ar-hcDLlzVSoTPs2MaywO",
+      "type": "arrow",
+      "x": 557.5999880535809,
+      "y": 352.71795654296875,
+      "width": 112.87813113656136,
+      "height": 106.3255615234375,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 1508782476,
+      "version": 331,
+      "versionNonce": 1137561908,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355153683,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          112.87813113656136,
+          106.3255615234375
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "mIu-d0lmShCxzMLD5iA_p",
+        "focus": 0.4358123206211808,
+        "gap": 8.280624389648438
+      },
+      "endBinding": {
+        "elementId": "gNHiZJKo0ap69ALDobtZ-",
+        "focus": 0.4783744286829653,
+        "gap": 12.410430908203125
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "WqHnv9_g3SdkLZuy4WSHa",
+      "type": "text",
+      "x": 186.5891876220703,
+      "y": 523.145263671875,
+      "width": 272,
+      "height": 19,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "seed": 1310147468,
+      "version": 281,
+      "versionNonce": 949157044,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355040939,
+      "text": "Alertmanagers in cluster mode",
+      "fontSize": 16,
+      "fontFamily": 3,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "baseline": 15
+    },
+    {
+      "id": "yA0kgvdF71wZbHJ7Cg2p8",
+      "type": "rectangle",
+      "x": 159.41685485839844,
+      "y": 440.7666015625,
+      "width": 625.9590148925781,
+      "height": 110.13494873046878,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "seed": 1465986996,
+      "version": 366,
+      "versionNonce": 196040844,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355040939
+    },
+    {
+      "id": "D6fkQH1E_MFbCuL697ArO",
+      "type": "arrow",
+      "x": 422.4853057861328,
+      "y": 494.8779132311229,
+      "width": 90.5517578125,
+      "height": 0.12427004064892344,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 2025208204,
+      "version": 316,
+      "versionNonce": 2144102156,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638355040939,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          90.5517578125,
+          0.12427004064892344
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "8-XFSbd6Zw96EUSJbJXZv",
+        "focus": 0.08064204025505255,
+        "gap": 8.706283569335938
+      },
+      "endBinding": {
+        "elementId": "gNHiZJKo0ap69ALDobtZ-",
+        "focus": -0.05976220061952895,
+        "gap": 5.5225830078125
+      },
+      "startArrowhead": "arrow",
+      "endArrowhead": "arrow"
+    },
+    {
+      "type": "text",
+      "version": 463,
+      "versionNonce": 2118914484,
+      "isDeleted": false,
+      "id": "p4EDvSKPqZzfM_SReybeq",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 274.6086883544922,
+      "y": 56.77886962890625,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 394,
+      "height": 19,
+      "seed": 1214462348,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355074872,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "VictoriaMetrics with deduplication enabled",
+      "baseline": 15,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 712,
+      "versionNonce": 737833612,
+      "isDeleted": false,
+      "id": "YfNFeOucMo8BJk2P2JRex",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 229.77923583984375,
+      "y": 227.84378051757812,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 448.5542297363281,
+      "height": 134.06329345703122,
+      "seed": 932207756,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355133244
+    },
+    {
+      "type": "text",
+      "version": 427,
+      "versionNonce": 1458527796,
+      "isDeleted": false,
+      "id": "oldM7Q_aJtpWd2jXQ0iKf",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 238.9442901611328,
+      "y": 230.862548828125,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 225,
+      "height": 38,
+      "seed": 2131899572,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638355259358,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "vmalerts with identical \nconfigurations",
+      "baseline": 34,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}
--- a/app/vmalert/vmalert_ha.png
+++ b/app/vmalert/vmalert_ha.png
--- a/app/vmalert/vmalert_multicluster.excalidraw
+++ b/app/vmalert/vmalert_multicluster.excalidraw
@ -0,0 +1,915 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 791,
+      "versionNonce": 48874036,
+      "isDeleted": false,
+      "id": "VgBUzo0blGR-Ijd2mQEEf",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 289.6802978515625,
+      "y": 399.3895568847656,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 123.7601318359375,
+      "height": 72.13211059570312,
+      "seed": 1194011660,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "miEbzHxOPXe4PEYvXiJp5",
+        "rcmiQfIWtfbTTlwxqr1sl",
+        "P-dpWlSTtnsux-zr5oqgF",
+        "oAToSPttH7aWoD_AqXGFX",
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "sxEhnxlbT7ldlSsmHDUHp",
+        "pD9DcILMxa6GaR1U5YyMO",
+        "HPEwr85wL4IedW0AgdArp",
+        "EyecK0YM9Cc8T6ju-nTOc"
+      ],
+      "updated": 1638347812431
+    },
+    {
+      "type": "text",
+      "version": 658,
+      "versionNonce": 1653816076,
+      "isDeleted": false,
+      "id": "e9TDm09y-GhPm84XWt0Jv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 311.22686767578125,
+      "y": 420.4738006591797,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 82,
+      "height": 24,
+      "seed": 327273100,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347796775,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmalert",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "middle"
+    },
+    {
+      "type": "rectangle",
+      "version": 802,
+      "versionNonce": 995326644,
+      "isDeleted": false,
+      "id": "Sa4OBd1ZjD6itohm7Ll8z",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 603.05322265625,
+      "y": 228.65371704101562,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 126267060,
+      "groupIds": [
+        "ek-pq3umtz1yN-J_-preq"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "he-SpFjCxEQEWpWny2kKP",
+        "-pjrKo16rOsasM8viZPJ-",
+        "MGdu6GDIPNBAaEUr0Gt-a"
+      ],
+      "updated": 1638347737174
+    },
+    {
+      "type": "text",
+      "version": 624,
+      "versionNonce": 707755700,
+      "isDeleted": false,
+      "id": "we766A079lfGYu2_aC4Pl",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 640.7635803222656,
+      "y": 241.02886962890625,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 478660236,
+      "groupIds": [
+        "ek-pq3umtz1yN-J_-preq"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347701075,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmselect",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "text",
+      "version": 802,
+      "versionNonce": 1974403340,
+      "isDeleted": false,
+      "id": "ia2QzZNl_tuvfY3ymLjyJ",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 401.241943359375,
+      "y": 342.4627990722656,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 178,
+      "height": 38,
+      "seed": 157304972,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh"
+      ],
+      "updated": 1638347863144,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "execute aggregating\nrecording rules",
+      "baseline": 34,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "id": "jrFZeFNsxlss7IsEZpA-J",
+      "type": "rectangle",
+      "x": 594.1509857177734,
+      "y": 192.09194946289062,
+      "width": 397.1286010742188,
+      "height": 209.62742614746097,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "seed": 48379060,
+      "version": 665,
+      "versionNonce": 617456652,
+      "isDeleted": false,
+      "boundElementIds": [
+        "pD9DcILMxa6GaR1U5YyMO"
+      ],
+      "updated": 1638347763716
+    },
+    {
+      "id": "6ibhLp94HJFIdfP5HCEv8",
+      "type": "text",
+      "x": 609.7205657958984,
+      "y": 199.81723022460938,
+      "width": 225,
+      "height": 19,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "seed": 1053164852,
+      "version": 256,
+      "versionNonce": 538595124,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638347761092,
+      "text": "VM cluster with raw data",
+      "fontSize": 16,
+      "fontFamily": 3,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "baseline": 15
+    },
+    {
+      "type": "rectangle",
+      "version": 914,
+      "versionNonce": 1576666164,
+      "isDeleted": false,
+      "id": "R5v-yZCVJ97BkJqr0Qb7H",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 807.5664215087891,
+      "y": 287.8628387451172,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 1794212620,
+      "groupIds": [
+        "BJNOAY1MY3Evr9B3qQtHf"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "he-SpFjCxEQEWpWny2kKP",
+        "-pjrKo16rOsasM8viZPJ-",
+        "bZXA8PH9gYu-clotqJ4f7",
+        "MGdu6GDIPNBAaEUr0Gt-a"
+      ],
+      "updated": 1638347737174
+    },
+    {
+      "type": "text",
+      "version": 725,
+      "versionNonce": 267455500,
+      "isDeleted": false,
+      "id": "pWRC_smX7TuOI8_8UrA4H",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 840.0209197998047,
+      "y": 300.2379913330078,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 105,
+      "height": 24,
+      "seed": 421856180,
+      "groupIds": [
+        "BJNOAY1MY3Evr9B3qQtHf"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347701076,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmstorage",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 843,
+      "versionNonce": 1244193972,
+      "isDeleted": false,
+      "id": "EqROOfYulSPsZm7ovxfQN",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 604.9091339111328,
+      "y": 345.2847442626953,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 2043521972,
+      "groupIds": [
+        "ls6uq-W9bbVBM_UxAuyba"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "bZXA8PH9gYu-clotqJ4f7"
+      ],
+      "updated": 1638347718537
+    },
+    {
+      "type": "text",
+      "version": 676,
+      "versionNonce": 143370892,
+      "isDeleted": false,
+      "id": "ddQH1nnmT7HbKW7Xmv4zx",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 642.6199798583984,
+      "y": 357.90354919433594,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 335223180,
+      "groupIds": [
+        "ls6uq-W9bbVBM_UxAuyba"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "bZXA8PH9gYu-clotqJ4f7"
+      ],
+      "updated": 1638347701076,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vminsert",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "id": "bZXA8PH9gYu-clotqJ4f7",
+      "type": "arrow",
+      "x": 785.2667708463345,
+      "y": 367.2342882272049,
+      "width": 99.87819315552736,
+      "height": 24.681162491468,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 1153029388,
+      "version": 420,
+      "versionNonce": 1726025228,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638347704644,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          99.87819315552736,
+          -24.681162491468
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "EqROOfYulSPsZm7ovxfQN",
+        "focus": 0.5247890891198189,
+        "gap": 8.36404562660789
+      },
+      "endBinding": {
+        "elementId": "R5v-yZCVJ97BkJqr0Qb7H",
+        "focus": -0.6931056940383433,
+        "gap": 9.943033572650961
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "type": "arrow",
+      "version": 528,
+      "versionNonce": 1908259468,
+      "isDeleted": false,
+      "id": "MGdu6GDIPNBAaEUr0Gt-a",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 781.4456641707259,
+      "y": 248.777857603323,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 104.0940537386266,
+      "height": 27.60697400233846,
+      "seed": 1695061516,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638347737174,
+      "startBinding": {
+        "elementId": "Sa4OBd1ZjD6itohm7Ll8z",
+        "focus": -0.5921495247360469,
+        "gap": 6.398850205882127
+      },
+      "endBinding": {
+        "elementId": "R5v-yZCVJ97BkJqr0Qb7H",
+        "focus": 0.7021471697457312,
+        "gap": 11.478007139455713
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          104.0940537386266,
+          27.60697400233846
+        ]
+      ]
+    },
+    {
+      "type": "rectangle",
+      "version": 883,
+      "versionNonce": 845352076,
+      "isDeleted": false,
+      "id": "4pW8hvBu3bo1eMtFvg_gS",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 601.6520690917969,
+      "y": 473.7677536010742,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 1678097076,
+      "groupIds": [
+        "3kSpFrIN3kg4jwjDKpNWw"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "he-SpFjCxEQEWpWny2kKP",
+        "-pjrKo16rOsasM8viZPJ-",
+        "5W90eDBjtfZvkSuQTG0Iw"
+      ],
+      "updated": 1638347777173
+    },
+    {
+      "type": "text",
+      "version": 703,
+      "versionNonce": 519371956,
+      "isDeleted": false,
+      "id": "w_i2hO06oLa0bWbyAfFzU",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 639.3624267578125,
+      "y": 486.14290618896484,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 340753036,
+      "groupIds": [
+        "3kSpFrIN3kg4jwjDKpNWw"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347776748,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmselect",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 745,
+      "versionNonce": 879581324,
+      "isDeleted": false,
+      "id": "U5U-67wL5fPwvzlxOAF5A",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 592.7498321533203,
+      "y": 437.2059860229492,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 397.1286010742188,
+      "height": 209.62742614746097,
+      "seed": 912540724,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "pD9DcILMxa6GaR1U5YyMO"
+      ],
+      "updated": 1638347776748
+    },
+    {
+      "type": "text",
+      "version": 345,
+      "versionNonce": 1628116148,
+      "isDeleted": false,
+      "id": "0IGVrvICMVeZp2RCaqTeP",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "dotted",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 608.3194122314453,
+      "y": 444.93126678466797,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 291,
+      "height": 19,
+      "seed": 68700428,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347784373,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "VM cluster with aggregated data",
+      "baseline": 15,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 996,
+      "versionNonce": 2112461580,
+      "isDeleted": false,
+      "id": "9QMf0HXPE1W4M9S-zMJVO",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 806.1652679443359,
+      "y": 532.9768753051758,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 523607476,
+      "groupIds": [
+        "Eb2kWfz3ZWBu8cNul0h_c"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "he-SpFjCxEQEWpWny2kKP",
+        "-pjrKo16rOsasM8viZPJ-",
+        "9WBH34em4CdU2OwzqYIl5",
+        "5W90eDBjtfZvkSuQTG0Iw"
+      ],
+      "updated": 1638347777173
+    },
+    {
+      "type": "text",
+      "version": 804,
+      "versionNonce": 1660832052,
+      "isDeleted": false,
+      "id": "WOrOD5vn6EtMUQRJomt3-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 838.6197662353516,
+      "y": 545.3520278930664,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 105,
+      "height": 24,
+      "seed": 1321420684,
+      "groupIds": [
+        "Eb2kWfz3ZWBu8cNul0h_c"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638347776748,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmstorage",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 925,
+      "versionNonce": 2052807604,
+      "isDeleted": false,
+      "id": "4dtJZpXEUxSK3biwFG7vd",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 603.5079803466797,
+      "y": 590.3987808227539,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 171.99359130859375,
+      "height": 44.74725341796875,
+      "seed": 1738524468,
+      "groupIds": [
+        "punXEDFtHkSpcd9seAkZj"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "9WBH34em4CdU2OwzqYIl5",
+        "EyecK0YM9Cc8T6ju-nTOc"
+      ],
+      "updated": 1638347812431
+    },
+    {
+      "type": "text",
+      "version": 756,
+      "versionNonce": 2040967820,
+      "isDeleted": false,
+      "id": "rAbyooo-X08-86qjoK0WR",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 641.2188262939453,
+      "y": 603.0175857543945,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 94,
+      "height": 24,
+      "seed": 948598284,
+      "groupIds": [
+        "punXEDFtHkSpcd9seAkZj"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "9WBH34em4CdU2OwzqYIl5"
+      ],
+      "updated": 1638347776748,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vminsert",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "arrow",
+      "version": 660,
+      "versionNonce": 1560678196,
+      "isDeleted": false,
+      "id": "9WBH34em4CdU2OwzqYIl5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 783.8656172818814,
+      "y": 612.3483247872634,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 99.87819315552736,
+      "height": 24.681162491468,
+      "seed": 1141424308,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638347777173,
+      "startBinding": {
+        "elementId": "4dtJZpXEUxSK3biwFG7vd",
+        "focus": 0.5247890891198189,
+        "gap": 8.364045626608004
+      },
+      "endBinding": {
+        "elementId": "9QMf0HXPE1W4M9S-zMJVO",
+        "focus": -0.6931056940383404,
+        "gap": 9.943033572650847
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          99.87819315552736,
+          -24.681162491468
+        ]
+      ]
+    },
+    {
+      "type": "arrow",
+      "version": 768,
+      "versionNonce": 1653264948,
+      "isDeleted": false,
+      "id": "5W90eDBjtfZvkSuQTG0Iw",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 780.0445106062728,
+      "y": 493.8918941633816,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 104.0940537386266,
+      "height": 27.60697400233846,
+      "seed": 1852298380,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638347777173,
+      "startBinding": {
+        "elementId": "4pW8hvBu3bo1eMtFvg_gS",
+        "focus": -0.5921495247360469,
+        "gap": 6.398850205882127
+      },
+      "endBinding": {
+        "elementId": "9QMf0HXPE1W4M9S-zMJVO",
+        "focus": 0.7021471697457312,
+        "gap": 11.478007139455713
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          104.0940537386266,
+          27.60697400233846
+        ]
+      ]
+    },
+    {
+      "id": "HPEwr85wL4IedW0AgdArp",
+      "type": "arrow",
+      "x": 423.70701599121094,
+      "y": 428.34434509277344,
+      "width": 179.54901123046875,
+      "height": 182.9937744140625,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 389863732,
+      "version": 47,
+      "versionNonce": 1364399028,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638347805500,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          179.54901123046875,
+          -182.9937744140625
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": 0.6700023593531782,
+        "gap": 10.266586303710938
+      },
+      "endBinding": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "id": "EyecK0YM9Cc8T6ju-nTOc",
+      "type": "arrow",
+      "x": 424.7585906982422,
+      "y": 441.12806701660156,
+      "width": 174.29449462890625,
+      "height": 170.56607055664062,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "seed": 981082124,
+      "version": 92,
+      "versionNonce": 1261546252,
+      "isDeleted": false,
+      "boundElementIds": null,
+      "updated": 1638347812431,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          174.29449462890625,
+          170.56607055664062
+        ]
+      ],
+      "lastCommittedPoint": null,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "focus": -0.6826568395144794,
+        "gap": 11.318161010742188
+      },
+      "endBinding": {
+        "elementId": "4dtJZpXEUxSK3biwFG7vd",
+        "focus": -0.8207814548026305,
+        "gap": 4.45489501953125
+      },
+      "startArrowhead": null,
+      "endArrowhead": "arrow"
+    },
+    {
+      "type": "text",
+      "version": 775,
+      "versionNonce": 95530764,
+      "isDeleted": false,
+      "id": "o-yIJ_WZzVubhbVODvhcC",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 407.27012634277344,
+      "y": 489.33091735839844,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 141,
+      "height": 19,
+      "seed": 775116812,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "wRO0q9xKPHc8e8XPPsQWh"
+      ],
+      "updated": 1638347824876,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "persist results",
+      "baseline": 15,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}
--- a/app/vmalert/vmalert_multicluster.png
+++ b/app/vmalert/vmalert_multicluster.png
--- a/app/vmalert/vmalert_single.excalidraw
+++ b/app/vmalert/vmalert_single.excalidraw
@ -0,0 +1,354 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 795,
+      "versionNonce": 1195460364,
+      "isDeleted": false,
+      "id": "VgBUzo0blGR-Ijd2mQEEf",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 422.3502197265625,
+      "y": 215.55953979492188,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 123.7601318359375,
+      "height": 72.13211059570312,
+      "seed": 1194011660,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "miEbzHxOPXe4PEYvXiJp5",
+        "rcmiQfIWtfbTTlwxqr1sl",
+        "P-dpWlSTtnsux-zr5oqgF",
+        "oAToSPttH7aWoD_AqXGFX",
+        "Bpy5by47XGKB4yS99ZkuA",
+        "wRO0q9xKPHc8e8XPPsQWh",
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348116633
+    },
+    {
+      "type": "text",
+      "version": 660,
+      "versionNonce": 1034125236,
+      "isDeleted": false,
+      "id": "e9TDm09y-GhPm84XWt0Jv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 443.89678955078125,
+      "y": 236.64378356933594,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 82,
+      "height": 24,
+      "seed": 327273100,
+      "groupIds": [
+        "iBaXgbpyifSwPplm_GO5b"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348116633,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "vmalert",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "middle"
+    },
+    {
+      "type": "rectangle",
+      "version": 1690,
+      "versionNonce": 236788620,
+      "isDeleted": false,
+      "id": "dd52BjHfPMPRji9Tws7U-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 344.9837951660156,
+      "y": 64.64248657226562,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 269.336669921875,
+      "height": 44.74725341796875,
+      "seed": 1779959692,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "Bpy5by47XGKB4yS99ZkuA"
+      ],
+      "updated": 1638348176044
+    },
+    {
+      "type": "text",
+      "version": 1331,
+      "versionNonce": 945335476,
+      "isDeleted": false,
+      "id": "9TEzv0sVCHAkc46ou0oNF",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 364.35296630859375,
+      "y": 76.77398681640625,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 234,
+      "height": 24,
+      "seed": 1617178804,
+      "groupIds": [
+        "2Lijjn3PwPQW_8KrcDmdu"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348176045,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "victoriametrics:8428",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "rectangle",
+      "version": 1099,
+      "versionNonce": 671872012,
+      "isDeleted": false,
+      "id": "8-XFSbd6Zw96EUSJbJXZv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 360.6495666503906,
+      "y": 382.2536315917969,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 240.10644531249997,
+      "height": 44.74725341796875,
+      "seed": 99322124,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348116633
+    },
+    {
+      "type": "text",
+      "version": 865,
+      "versionNonce": 635839156,
+      "isDeleted": false,
+      "id": "GUs816aggGqUSdoEsSmea",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 382.64422607421875,
+      "y": 394.3433837890625,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 199,
+      "height": 24,
+      "seed": 1194745268,
+      "groupIds": [
+        "6obQBPHIfExBKfejeLLVO"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348116633,
+      "fontSize": 20,
+      "fontFamily": 3,
+      "text": "alertmanager:9093",
+      "baseline": 19,
+      "textAlign": "center",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "arrow",
+      "version": 2444,
+      "versionNonce": 361351692,
+      "isDeleted": false,
+      "id": "Bpy5by47XGKB4yS99ZkuA",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 486.0465703465111,
+      "y": 204.98379516601562,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 0.7962088410123442,
+      "height": 86.86798095703125,
+      "seed": 357577356,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638348176045,
+      "startBinding": {
+        "elementId": "VgBUzo0blGR-Ijd2mQEEf",
+        "gap": 10.57574462890625,
+        "focus": 0.0344528515859526
+      },
+      "endBinding": {
+        "elementId": "dd52BjHfPMPRji9Tws7U-",
+        "gap": 8.72607421875,
+        "focus": -0.039393828258510157
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -0.7962088410123442,
+          -86.86798095703125
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 640,
+      "versionNonce": 1892951180,
+      "isDeleted": false,
+      "id": "RbVSa4PnOgAMtzoKb-DhW",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 500.609619140625,
+      "y": 134.11544799804688,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 188,
+      "height": 95,
+      "seed": 1989838604,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [],
+      "updated": 1638348163424,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "execute rules,\npersist alerts \nand recording rules,\nrestore state\n",
+      "baseline": 91,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    },
+    {
+      "type": "arrow",
+      "version": 1472,
+      "versionNonce": 768565516,
+      "isDeleted": false,
+      "id": "sxEhnxlbT7ldlSsmHDUHp",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 486.5245361328125,
+      "y": 300.5478957313172,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 0.3521007656264601,
+      "height": 70.26802223743277,
+      "seed": 1818348300,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElementIds": [],
+      "updated": 1638348116633,
+      "startBinding": {
+        "elementId": "E9Run6wCm2chQ6JHrmc_y",
+        "focus": 1.1925203824459496,
+        "gap": 11.77154541015625
+      },
+      "endBinding": {
+        "elementId": "8-XFSbd6Zw96EUSJbJXZv",
+        "focus": 0.0441077573536454,
+        "gap": 11.437713623046875
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -0.3521007656264601,
+          70.26802223743277
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 577,
+      "versionNonce": 1646003636,
+      "isDeleted": false,
+      "id": "E9Run6wCm2chQ6JHrmc_y",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 0,
+      "opacity": 100,
+      "angle": 0,
+      "x": 498.29608154296875,
+      "y": 298.6573791503906,
+      "strokeColor": "#000000",
+      "backgroundColor": "transparent",
+      "width": 122,
+      "height": 38,
+      "seed": 1836541708,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElementIds": [
+        "sxEhnxlbT7ldlSsmHDUHp"
+      ],
+      "updated": 1638348116633,
+      "fontSize": 16,
+      "fontFamily": 3,
+      "text": "send alert \nnotifications",
+      "baseline": 34,
+      "textAlign": "left",
+      "verticalAlign": "top"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}
--- a/app/vmalert/vmalert_single.png
+++ b/app/vmalert/vmalert_single.png
--- a/app/vmalert/web.go
+++ b/app/vmalert/web.go
@ -68,7 +68,7 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.Write(data)
 		return true
 	case "/api/v1/alerts":
@ -77,7 +77,7 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 			httpserver.Errorf(w, r, "%s", err)
 			return true
 		}
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.Write(data)
 		return true
 	case "/-/reload":
@ -102,7 +102,7 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 				httpserver.Errorf(w, r, "failed to marshal alert: %s", err)
 				return true
 			}
-			w.Header().Set("Content-Type", "application/json; charset=utf-8")
+			w.Header().Set("Content-Type", "application/json")
 			w.Write(data)
 			return true
 		}
--- a/app/vmalert/web.qtpl
+++ b/app/vmalert/web.qtpl
@ -54,10 +54,10 @@
                 {% if rNotOk[g.Name] > 0 %}<span class="badge bg-danger" title="Number of rules with status Error">{%d rNotOk[g.Name] %}</span> {% endif %}
                <span class="badge bg-success" title="Number of rules withs status Ok">{%d rOk[g.Name] %}</span>
                <p class="fs-6 fw-lighter">{%s g.File %}</p>
-                {% if len(g.ExtraFilterLabels) > 0 %}
-                    <div class="fs-6 fw-lighter">Extra filter labels
-                    {% for k, v := range g.ExtraFilterLabels %}
-                            <span class="float-left badge bg-primary">{%s k %}={%s v %}</span>
+                {% if len(g.Params) > 0 %}
+                    <div class="fs-6 fw-lighter">Extra params
+                    {% for _, param := range g.Params %}
+                            <span class="float-left badge bg-primary">{%s param %}</span>
                    {% endfor %}
                    </div>
                {% endif %}
--- a/app/vmalert/web.qtpl.go
+++ b/app/vmalert/web.qtpl.go
@ -211,22 +211,18 @@ func StreamListGroups(qw422016 *qt422016.Writer, groups []APIGroup) {
 			qw422016.N().S(`</p>
                `)
 //line app/vmalert/web.qtpl:57
-			if len(g.ExtraFilterLabels) > 0 {
+			if len(g.Params) > 0 {
 //line app/vmalert/web.qtpl:57
 				qw422016.N().S(`
-                    <div class="fs-6 fw-lighter">Extra filter labels
+                    <div class="fs-6 fw-lighter">Extra params
                    `)
 //line app/vmalert/web.qtpl:59
-				for k, v := range g.ExtraFilterLabels {
+				for _, param := range g.Params {
 //line app/vmalert/web.qtpl:59
 					qw422016.N().S(`
                            <span class="float-left badge bg-primary">`)
 //line app/vmalert/web.qtpl:60
-					qw422016.E().S(k)
-//line app/vmalert/web.qtpl:60
-					qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:60
-					qw422016.E().S(v)
+					qw422016.E().S(param)
 //line app/vmalert/web.qtpl:60
 					qw422016.N().S(`</span>
                    `)
--- a/app/vmalert/web_types.go
+++ b/app/vmalert/web_types.go
@ -29,7 +29,7 @@ type APIGroup struct {
 	File           string             `json:"file"`
 	Interval       string             `json:"interval"`
 	Concurrency    int                `json:"concurrency"`
-	ExtraFilterLabels map[string]string  `json:"extra_filter_labels"`
+	Params         []string           `json:"params"`
 	Labels         map[string]string  `json:"labels,omitempty"`
 	AlertingRules  []APIAlertingRule  `json:"alerting_rules"`
 	RecordingRules []APIRecordingRule `json:"recording_rules"`
--- a/app/vmauth/README.md
+++ b/app/vmauth/README.md
@ -43,6 +43,7 @@ Each `url_prefix` in the [-auth.config](#auth-config) may contain either a singl
 users:
  # Requests with the 'Authorization: Bearer XXXX' header are proxied to http://localhost:8428 .
  # For example, http://vmauth:8427/api/v1/query is proxied to http://localhost:8428/api/v1/query
+  # Requests with the Basic Auth username=XXXX are proxied to http://localhost:8428 as well.
 - bearer_token: "XXXX"
  url_prefix: "http://localhost:8428"

@ -147,6 +148,14 @@ It is recommended protecting `/-/reload` endpoint with `-reloadAuthKey` command-
 `vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page
 either via [vmagent](https://docs.victoriametrics.com/vmagent.html) or via Prometheus, so the exported metrics could be analyzed later.

+`vmauth` exports `vmauth_user_requests_total` metric with `username` label. The `username` label value equals to `username` field value set in the `-auth.config` file. It is possible to override or hide the value in the label by specifying `name` field. For example, the following config will result in `vmauth_user_requests_total{username="foobar"}` instead of `vmauth_user_requests_total{username="secret_user"}`:
+
+```yml
+users:
+- username: "secret_user"
+  name: "foobar"
+  # other config options here
+```

 ## How to build from sources

--- a/app/vmauth/auth_config.go
+++ b/app/vmauth/auth_config.go
@ -32,6 +32,7 @@ type AuthConfig struct {

 // UserInfo is user information read from authConfigPath
 type UserInfo struct {
+	Name        string     `yaml:"name,omitempty"`
 	BearerToken string     `yaml:"bearer_token,omitempty"`
 	Username    string     `yaml:"username,omitempty"`
 	Password    string     `yaml:"password,omitempty"`
@ -275,9 +276,12 @@ func parseAuthConfig(data []byte) (map[string]*UserInfo, error) {
 		if byUsername[ui.Username] {
 			return nil, fmt.Errorf("duplicate username found; username: %q", ui.Username)
 		}
-		authToken := getAuthToken(ui.BearerToken, ui.Username, ui.Password)
-		if byAuthToken[authToken] != nil {
-			return nil, fmt.Errorf("duplicate auth token found for bearer_token=%q, username=%q: %q", authToken, ui.BearerToken, ui.Username)
+		at1, at2 := getAuthTokens(ui.BearerToken, ui.Username, ui.Password)
+		if byAuthToken[at1] != nil {
+			return nil, fmt.Errorf("duplicate auth token found for bearer_token=%q, username=%q: %q", ui.BearerToken, ui.Username, at1)
+		}
+		if byAuthToken[at2] != nil {
+			return nil, fmt.Errorf("duplicate auth token found for bearer_token=%q, username=%q: %q", ui.BearerToken, ui.Username, at2)
 		}
 		if ui.URLPrefix != nil {
 			if err := ui.URLPrefix.sanitize(); err != nil {
@ -299,21 +303,41 @@ func parseAuthConfig(data []byte) (map[string]*UserInfo, error) {
 			return nil, fmt.Errorf("missing `url_prefix`")
 		}
 		if ui.BearerToken != "" {
+			name := "bearer_token"
+			if ui.Name != "" {
+				name = ui.Name
+			}
 			if ui.Password != "" {
 				return nil, fmt.Errorf("password shouldn't be set for bearer_token %q", ui.BearerToken)
 			}
-			ui.requests = metrics.GetOrCreateCounter(`vmauth_user_requests_total{username="bearer_token"}`)
+			ui.requests = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_requests_total{username=%q}`, name))
 			byBearerToken[ui.BearerToken] = true
 		}
 		if ui.Username != "" {
-			ui.requests = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_requests_total{username=%q}`, ui.Username))
+			name := ui.Username
+			if ui.Name != "" {
+				name = ui.Name
+			}
+			ui.requests = metrics.GetOrCreateCounter(fmt.Sprintf(`vmauth_user_requests_total{username=%q}`, name))
 			byUsername[ui.Username] = true
 		}
-		byAuthToken[authToken] = ui
+		byAuthToken[at1] = ui
+		byAuthToken[at2] = ui
 	}
 	return byAuthToken, nil
 }

+func getAuthTokens(bearerToken, username, password string) (string, string) {
+	if bearerToken != "" {
+		// Accept the bearerToken as Basic Auth username with empty password
+		at1 := getAuthToken(bearerToken, "", "")
+		at2 := getAuthToken("", bearerToken, "")
+		return at1, at2
+	}
+	at := getAuthToken("", username, password)
+	return at, at
+}
+
 func getAuthToken(bearerToken, username, password string) string {
 	if bearerToken != "" {
 		return "Bearer " + bearerToken
--- a/app/vmauth/auth_config_test.go
+++ b/app/vmauth/auth_config_test.go
@ -290,6 +290,32 @@ users:
 				},
 			},
 		},
+		getAuthToken("", "foo", ""): {
+			BearerToken: "foo",
+			URLMap: []URLMap{
+				{
+					SrcPaths:  getSrcPaths([]string{"/api/v1/query", "/api/v1/query_range", "/api/v1/label/[^./]+/.+"}),
+					URLPrefix: mustParseURL("http://vmselect/select/0/prometheus"),
+				},
+				{
+					SrcPaths: getSrcPaths([]string{"/api/v1/write"}),
+					URLPrefix: mustParseURLs([]string{
+						"http://vminsert1/insert/0/prometheus",
+						"http://vminsert2/insert/0/prometheus",
+					}),
+					Headers: []Header{
+						{
+							Name:  "foo",
+							Value: "bar",
+						},
+						{
+							Name:  "xxx",
+							Value: "y",
+						},
+					},
+				},
+			},
+		},
 	})
 }

--- a/app/vmauth/example_config.yml
+++ b/app/vmauth/example_config.yml
@ -4,6 +4,7 @@
 users:
  # Requests with the 'Authorization: Bearer XXXX' header are proxied to http://localhost:8428 .
  # For example, http://vmauth:8427/api/v1/query is proxied to http://localhost:8428/api/v1/query
+  # Requests with the Basic Auth username=XXXX are proxied to http://localhost:8428 as well.
 - bearer_token: "XXXX"
  url_prefix: "http://localhost:8428"

--- a/app/vmauth/main.go
+++ b/app/vmauth/main.go
@ -7,6 +7,7 @@ import (
 	"net/http/httputil"
 	"net/url"
 	"os"
+	"sync"
 	"time"

 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
@ -108,7 +109,7 @@ func proxyRequest(w http.ResponseWriter, r *http.Request) {
 		// Forward other panics to the caller.
 		panic(err)
 	}()
-	reverseProxy.ServeHTTP(w, r)
+	getReverseProxy().ServeHTTP(w, r)
 }

 var (
@ -117,7 +118,19 @@ var (
 	missingRouteRequests     = metrics.NewCounter(`vmauth_http_request_errors_total{reason="missing_route"}`)
 )

-var reverseProxy = &httputil.ReverseProxy{
+var (
+	reverseProxy     *httputil.ReverseProxy
+	reverseProxyOnce sync.Once
+)
+
+func getReverseProxy() *httputil.ReverseProxy {
+	reverseProxyOnce.Do(initReverseProxy)
+	return reverseProxy
+}
+
+// initReverseProxy must be called after flag.Parse(), since it uses command-line flags.
+func initReverseProxy() {
+	reverseProxy = &httputil.ReverseProxy{
 		Director: func(r *http.Request) {
 			targetURL := r.Header.Get("vm-target-url")
 			target, err := url.Parse(targetURL)
@ -140,6 +153,7 @@ var reverseProxy = &httputil.ReverseProxy{
 		}(),
 		FlushInterval: time.Second,
 		ErrorLog:      logger.StdErrorLogger(),
+	}
 }

 func usage() {
--- a/app/vmbackup/README.md
+++ b/app/vmbackup/README.md
@ -185,12 +185,32 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
  -dst string
    	Where to put the backup on the remote storage. Example: gs://bucket/path/to/backup/dir, s3://bucket/path/to/backup/dir or fs:///path/to/local/backup/dir
    	-dst can point to the previous backup. In this case incremental backup is performed, i.e. only changed data is uploaded
+  -enableTCP6
+    	Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP and UDP is used
  -envflag.enable
    	Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
  -envflag.prefix string
    	Prefix for environment variables if -envflag.enable is set
  -fs.disableMmap
    	Whether to use pread() instead of mmap() for reading data files. By default mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help to spread the incoming load among a cluster of services behind a load balancer. Please note that the real timeout may be bigger by up to 10% as a protection against the thundering herd problem (default 2m0s)
+  -http.disableResponseCompression
+    	Disable compression of HTTP responses to save CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
+  -http.maxGracefulShutdownDuration duration
+    	The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
+  -http.pathPrefix string
+    	An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
+  -http.shutdownDelay duration
+    	Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
+  -httpAuth.password string
+    	Password for HTTP Basic Auth. The authentication is disabled if -httpAuth.username is empty
+  -httpAuth.username string
+    	Username for HTTP Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
+  -httpListenAddr string
+    	TCP address for exporting metrics at /metrics page (default ":8420")
  -loggerDisableTimestamps
    	Whether to disable writing timestamps in logs
  -loggerErrorsPerSecondLimit int
@ -213,8 +233,14 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
    	Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
  -memory.allowedPercent float
    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from OS page cache which will result in higher disk IO usage (default 60)
+  -metricsAuthKey string
+    	Auth key for /metrics. It must be passed via authKey query arg. It overrides httpAuth.* settings
  -origin string
    	Optional origin directory on the remote storage with old backup for server-side copying when performing full backup. This speeds up full backups
+  -pprofAuthKey string
+    	Auth key for /debug/pprof. It must be passed via authKey query arg. It overrides httpAuth.* settings
+  -s3ForcePathStyle
+    	Prefixing endpoint with bucket name when set false, true by default. (default true)
  -snapshot.createURL string
    	VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup. Example: http://victoriametrics:8428/snapshot/create . There is no need in setting -snapshotName if -snapshot.createURL is set
  -snapshot.deleteURL string
@ -223,6 +249,12 @@ See [this article](https://medium.com/@valyala/speeding-up-backups-for-big-time-
    	Name for the snapshot to backup. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-work-with-snapshots. There is no need in setting -snapshotName if -snapshot.createURL is set
  -storageDataPath string
    	Path to VictoriaMetrics data. Must match -storageDataPath from VictoriaMetrics or vmstorage (default "victoria-metrics-data")
+  -tls
+    	Whether to enable TLS (aka HTTPS) for incoming requests. -tlsCertFile and -tlsKeyFile must be set if -tls is set
+  -tlsCertFile string
+    	Path to file with TLS certificate. Used only if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower
+  -tlsKeyFile string
+    	Path to file with TLS key. Used only if -tls is set
  -version
    	Show VictoriaMetrics version
 ```
--- a/app/vmbackup/main.go
+++ b/app/vmbackup/main.go
@ -5,6 +5,7 @@ import (
 	"fmt"
 	"os"
 	"strings"
+	"time"

 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmbackup/snapshot"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/actions"
@ -14,10 +15,12 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )

 var (
+	httpListenAddr    = flag.String("httpListenAddr", ":8420", "TCP address for exporting metrics at /metrics page")
 	storageDataPath   = flag.String("storageDataPath", "victoria-metrics-data", "Path to VictoriaMetrics data. Must match -storageDataPath from VictoriaMetrics or vmstorage")
 	snapshotName      = flag.String("snapshotName", "", "Name for the snapshot to backup. See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-work-with-snapshots. There is no need in setting -snapshotName if -snapshot.createURL is set")
 	snapshotCreateURL = flag.String("snapshot.createURL", "", "VictoriaMetrics create snapshot url. When this is given a snapshot will automatically be created during backup. "+
@ -70,6 +73,9 @@ func main() {
 		}()
 	}

+	logger.Infof("starting http server for exporting metrics at http://%q/metrics", *httpListenAddr)
+	go httpserver.Serve(*httpListenAddr, nil)
+
 	srcFS, err := newSrcFS()
 	if err != nil {
 		logger.Fatalf("%s", err)
@ -94,6 +100,13 @@ func main() {
 	srcFS.MustStop()
 	dstFS.MustStop()
 	originFS.MustStop()
+
+	startTime := time.Now()
+	logger.Infof("gracefully shutting down http server for metrics at %q", *httpListenAddr)
+	if err := httpserver.Stop(*httpListenAddr); err != nil {
+		logger.Fatalf("cannot stop http server for metrics: %s", err)
+	}
+	logger.Infof("successfully shut down http server for metrics in %.3f seconds", time.Since(startTime).Seconds())
 }

 func usage() {
--- a/app/vmctl/flags.go
+++ b/app/vmctl/flags.go
@ -155,7 +155,7 @@ var (
 		&cli.IntFlag{
 			Name:  otsdbQueryLimit,
 			Usage: "Result limit on meta queries to OpenTSDB (affects both metric name and tag value queries, recommended to use a value exceeding your largest series)",
-			Value: 100e3,
+			Value: 100e6,
 		},
 		&cli.BoolFlag{
 			Name:  otsdbMsecsTime,
--- a/app/vmctl/opentsdb/opentsdb.go
+++ b/app/vmctl/opentsdb/opentsdb.go
@ -1,7 +1,6 @@
 package opentsdb

 import (
-	"bytes"
 	"encoding/json"
 	"fmt"
 	"io/ioutil"
@ -88,7 +87,15 @@ type Meta struct {
 	Tags   map[string]string `json:"tags"`
 }

-// Metric holds the time series data
+// OtsdbMetric is a single series in OpenTSDB's returned format
+type OtsdbMetric struct {
+	Metric        string
+	Tags          map[string]string
+	AggregateTags []string
+	Dps           map[int64]float64
+}
+
+// Metric holds the time series data in VictoriaMetrics format
 type Metric struct {
 	Metric     string
 	Tags       map[string]string
@ -96,83 +103,6 @@ type Metric struct {
 	Values     []float64
 }

-// ExpressionOutput contains results from actual data queries
-type ExpressionOutput struct {
-	Outputs []qoObj     `json:"outputs"`
-	Query   interface{} `json:"query"`
-}
-
-// QoObj contains actual timeseries data from the returned data query
-type qoObj struct {
-	ID    string      `json:"id"`
-	Alias string      `json:"alias"`
-	Dps   [][]float64 `json:"dps"`
-	//dpsMeta interface{}
-	//meta    interface{}
-}
-
-// Expression objects format our data queries
-/*
-All of the following structs are to build a OpenTSDB expression object
-*/
-type Expression struct {
-	Time    timeObj     `json:"time"`
-	Filters []filterObj `json:"filters"`
-	Metrics []metricObj `json:"metrics"`
-	// this just needs to be an empty object, so the value doesn't matter
-	Expressions []int       `json:"expressions"`
-	Outputs     []outputObj `json:"outputs"`
-}
-
-type timeObj struct {
-	Start       int64  `json:"start"`
-	End         int64  `json:"end"`
-	Aggregator  string `json:"aggregator"`
-	Downsampler dSObj  `json:"downsampler"`
-}
-
-type dSObj struct {
-	Interval   string  `json:"interval"`
-	Aggregator string  `json:"aggregator"`
-	FillPolicy fillObj `json:"fillPolicy"`
-}
-
-type fillObj struct {
-	// we'll always hard-code to NaN here, so we don't need value
-	Policy string `json:"policy"`
-}
-
-type filterObj struct {
-	Tags []tagObj `json:"tags"`
-	ID   string   `json:"id"`
-}
-
-type tagObj struct {
-	Type    string `json:"type"`
-	Tagk    string `json:"tagk"`
-	Filter  string `json:"filter"`
-	GroupBy bool   `json:"groupBy"`
-}
-
-type metricObj struct {
-	ID         string  `json:"id"`
-	Metric     string  `json:"metric"`
-	Filter     string  `json:"filter"`
-	FillPolicy fillObj `json:"fillPolicy"`
-}
-
-type outputObj struct {
-	ID    string `json:"id"`
-	Alias string `json:"alias"`
-}
-
-/* End expression object structs */
-
-var (
-	exprOutput     = outputObj{ID: "a", Alias: "query"}
-	exprFillPolicy = fillObj{Policy: "nan"}
-)
-
 // FindMetrics discovers all metrics that OpenTSDB knows about (given a filter)
 // e.g. /api/suggest?type=metrics&q=system&max=100000
 func (c Client) FindMetrics(q string) ([]string, error) {
@ -221,41 +151,39 @@ func (c Client) FindSeries(metric string) ([]Meta, error) {
 }

 // GetData actually retrieves data for a series at a specified time range
+// e.g. /api/query?start=1&end=200&m=sum:1m-avg-none:system.load5{host=host1}
 func (c Client) GetData(series Meta, rt RetentionMeta, start int64, end int64) (Metric, error) {
 	/*
-		Here we build the actual exp query we'll send to OpenTSDB
-
-		This is comprised of a number of different settings. We hard-code
-		a few to simplify the JSON object creation.
-		There are examples queries available, so not too much detail here...
+		First, build our tag string.
+		It's literally just key=value,key=value,...
 	*/
-	expr := Expression{}
-	expr.Outputs = []outputObj{exprOutput}
-	expr.Metrics = append(expr.Metrics, metricObj{ID: "a", Metric: series.Metric,
-		Filter: "f1", FillPolicy: exprFillPolicy})
-	expr.Time = timeObj{Start: start, End: end, Aggregator: rt.FirstOrder,
-		Downsampler: dSObj{Interval: rt.AggTime,
-			Aggregator: rt.SecondOrder,
-			FillPolicy: exprFillPolicy}}
-	var TagList []tagObj
+	tagStr := ""
 	for k, v := range series.Tags {
-		/*
-			every tag should be a literal_or because that's the closest to a full "==" that
-			this endpoint allows for
-		*/
-		TagList = append(TagList, tagObj{Type: "literal_or", Tagk: k,
-			Filter: v, GroupBy: true})
-	}
-	expr.Filters = append(expr.Filters, filterObj{ID: "f1", Tags: TagList})
-	// "expressions" is required in the query object or we get a 5xx, so force it to exist
-	expr.Expressions = make([]int, 0)
-	inputData, err := json.Marshal(expr)
-	if err != nil {
-		return Metric{}, fmt.Errorf("failed to marshal query JSON %s", err)
+		tagStr += fmt.Sprintf("%s=%s,", k, v)
 	}
+	// obviously we don't want trailing commas...
+	tagStr = strings.Trim(tagStr, ",")

-	q := fmt.Sprintf("%s/api/query/exp", c.Addr)
-	resp, err := http.Post(q, "application/json", bytes.NewBuffer(inputData))
+	/*
+		The aggregation policy should already be somewhat formatted:
+		FirstOrder (e.g. sum/avg/max/etc.)
+		SecondOrder (e.g. sum/avg/max/etc.)
+		AggTime	(e.g. 1m/10m/1d/etc.)
+		This will build into m=<FirstOrder>:<AggTime>-<SecondOrder>-none:
+		Or an example: m=sum:1m-avg-none
+	*/
+	aggPol := fmt.Sprintf("%s:%s-%s-none", rt.FirstOrder, rt.AggTime, rt.SecondOrder)
+
+	/*
+		Our actual query string:
+		Start and End are just timestamps
+		We then add the aggregation policy, the metric, and the tag set
+	*/
+	queryStr := fmt.Sprintf("start=%v&end=%v&m=%s:%s{%s}", start, end, aggPol,
+		series.Metric, tagStr)
+
+	q := fmt.Sprintf("%s/api/query?%s", c.Addr, queryStr)
+	resp, err := http.Get(q)
 	if err != nil {
 		return Metric{}, fmt.Errorf("failed to send GET request to %q: %s", q, err)
 	}
@ -267,28 +195,63 @@ func (c Client) GetData(series Meta, rt RetentionMeta, start int64, end int64) (
 	if err != nil {
 		return Metric{}, fmt.Errorf("could not retrieve series data from %q: %s", q, err)
 	}
-	var output ExpressionOutput
+	var output []OtsdbMetric
 	err = json.Unmarshal(body, &output)
 	if err != nil {
-		return Metric{}, fmt.Errorf("failed to unmarshal response from %q: %s", q, err)
+		return Metric{}, fmt.Errorf("failed to unmarshal response from %q [%v]: %s", q, body, err)
 	}
-	if len(output.Outputs) < 1 {
+	/*
+		We expect results to look like:
+		[
+		  {
+			"metric": "zfs_filesystem.available",
+			"tags": {
+			  "rack": "6",
+			  "replica": "1",
+			  "host": "c7-bfyii-115",
+			  "pool": "dattoarray",
+			  "row": "c",
+			  "dc": "us-west-3",
+			  "group": "legonode"
+			},
+			"aggregateTags": [],
+			"dps": {
+			  "1626019200": 32490602877610.668,
+			  "1626033600": 32486439014058.668
+			}
+		  }
+		]
+		There are two things that could be bad here:
+			1. There are no actual stats returned (an empty array -> [])
+			2. There are aggregate tags in the results
+		An empty array doesn't cast to a OtsdbMetric struct well, and there's no reason to try, so we should just skip it
+		Because we're trying to migrate data without transformations, seeing aggregate tags could mean
+		we're dropping series on the floor.
+	*/
+	if len(output) < 1 {
 		// no results returned...return an empty object without error
 		return Metric{}, nil
 	}
+	if len(output) > 1 {
+		// multiple series returned for a single query. We can't process this right, so...
+		return Metric{}, fmt.Errorf("Query returned multiple results: %v", output)
+	}
+	if len(output[0].AggregateTags) > 0 {
+		// This failure means we've suppressed potential series somehow...
+		return Metric{}, fmt.Errorf("Query somehow has aggregate tags: %v", output[0].AggregateTags)
+	}
 	data := Metric{}
-	data.Metric = series.Metric
-	data.Tags = series.Tags
+	data.Metric = output[0].Metric
+	data.Tags = output[0].Tags
 	/*
 		We evaluate data for correctness before formatting the actual values
 		to skip a little bit of time if the series has invalid formatting
-
-		First step is to enforce Prometheus' data model
 	*/
 	data, err = modifyData(data, c.Normalize)
 	if err != nil {
 		return Metric{}, fmt.Errorf("invalid series data from %q: %s", q, err)
 	}
+
 	/*
 		Convert data from OpenTSDB's output format ([[ts,val],[ts,val]...])
 		to VictoriaMetrics format: {"timestamps": [ts,ts,ts...], "values": [val,val,val...]}
@ -296,9 +259,9 @@ func (c Client) GetData(series Meta, rt RetentionMeta, start int64, end int64) (
 		can be a float64, we have to initially cast _all_ objects that way
 		then convert the timestamp back to something reasonable.
 	*/
-	for _, tsobj := range output.Outputs[0].Dps {
-		data.Timestamps = append(data.Timestamps, int64(tsobj[0]))
-		data.Values = append(data.Values, tsobj[1])
+	for ts, val := range output[0].Dps {
+		data.Timestamps = append(data.Timestamps, ts)
+		data.Values = append(data.Values, val)
 	}
 	return data, nil
 }
@ -308,9 +271,12 @@ func (c Client) GetData(series Meta, rt RetentionMeta, start int64, end int64) (
 func NewClient(cfg Config) (*Client, error) {
 	var retentions []Retention
 	offsetPrint := int64(time.Now().Unix())
+	offsetSecs := cfg.Offset * 24 * 60 * 60
 	if cfg.MsecsTime {
 		// 1000000 == Nanoseconds -> Milliseconds difference
 		offsetPrint = int64(time.Now().UnixNano() / 1000000)
+		// also bump offsetSecs to milliseconds
+		offsetSecs = offsetSecs * 1000
 	}
 	if cfg.HardTS > 0 {
 		/*
@ -318,20 +284,16 @@ func NewClient(cfg Config) (*Client, error) {
 			Just present that if it is defined
 		*/
 		offsetPrint = cfg.HardTS
-	} else if cfg.Offset > 0 {
+	} else if offsetSecs > 0 {
 		/*
-			Our "offset" is the number of days we should step
+			Our "offset" is the number of days (in seconds) we should step
 			back before starting to scan for data
 		*/
-		if cfg.MsecsTime {
-			offsetPrint = offsetPrint - (cfg.Offset * 24 * 60 * 60 * 1000)
-		} else {
-			offsetPrint = offsetPrint - (cfg.Offset * 24 * 60 * 60)
-		}
+		offsetPrint = offsetPrint - offsetSecs
 	}
 	log.Println(fmt.Sprintf("Will collect data starting at TS %v", offsetPrint))
 	for _, r := range cfg.Retentions {
-		ret, err := convertRetention(r, cfg.Offset, cfg.MsecsTime)
+		ret, err := convertRetention(r, offsetSecs, cfg.MsecsTime)
 		if err != nil {
 			return &Client{}, fmt.Errorf("Couldn't parse retention %q :: %v", r, err)
 		}
--- a/app/vmctl/opentsdb/parser.go
+++ b/app/vmctl/opentsdb/parser.go
@ -107,6 +107,7 @@ func convertRetention(retention string, offset int64, msecTime bool) (Retention,
 	}
 	// bump by the offset so we don't look at empty ranges any time offset > ttl
 	ttl += offset
+
 	var timeChunks []TimeRange
 	var i int64
 	for i = offset; i <= ttl; i = i + rowLength {
--- a/app/vmctl/opentsdb/testdata/exampleOutput.json
+++ b/app/vmctl/opentsdb/testdata/exampleOutput.json
@ -1,398 +0,0 @@
-{
-  "outputs": [
-    {
-      "id": "a",
-      "alias": "query",
-      "dps": [
-        [
-          1614099600000,
-          0.28
-        ],
-        [
-          1614099660000,
-          0.22
-        ],
-        [
-          1614099720000,
-          0.18
-        ],
-        [
-          1614099780000,
-          0.14
-        ],
-        [
-          1614099840000,
-          0.24
-        ],
-        [
-          1614099900000,
-          0.19
-        ],
-        [
-          1614099960000,
-          0.22
-        ],
-        [
-          1614100020000,
-          0.2
-        ],
-        [
-          1614100080000,
-          0.18
-        ],
-        [
-          1614100140000,
-          0.22
-        ],
-        [
-          1614100200000,
-          0.17
-        ],
-        [
-          1614100260000,
-          0.16
-        ],
-        [
-          1614100320000,
-          0.22
-        ],
-        [
-          1614100380000,
-          0.3
-        ],
-        [
-          1614100440000,
-          0.28
-        ],
-        [
-          1614100500000,
-          0.27
-        ],
-        [
-          1614100560000,
-          0.26
-        ],
-        [
-          1614100620000,
-          0.23
-        ],
-        [
-          1614100680000,
-          0.18
-        ],
-        [
-          1614100740000,
-          0.3
-        ],
-        [
-          1614100800000,
-          0.24
-        ],
-        [
-          1614100860000,
-          0.19
-        ],
-        [
-          1614100920000,
-          0.16
-        ],
-        [
-          1614100980000,
-          0.19
-        ],
-        [
-          1614101040000,
-          0.23
-        ],
-        [
-          1614101100000,
-          0.18
-        ],
-        [
-          1614101160000,
-          0.15
-        ],
-        [
-          1614101220000,
-          0.12
-        ],
-        [
-          1614101280000,
-          0.1
-        ],
-        [
-          1614101340000,
-          0.24
-        ],
-        [
-          1614101400000,
-          0.19
-        ],
-        [
-          1614101460000,
-          0.16
-        ],
-        [
-          1614101520000,
-          0.14
-        ],
-        [
-          1614101580000,
-          0.12
-        ],
-        [
-          1614101640000,
-          0.14
-        ],
-        [
-          1614101700000,
-          0.12
-        ],
-        [
-          1614101760000,
-          0.13
-        ],
-        [
-          1614101820000,
-          0.12
-        ],
-        [
-          1614101880000,
-          0.11
-        ],
-        [
-          1614101940000,
-          0.36
-        ],
-        [
-          1614102000000,
-          0.35
-        ],
-        [
-          1614102060000,
-          0.3
-        ],
-        [
-          1614102120000,
-          0.32
-        ],
-        [
-          1614102180000,
-          0.27
-        ],
-        [
-          1614102240000,
-          0.26
-        ],
-        [
-          1614102300000,
-          0.21
-        ],
-        [
-          1614102360000,
-          0.18
-        ],
-        [
-          1614102420000,
-          0.15
-        ],
-        [
-          1614102480000,
-          0.12
-        ],
-        [
-          1614102540000,
-          0.24
-        ],
-        [
-          1614102600000,
-          0.2
-        ],
-        [
-          1614102660000,
-          0.17
-        ],
-        [
-          1614102720000,
-          0.18
-        ],
-        [
-          1614102780000,
-          0.14
-        ],
-        [
-          1614102840000,
-          0.39
-        ],
-        [
-          1614102900000,
-          0.31
-        ],
-        [
-          1614102960000,
-          0.3
-        ],
-        [
-          1614103020000,
-          0.24
-        ],
-        [
-          1614103080000,
-          0.26
-        ],
-        [
-          1614103140000,
-          0.21
-        ],
-        [
-          1614103200000,
-          0.17
-        ],
-        [
-          1614103260000,
-          0.15
-        ],
-        [
-          1614103320000,
-          0.2
-        ],
-        [
-          1614103380000,
-          0.2
-        ],
-        [
-          1614103440000,
-          0.22
-        ],
-        [
-          1614103500000,
-          0.19
-        ],
-        [
-          1614103560000,
-          0.22
-        ],
-        [
-          1614103620000,
-          0.29
-        ],
-        [
-          1614103680000,
-          0.31
-        ],
-        [
-          1614103740000,
-          0.28
-        ],
-        [
-          1614103800000,
-          0.23
-        ]
-      ],
-      "dpsMeta": {
-        "firstTimestamp": 1614099600000,
-        "lastTimestamp": 1614103800000,
-        "setCount": 71,
-        "series": 1
-      },
-      "meta": [
-        {
-          "index": 0,
-          "metrics": [
-            "timestamp"
-          ]
-        },
-        {
-          "index": 1,
-          "metrics": [
-            "system.load5"
-          ],
-          "commonTags": {
-            "rack": "undef",
-            "host": "use1-mon-metrics-1",
-            "row": "undef",
-            "dc": "us-east-1",
-            "group": "monitoring"
-          },
-          "aggregatedTags": []
-        }
-      ]
-    }
-  ],
-  "query": {
-    "name": null,
-    "time": {
-      "start": "1h-ago",
-      "end": null,
-      "timezone": null,
-      "downsampler": {
-        "interval": "1m",
-        "aggregator": "avg",
-        "fillPolicy": {
-          "policy": "nan",
-          "value": "NaN"
-        }
-      },
-      "aggregator": "sum",
-      "rate": false
-    },
-    "filters": [
-      {
-        "id": "f1",
-        "tags": [
-          {
-            "tagk": "host",
-            "filter": "use1-mon-metrics-1",
-            "group_by": true,
-            "type": "literal_or"
-          },
-          {
-            "tagk": "group",
-            "filter": "monitoring",
-            "group_by": true,
-            "type": "literal_or"
-          },
-          {
-            "tagk": "dc",
-            "filter": "us-east-1",
-            "group_by": true,
-            "type": "literal_or"
-          },
-          {
-            "tagk": "rack",
-            "filter": "undef",
-            "group_by": true,
-            "type": "literal_or"
-          },
-          {
-            "tagk": "row",
-            "filter": "undef",
-            "group_by": true,
-            "type": "literal_or"
-          }
-        ],
-        "explicitTags": false
-      }
-    ],
-    "metrics": [
-      {
-        "metric": "system.load5",
-        "id": "a",
-        "filter": "f1",
-        "aggregator": null,
-        "timeOffset": null,
-        "fillPolicy": {
-          "policy": "nan",
-          "value": "NaN"
-        }
-      }
-    ],
-    "expressions": [],
-    "outputs": [
-      {
-        "id": "a",
-        "alias": "query"
-      }
-    ]
-  }
-}
--- a/app/vmctl/opentsdb/testdata/exampleQuery.json
+++ b/app/vmctl/opentsdb/testdata/exampleQuery.json
@ -1,62 +0,0 @@
-{
-   "time": {
-       "start": "1h-ago",
-       "aggregator":"sum",
-       "downsampler": {
-           "interval": "1m",
-           "aggregator": "avg",
-           "fillPolicy": {
-               "policy": "nan"
-           }
-       }
-   },
-   "filters": [
-       {
-           "tags": [
-               {
-                   "type": "literal_or",
-                   "tagk": "host",
-                   "filter": "use1-mon-metrics-1",
-                   "groupBy": true
-               },
-               {
-                   "type": "literal_or",
-                   "tagk": "group",
-                   "filter": "monitoring",
-                   "groupBy": true
-               },
-               {
-                   "type": "literal_or",
-                   "tagk": "dc",
-                   "filter": "us-east-1",
-                   "groupBy": true
-               },
-               {
-                   "type": "literal_or",
-                   "tagk": "rack",
-                   "filter": "undef",
-                   "groupBy": true
-               },
-               {
-                   "type": "literal_or",
-                   "tagk": "row",
-                   "filter": "undef",
-                   "groupBy": true
-               }
-           ],
-           "id": "f1"
-       }
-   ],
-   "metrics": [
-       {
-           "id": "a",
-           "metric": "system.load5",
-           "filter": "f1",
-           "fillPolicy":{"policy":"nan"}
-       }
-   ],
-   "expressions": [],
-   "outputs":[
-     {"id":"a", "alias":"query"}
-   ]
-}
--- a/app/vminsert/main.go
+++ b/app/vminsert/main.go
@ -165,26 +165,26 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 			return true
 		}
 		// See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(202)
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "/datadog/api/v1/validate":
 		datadogValidateRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/authentication/#validate-api-key
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{"valid":true}`)
 		return true
 	case "/datadog/api/v1/check_run":
 		datadogCheckRunRequests.Inc()
 		// See https://docs.datadoghq.com/api/latest/service-checks/#submit-a-service-check
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(202)
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "/datadog/intake/":
 		datadogIntakeRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, `{}`)
 		return true
 	case "/prometheus/targets", "/targets":
@ -193,7 +193,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/prometheus/api/v1/targets", "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		state := r.FormValue("state")
 		promscrape.WriteAPIV1Targets(w, state)
 		return true
--- a/app/vmrestore/README.md
+++ b/app/vmrestore/README.md
@ -85,12 +85,32 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
    	See https://cloud.google.com/iam/docs/creating-managing-service-account-keys and https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
  -customS3Endpoint string
    	Custom S3 endpoint for use with S3-compatible storages (e.g. MinIO). S3 is used if not set
+  -enableTCP6
+    	Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP and UDP is used
  -envflag.enable
    	Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details
  -envflag.prefix string
    	Prefix for environment variables if -envflag.enable is set
  -fs.disableMmap
    	Whether to use pread() instead of mmap() for reading data files. By default mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
+  -http.connTimeout duration
+    	Incoming http connections are closed after the configured timeout. This may help to spread the incoming load among a cluster of services behind a load balancer. Please note that the real timeout may be bigger by up to 10% as a protection against the thundering herd problem (default 2m0s)
+  -http.disableResponseCompression
+    	Disable compression of HTTP responses to save CPU resources. By default compression is enabled to save network bandwidth
+  -http.idleConnTimeout duration
+    	Timeout for incoming idle http connections (default 1m0s)
+  -http.maxGracefulShutdownDuration duration
+    	The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s)
+  -http.pathPrefix string
+    	An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus
+  -http.shutdownDelay duration
+    	Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers
+  -httpAuth.password string
+    	Password for HTTP Basic Auth. The authentication is disabled if -httpAuth.username is empty
+  -httpAuth.username string
+    	Username for HTTP Basic Auth. The authentication is disabled if empty. See also -httpAuth.password
+  -httpListenAddr string
+    	TCP address for exporting metrics at /metrics page (default ":8421")
  -loggerDisableTimestamps
    	Whether to disable writing timestamps in logs
  -loggerErrorsPerSecondLimit int
@ -113,12 +133,24 @@ i.e. the end result would be similar to [rsync --delete](https://askubuntu.com/q
    	Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 0)
  -memory.allowedPercent float
    	Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from OS page cache which will result in higher disk IO usage (default 60)
+  -metricsAuthKey string
+    	Auth key for /metrics. It must be passed via authKey query arg. It overrides httpAuth.* settings
+  -pprofAuthKey string
+    	Auth key for /debug/pprof. It must be passed via authKey query arg. It overrides httpAuth.* settings
+  -s3ForcePathStyle
+    	Prefixing endpoint with bucket name when set false, true by default. (default true)
  -skipBackupCompleteCheck
    	Whether to skip checking for 'backup complete' file in -src. This may be useful for restoring from old backups, which were created without 'backup complete' file
  -src string
    	Source path with backup on the remote storage. Example: gs://bucket/path/to/backup/dir, s3://bucket/path/to/backup/dir or fs:///path/to/local/backup/dir
  -storageDataPath string
    	Destination path where backup must be restored. VictoriaMetrics must be stopped when restoring from backup. -storageDataPath dir can be non-empty. In this case the contents of -storageDataPath dir is synchronized with -src contents, i.e. it works like 'rsync --delete' (default "victoria-metrics-data")
+  -tls
+    	Whether to enable TLS (aka HTTPS) for incoming requests. -tlsCertFile and -tlsKeyFile must be set if -tls is set
+  -tlsCertFile string
+    	Path to file with TLS certificate. Used only if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower
+  -tlsKeyFile string
+    	Path to file with TLS key. Used only if -tls is set
  -version
    	Show VictoriaMetrics version
 ```
--- a/app/vmrestore/main.go
+++ b/app/vmrestore/main.go
@ -4,6 +4,7 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"time"

 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/actions"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/backup/common"
@ -11,10 +12,12 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/flagutil"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 )

 var (
+	httpListenAddr = flag.String("httpListenAddr", ":8421", "TCP address for exporting metrics at /metrics page")
 	src            = flag.String("src", "", "Source path with backup on the remote storage. "+
 		"Example: gs://bucket/path/to/backup/dir, s3://bucket/path/to/backup/dir or fs:///path/to/local/backup/dir")
 	storageDataPath = flag.String("storageDataPath", "victoria-metrics-data", "Destination path where backup must be restored. "+
@ -33,6 +36,9 @@ func main() {
 	buildinfo.Init()
 	logger.Init()

+	logger.Infof("starting http server for exporting metrics at http://%q/metrics", *httpListenAddr)
+	go httpserver.Serve(*httpListenAddr, nil)
+
 	srcFS, err := newSrcFS()
 	if err != nil {
 		logger.Fatalf("%s", err)
@ -52,6 +58,13 @@ func main() {
 	}
 	srcFS.MustStop()
 	dstFS.MustStop()
+
+	startTime := time.Now()
+	logger.Infof("gracefully shutting down http server for metrics at %q", *httpListenAddr)
+	if err := httpserver.Stop(*httpListenAddr); err != nil {
+		logger.Fatalf("cannot stop http server for metrics: %s", err)
+	}
+	logger.Infof("successfully shut down http server for metrics in %.3f seconds", time.Since(startTime).Seconds())
 }

 func usage() {
--- a/app/vmselect/graphite/metrics_api.go
+++ b/app/vmselect/graphite/metrics_api.go
@ -456,7 +456,7 @@ const maxRegexpCacheSize = 10000

 func getContentType(jsonp string) string {
 	if jsonp == "" {
-		return "application/json; charset=utf-8"
+		return "application/json"
 	}
 	return "text/javascript; charset=utf-8"
 }
--- a/app/vmselect/graphite/tags_api.go
+++ b/app/vmselect/graphite/tags_api.go
@ -62,7 +62,7 @@ func TagsDelSeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Re
 		totalDeleted += n
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	if totalDeleted > 0 {
 		fmt.Fprintf(w, "true")
 	} else {
@ -141,7 +141,7 @@ func registerMetrics(startTime time.Time, w http.ResponseWriter, r *http.Request
 	// Return response
 	contentType := "text/plain; charset=utf-8"
 	if isJSONResponse {
-		contentType = "application/json; charset=utf-8"
+		contentType = "application/json"
 	}
 	w.Header().Set("Content-Type", contentType)
 	WriteTagsTagMultiSeriesResponse(w, canonicalPaths, isJSONResponse)
@ -362,7 +362,7 @@ func TagsFindSeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.R
 		paths = paths[:limit]
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteTagsFindSeriesResponse(bw, paths)
@ -418,7 +418,7 @@ func TagValuesHandler(startTime time.Time, tagName string, w http.ResponseWriter
 		return err
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteTagValuesResponse(bw, tagName, tagValues)
@ -449,7 +449,7 @@ func TagsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) er
 		return err
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteTagsResponse(bw, labels)
--- a/app/vmselect/main.go
+++ b/app/vmselect/main.go
@ -200,7 +200,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 	if strings.HasPrefix(path, "/functions") {
 		graphiteFunctionsRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, "%s", `{}`)
 		return true
 	}
@ -404,25 +404,25 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 	case "/api/v1/rules", "/rules":
 		// Return dumb placeholder for https://prometheus.io/docs/prometheus/latest/querying/api/#rules
 		rulesRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, "%s", `{"status":"success","data":{"groups":[]}}`)
 		return true
 	case "/api/v1/alerts", "/alerts":
 		// Return dumb placeholder for https://prometheus.io/docs/prometheus/latest/querying/api/#alerts
 		alertsRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, "%s", `{"status":"success","data":{"alerts":[]}}`)
 		return true
 	case "/api/v1/metadata":
 		// Return dumb placeholder for https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata
 		metadataRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, "%s", `{"status":"success","data":{}}`)
 		return true
 	case "/api/v1/query_exemplars":
 		// Return dumb placeholder for https://prometheus.io/docs/prometheus/latest/querying/api/#querying-exemplars
 		queryExemplarsRequests.Inc()
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		fmt.Fprintf(w, "%s", `{"status":"success","data":null}`)
 		return true
 	case "/api/v1/admin/tsdb/delete_series":
@ -459,7 +459,7 @@ func isGraphiteTagsPath(path string) bool {
 func sendPrometheusError(w http.ResponseWriter, r *http.Request, err error) {
 	logger.Warnf("error in %q: %s", httpserver.GetRequestURI(r), err)

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	statusCode := http.StatusUnprocessableEntity
 	var esc *httpserver.ErrorWithStatusCode
 	if errors.As(err, &esc) {
--- a/app/vmselect/prometheus/prometheus.go
+++ b/app/vmselect/prometheus/prometheus.go
@ -533,7 +533,7 @@ func LabelValuesHandler(startTime time.Time, labelName string, w http.ResponseWr
 		}
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteLabelValuesResponse(bw, labelValues)
@ -622,7 +622,7 @@ func LabelsCountHandler(startTime time.Time, w http.ResponseWriter, r *http.Requ
 	if err != nil {
 		return fmt.Errorf(`cannot obtain label entries: %w`, err)
 	}
-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteLabelsCountResponse(bw, labelEntries)
@ -690,7 +690,7 @@ func TSDBStatusHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
 			return fmt.Errorf("cannot obtain tsdb status with matches for date=%d, topN=%d: %w", date, topN, err)
 		}
 	}
-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteTSDBStatusResponse(bw, status)
@ -784,7 +784,7 @@ func LabelsHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 		}
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteLabelsResponse(bw, labels)
@ -860,7 +860,7 @@ func SeriesCountHandler(startTime time.Time, w http.ResponseWriter, r *http.Requ
 	if err != nil {
 		return fmt.Errorf("cannot obtain series count: %w", err)
 	}
-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteSeriesCountResponse(bw, n)
@ -911,7 +911,7 @@ func SeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 		if err != nil {
 			return fmt.Errorf("cannot fetch time series for %q: %w", sq, err)
 		}
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		bw := bufferedwriter.Get(w)
 		defer bufferedwriter.Put(bw)
 		resultsCh := make(chan *quicktemplate.ByteBuffer)
@ -936,7 +936,7 @@ func SeriesHandler(startTime time.Time, w http.ResponseWriter, r *http.Request)
 		return fmt.Errorf("cannot fetch data for %q: %w", sq, err)
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	resultsCh := make(chan *quicktemplate.ByteBuffer)
@ -1070,7 +1070,7 @@ func QueryHandler(startTime time.Time, w http.ResponseWriter, r *http.Request) e
 		}
 	}

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteQueryResponse(bw, result)
@ -1163,7 +1163,7 @@ func queryRangeHandler(startTime time.Time, w http.ResponseWriter, query string,
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
 	result = removeEmptyValuesAndTimeseries(result)

-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	WriteQueryRangeResponse(bw, result)
@ -1343,7 +1343,7 @@ func QueryStatsHandler(startTime time.Time, w http.ResponseWriter, r *http.Reque
 		return fmt.Errorf("cannot parse `maxLifetime` arg: %w", err)
 	}
 	maxLifetime := time.Duration(maxLifetimeMsecs) * time.Millisecond
-	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.Header().Set("Content-Type", "application/json")
 	bw := bufferedwriter.Get(w)
 	defer bufferedwriter.Put(bw)
 	querystats.WriteJSONQueryStats(bw, topN, maxLifetime)
--- a/app/vmselect/promql/aggr.go
+++ b/app/vmselect/promql/aggr.go
@ -15,45 +15,42 @@ import (
 )

 var aggrFuncs = map[string]aggrFunc{
-	// See https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators
-	"sum":          newAggrFunc(aggrFuncSum),
-	"min":          newAggrFunc(aggrFuncMin),
-	"max":          newAggrFunc(aggrFuncMax),
+	"any":            aggrFuncAny,
 	"avg":            newAggrFunc(aggrFuncAvg),
-	"stddev":       newAggrFunc(aggrFuncStddev),
-	"stdvar":       newAggrFunc(aggrFuncStdvar),
-	"count":        newAggrFunc(aggrFuncCount),
-	"count_values": aggrFuncCountValues,
 	"bottomk":        newAggrFuncTopK(true),
-	"topk":         newAggrFuncTopK(false),
-	"quantile":     aggrFuncQuantile,
-	"group":        newAggrFunc(aggrFuncGroup),
-
-	// PromQL extension funcs
-	"median":         aggrFuncMedian,
-	"limitk":         aggrFuncLimitK,
-	"limit_offset":   aggrFuncLimitOffset,
-	"distinct":       newAggrFunc(aggrFuncDistinct),
-	"sum2":           newAggrFunc(aggrFuncSum2),
-	"geomean":        newAggrFunc(aggrFuncGeomean),
-	"histogram":      newAggrFunc(aggrFuncHistogram),
-	"topk_min":       newAggrFuncRangeTopK(minValue, false),
-	"topk_max":       newAggrFuncRangeTopK(maxValue, false),
-	"topk_avg":       newAggrFuncRangeTopK(avgValue, false),
-	"topk_median":    newAggrFuncRangeTopK(medianValue, false),
-	"topk_last":      newAggrFuncRangeTopK(lastValue, false),
-	"bottomk_min":    newAggrFuncRangeTopK(minValue, true),
-	"bottomk_max":    newAggrFuncRangeTopK(maxValue, true),
 	"bottomk_avg":    newAggrFuncRangeTopK(avgValue, true),
+	"bottomk_max":    newAggrFuncRangeTopK(maxValue, true),
 	"bottomk_median": newAggrFuncRangeTopK(medianValue, true),
 	"bottomk_last":   newAggrFuncRangeTopK(lastValue, true),
-	"any":            aggrFuncAny,
+	"bottomk_min":    newAggrFuncRangeTopK(minValue, true),
+	"count":          newAggrFunc(aggrFuncCount),
+	"count_values":   aggrFuncCountValues,
+	"distinct":       newAggrFunc(aggrFuncDistinct),
+	"geomean":        newAggrFunc(aggrFuncGeomean),
+	"group":          newAggrFunc(aggrFuncGroup),
+	"histogram":      newAggrFunc(aggrFuncHistogram),
+	"limit_offset":   aggrFuncLimitOffset,
+	"limitk":         aggrFuncLimitK,
 	"mad":            newAggrFunc(aggrFuncMAD),
+	"max":            newAggrFunc(aggrFuncMax),
+	"median":         aggrFuncMedian,
+	"min":            newAggrFunc(aggrFuncMin),
+	"mode":           newAggrFunc(aggrFuncMode),
 	"outliers_mad":   aggrFuncOutliersMAD,
 	"outliersk":      aggrFuncOutliersK,
-	"mode":           newAggrFunc(aggrFuncMode),
-	"zscore":         aggrFuncZScore,
+	"quantile":       aggrFuncQuantile,
 	"quantiles":      aggrFuncQuantiles,
+	"stddev":         newAggrFunc(aggrFuncStddev),
+	"stdvar":         newAggrFunc(aggrFuncStdvar),
+	"sum":            newAggrFunc(aggrFuncSum),
+	"sum2":           newAggrFunc(aggrFuncSum2),
+	"topk":           newAggrFuncTopK(false),
+	"topk_avg":       newAggrFuncRangeTopK(avgValue, false),
+	"topk_max":       newAggrFuncRangeTopK(maxValue, false),
+	"topk_median":    newAggrFuncRangeTopK(medianValue, false),
+	"topk_last":      newAggrFuncRangeTopK(lastValue, false),
+	"topk_min":       newAggrFuncRangeTopK(minValue, false),
+	"zscore":         aggrFuncZScore,
 }

 type aggrFunc func(afa *aggrFuncArg) ([]*timeseries, error)
--- a/app/vmselect/promql/eval.go
+++ b/app/vmselect/promql/eval.go
@ -729,9 +729,10 @@ func evalRollupFuncWithMetricExpr(ec *EvalConfig, funcName string, rf rollupFunc
 		rss.Cancel()
 		return nil, fmt.Errorf("not enough memory for processing %d data points across %d time series with %d points in each time series; "+
 			"total available memory for concurrent requests: %d bytes; "+
+			"requested memory: %d bytes; "+
 			"possible solutions are: reducing the number of matching time series; switching to node with more RAM; "+
 			"increasing -memory.allowedPercent; increasing `step` query arg (%gs)",
-			rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, rml.MaxSize, float64(ec.Step)/1e3)
+			rollupPoints, timeseriesLen*len(rcs), pointsPerTimeseries, rml.MaxSize, uint64(rollupMemorySize), float64(ec.Step)/1e3)
 	}
 	defer rml.Put(uint64(rollupMemorySize))

--- a/app/vmselect/promql/exec_test.go
+++ b/app/vmselect/promql/exec_test.go
@ -1017,6 +1017,17 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
+	t.Run("now()", func(t *testing.T) {
+		t.Parallel()
+		q := `round(now()/now())`
+		r := netstorage.Result{
+			MetricName: metricNameExpected,
+			Values:     []float64{1, 1, 1, 1, 1, 1},
+			Timestamps: timestampsExpected,
+		}
+		resultExpected := []netstorage.Result{r}
+		f(q, resultExpected)
+	})
 	t.Run("pi()", func(t *testing.T) {
 		t.Parallel()
 		q := `pi()`
@ -6398,9 +6409,9 @@ func TestExecSuccess(t *testing.T) {
 		resultExpected := []netstorage.Result{r}
 		f(q, resultExpected)
 	})
-	t.Run(`deriv(1)`, func(t *testing.T) {
+	t.Run(`deriv(N)`, func(t *testing.T) {
 		t.Parallel()
-		q := `deriv(1)`
+		q := `deriv(1000)`
 		r := netstorage.Result{
 			MetricName: metricNameExpected,
 			Values:     []float64{0, 0, 0, 0, 0, 0},
@ -7412,6 +7423,7 @@ func TestExecError(t *testing.T) {
 	f(`rand_normal(123, 456)`)
 	f(`rand_exponential(122, 456)`)
 	f(`pi(123)`)
+	f(`now(123)`)
 	f(`label_copy()`)
 	f(`label_move()`)
 	f(`median_over_time()`)
--- a/app/vmselect/promql/rollup.go
+++ b/app/vmselect/promql/rollup.go
@ -19,131 +19,121 @@ var minStalenessInterval = flag.Duration("search.minStalenessInterval", 0, "The
 	"See also '-search.maxStalenessInterval'")

 var rollupFuncs = map[string]newRollupFunc{
-	// Standard rollup funcs from PromQL.
-	// See funcs accepting range-vector on https://prometheus.io/docs/prometheus/latest/querying/functions/ .
+	"absent_over_time":       newRollupFuncOneArg(rollupAbsent),
+	"aggr_over_time":         newRollupFuncTwoArgs(rollupFake),
+	"ascent_over_time":       newRollupFuncOneArg(rollupAscentOverTime),
+	"avg_over_time":          newRollupFuncOneArg(rollupAvg),
 	"changes":                newRollupFuncOneArg(rollupChanges),
+	"count_eq_over_time":     newRollupCountEQ,
+	"count_gt_over_time":     newRollupCountGT,
+	"count_le_over_time":     newRollupCountLE,
+	"count_ne_over_time":     newRollupCountNE,
+	"count_over_time":        newRollupFuncOneArg(rollupCount),
+	"decreases_over_time":    newRollupFuncOneArg(rollupDecreases),
+	"default_rollup":         newRollupFuncOneArg(rollupDefault), // default rollup func
 	"delta":                  newRollupFuncOneArg(rollupDelta),
 	"deriv":                  newRollupFuncOneArg(rollupDerivSlow),
 	"deriv_fast":             newRollupFuncOneArg(rollupDerivFast),
+	"descent_over_time":      newRollupFuncOneArg(rollupDescentOverTime),
+	"distinct_over_time":     newRollupFuncOneArg(rollupDistinct),
+	"duration_over_time":     newRollupDurationOverTime,
+	"first_over_time":        newRollupFuncOneArg(rollupFirst),
+	"geomean_over_time":      newRollupFuncOneArg(rollupGeomean),
+	"histogram_over_time":    newRollupFuncOneArg(rollupHistogram),
+	"hoeffding_bound_lower":  newRollupHoeffdingBoundLower,
+	"hoeffding_bound_upper":  newRollupHoeffdingBoundUpper,
 	"holt_winters":           newRollupHoltWinters,
 	"idelta":                 newRollupFuncOneArg(rollupIdelta),
+	"ideriv":                 newRollupFuncOneArg(rollupIderiv),
 	"increase":               newRollupFuncOneArg(rollupDelta),        // + rollupFuncsRemoveCounterResets
+	"increase_pure":          newRollupFuncOneArg(rollupIncreasePure), // + rollupFuncsRemoveCounterResets
+	"increases_over_time":    newRollupFuncOneArg(rollupIncreases),
+	"integrate":              newRollupFuncOneArg(rollupIntegrate),
 	"irate":                  newRollupFuncOneArg(rollupIderiv), // + rollupFuncsRemoveCounterResets
-	"predict_linear":     newRollupPredictLinear,
-	"rate":               newRollupFuncOneArg(rollupDerivFast), // + rollupFuncsRemoveCounterResets
-	"resets":             newRollupFuncOneArg(rollupResets),
-	"avg_over_time":      newRollupFuncOneArg(rollupAvg),
-	"min_over_time":      newRollupFuncOneArg(rollupMin),
+	"lag":                    newRollupFuncOneArg(rollupLag),
+	"last_over_time":         newRollupFuncOneArg(rollupLast),
+	"lifetime":               newRollupFuncOneArg(rollupLifetime),
 	"max_over_time":          newRollupFuncOneArg(rollupMax),
-	"sum_over_time":      newRollupFuncOneArg(rollupSum),
-	"count_over_time":    newRollupFuncOneArg(rollupCount),
+	"min_over_time":          newRollupFuncOneArg(rollupMin),
+	"mode_over_time":         newRollupFuncOneArg(rollupModeOverTime),
+	"predict_linear":         newRollupPredictLinear,
+	"present_over_time":      newRollupFuncOneArg(rollupPresent),
 	"quantile_over_time":     newRollupQuantile,
+	"quantiles_over_time":    newRollupQuantiles,
+	"range_over_time":        newRollupFuncOneArg(rollupRange),
+	"rate":                   newRollupFuncOneArg(rollupDerivFast), // + rollupFuncsRemoveCounterResets
+	"rate_over_sum":          newRollupFuncOneArg(rollupRateOverSum),
+	"resets":                 newRollupFuncOneArg(rollupResets),
+	"rollup":                 newRollupFuncOneArg(rollupFake),
+	"rollup_candlestick":     newRollupFuncOneArg(rollupFake),
+	"rollup_delta":           newRollupFuncOneArg(rollupFake),
+	"rollup_deriv":           newRollupFuncOneArg(rollupFake),
+	"rollup_increase":        newRollupFuncOneArg(rollupFake), // + rollupFuncsRemoveCounterResets
+	"rollup_rate":            newRollupFuncOneArg(rollupFake), // + rollupFuncsRemoveCounterResets
+	"rollup_scrape_interval": newRollupFuncOneArg(rollupFake),
+	"scrape_interval":        newRollupFuncOneArg(rollupScrapeInterval),
+	"share_gt_over_time":     newRollupShareGT,
+	"share_le_over_time":     newRollupShareLE,
 	"stddev_over_time":       newRollupFuncOneArg(rollupStddev),
 	"stdvar_over_time":       newRollupFuncOneArg(rollupStdvar),
-	"absent_over_time":   newRollupFuncOneArg(rollupAbsent),
-	"present_over_time":  newRollupFuncOneArg(rollupPresent),
-	"last_over_time":     newRollupFuncOneArg(rollupLast),
-
-	// Additional rollup funcs.
-	"default_rollup":         newRollupFuncOneArg(rollupDefault), // default rollup func
-	"range_over_time":        newRollupFuncOneArg(rollupRange),
+	"sum_over_time":          newRollupFuncOneArg(rollupSum),
 	"sum2_over_time":         newRollupFuncOneArg(rollupSum2),
-	"geomean_over_time":      newRollupFuncOneArg(rollupGeomean),
-	"first_over_time":        newRollupFuncOneArg(rollupFirst),
-	"distinct_over_time":     newRollupFuncOneArg(rollupDistinct),
-	"increases_over_time":    newRollupFuncOneArg(rollupIncreases),
-	"decreases_over_time":    newRollupFuncOneArg(rollupDecreases),
-	"increase_pure":          newRollupFuncOneArg(rollupIncreasePure), // + rollupFuncsRemoveCounterResets
-	"integrate":              newRollupFuncOneArg(rollupIntegrate),
-	"ideriv":                 newRollupFuncOneArg(rollupIderiv),
-	"lifetime":               newRollupFuncOneArg(rollupLifetime),
-	"lag":                    newRollupFuncOneArg(rollupLag),
-	"scrape_interval":        newRollupFuncOneArg(rollupScrapeInterval),
-	"tmin_over_time":         newRollupFuncOneArg(rollupTmin),
-	"tmax_over_time":         newRollupFuncOneArg(rollupTmax),
 	"tfirst_over_time":       newRollupFuncOneArg(rollupTfirst),
-	"tlast_over_time":        newRollupFuncOneArg(rollupTlast),
-	"duration_over_time":     newRollupDurationOverTime,
-	"share_le_over_time":     newRollupShareLE,
-	"share_gt_over_time":     newRollupShareGT,
-	"count_le_over_time":     newRollupCountLE,
-	"count_gt_over_time":     newRollupCountGT,
-	"count_eq_over_time":     newRollupCountEQ,
-	"count_ne_over_time":     newRollupCountNE,
-	"histogram_over_time":    newRollupFuncOneArg(rollupHistogram),
-	"rollup":                 newRollupFuncOneArg(rollupFake),
-	"rollup_rate":            newRollupFuncOneArg(rollupFake), // + rollupFuncsRemoveCounterResets
-	"rollup_deriv":           newRollupFuncOneArg(rollupFake),
-	"rollup_delta":           newRollupFuncOneArg(rollupFake),
-	"rollup_increase":        newRollupFuncOneArg(rollupFake), // + rollupFuncsRemoveCounterResets
-	"rollup_candlestick":     newRollupFuncOneArg(rollupFake),
-	"rollup_scrape_interval": newRollupFuncOneArg(rollupFake),
-	"aggr_over_time":         newRollupFuncTwoArgs(rollupFake),
-	"hoeffding_bound_upper":  newRollupHoeffdingBoundUpper,
-	"hoeffding_bound_lower":  newRollupHoeffdingBoundLower,
-	"ascent_over_time":       newRollupFuncOneArg(rollupAscentOverTime),
-	"descent_over_time":      newRollupFuncOneArg(rollupDescentOverTime),
-	"zscore_over_time":       newRollupFuncOneArg(rollupZScoreOverTime),
-	"quantiles_over_time":    newRollupQuantiles,
-
 	// `timestamp` function must return timestamp for the last datapoint on the current window
 	// in order to properly handle offset and timestamps unaligned to the current step.
 	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415 for details.
 	"timestamp":        newRollupFuncOneArg(rollupTlast),
-
-	// See https://en.wikipedia.org/wiki/Mode_(statistics)
-	"mode_over_time": newRollupFuncOneArg(rollupModeOverTime),
-
-	"rate_over_sum": newRollupFuncOneArg(rollupRateOverSum),
+	"tlast_over_time":  newRollupFuncOneArg(rollupTlast),
+	"tmax_over_time":   newRollupFuncOneArg(rollupTmax),
+	"tmin_over_time":   newRollupFuncOneArg(rollupTmin),
+	"zscore_over_time": newRollupFuncOneArg(rollupZScoreOverTime),
 }

 // rollupAggrFuncs are functions that can be passed to `aggr_over_time()`
 var rollupAggrFuncs = map[string]rollupFunc{
-	// Standard rollup funcs from PromQL.
+	"absent_over_time":    rollupAbsent,
+	"ascent_over_time":    rollupAscentOverTime,
+	"avg_over_time":       rollupAvg,
 	"changes":             rollupChanges,
+	"count_over_time":     rollupCount,
+	"decreases_over_time": rollupDecreases,
+	"default_rollup":      rollupDefault,
 	"delta":               rollupDelta,
 	"deriv":               rollupDerivSlow,
 	"deriv_fast":          rollupDerivFast,
+	"descent_over_time":   rollupDescentOverTime,
+	"distinct_over_time":  rollupDistinct,
+	"first_over_time":     rollupFirst,
+	"geomean_over_time":   rollupGeomean,
 	"idelta":              rollupIdelta,
-	"increase":          rollupDelta,     // + rollupFuncsRemoveCounterResets
-	"irate":             rollupIderiv,    // + rollupFuncsRemoveCounterResets
-	"rate":              rollupDerivFast, // + rollupFuncsRemoveCounterResets
-	"resets":            rollupResets,
-	"avg_over_time":     rollupAvg,
-	"min_over_time":     rollupMin,
+	"ideriv":              rollupIderiv,
+	"increase":            rollupDelta,
+	"increase_pure":       rollupIncreasePure,
+	"increases_over_time": rollupIncreases,
+	"integrate":           rollupIntegrate,
+	"irate":               rollupIderiv,
+	"lag":                 rollupLag,
+	"last_over_time":      rollupLast,
+	"lifetime":            rollupLifetime,
 	"max_over_time":       rollupMax,
-	"sum_over_time":     rollupSum,
-	"count_over_time":   rollupCount,
+	"min_over_time":       rollupMin,
+	"mode_over_time":      rollupModeOverTime,
+	"present_over_time":   rollupPresent,
+	"range_over_time":     rollupRange,
+	"rate":                rollupDerivFast,
+	"rate_over_sum":       rollupRateOverSum,
+	"resets":              rollupResets,
+	"scrape_interval":     rollupScrapeInterval,
 	"stddev_over_time":    rollupStddev,
 	"stdvar_over_time":    rollupStdvar,
-	"absent_over_time":  rollupAbsent,
-	"present_over_time": rollupPresent,
-
-	// Additional rollup funcs.
-	"range_over_time":     rollupRange,
+	"sum_over_time":       rollupSum,
 	"sum2_over_time":      rollupSum2,
-	"geomean_over_time":   rollupGeomean,
-	"first_over_time":     rollupFirst,
-	"last_over_time":      rollupLast,
-	"distinct_over_time":  rollupDistinct,
-	"increases_over_time": rollupIncreases,
-	"decreases_over_time": rollupDecreases,
-	"increase_pure":       rollupIncreasePure,
-	"integrate":           rollupIntegrate,
-	"ideriv":              rollupIderiv,
-	"lifetime":            rollupLifetime,
-	"lag":                 rollupLag,
-	"scrape_interval":     rollupScrapeInterval,
-	"tmin_over_time":      rollupTmin,
-	"tmax_over_time":      rollupTmax,
 	"tfirst_over_time":    rollupTfirst,
-	"tlast_over_time":     rollupTlast,
-	"ascent_over_time":    rollupAscentOverTime,
-	"descent_over_time":   rollupDescentOverTime,
-	"zscore_over_time":    rollupZScoreOverTime,
 	"timestamp":           rollupTlast,
-	"mode_over_time":      rollupModeOverTime,
-	"rate_over_sum":       rollupRateOverSum,
+	"tlast_over_time":     rollupTlast,
+	"tmax_over_time":      rollupTmax,
+	"tmin_over_time":      rollupTmin,
+	"zscore_over_time":    rollupZScoreOverTime,
 }

 // VictoriaMetrics can increase lookbehind window in square brackets for these functions
@ -170,29 +160,31 @@ var rollupFuncsCanAdjustWindow = map[string]bool{

 var rollupFuncsRemoveCounterResets = map[string]bool{
 	"increase":        true,
+	"increase_pure":   true,
 	"irate":           true,
 	"rate":            true,
-	"rollup_rate":     true,
 	"rollup_increase": true,
-	"increase_pure":   true,
+	"rollup_rate":     true,
 }

+// These functions don't change physical meaning of input time series,
+// so they don't drop metric name
 var rollupFuncsKeepMetricGroup = map[string]bool{
-	"holt_winters":          true,
-	"predict_linear":        true,
-	"default_rollup":        true,
 	"avg_over_time":         true,
-	"min_over_time":         true,
-	"max_over_time":         true,
-	"quantile_over_time":    true,
-	"quantiles_over_time":   true,
-	"rollup":                true,
+	"default_rollup":        true,
+	"first_over_time":       true,
 	"geomean_over_time":     true,
 	"hoeffding_bound_lower": true,
 	"hoeffding_bound_upper": true,
-	"first_over_time":       true,
+	"holt_winters":          true,
 	"last_over_time":        true,
+	"max_over_time":         true,
+	"min_over_time":         true,
 	"mode_over_time":        true,
+	"predict_linear":        true,
+	"quantile_over_time":    true,
+	"quantiles_over_time":   true,
+	"rollup":                true,
 	"rollup_candlestick":    true,
 }

@ -858,7 +850,7 @@ func linearRegression(rfa *rollupFuncArg) (float64, float64) {
 	if n == 0 {
 		return nan, nan
 	}
-	if n == 1 {
+	if areConstValues(values) {
 		return values[0], 0
 	}

@ -875,11 +867,30 @@ func linearRegression(rfa *rollupFuncArg) (float64, float64) {
 		tvSum += dt * v
 		ttSum += dt * dt
 	}
-	k := (tvSum - tSum*vSum/n) / (ttSum - tSum*tSum/n)
+	k := float64(0)
+	tDiff := ttSum - tSum*tSum/n
+	if math.Abs(tDiff) >= 1e-6 {
+		// Prevent from incorrect division for too small tDiff values.
+		k = (tvSum - tSum*vSum/n) / tDiff
+	}
 	v := vSum/n - k*tSum/n
 	return v, k
 }

+func areConstValues(values []float64) bool {
+	if len(values) <= 1 {
+		return true
+	}
+	vPrev := values[0]
+	for _, v := range values[1:] {
+		if v != vPrev {
+			return false
+		}
+		vPrev = v
+	}
+	return true
+}
+
 func newRollupDurationOverTime(args []interface{}) (rollupFunc, error) {
 	if err := expectRollupArgsNum(args, 2); err != nil {
 		return nil, err
@ -891,11 +902,10 @@ func newRollupDurationOverTime(args []interface{}) (rollupFunc, error) {
 	rf := func(rfa *rollupFuncArg) float64 {
 		// There is no need in handling NaNs here, since they must be cleaned up
 		// before calling rollup funcs.
-		values := rfa.values
-		if len(values) == 0 {
+		timestamps := rfa.timestamps
+		if len(timestamps) == 0 {
 			return nan
 		}
-		timestamps := rfa.timestamps
 		tPrev := timestamps[0]
 		dSum := int64(0)
 		dMax := int64(dMaxs[rfa.idx] * 1000)
@ -906,7 +916,7 @@ func newRollupDurationOverTime(args []interface{}) (rollupFunc, error) {
 			}
 			tPrev = t
 		}
-		return float64(dSum / 1000)
+		return float64(dSum) / 1000
 	}
 	return rf, nil
 }
--- a/app/vmselect/promql/rollup_test.go
+++ b/app/vmselect/promql/rollup_test.go
@ -193,6 +193,27 @@ func testRollupFunc(t *testing.T, funcName string, args []interface{}, meExpecte
 	}
 }

+func TestRollupDurationOverTime(t *testing.T) {
+	f := func(maxInterval, dExpected float64) {
+		t.Helper()
+		maxIntervals := []*timeseries{{
+			Values:     []float64{maxInterval},
+			Timestamps: []int64{123},
+		}}
+		var me metricsql.MetricExpr
+		args := []interface{}{&metricsql.RollupExpr{Expr: &me}, maxIntervals}
+		testRollupFunc(t, "duration_over_time", args, &me, dExpected)
+	}
+	f(-123, 0)
+	f(0, 0)
+	f(0.001, 0)
+	f(0.005, 0.007)
+	f(0.01, 0.036)
+	f(0.02, 0.125)
+	f(1, 0.125)
+	f(100, 0.125)
+}
+
 func TestRollupShareLEOverTime(t *testing.T) {
 	f := func(le, vExpected float64) {
 		t.Helper()
--- a/app/vmselect/promql/transform.go
+++ b/app/vmselect/promql/transform.go
@ -17,31 +17,8 @@ import (
 	"github.com/VictoriaMetrics/metricsql"
 )

-var transformFuncsKeepMetricGroup = map[string]bool{
-	"ceil":               true,
-	"clamp":              true,
-	"clamp_max":          true,
-	"clamp_min":          true,
-	"floor":              true,
-	"round":              true,
-	"keep_last_value":    true,
-	"keep_next_value":    true,
-	"interpolate":        true,
-	"running_min":        true,
-	"running_max":        true,
-	"running_avg":        true,
-	"range_min":          true,
-	"range_max":          true,
-	"range_avg":          true,
-	"range_first":        true,
-	"range_last":         true,
-	"range_quantile":     true,
-	"smooth_exponential": true,
-}
-
 var transformFuncs = map[string]transformFunc{
-	// Standard promql funcs
-	// See funcs accepting instant-vector on https://prometheus.io/docs/prometheus/latest/querying/functions/ .
+	"":                     transformUnion, // empty func is a synonym to union
 	"abs":                  newTransformFuncOneArg(transformAbs),
 	"absent":               transformAbsent,
 	"acos":                 newTransformFuncOneArg(transformAcos),
@ -50,6 +27,10 @@ var transformFuncs = map[string]transformFunc{
 	"asinh":                newTransformFuncOneArg(transformAsinh),
 	"atan":                 newTransformFuncOneArg(transformAtan),
 	"atanh":                newTransformFuncOneArg(transformAtanh),
+	"bitmap_and":           newTransformBitmap(bitmapAnd),
+	"bitmap_or":            newTransformBitmap(bitmapOr),
+	"bitmap_xor":           newTransformBitmap(bitmapXor),
+	"buckets_limit":        transformBucketsLimit,
 	"ceil":                 newTransformFuncOneArg(transformCeil),
 	"clamp":                transformClamp,
 	"clamp_max":            transformClampMax,
@ -60,86 +41,103 @@ var transformFuncs = map[string]transformFunc{
 	"day_of_week":          newTransformFuncDateTime(transformDayOfWeek),
 	"days_in_month":        newTransformFuncDateTime(transformDaysInMonth),
 	"deg":                  newTransformFuncOneArg(transformDeg),
+	"end":                  newTransformFuncZeroArgs(transformEnd),
 	"exp":                  newTransformFuncOneArg(transformExp),
 	"floor":                newTransformFuncOneArg(transformFloor),
+	"histogram_avg":        transformHistogramAvg,
 	"histogram_quantile":   transformHistogramQuantile,
+	"histogram_quantiles":  transformHistogramQuantiles,
+	"histogram_share":      transformHistogramShare,
+	"histogram_stddev":     transformHistogramStddev,
+	"histogram_stdvar":     transformHistogramStdvar,
 	"hour":                 newTransformFuncDateTime(transformHour),
+	"interpolate":          transformInterpolate,
+	"keep_last_value":      transformKeepLastValue,
+	"keep_next_value":      transformKeepNextValue,
+	"label_copy":           transformLabelCopy,
+	"label_del":            transformLabelDel,
+	"label_graphite_group": transformLabelGraphiteGroup,
 	"label_join":           transformLabelJoin,
+	"label_keep":           transformLabelKeep,
+	"label_lowercase":      transformLabelLowercase,
+	"label_map":            transformLabelMap,
+	"label_match":          transformLabelMatch,
+	"label_mismatch":       transformLabelMismatch,
+	"label_move":           transformLabelMove,
 	"label_replace":        transformLabelReplace,
+	"label_set":            transformLabelSet,
+	"label_transform":      transformLabelTransform,
+	"label_uppercase":      transformLabelUppercase,
+	"label_value":          transformLabelValue,
 	"ln":                   newTransformFuncOneArg(transformLn),
 	"log2":                 newTransformFuncOneArg(transformLog2),
 	"log10":                newTransformFuncOneArg(transformLog10),
 	"minute":               newTransformFuncDateTime(transformMinute),
 	"month":                newTransformFuncDateTime(transformMonth),
+	"now":                  transformNow,
 	"pi":                   transformPi,
+	"prometheus_buckets":   transformPrometheusBuckets,
 	"rad":                  newTransformFuncOneArg(transformRad),
+	"rand":                 newTransformRand(newRandFloat64),
+	"rand_exponential":     newTransformRand(newRandExpFloat64),
+	"rand_normal":          newTransformRand(newRandNormFloat64),
+	"range_avg":            newTransformFuncRange(runningAvg),
+	"range_first":          transformRangeFirst,
+	"range_last":           transformRangeLast,
+	"range_max":            newTransformFuncRange(runningMax),
+	"range_min":            newTransformFuncRange(runningMin),
+	"range_quantile":       transformRangeQuantile,
+	"range_sum":            newTransformFuncRange(runningSum),
+	"remove_resets":        transformRemoveResets,
 	"round":                transformRound,
+	"running_avg":          newTransformFuncRunning(runningAvg),
+	"running_max":          newTransformFuncRunning(runningMax),
+	"running_min":          newTransformFuncRunning(runningMin),
+	"running_sum":          newTransformFuncRunning(runningSum),
 	"scalar":               transformScalar,
 	"sgn":                  transformSgn,
 	"sin":                  newTransformFuncOneArg(transformSin),
 	"sinh":                 newTransformFuncOneArg(transformSinh),
+	"smooth_exponential":   transformSmoothExponential,
 	"sort":                 newTransformFuncSort(false),
+	"sort_by_label":        newTransformFuncSortByLabel(false),
+	"sort_by_label_desc":   newTransformFuncSortByLabel(true),
 	"sort_desc":            newTransformFuncSort(true),
 	"sqrt":                 newTransformFuncOneArg(transformSqrt),
+	"start":                newTransformFuncZeroArgs(transformStart),
+	"step":                 newTransformFuncZeroArgs(transformStep),
 	"tan":                  newTransformFuncOneArg(transformTan),
 	"tanh":                 newTransformFuncOneArg(transformTanh),
 	"time":                 transformTime,
 	// "timestamp" has been moved to rollup funcs. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
+	"timezone_offset": transformTimezoneOffset,
+	"union":           transformUnion,
 	"vector":          transformVector,
 	"year":            newTransformFuncDateTime(transformYear),
+}

-	// New funcs
-	"label_set":            transformLabelSet,
-	"label_map":            transformLabelMap,
-	"label_uppercase":      transformLabelUppercase,
-	"label_lowercase":      transformLabelLowercase,
-	"label_del":            transformLabelDel,
-	"label_keep":           transformLabelKeep,
-	"label_copy":           transformLabelCopy,
-	"label_move":           transformLabelMove,
-	"label_transform":      transformLabelTransform,
-	"label_value":          transformLabelValue,
-	"label_match":          transformLabelMatch,
-	"label_mismatch":       transformLabelMismatch,
-	"label_graphite_group": transformLabelGraphiteGroup,
-
-	"union":               transformUnion,
-	"":                    transformUnion, // empty func is a synonym to union
-	"keep_last_value":     transformKeepLastValue,
-	"keep_next_value":     transformKeepNextValue,
-	"interpolate":         transformInterpolate,
-	"start":               newTransformFuncZeroArgs(transformStart),
-	"end":                 newTransformFuncZeroArgs(transformEnd),
-	"step":                newTransformFuncZeroArgs(transformStep),
-	"running_sum":         newTransformFuncRunning(runningSum),
-	"running_max":         newTransformFuncRunning(runningMax),
-	"running_min":         newTransformFuncRunning(runningMin),
-	"running_avg":         newTransformFuncRunning(runningAvg),
-	"range_sum":           newTransformFuncRange(runningSum),
-	"range_max":           newTransformFuncRange(runningMax),
-	"range_min":           newTransformFuncRange(runningMin),
-	"range_avg":           newTransformFuncRange(runningAvg),
-	"range_first":         transformRangeFirst,
-	"range_last":          transformRangeLast,
-	"range_quantile":      transformRangeQuantile,
-	"smooth_exponential":  transformSmoothExponential,
-	"remove_resets":       transformRemoveResets,
-	"rand":                newTransformRand(newRandFloat64),
-	"rand_normal":         newTransformRand(newRandNormFloat64),
-	"rand_exponential":    newTransformRand(newRandExpFloat64),
-	"prometheus_buckets":  transformPrometheusBuckets,
-	"buckets_limit":       transformBucketsLimit,
-	"histogram_share":     transformHistogramShare,
-	"histogram_avg":       transformHistogramAvg,
-	"histogram_stdvar":    transformHistogramStdvar,
-	"histogram_stddev":    transformHistogramStddev,
-	"sort_by_label":       newTransformFuncSortByLabel(false),
-	"sort_by_label_desc":  newTransformFuncSortByLabel(true),
-	"timezone_offset":     transformTimezoneOffset,
-	"bitmap_and":          newTransformBitmap(bitmapAnd),
-	"bitmap_or":           newTransformBitmap(bitmapOr),
-	"bitmap_xor":          newTransformBitmap(bitmapXor),
-	"histogram_quantiles": transformHistogramQuantiles,
+// These functions don't change physical meaning of input time series,
+// so they don't drop metric name
+var transformFuncsKeepMetricGroup = map[string]bool{
+	"ceil":               true,
+	"clamp":              true,
+	"clamp_max":          true,
+	"clamp_min":          true,
+	"floor":              true,
+	"interpolate":        true,
+	"keep_last_value":    true,
+	"keep_next_value":    true,
+	"range_avg":          true,
+	"range_first":        true,
+	"range_last":         true,
+	"range_max":          true,
+	"range_min":          true,
+	"range_quantile":     true,
+	"round":              true,
+	"running_avg":        true,
+	"running_max":        true,
+	"running_min":        true,
+	"smooth_exponential": true,
 }

 func getTransformFunc(s string) transformFunc {
@ -2048,6 +2046,14 @@ func transformPi(tfa *transformFuncArg) ([]*timeseries, error) {
 	return evalNumber(tfa.ec, math.Pi), nil
 }

+func transformNow(tfa *transformFuncArg) ([]*timeseries, error) {
+	if err := expectTransformArgsNum(tfa.args, 0); err != nil {
+		return nil, err
+	}
+	now := float64(time.Now().UnixNano()) / 1e9
+	return evalNumber(tfa.ec, now), nil
+}
+
 func bitmapAnd(a, b uint64) uint64 {
 	return a & b
 }
--- a/app/vmselect/vmui/asset-manifest.json
+++ b/app/vmselect/vmui/asset-manifest.json
@ -1,19 +1,19 @@
 {
  "files": {
    "main.css": "./static/css/main.674f8c98.chunk.css",
-    "main.js": "./static/js/main.9d24c3b2.chunk.js",
-    "runtime-main.js": "./static/js/runtime-main.c0002ac8.js",
-    "static/css/2.81b2a0ac.chunk.css": "./static/css/2.81b2a0ac.chunk.css",
-    "static/js/2.8fa069e1.chunk.js": "./static/js/2.8fa069e1.chunk.js",
-    "static/js/3.0dc73915.chunk.js": "./static/js/3.0dc73915.chunk.js",
+    "main.js": "./static/js/main.f4cab8bc.chunk.js",
+    "runtime-main.js": "./static/js/runtime-main.f698388d.js",
+    "static/css/2.77671664.chunk.css": "./static/css/2.77671664.chunk.css",
+    "static/js/2.bfcf9c30.chunk.js": "./static/js/2.bfcf9c30.chunk.js",
+    "static/js/3.e51afffb.chunk.js": "./static/js/3.e51afffb.chunk.js",
    "index.html": "./index.html",
-    "static/js/2.8fa069e1.chunk.js.LICENSE.txt": "./static/js/2.8fa069e1.chunk.js.LICENSE.txt"
+    "static/js/2.bfcf9c30.chunk.js.LICENSE.txt": "./static/js/2.bfcf9c30.chunk.js.LICENSE.txt"
  },
  "entrypoints": [
-    "static/js/runtime-main.c0002ac8.js",
-    "static/css/2.81b2a0ac.chunk.css",
-    "static/js/2.8fa069e1.chunk.js",
+    "static/js/runtime-main.f698388d.js",
+    "static/css/2.77671664.chunk.css",
+    "static/js/2.bfcf9c30.chunk.js",
    "static/css/main.674f8c98.chunk.css",
-    "static/js/main.9d24c3b2.chunk.js"
+    "static/js/main.f4cab8bc.chunk.js"
  ]
 }
--- a/app/vmselect/vmui/index.html
+++ b/app/vmselect/vmui/index.html
@ -1 +1 @@
-<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><link href="./static/css/2.81b2a0ac.chunk.css" rel="stylesheet"><link href="./static/css/main.674f8c98.chunk.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function r(r){for(var n,i,a=r[0],c=r[1],l=r[2],s=0,p=[];s<a.length;s++)i=a[s],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&p.push(o[i][0]),o[i]=0;for(n in c)Object.prototype.hasOwnProperty.call(c,n)&&(e[n]=c[n]);for(f&&f(r);p.length;)p.shift()();return u.push.apply(u,l||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var c=t[a];0!==o[c]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.e=function(e){var r=[],t=o[e];if(0!==t)if(t)r.push(t[2]);else{var n=new Promise((function(r,n){t=o[e]=[r,n]}));r.push(t[2]=n);var u,a=document.createElement("script");a.charset="utf-8",a.timeout=120,i.nc&&a.setAttribute("nonce",i.nc),a.src=function(e){return i.p+"static/js/"+({}[e]||e)+"."+{3:"0dc73915"}[e]+".chunk.js"}(e);var c=new Error;u=function(r){a.onerror=a.onload=null,clearTimeout(l);var t=o[e];if(0!==t){if(t){var n=r&&("load"===r.type?"missing":r.type),u=r&&r.target&&r.target.src;c.message="Loading chunk "+e+" failed.\n("+n+": "+u+")",c.name="ChunkLoadError",c.type=n,c.request=u,t[1](c)}o[e]=void 0}};var l=setTimeout((function(){u({type:"timeout",target:a})}),12e4);a.onerror=a.onload=u,document.head.appendChild(a)}return Promise.all(r)},i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="./",i.oe=function(e){throw console.error(e),e};var a=this.webpackJsonpvmui=this.webpackJsonpvmui||[],c=a.push.bind(a);a.push=r,a=a.slice();for(var l=0;l<a.length;l++)r(a[l]);var f=c;t()}([])</script><script src="./static/js/2.8fa069e1.chunk.js"></script><script src="./static/js/main.9d24c3b2.chunk.js"></script></body></html>
+<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><link href="./static/css/2.77671664.chunk.css" rel="stylesheet"><link href="./static/css/main.674f8c98.chunk.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function r(r){for(var n,i,a=r[0],c=r[1],f=r[2],s=0,p=[];s<a.length;s++)i=a[s],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&p.push(o[i][0]),o[i]=0;for(n in c)Object.prototype.hasOwnProperty.call(c,n)&&(e[n]=c[n]);for(l&&l(r);p.length;)p.shift()();return u.push.apply(u,f||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var c=t[a];0!==o[c]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.e=function(e){var r=[],t=o[e];if(0!==t)if(t)r.push(t[2]);else{var n=new Promise((function(r,n){t=o[e]=[r,n]}));r.push(t[2]=n);var u,a=document.createElement("script");a.charset="utf-8",a.timeout=120,i.nc&&a.setAttribute("nonce",i.nc),a.src=function(e){return i.p+"static/js/"+({}[e]||e)+"."+{3:"e51afffb"}[e]+".chunk.js"}(e);var c=new Error;u=function(r){a.onerror=a.onload=null,clearTimeout(f);var t=o[e];if(0!==t){if(t){var n=r&&("load"===r.type?"missing":r.type),u=r&&r.target&&r.target.src;c.message="Loading chunk "+e+" failed.\n("+n+": "+u+")",c.name="ChunkLoadError",c.type=n,c.request=u,t[1](c)}o[e]=void 0}};var f=setTimeout((function(){u({type:"timeout",target:a})}),12e4);a.onerror=a.onload=u,document.head.appendChild(a)}return Promise.all(r)},i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="./",i.oe=function(e){throw console.error(e),e};var a=this.webpackJsonpvmui=this.webpackJsonpvmui||[],c=a.push.bind(a);a.push=r,a=a.slice();for(var f=0;f<a.length;f++)r(a[f]);var l=c;t()}([])</script><script src="./static/js/2.bfcf9c30.chunk.js"></script><script src="./static/js/main.f4cab8bc.chunk.js"></script></body></html>
--- a/app/vmselect/vmui/static/css/2.77671664.chunk.css
+++ b/app/vmselect/vmui/static/css/2.77671664.chunk.css
@ -0,0 +1 @@
+.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;position:relative;width:100%;height:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{height:100%;border-right:1px dashed #607d8b}.u-hz .u-cursor-y,.u-vt .u-cursor-x{width:100%;border-bottom:1px dashed #607d8b}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0 solid;pointer-events:none;will-change:transform;z-index:100;background-clip:padding-box!important}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off{display:none}
--- a/app/vmselect/vmui/static/css/2.81b2a0ac.chunk.css
+++ b/app/vmselect/vmui/static/css/2.81b2a0ac.chunk.css
@ -1 +0,0 @@
-.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;position:relative;width:100%;height:100%}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{height:100%;border-right:1px dashed #607d8b}.u-hz .u-cursor-y,.u-vt .u-cursor-x{width:100%;border-bottom:1px dashed #607d8b}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0 solid;pointer-events:none;will-change:transform;z-index:100;background-clip:padding-box!important}.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off{display:none}
--- a/app/vmselect/vmui/static/js/2.8fa069e1.chunk.js
+++ b/app/vmselect/vmui/static/js/2.8fa069e1.chunk.js
--- a/app/vmselect/vmui/static/js/2.bfcf9c30.chunk.js
+++ b/app/vmselect/vmui/static/js/2.bfcf9c30.chunk.js
--- a/app/vmselect/vmui/static/js/2.bfcf9c30.chunk.js.LICENSE.txt
+++ b/app/vmselect/vmui/static/js/2.bfcf9c30.chunk.js.LICENSE.txt
@ -12,34 +12,6 @@ object-assign
 * http://adamwdraper.github.com/Numeral-js/
 */

-/*! exports provided: default */
-
-/*! exports provided: optionsUpdateState, dataMatch */
-
-/*! no static exports found */
-
-/*! react */
-
-/*! uplot */
-
-/*! uplot-wrappers-common */
-
-/*!*************************!*\
-        !*** ./common/index.ts ***!
-        \*************************/
-
-/*!*******************************!*\
-        !*** ./react/uplot-react.tsx ***!
-        \*******************************/
-
-/*!**************************************************************************************!*\
-        !*** external {"amd":"react","commonjs":"react","commonjs2":"react","root":"React"} ***!
-        \**************************************************************************************/
-
-/*!**************************************************************************************!*\
-        !*** external {"amd":"uplot","commonjs":"uplot","commonjs2":"uplot","root":"uPlot"} ***!
-        \**************************************************************************************/
-
 /**
 * A better abstraction over CSS.
 *
@ -48,7 +20,7 @@ object-assign
 * @license MIT
 */

-/** @license MUI v5.0.2
+/** @license MUI v5.2.0
 *
 * This source code is licensed under the MIT license found in the
 * LICENSE file in the root directory of this source tree.
--- a/app/vmselect/vmui/static/js/3.e51afffb.chunk.js
+++ b/app/vmselect/vmui/static/js/3.e51afffb.chunk.js
@ -1 +1 @@
-(this.webpackJsonpvmui=this.webpackJsonpvmui||[]).push([[3],{341:function(e,t,n){"use strict";n.r(t),n.d(t,"getCLS",(function(){return y})),n.d(t,"getFCP",(function(){return g})),n.d(t,"getFID",(function(){return C})),n.d(t,"getLCP",(function(){return k})),n.d(t,"getTTFB",(function(){return D}));var i,r,a,o,u=function(e,t){return{name:e,value:void 0===t?-1:t,delta:0,entries:[],id:"v2-".concat(Date.now(),"-").concat(Math.floor(8999999999999*Math.random())+1e12)}},c=function(e,t){try{if(PerformanceObserver.supportedEntryTypes.includes(e)){if("first-input"===e&&!("PerformanceEventTiming"in self))return;var n=new PerformanceObserver((function(e){return e.getEntries().map(t)}));return n.observe({type:e,buffered:!0}),n}}catch(e){}},f=function(e,t){var n=function n(i){"pagehide"!==i.type&&"hidden"!==document.visibilityState||(e(i),t&&(removeEventListener("visibilitychange",n,!0),removeEventListener("pagehide",n,!0)))};addEventListener("visibilitychange",n,!0),addEventListener("pagehide",n,!0)},s=function(e){addEventListener("pageshow",(function(t){t.persisted&&e(t)}),!0)},m=function(e,t,n){var i;return function(r){t.value>=0&&(r||n)&&(t.delta=t.value-(i||0),(t.delta||void 0===i)&&(i=t.value,e(t)))}},v=-1,p=function(){return"hidden"===document.visibilityState?0:1/0},d=function(){f((function(e){var t=e.timeStamp;v=t}),!0)},l=function(){return v<0&&(v=p(),d(),s((function(){setTimeout((function(){v=p(),d()}),0)}))),{get firstHiddenTime(){return v}}},g=function(e,t){var n,i=l(),r=u("FCP"),a=function(e){"first-contentful-paint"===e.name&&(f&&f.disconnect(),e.startTime<i.firstHiddenTime&&(r.value=e.startTime,r.entries.push(e),n(!0)))},o=window.performance&&performance.getEntriesByName&&performance.getEntriesByName("first-contentful-paint")[0],f=o?null:c("paint",a);(o||f)&&(n=m(e,r,t),o&&a(o),s((function(i){r=u("FCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,n(!0)}))}))})))},h=!1,T=-1,y=function(e,t){h||(g((function(e){T=e.value})),h=!0);var n,i=function(t){T>-1&&e(t)},r=u("CLS",0),a=0,o=[],v=function(e){if(!e.hadRecentInput){var t=o[0],i=o[o.length-1];a&&e.startTime-i.startTime<1e3&&e.startTime-t.startTime<5e3?(a+=e.value,o.push(e)):(a=e.value,o=[e]),a>r.value&&(r.value=a,r.entries=o,n())}},p=c("layout-shift",v);p&&(n=m(i,r,t),f((function(){p.takeRecords().map(v),n(!0)})),s((function(){a=0,T=-1,r=u("CLS",0),n=m(i,r,t)})))},E={passive:!0,capture:!0},w=new Date,L=function(e,t){i||(i=t,r=e,a=new Date,F(removeEventListener),S())},S=function(){if(r>=0&&r<a-w){var e={entryType:"first-input",name:i.type,target:i.target,cancelable:i.cancelable,startTime:i.timeStamp,processingStart:i.timeStamp+r};o.forEach((function(t){t(e)})),o=[]}},b=function(e){if(e.cancelable){var t=(e.timeStamp>1e12?new Date:performance.now())-e.timeStamp;"pointerdown"==e.type?function(e,t){var n=function(){L(e,t),r()},i=function(){r()},r=function(){removeEventListener("pointerup",n,E),removeEventListener("pointercancel",i,E)};addEventListener("pointerup",n,E),addEventListener("pointercancel",i,E)}(t,e):L(t,e)}},F=function(e){["mousedown","keydown","touchstart","pointerdown"].forEach((function(t){return e(t,b,E)}))},C=function(e,t){var n,a=l(),v=u("FID"),p=function(e){e.startTime<a.firstHiddenTime&&(v.value=e.processingStart-e.startTime,v.entries.push(e),n(!0))},d=c("first-input",p);n=m(e,v,t),d&&f((function(){d.takeRecords().map(p),d.disconnect()}),!0),d&&s((function(){var a;v=u("FID"),n=m(e,v,t),o=[],r=-1,i=null,F(addEventListener),a=p,o.push(a),S()}))},P={},k=function(e,t){var n,i=l(),r=u("LCP"),a=function(e){var t=e.startTime;t<i.firstHiddenTime&&(r.value=t,r.entries.push(e)),n()},o=c("largest-contentful-paint",a);if(o){n=m(e,r,t);var v=function(){P[r.id]||(o.takeRecords().map(a),o.disconnect(),P[r.id]=!0,n(!0))};["keydown","click"].forEach((function(e){addEventListener(e,v,{once:!0,capture:!0})})),f(v,!0),s((function(i){r=u("LCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,P[r.id]=!0,n(!0)}))}))}))}},D=function(e){var t,n=u("TTFB");t=function(){try{var t=performance.getEntriesByType("navigation")[0]||function(){var e=performance.timing,t={entryType:"navigation",startTime:0};for(var n in e)"navigationStart"!==n&&"toJSON"!==n&&(t[n]=Math.max(e[n]-e.navigationStart,0));return t}();if(n.value=n.delta=t.responseStart,n.value<0||n.value>performance.now())return;n.entries=[t],e(n)}catch(e){}},"complete"===document.readyState?setTimeout(t,0):addEventListener("pageshow",t)}}}]);
+(this.webpackJsonpvmui=this.webpackJsonpvmui||[]).push([[3],{351:function(e,t,n){"use strict";n.r(t),n.d(t,"getCLS",(function(){return y})),n.d(t,"getFCP",(function(){return g})),n.d(t,"getFID",(function(){return C})),n.d(t,"getLCP",(function(){return k})),n.d(t,"getTTFB",(function(){return D}));var i,r,a,o,u=function(e,t){return{name:e,value:void 0===t?-1:t,delta:0,entries:[],id:"v2-".concat(Date.now(),"-").concat(Math.floor(8999999999999*Math.random())+1e12)}},c=function(e,t){try{if(PerformanceObserver.supportedEntryTypes.includes(e)){if("first-input"===e&&!("PerformanceEventTiming"in self))return;var n=new PerformanceObserver((function(e){return e.getEntries().map(t)}));return n.observe({type:e,buffered:!0}),n}}catch(e){}},f=function(e,t){var n=function n(i){"pagehide"!==i.type&&"hidden"!==document.visibilityState||(e(i),t&&(removeEventListener("visibilitychange",n,!0),removeEventListener("pagehide",n,!0)))};addEventListener("visibilitychange",n,!0),addEventListener("pagehide",n,!0)},s=function(e){addEventListener("pageshow",(function(t){t.persisted&&e(t)}),!0)},m=function(e,t,n){var i;return function(r){t.value>=0&&(r||n)&&(t.delta=t.value-(i||0),(t.delta||void 0===i)&&(i=t.value,e(t)))}},v=-1,p=function(){return"hidden"===document.visibilityState?0:1/0},d=function(){f((function(e){var t=e.timeStamp;v=t}),!0)},l=function(){return v<0&&(v=p(),d(),s((function(){setTimeout((function(){v=p(),d()}),0)}))),{get firstHiddenTime(){return v}}},g=function(e,t){var n,i=l(),r=u("FCP"),a=function(e){"first-contentful-paint"===e.name&&(f&&f.disconnect(),e.startTime<i.firstHiddenTime&&(r.value=e.startTime,r.entries.push(e),n(!0)))},o=window.performance&&performance.getEntriesByName&&performance.getEntriesByName("first-contentful-paint")[0],f=o?null:c("paint",a);(o||f)&&(n=m(e,r,t),o&&a(o),s((function(i){r=u("FCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,n(!0)}))}))})))},h=!1,T=-1,y=function(e,t){h||(g((function(e){T=e.value})),h=!0);var n,i=function(t){T>-1&&e(t)},r=u("CLS",0),a=0,o=[],v=function(e){if(!e.hadRecentInput){var t=o[0],i=o[o.length-1];a&&e.startTime-i.startTime<1e3&&e.startTime-t.startTime<5e3?(a+=e.value,o.push(e)):(a=e.value,o=[e]),a>r.value&&(r.value=a,r.entries=o,n())}},p=c("layout-shift",v);p&&(n=m(i,r,t),f((function(){p.takeRecords().map(v),n(!0)})),s((function(){a=0,T=-1,r=u("CLS",0),n=m(i,r,t)})))},E={passive:!0,capture:!0},w=new Date,L=function(e,t){i||(i=t,r=e,a=new Date,F(removeEventListener),S())},S=function(){if(r>=0&&r<a-w){var e={entryType:"first-input",name:i.type,target:i.target,cancelable:i.cancelable,startTime:i.timeStamp,processingStart:i.timeStamp+r};o.forEach((function(t){t(e)})),o=[]}},b=function(e){if(e.cancelable){var t=(e.timeStamp>1e12?new Date:performance.now())-e.timeStamp;"pointerdown"==e.type?function(e,t){var n=function(){L(e,t),r()},i=function(){r()},r=function(){removeEventListener("pointerup",n,E),removeEventListener("pointercancel",i,E)};addEventListener("pointerup",n,E),addEventListener("pointercancel",i,E)}(t,e):L(t,e)}},F=function(e){["mousedown","keydown","touchstart","pointerdown"].forEach((function(t){return e(t,b,E)}))},C=function(e,t){var n,a=l(),v=u("FID"),p=function(e){e.startTime<a.firstHiddenTime&&(v.value=e.processingStart-e.startTime,v.entries.push(e),n(!0))},d=c("first-input",p);n=m(e,v,t),d&&f((function(){d.takeRecords().map(p),d.disconnect()}),!0),d&&s((function(){var a;v=u("FID"),n=m(e,v,t),o=[],r=-1,i=null,F(addEventListener),a=p,o.push(a),S()}))},P={},k=function(e,t){var n,i=l(),r=u("LCP"),a=function(e){var t=e.startTime;t<i.firstHiddenTime&&(r.value=t,r.entries.push(e)),n()},o=c("largest-contentful-paint",a);if(o){n=m(e,r,t);var v=function(){P[r.id]||(o.takeRecords().map(a),o.disconnect(),P[r.id]=!0,n(!0))};["keydown","click"].forEach((function(e){addEventListener(e,v,{once:!0,capture:!0})})),f(v,!0),s((function(i){r=u("LCP"),n=m(e,r,t),requestAnimationFrame((function(){requestAnimationFrame((function(){r.value=performance.now()-i.timeStamp,P[r.id]=!0,n(!0)}))}))}))}},D=function(e){var t,n=u("TTFB");t=function(){try{var t=performance.getEntriesByType("navigation")[0]||function(){var e=performance.timing,t={entryType:"navigation",startTime:0};for(var n in e)"navigationStart"!==n&&"toJSON"!==n&&(t[n]=Math.max(e[n]-e.navigationStart,0));return t}();if(n.value=n.delta=t.responseStart,n.value<0||n.value>performance.now())return;n.entries=[t],e(n)}catch(e){}},"complete"===document.readyState?setTimeout(t,0):addEventListener("pageshow",t)}}}]);
--- a/app/vmselect/vmui/static/js/main.9d24c3b2.chunk.js
+++ b/app/vmselect/vmui/static/js/main.9d24c3b2.chunk.js
--- a/app/vmselect/vmui/static/js/main.f4cab8bc.chunk.js
+++ b/app/vmselect/vmui/static/js/main.f4cab8bc.chunk.js
--- a/app/vmselect/vmui/static/js/runtime-main.f698388d.js
+++ b/app/vmselect/vmui/static/js/runtime-main.f698388d.js
@ -1 +1 @@
-!function(e){function r(r){for(var n,i,a=r[0],c=r[1],l=r[2],s=0,p=[];s<a.length;s++)i=a[s],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&p.push(o[i][0]),o[i]=0;for(n in c)Object.prototype.hasOwnProperty.call(c,n)&&(e[n]=c[n]);for(f&&f(r);p.length;)p.shift()();return u.push.apply(u,l||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var c=t[a];0!==o[c]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.e=function(e){var r=[],t=o[e];if(0!==t)if(t)r.push(t[2]);else{var n=new Promise((function(r,n){t=o[e]=[r,n]}));r.push(t[2]=n);var u,a=document.createElement("script");a.charset="utf-8",a.timeout=120,i.nc&&a.setAttribute("nonce",i.nc),a.src=function(e){return i.p+"static/js/"+({}[e]||e)+"."+{3:"0dc73915"}[e]+".chunk.js"}(e);var c=new Error;u=function(r){a.onerror=a.onload=null,clearTimeout(l);var t=o[e];if(0!==t){if(t){var n=r&&("load"===r.type?"missing":r.type),u=r&&r.target&&r.target.src;c.message="Loading chunk "+e+" failed.\n("+n+": "+u+")",c.name="ChunkLoadError",c.type=n,c.request=u,t[1](c)}o[e]=void 0}};var l=setTimeout((function(){u({type:"timeout",target:a})}),12e4);a.onerror=a.onload=u,document.head.appendChild(a)}return Promise.all(r)},i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!==typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"===typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="./",i.oe=function(e){throw console.error(e),e};var a=this.webpackJsonpvmui=this.webpackJsonpvmui||[],c=a.push.bind(a);a.push=r,a=a.slice();for(var l=0;l<a.length;l++)r(a[l]);var f=c;t()}([]);
+!function(e){function r(r){for(var n,i,a=r[0],c=r[1],f=r[2],s=0,p=[];s<a.length;s++)i=a[s],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&p.push(o[i][0]),o[i]=0;for(n in c)Object.prototype.hasOwnProperty.call(c,n)&&(e[n]=c[n]);for(l&&l(r);p.length;)p.shift()();return u.push.apply(u,f||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var c=t[a];0!==o[c]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.e=function(e){var r=[],t=o[e];if(0!==t)if(t)r.push(t[2]);else{var n=new Promise((function(r,n){t=o[e]=[r,n]}));r.push(t[2]=n);var u,a=document.createElement("script");a.charset="utf-8",a.timeout=120,i.nc&&a.setAttribute("nonce",i.nc),a.src=function(e){return i.p+"static/js/"+({}[e]||e)+"."+{3:"e51afffb"}[e]+".chunk.js"}(e);var c=new Error;u=function(r){a.onerror=a.onload=null,clearTimeout(f);var t=o[e];if(0!==t){if(t){var n=r&&("load"===r.type?"missing":r.type),u=r&&r.target&&r.target.src;c.message="Loading chunk "+e+" failed.\n("+n+": "+u+")",c.name="ChunkLoadError",c.type=n,c.request=u,t[1](c)}o[e]=void 0}};var f=setTimeout((function(){u({type:"timeout",target:a})}),12e4);a.onerror=a.onload=u,document.head.appendChild(a)}return Promise.all(r)},i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!==typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"===typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="./",i.oe=function(e){throw console.error(e),e};var a=this.webpackJsonpvmui=this.webpackJsonpvmui||[],c=a.push.bind(a);a.push=r,a=a.slice();for(var f=0;f<a.length;f++)r(a[f]);var l=c;t()}([]);
--- a/app/vmstorage/main.go
+++ b/app/vmstorage/main.go
@ -308,7 +308,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {

 	switch path {
 	case "/create":
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		snapshotPath, err := Storage.CreateSnapshot()
 		if err != nil {
 			err = fmt.Errorf("cannot create snapshot: %w", err)
@ -322,7 +322,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		return true
 	case "/list":
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		snapshots, err := Storage.ListSnapshots()
 		if err != nil {
 			err = fmt.Errorf("cannot list snapshots: %w", err)
@ -339,7 +339,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, `]}`)
 		return true
 	case "/delete":
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		snapshotName := r.FormValue("snapshot")
 		if err := Storage.DeleteSnapshot(snapshotName); err != nil {
 			err = fmt.Errorf("cannot delete snapshot %q: %w", snapshotName, err)
@ -349,7 +349,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		fmt.Fprintf(w, `{"status":"ok"}`)
 		return true
 	case "/delete_all":
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.Header().Set("Content-Type", "application/json")
 		snapshots, err := Storage.ListSnapshots()
 		if err != nil {
 			err = fmt.Errorf("cannot list snapshots: %w", err)
@ -717,6 +717,31 @@ func registerStorageMetrics() {
 		return float64(m().PrefetchedMetricIDsSizeBytes)
 	})

+	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/tsid"}`, func() float64 {
+		return float64(m().TSIDCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/metricIDs"}`, func() float64 {
+		return float64(m().MetricIDCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/metricName"}`, func() float64 {
+		return float64(m().MetricNameCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/bigIndexBlocks"}`, func() float64 {
+		return float64(tm().BigIndexBlocksCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/smallIndexBlocks"}`, func() float64 {
+		return float64(tm().SmallIndexBlocksCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/dataBlocks"}`, func() float64 {
+		return float64(idbm().DataBlocksCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/indexBlocks"}`, func() float64 {
+		return float64(idbm().IndexBlocksCacheSizeMaxBytes)
+	})
+	metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/tagFilters"}`, func() float64 {
+		return float64(idbm().TagFiltersCacheSizeMaxBytes)
+	})
+
 	metrics.NewGauge(`vm_cache_requests_total{type="storage/tsid"}`, func() float64 {
 		return float64(m().TSIDCacheRequests)
 	})
--- a/app/vmui/Dockerfile-web
+++ b/app/vmui/Dockerfile-web
@ -1,4 +1,4 @@
-FROM golang:1.17.1 as build-web-stage
+FROM golang:1.17.3 as build-web-stage
 COPY build /build

 WORKDIR /build
@ -6,7 +6,7 @@ COPY web/ /build/
 RUN GOOS=linux GOARCH=amd64 GO111MODULE=on CGO_ENABLED=0 go build -o web-amd64 github.com/VictoriMetrics/vmui/ && \
    GOOS=windows GOARCH=amd64 GO111MODULE=on CGO_ENABLED=0 go build -o web-windows github.com/VictoriMetrics/vmui/

-FROM alpine:3.14.2
+FROM alpine:3.15.0
 USER root

 COPY --from=build-web-stage /build/web-amd64 /app/web
--- a/app/vmui/packages/vmui/package-lock.json
+++ b/app/vmui/packages/vmui/package-lock.json
--- a/app/vmui/packages/vmui/package.json
+++ b/app/vmui/packages/vmui/package.json
@ -4,44 +4,45 @@
  "private": true,
  "homepage": "./",
  "dependencies": {
-    "@codemirror/autocomplete": "^0.19.4",
+    "@codemirror/autocomplete": "^0.19.9",
    "@codemirror/basic-setup": "^0.19.0",
    "@codemirror/commands": "^0.19.5",
    "@codemirror/highlight": "^0.19.6",
-    "@codemirror/state": "^0.19.4",
-    "@codemirror/view": "^0.19.14",
+    "@codemirror/state": "^0.19.6",
+    "@codemirror/view": "^0.19.21",
    "@date-io/dayjs": "^2.11.0",
-    "@emotion/react": "^11.5.0",
-    "@emotion/styled": "^11.3.0",
-    "@mui/icons-material": "^5.0.5",
-    "@mui/lab": "^5.0.0-alpha.53",
-    "@mui/material": "^5.0.6",
-    "@mui/styles": "^5.0.2",
-    "@testing-library/jest-dom": "^5.15.0",
+    "@emotion/react": "^11.7.0",
+    "@emotion/styled": "^11.6.0",
+    "@mui/icons-material": "^5.2.0",
+    "@mui/lab": "^5.0.0-alpha.58",
+    "@mui/material": "^5.2.2",
+    "@mui/styles": "^5.2.2",
+    "@testing-library/jest-dom": "^5.15.1",
    "@testing-library/react": "^12.1.2",
    "@testing-library/user-event": "^13.5.0",
-    "@types/jest": "^27.0.2",
+    "@types/jest": "^27.0.3",
    "@types/lodash.debounce": "^4.0.6",
    "@types/lodash.get": "^4.4.6",
-    "@types/node": "^16.11.6",
+    "@types/lodash.throttle": "^4.1.6",
+    "@types/node": "^16.11.10",
    "@types/numeral": "^2.0.2",
    "@types/qs": "^6.9.7",
-    "@types/react": "^17.0.34",
+    "@types/react": "^17.0.37",
    "@types/react-dom": "^17.0.11",
    "@types/react-measure": "^2.0.8",
    "codemirror-promql": "^0.18.0",
    "dayjs": "^1.10.7",
    "lodash.debounce": "^4.0.8",
    "lodash.get": "^4.4.2",
+    "lodash.throttle": "^4.1.1",
    "numeral": "^2.0.6",
    "qs": "^6.10.1",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-measure": "^2.5.2",
    "react-scripts": "4.0.3",
-    "typescript": "~4.4.4",
-    "uplot": "^1.6.16",
-    "uplot-react": "^1.1.1",
+    "typescript": "~4.5.2",
+    "uplot": "^1.6.17",
    "web-vitals": "^2.1.2"
  },
  "scripts": {
@ -72,10 +73,10 @@
  },
  "devDependencies": {
    "@babel/plugin-proposal-nullish-coalescing-operator": "^7.16.0",
-    "@typescript-eslint/eslint-plugin": "^5.3.0",
-    "@typescript-eslint/parser": "^5.3.0",
+    "@typescript-eslint/eslint-plugin": "^5.4.0",
+    "@typescript-eslint/parser": "^5.4.0",
    "customize-cra": "^1.0.0",
-    "eslint-plugin-react": "^7.26.1",
+    "eslint-plugin-react": "^7.27.1",
    "react-app-rewired": "^2.1.8"
  }
 }
--- a/app/vmui/packages/vmui/src/components/Home/Configurator/DisplayTypeSwitch.tsx
+++ b/app/vmui/packages/vmui/src/components/Home/Configurator/DisplayTypeSwitch.tsx
@ -12,8 +12,12 @@ export type DisplayType = "table" | "chart" | "code";

 const StylizedToggleButton = withStyles({
  root: {
-    padding: 6,
+    display: "grid",
+    gridTemplateColumns: "18px auto",
+    gridGap: 6,
+    padding: "8px 12px",
    color: "white",
+    lineHeight: "19px",
    "&.Mui-selected": {
      color: "white"
    }
@ -34,13 +38,13 @@ export const DisplayTypeSwitch: FC = () => {
        dispatch({type: "SET_DISPLAY_TYPE", payload: val ?? displayType})
    }>
    <StylizedToggleButton value="chart" aria-label="display as chart">
-      <ShowChartIcon/>&nbsp;Query Range as Chart
+      <ShowChartIcon/><span>Query Range as Chart</span>
    </StylizedToggleButton>
    <StylizedToggleButton value="code" aria-label="display as code">
-      <CodeIcon/>&nbsp;Instant Query as JSON
+      <CodeIcon/><span>Instant Query as JSON</span>
    </StylizedToggleButton>
    <StylizedToggleButton value="table" aria-label="display as table">
-      <TableChartIcon/>&nbsp;Instant Query as Table
+      <TableChartIcon/><span>Instant Query as Table</span>
    </StylizedToggleButton>
  </ToggleButtonGroup>;
 };
--- a/app/vmui/packages/vmui/src/components/Home/Configurator/TimeSelector.tsx
+++ b/app/vmui/packages/vmui/src/components/Home/Configurator/TimeSelector.tsx
@ -51,6 +51,13 @@ export const TimeSelector: FC<TimeSelectorProps> = ({setDuration}) => {
    setAnchorEl(null);
  };

+  const onKeyUp = (event: React.KeyboardEvent<HTMLInputElement>) => {
+    if (event.key !== "Enter") return;
+    const target = event.target as HTMLInputElement;
+    target.blur();
+    setDurationString(target.value);
+  };
+
  const open = Boolean(anchorEl);

  return <Box m={1} flexDirection="row" display="flex">
@ -60,12 +67,9 @@ export const TimeSelector: FC<TimeSelectorProps> = ({setDuration}) => {
        <TextField label="Duration" value={durationString} onChange={handleDurationChange}
          variant="standard"
          fullWidth={true}
-          onBlur={() => {
-            setFocused(false);
-          }}
-          onFocus={() => {
-            setFocused(true);
-          }}
+          onKeyUp={onKeyUp}
+          onBlur={() => {setFocused(false);}}
+          onFocus={() => {setFocused(true);}}
        />
      </Box>
      <Box my={2}>
--- a/app/vmui/packages/vmui/src/components/Home/Configurator/useFetchQuery.ts
+++ b/app/vmui/packages/vmui/src/components/Home/Configurator/useFetchQuery.ts
@ -2,7 +2,6 @@ import {useEffect, useMemo, useState} from "react";
 import {getQueryRangeUrl, getQueryUrl} from "../../../api/query-range";
 import {useAppState} from "../../../state/common/StateContext";
 import {InstantMetricResult, MetricResult} from "../../../api/types";
-import {saveToStorage} from "../../../utils/storage";
 import {isValidHttpUrl} from "../../../utils/url";
 import {useAuthState} from "../../../state/auth/AuthStateContext";
 import {TimeParams} from "../../../types";
@ -56,7 +55,6 @@ export const useFetchQuery = (): {
    try {
      const response = await fetch(fetchUrl, { headers });
      if (response.ok) {
-        saveToStorage("LAST_QUERY", query);
        const resp = await response.json();
        setError(undefined);
        displayType === "chart" ? setGraphData(resp.data.result) : setLiveData(resp.data.result);
--- a/app/vmui/packages/vmui/src/components/Home/Views/GraphView.tsx
+++ b/app/vmui/packages/vmui/src/components/Home/Views/GraphView.tsx
@ -3,60 +3,31 @@ import {MetricResult} from "../../../api/types";
 import LineChart from "../../LineChart/LineChart";
 import {AlignedData as uPlotData, Series as uPlotSeries} from "uplot";
 import {Legend, LegendItem} from "../../Legend/Legend";
-import {useAppState} from "../../../state/common/StateContext";
-import {getNameForMetric} from "../../../utils/metric";
-import {getColorFromString} from "../../../utils/color";
 import {useGraphDispatch, useGraphState} from "../../../state/graph/GraphStateContext";
-import {getHideSeries} from "../../../utils/uPlot";
+import {getHideSeries, getLegendItem, getLimitsYAxis, getSeriesItem, getTimeSeries} from "../../../utils/uPlot";

 export interface GraphViewProps {
  data?: MetricResult[];
 }

 const GraphView: FC<GraphViewProps> = ({data = []}) => {
-  const {time: {period}} = useAppState();
-
  const { yaxis } = useGraphState();
  const graphDispatch = useGraphDispatch();

-  const [timeArray, setTimeArray] = useState<number[]>([]);
  const [dataChart, setDataChart] = useState<uPlotData>([[]]);
  const [series, setSeries] = useState<uPlotSeries[]>([]);
  const [legend, setLegend] = useState<LegendItem[]>([]);
  const [hideSeries, setHideSeries] = useState<string[]>([]);
-
-  const setTimes = (times: number[]) => {
-    const allTimes = times.sort((a,b) => a-b);
-    const startTime = allTimes[0] || 0;
-    const endTime = allTimes[allTimes.length - 1] || 0;
-    const step = period.step || 1;
-    const length = Math.round((endTime - startTime) / step);
-    setTimeArray(new Array(length).fill(0).map((d, i) => startTime + (step * i)));
-  };
+  const [valuesLimit, setValuesLimit] = useState<[number, number]>([0, 1]);

  const setLimitsYaxis = (values: number[]) => {
    if (!yaxis.limits.enable || (yaxis.limits.range.every(item => !item))) {
-      const allValues = values.flat().sort((a,b) => a-b);
-      graphDispatch({type: "SET_YAXIS_LIMITS", payload: [allValues[0], allValues[allValues.length - 1]]});
+      const limits = getLimitsYAxis(values);
+      setValuesLimit(limits);
+      graphDispatch({type: "SET_YAXIS_LIMITS", payload: limits});
    }
  };

-  const getSeriesItem = (d: MetricResult) => {
-    const label = getNameForMetric(d);
-    return {
-      label,
-      width: 1.5,
-      stroke: getColorFromString(label),
-      show: !hideSeries.includes(label)
-    };
-  };
-
-  const getLegendItem = (s: uPlotSeries): LegendItem => ({
-    label: s.label || "",
-    color: s.stroke as string,
-    checked: s.show || false
-  });
-
  const onChangeLegend = (label: string, metaKey: boolean) => {
    setHideSeries(getHideSeries({hideSeries, label, metaKey, series}));
  };
@ -68,27 +39,34 @@ const GraphView: FC<GraphViewProps> = ({data = []}) => {
    const tempSeries: uPlotSeries[] = [];

    data?.forEach(d => {
-      const seriesItem = getSeriesItem(d);
+      const seriesItem = getSeriesItem(d, hideSeries);
      tempSeries.push(seriesItem);
      tempLegend.push(getLegendItem(seriesItem));

      d.values.forEach(v => {
-        if (tempTimes.indexOf(v[0]) === -1) tempTimes.push(v[0]);
+        tempTimes.push(v[0]);
        tempValues.push(+v[1]);
      });
    });

-    setTimes(tempTimes);
+    const timeSeries = getTimeSeries(tempTimes);
+    setDataChart([timeSeries, ...data.map(d => {
+      return new Array(timeSeries.length).fill(1).map((v, i) => d.values[i] ? +d.values[i][1] : null);
+    })] as uPlotData);
    setLimitsYaxis(tempValues);
-    setSeries([{}, ...tempSeries]);
+
+    const newSeries = [{}, ...tempSeries];
+    if (JSON.stringify(newSeries) !== JSON.stringify(series)) {
+      setSeries(newSeries);
      setLegend(tempLegend);
+    }
  }, [data]);

  useEffect(() => {
    const tempLegend: LegendItem[] = [];
    const tempSeries: uPlotSeries[] = [];
    data?.forEach(d => {
-      const seriesItem = getSeriesItem(d);
+      const seriesItem = getSeriesItem(d, hideSeries);
      tempSeries.push(seriesItem);
      tempLegend.push(getLegendItem(seriesItem));
    });
@ -96,17 +74,10 @@ const GraphView: FC<GraphViewProps> = ({data = []}) => {
    setLegend(tempLegend);
  }, [hideSeries]);

-  useEffect(() => {
-    setDataChart([timeArray, ...data.map(d => timeArray.map(t => {
-      const v = d.values.find(v => v[0] === t);
-      return v ? +v[1] : null;
-    }))]);
-  }, [timeArray]);
-
  return <>
    {(data.length > 0)
      ? <div>
-        <LineChart data={dataChart} series={series} metrics={data}/>
+        <LineChart data={dataChart} series={series} metrics={data} limits={valuesLimit}/>
        <Legend labels={legend} onChange={onChangeLegend}/>
      </div>
      : <div style={{textAlign: "center"}}>No data to show</div>}
--- a/app/vmui/packages/vmui/src/components/Home/Views/TableView.tsx
+++ b/app/vmui/packages/vmui/src/components/Home/Views/TableView.tsx
@ -24,7 +24,7 @@ const TableView: FC<GraphViewProps> = ({data}) => {
  const rows: InstantDataSeries[] = useMemo(() => {
    return data?.map(d => ({
      metadata: sortedColumns.map(c => d.metric[c.key] || "-"),
-      value: d.value[1]
+      value: d.value ? d.value[1] : "-"
    }));
  }, [sortedColumns, data]);

--- a/app/vmui/packages/vmui/src/components/LineChart/LineChart.tsx
+++ b/app/vmui/packages/vmui/src/components/LineChart/LineChart.tsx
@ -1,60 +1,63 @@
-import React, {FC, useRef, useState} from "react";
+import React, {FC, useCallback, useEffect, useRef, useState} from "react";
 import {useAppDispatch, useAppState} from "../../state/common/StateContext";
-import uPlot, {AlignedData as uPlotData, Options as uPlotOptions, Series as uPlotSeries} from "uplot";
-import UplotReact from "uplot-react";
-import "uplot/dist/uPlot.min.css";
-import numeral from "numeral";
-import "./tooltip.css";
+import uPlot, {AlignedData as uPlotData, Options as uPlotOptions, Series as uPlotSeries, Range} from "uplot";
 import {useGraphState} from "../../state/graph/GraphStateContext";
-import {setTooltip } from "../../utils/uPlot";
+import {defaultOptions, dragChart, setTooltip} from "../../utils/uPlot";
 import {MetricResult} from "../../api/types";
+import {limitsDurations} from "../../utils/time";
+import throttle from "lodash.throttle";
+import "uplot/dist/uPlot.min.css";
+import "./tooltip.css";

 export interface LineChartProps {
  metrics: MetricResult[]
  data: uPlotData;
-  series: uPlotSeries[]
+  series: uPlotSeries[],
+  limits: [number, number]
 }

-const LineChart: FC<LineChartProps> = ({data, series, metrics = []}) => {
+enum typeChartUpdate { xRange = "xRange", yRange = "yRange", data = "data" }
+
+const LineChart: FC<LineChartProps> = ({data, series, metrics = [], limits}) => {
  const dispatch = useAppDispatch();
  const {time: {period}} = useAppState();
  const { yaxis } = useGraphState();
-  const refContainer = useRef<HTMLDivElement>(null);
-  const [isPanning, setIsPanning] = useState(false);
+  const containerRef = useRef<HTMLDivElement>(null);
+  const uPlotRef = useRef<HTMLDivElement>(null);
+  const [isPanning, setPanning] = useState(false);
  const [zoomPos, setZoomPos] = useState(0);
-  const tooltipIdx = { seriesIdx: 1, dataIdx: 0 };
-  const tooltipOffset = { left: 0, top: 0 };
+  const [xRange, setXRange] = useState({ min: period.start, max: period.end });
+  const [uPlotInst, setUPlotInst] = useState<uPlot>();

  const tooltip = document.createElement("div");
  tooltip.className = "u-tooltip";
+  const tooltipIdx = { seriesIdx: 1, dataIdx: 0 };
+  const tooltipOffset = { left: 0, top: 0 };
+
+  const setScale = ({min, max}: {min: number, max: number}): void => {
+    dispatch({type: "SET_PERIOD", payload: {from: new Date(min * 1000), to: new Date(max * 1000)}});
+  };
+  const throttledSetScale = useCallback(throttle(setScale, 500), []);
+
+  const setPlotScale = ({u, min, max}: {u: uPlot, min: number, max: number}) => {
+    const delta = (max - min) * 1000;
+    if ((delta < limitsDurations.min) || (delta > limitsDurations.max)) return;
+    u.setScale("x", {min, max});
+    setXRange({min, max});
+    throttledSetScale({min, max});
+  };

  const onReadyChart = (u: uPlot) => {
    const factor = 0.85;
    tooltipOffset.left = parseFloat(u.over.style.left);
    tooltipOffset.top = parseFloat(u.over.style.top);
    u.root.querySelector(".u-wrap")?.appendChild(tooltip);
+
    // wheel drag pan
    u.over.addEventListener("mousedown", e => {
-      if (e.button !== 0) return;
-      setIsPanning(true);
-      e.preventDefault();
-      const left0 = e.clientX;
-      const onmove = (e: MouseEvent) => {
-        e.preventDefault();
-        const dx = (u.posToVal(1, "x") - u.posToVal(0, "x")) * (e.clientX - left0);
-        const min = (u.scales.x.min || 1) - dx;
-        const max = (u.scales.x.max || 1) - dx;
-        u.setScale("x", {min, max});
-        setScale({min, max});
-      };
-      const onup = () => {
-        setIsPanning(false);
-        document.removeEventListener("mousemove", onmove);
-        document.removeEventListener("mouseup", onup);
-      };
-      document.addEventListener("mousemove", onmove);
-      document.addEventListener("mouseup", onup);
+      dragChart({u, e, setPanning, setPlotScale, factor});
    });
+
    // wheel scroll zoom
    u.over.addEventListener("wheel", e => {
      if (!e.ctrlKey && !e.metaKey) return;
@ -66,10 +69,7 @@ const LineChart: FC<LineChartProps> = ({data, series, metrics = []}) => {
      const nxRange = e.deltaY < 0 ? oxRange * factor : oxRange / factor;
      const min = xVal - (zoomPos/width) * nxRange;
      const max = min + nxRange;
-      u.batch(() => {
-        u.setScale("x", {min, max});
-        setScale({min, max});
-      });
+      u.batch(() => setPlotScale({u, min, max}));
    });
  };

@ -89,43 +89,58 @@ const LineChart: FC<LineChartProps> = ({data, series, metrics = []}) => {
      : tooltip.style.display = "none";
  };

-  const setScale = ({min, max}: {min: number, max: number}): void => {
-    dispatch({type: "SET_PERIOD", payload: {from: new Date(min * 1000), to: new Date(max * 1000)}});
+  const getRangeY = (u: uPlot, min =  0, max =  1): Range.MinMax => {
+    if (yaxis.limits.enable) return yaxis.limits.range;
+    return min && max ? [min - (min * 0.05), max + (max * 0.05)] : limits;
+  };
+
+  const getRangeX = (): Range.MinMax => {
+    return [xRange.min, xRange.max];
  };

  const options: uPlotOptions = {
-    width: refContainer.current ? refContainer.current.offsetWidth : 400, height: 500, series: series,
+    ...defaultOptions,
+    width: containerRef.current ? containerRef.current.offsetWidth : 400,
+    series,
    plugins: [{ hooks: { ready: onReadyChart, setCursor, setSeries: seriesFocus }}],
-    cursor: {
-      drag: { x: false, y: false },
-      focus: { prox: 30 },
-      bind: {
-        mouseup: () => null,
-        mousedown: () => null,
-        click: () => null,
-        dblclick: () => null,
-        mouseenter: () => null,
-      }
-    },
-    legend: { show: false },
-    axes: [
-      { space: 80 },
-      { show: true, font: "10px Arial",
-        values: (self, ticks) => ticks.map(n => n > 1000 ? numeral(n).format("0.0a") : n) }
-    ],
    scales: {
-      x: { range: () => [period.start, period.end] },
-      y: {
-        range: (self, min, max) => {
-          const offsetFactor = 0.05; // 5%
-          return yaxis.limits.enable ? yaxis.limits.range : [min - (min * offsetFactor), max + (max * offsetFactor)];
-        }
-      }
+      x: { range: getRangeX },
+      y: { range: getRangeY }
    }
  };

-  return <div ref={refContainer} style={{pointerEvents: isPanning ? "none" : "auto", height: "500px"}}>
-    <UplotReact options={options} data={data}/>
+  const updateChart = (type: typeChartUpdate): void => {
+    if (!uPlotInst) return;
+    switch (type) {
+      case typeChartUpdate.xRange:
+        uPlotInst.scales.x.range = getRangeX;
+        break;
+      case typeChartUpdate.yRange:
+        uPlotInst.scales.y.range = getRangeY;
+        break;
+      case typeChartUpdate.data:
+        uPlotInst.setData(data);
+        break;
+    }
+    uPlotInst.redraw();
+  };
+
+  useEffect(() => setXRange({ min: period.start, max: period.end }), [period]);
+
+  useEffect(() => {
+    if (!uPlotRef.current) return;
+    const u = new uPlot(options, data, uPlotRef.current);
+    setUPlotInst(u);
+    setXRange({ min: period.start, max: period.end });
+    return u.destroy;
+  }, [uPlotRef.current, series]);
+
+  useEffect(() => updateChart(typeChartUpdate.data), [data]);
+  useEffect(() => updateChart(typeChartUpdate.xRange), [xRange]);
+  useEffect(() => updateChart(typeChartUpdate.yRange), [yaxis]);
+
+  return <div ref={containerRef} style={{pointerEvents: isPanning ? "none" : "auto", height: "500px"}}>
+    <div ref={uPlotRef}/>
  </div>;
 };

--- a/app/vmui/packages/vmui/src/state/common/reducer.ts
+++ b/app/vmui/packages/vmui/src/state/common/reducer.ts
@ -46,11 +46,11 @@ export type Action =

 const duration = getQueryStringValue("g0.range_input", "1h") as string;
 const endInput = formatDateToLocal(getQueryStringValue("g0.end_input", getDateNowUTC()) as Date);
-const query = getQueryStringValue("g0.expr", getFromStorage("LAST_QUERY") as string || "\n") as string;
+const query = getQueryStringValue("g0.expr", "") as string;

 export const initialState: AppState = {
  serverUrl: getDefaultServer(),
-  displayType: "chart",
+  displayType: getQueryStringValue("tab", "chart") as DisplayType,
  query: query, // demo_memory_usage_bytes
  queryHistory: { index: 0, values: [query] },
  time: {
--- a/app/vmui/packages/vmui/src/utils/math.ts
+++ b/app/vmui/packages/vmui/src/utils/math.ts
@ -0,0 +1,17 @@
+export const getMaxFromArray = (arr: number[]): number => {
+  let len = arr.length;
+  let max = -Infinity;
+  while (len--) {
+    if (arr[len] > max) max = arr[len];
+  }
+  return max;
+};
+
+export const getMinFromArray = (arr: number[]): number => {
+  let len = arr.length;
+  let min = Infinity;
+  while (len--) {
+    if (arr[len] < min) min = arr[len];
+  }
+  return min;
+};
--- a/app/vmui/packages/vmui/src/utils/query-string.ts
+++ b/app/vmui/packages/vmui/src/utils/query-string.ts
@ -6,7 +6,7 @@ const stateToUrlParams = {
  "time.duration": "g0.range_input",
  "time.period.date": "g0.end_input",
  "time.period.step": "g0.step_input",
-  "stacked": "g0.stacked",
+  "displayType": "tab"
 };

 // TODO need function for detect types.
--- a/app/vmui/packages/vmui/src/utils/storage.ts
+++ b/app/vmui/packages/vmui/src/utils/storage.ts
@ -1,5 +1,4 @@
-export type StorageKeys = "LAST_QUERY"
-    | "BASIC_AUTH_DATA"
+export type StorageKeys = "BASIC_AUTH_DATA"
    | "BEARER_AUTH_DATA"
    | "AUTH_TYPE"
    | "AUTOCOMPLETE"
--- a/app/vmui/packages/vmui/src/utils/time.ts
+++ b/app/vmui/packages/vmui/src/utils/time.ts
@ -1,15 +1,15 @@
 import {TimeParams, TimePeriod} from "../types";
-
 import dayjs, {UnitTypeShort} from "dayjs";
 import duration from "dayjs/plugin/duration";
 import utc from "dayjs/plugin/utc";
+import numeral from "numeral";

 dayjs.extend(duration);
 dayjs.extend(utc);

-const MAX_ITEMS_PER_CHART = window.screen.availWidth / 2;
+const MAX_ITEMS_PER_CHART = window.innerWidth / 2;

-export const limitsDurations = {min: 1000, max: 1.578e+11}; // min: 1 seconds, max: 5 years
+export const limitsDurations = {min: 1, max: 1.578e+11}; // min: 1 ms, max: 5 years

 export const dateIsoFormat = "YYYY-MM-DD[T]HH:mm:ss";

@ -26,6 +26,8 @@ export const supportedDurations = [

 const shortDurations = supportedDurations.map(d => d.short);

+export const roundTimeSeconds = (num: number): number => +(numeral(num).format("0.000"));
+
 export const isSupportedDuration = (str: string): Partial<Record<UnitTypeShort, string>> | undefined => {

  const digits = str.match(/\d+/g);
@ -57,7 +59,7 @@ export const getTimeperiodForDuration = (dur: string, date?: Date): TimeParams =
  }, {});

  const delta = dayjs.duration(durObject).asSeconds();
-  const step = Math.ceil(delta / MAX_ITEMS_PER_CHART);
+  const step = roundTimeSeconds(delta / MAX_ITEMS_PER_CHART) || 0.001;

  return {
    start: n - delta,
--- a/app/vmui/packages/vmui/src/utils/uPlot.ts
+++ b/app/vmui/packages/vmui/src/utils/uPlot.ts
@ -1,7 +1,12 @@
-import uPlot, {Series} from "uplot";
+import uPlot, {Series as uPlotSeries, Series} from "uplot";
 import {getColorFromString} from "./color";
 import dayjs from "dayjs";
 import {MetricResult} from "../api/types";
+import {LegendItem} from "../components/Legend/Legend";
+import {getNameForMetric} from "./metric";
+import {getMaxFromArray, getMinFromArray} from "./math";
+import {roundTimeSeconds} from "./time";
+import numeral from "numeral";

 interface SetupTooltip {
    u: uPlot,
@ -19,6 +24,34 @@ interface HideSeriesArgs {
  series: Series[]
 }

+interface DragArgs {
+  e: MouseEvent,
+  u: uPlot,
+  factor: number,
+  setPanning: (enable: boolean) => void,
+  setPlotScale: ({u, min, max}: {u: uPlot, min: number, max: number}) => void
+}
+
+const stub = (): null => null;
+
+export const defaultOptions = {
+  height: 500,
+  legend: { show: false },
+  axes: [
+    { space: 80 },
+    {
+      show: true,
+      font: "10px Arial",
+      values: (self: uPlot, ticks: number[]): (string | number)[] => ticks.map(n => n > 1000 ? numeral(n).format("0.0a") : n)
+    }
+  ],
+  cursor: {
+    drag: { x: false, y: false },
+    focus: { prox: 30 },
+    bind: { mouseup: stub, mousedown: stub, click: stub, dblclick: stub, mouseenter: stub }
+  },
+};
+
 export const setTooltip = ({ u, tooltipIdx, metrics, series, tooltip, tooltipOffset }: SetupTooltip) : void => {
  const {seriesIdx, dataIdx} = tooltipIdx;
  const dataSeries = u.data[seriesIdx][dataIdx];
@ -56,3 +89,59 @@ export const getHideSeries = ({hideSeries, label, metaKey, series}: HideSeriesAr
  }
  return include ? hideSeries.filter(l => l !== label) : [...hideSeries, label];
 };
+
+export const getTimeSeries = (times: number[]): number[] => {
+  const allTimes = Array.from(new Set(times)).sort((a,b) => a-b);
+  const step = getMinFromArray(allTimes.map((t, i) => allTimes[i + 1] - t));
+  const length = allTimes.length;
+  const startTime = allTimes[0] || 0;
+  return new Array(length).fill(startTime).map((d, i) => roundTimeSeconds(d + (step * i)));
+};
+
+export const getLimitsYAxis = (values: number[]): [number, number] => {
+  const min = getMinFromArray(values);
+  const max = getMaxFromArray(values);
+  return [min - (min * 0.05), max + (max * 0.05)];
+};
+
+export const getSeriesItem = (d: MetricResult, hideSeries: string[]): Series  => {
+  const label = getNameForMetric(d);
+  return {
+    label,
+    width: 1.5,
+    stroke: getColorFromString(label),
+    show: !hideSeries.includes(label),
+    scale: "y"
+  };
+};
+
+export const getLegendItem = (s: uPlotSeries): LegendItem => ({
+  label: s.label || "",
+  color: s.stroke as string,
+  checked: s.show || false
+});
+
+export const dragChart = ({e, factor = 0.85, u, setPanning, setPlotScale}: DragArgs): void => {
+  if (e.button !== 0) return;
+  e.preventDefault();
+  setPanning(true);
+  const leftStart = e.clientX;
+  const xUnitsPerPx = u.posToVal(1, "x") - u.posToVal(0, "x");
+  const scXMin = u.scales.x.min || 0;
+  const scXMax = u.scales.x.max || 0;
+
+  const mouseMove = (e: MouseEvent) => {
+    e.preventDefault();
+    const dx = xUnitsPerPx * ((e.clientX - leftStart) * factor);
+    setPlotScale({u, min: scXMin - dx, max: scXMax - dx});
+  };
+
+  const mouseUp = () => {
+    setPanning(false);
+    document.removeEventListener("mousemove", mouseMove);
+    document.removeEventListener("mouseup", mouseUp);
+  };
+
+  document.addEventListener("mousemove", mouseMove);
+  document.addEventListener("mouseup", mouseUp);
+};
--- a/dashboards/clusterbytenant.json
+++ b/dashboards/clusterbytenant.json
@ -475,7 +475,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_tenant_used_tenant_bytes{job=\"$job_storage\", instance=~\"$instance\",accountID=~\"$accountID\",projectID=~\"$projectID\"}) by(accountID,projectID)",
+          "expr": "sum(vm_tenant_used_tenant_bytes{job=~\"$job_storage\", instance=~\"$instance\",accountID=~\"$accountID\",projectID=~\"$projectID\"}) by(accountID,projectID)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
--- a/dashboards/victoriametrics.json
+++ b/dashboards/victoriametrics.json
@ -160,7 +160,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_rows{job=\"$job\", instance=~\"$instance\", type!=\"indexdb\"})",
+              "expr": "sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type!=\"indexdb\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -226,7 +226,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_data_size_bytes{job=\"$job\", type!=\"indexdb\"})",
+              "expr": "sum(vm_data_size_bytes{job=~\"$job\", type!=\"indexdb\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -292,7 +292,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_data_size_bytes{job=\"$job\", type!=\"indexdb\"}) / sum(vm_rows{job=\"$job\", type!=\"indexdb\"})",
+              "expr": "sum(vm_data_size_bytes{job=~\"$job\", type!=\"indexdb\"}) / sum(vm_rows{job=~\"$job\", type!=\"indexdb\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -358,7 +358,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_allowed_memory_bytes{job=\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(vm_allowed_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -423,7 +423,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "vm_app_uptime_seconds{job=\"$job\", instance=\"$instance\"}",
+              "expr": "vm_app_uptime_seconds{job=~\"$job\", instance=~\"$instance\"}",
              "instant": true,
              "interval": "",
              "legendFormat": "",
@ -485,7 +485,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_rows{job=\"$job\", instance=~\"$instance\", type=\"indexdb\"})",
+              "expr": "sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type=\"indexdb\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -551,7 +551,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "min(vm_free_disk_space_bytes{job=\"$job\", instance=~\"$instance\"})",
+              "expr": "min(vm_free_disk_space_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -621,7 +621,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_available_cpu_cores{job=\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(vm_available_cpu_cores{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -687,7 +687,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(vm_available_memory_bytes{job=\"$job\", instance=~\"$instance\"})",
+              "expr": "sum(vm_available_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "instant": true,
              "interval": "",
@ -772,7 +772,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(rate(vm_http_requests_total{job=\"$job\", instance=~\"$instance\", path!~\"/favicon.ico\"}[$__interval])) by (path) > 0",
+          "expr": "sum(rate(vm_http_requests_total{job=~\"$job\", instance=~\"$instance\", path!~\"/favicon.ico\"}[$__interval])) by (path) > 0",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -874,7 +874,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "max(vm_request_duration_seconds{job=\"$job\", instance=~\"$instance\", quantile=~\"(0.5|0.99)\"}) by (path, quantile) > 0",
+          "expr": "max(vm_request_duration_seconds{job=~\"$job\", instance=~\"$instance\", quantile=~\"(0.5|0.99)\"}) by (path, quantile) > 0",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{quantile}} ({{path}})",
@ -981,7 +981,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "vm_cache_entries{job=\"$job\", instance=~\"$instance\", type=\"storage/hour_metric_ids\"}",
+          "expr": "vm_cache_entries{job=~\"$job\", instance=~\"$instance\", type=\"storage/hour_metric_ids\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Active time series",
@ -1087,7 +1087,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_cache_size_bytes{job=\"$job\", instance=\"$instance\"})",
+          "expr": "sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -1095,7 +1095,7 @@
          "refId": "A"
        },
        {
-          "expr": "max(vm_allowed_memory_bytes{job=\"$job\", instance=\"$instance\"})",
+          "expr": "max(vm_allowed_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -1204,7 +1204,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_concurrent_addrows_capacity{job=\"$job\", instance=\"$instance\"})",
+          "expr": "sum(vm_concurrent_addrows_capacity{job=~\"$job\", instance=~\"$instance\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1212,7 +1212,7 @@
          "refId": "A"
        },
        {
-          "expr": "sum(vm_concurrent_addrows_current{job=\"$job\", instance=\"$instance\"})",
+          "expr": "sum(vm_concurrent_addrows_current{job=~\"$job\", instance=~\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "current",
@ -1316,7 +1316,7 @@
      "targets": [
        {
          "exemplar": true,
-          "expr": "sum(rate(vm_http_request_errors_total{job=\"$job\", instance=\"$instance\"}[$__interval])) by (path) > 0",
+          "expr": "sum(rate(vm_http_request_errors_total{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by (path) > 0",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1433,7 +1433,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(rate(vm_rows_inserted_total{job=\"$job\", instance=\"$instance\"}[$__interval])) by (type) > 0",
+          "expr": "sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[$__interval])) by (type) > 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -1537,7 +1537,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "vm_free_disk_space_bytes{job=\"$job\", instance=\"$instance\"} / ignoring(path) ((rate(vm_rows_added_to_storage_total{job=\"$job\", instance=\"$instance\"}[1d]) - ignoring(type) rate(vm_deduplicated_samples_total{job=\"$job\", instance=\"$instance\", type=\"merge\"}[1d])) * scalar(sum(vm_data_size_bytes{job=\"$job\", instance=\"$instance\", type!=\"indexdb\"}) / sum(vm_rows{job=\"$job\", instance=\"$instance\", type!=\"indexdb\"})))",
+          "expr": "vm_free_disk_space_bytes{job=~\"$job\", instance=~\"$instance\"} / ignoring(path) ((rate(vm_rows_added_to_storage_total{job=~\"$job\", instance=~\"$instance\"}[1d]) - ignoring(type) rate(vm_deduplicated_samples_total{job=~\"$job\", instance=~\"$instance\", type=\"merge\"}[1d])) * scalar(sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type!=\"indexdb\"}) / sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type!=\"indexdb\"})))",
          "format": "time_series",
          "hide": false,
          "interval": "",
@ -1646,7 +1646,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_rows{job=\"$job\", instance=~\"$instance\", type != \"indexdb\"})",
+          "expr": "sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type != \"indexdb\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1654,7 +1654,7 @@
          "refId": "A"
        },
        {
-          "expr": "sum(vm_data_size_bytes{job=\"$job\", instance=~\"$instance\", type!=\"indexdb\"}) / sum(vm_rows{job=\"$job\", instance=~\"$instance\", type != \"indexdb\"})",
+          "expr": "sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type!=\"indexdb\"}) / sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type != \"indexdb\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1762,7 +1762,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "vm_pending_rows{job=\"$job\", instance=~\"$instance\", type=\"storage\"}",
+          "expr": "vm_pending_rows{job=~\"$job\", instance=~\"$instance\", type=\"storage\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -1770,7 +1770,7 @@
          "refId": "A"
        },
        {
-          "expr": "vm_pending_rows{job=\"$job\", instance=~\"$instance\", type=\"indexdb\"}",
+          "expr": "vm_pending_rows{job=~\"$job\", instance=~\"$instance\", type=\"indexdb\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -1874,7 +1874,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_data_size_bytes{job=\"$job\", instance=~\"$instance\", type!=\"indexdb\"})",
+          "expr": "sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type!=\"indexdb\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1882,7 +1882,7 @@
          "refId": "A"
        },
        {
-          "expr": "vm_free_disk_space_bytes{job=\"$job\", instance=\"$instance\"}",
+          "expr": "vm_free_disk_space_bytes{job=~\"$job\", instance=~\"$instance\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -1984,7 +1984,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_parts{job=\"$job\", instance=\"$instance\"}) by (type)",
+          "expr": "sum(vm_parts{job=~\"$job\", instance=~\"$instance\"}) by (type)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{type}}",
@ -2086,7 +2086,7 @@
      "targets": [
        {
          "exemplar": true,
-          "expr": "vm_data_size_bytes{job=\"$job\", instance=~\"$instance\", type=\"indexdb\"}",
+          "expr": "vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type=\"indexdb\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@ -2187,7 +2187,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(vm_active_merges{job=\"$job\", instance=\"$instance\"}) by(type)",
+          "expr": "sum(vm_active_merges{job=~\"$job\", instance=~\"$instance\"}) by(type)",
          "legendFormat": "{{type}}",
          "refId": "A"
        }
@ -2288,7 +2288,7 @@
      "targets": [
        {
          "exemplar": true,
-          "expr": "sum(vm_rows_ignored_total{job=\"$job\", instance=\"$instance\"}) by (reason)",
+          "expr": "sum(vm_rows_ignored_total{job=~\"$job\", instance=~\"$instance\"}) by (reason)",
          "format": "time_series",
          "hide": false,
          "interval": "",
@ -2391,7 +2391,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(rate(vm_rows_merged_total{job=\"$job\", instance=\"$instance\"}[5m])) by(type)",
+          "expr": "sum(rate(vm_rows_merged_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by(type)",
          "legendFormat": "{{type}}",
          "refId": "A"
        }
@ -2493,7 +2493,7 @@
      "steppedLine": false,
      "targets": [
        {
-          "expr": "sum(rate(vm_log_messages_total{job=\"$job\", instance=\"$instance\"}[5m])) by (level) ",
+          "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (level) ",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
@ -2611,13 +2611,13 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(vm_new_timeseries_created_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_new_timeseries_created_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "interval": "",
              "legendFormat": "churn rate",
              "refId": "A"
            },
            {
-              "expr": "sum(increase(vm_new_timeseries_created_total{job=\"$job\", instance=\"$instance\"}[24h]))",
+              "expr": "sum(increase(vm_new_timeseries_created_total{job=~\"$job\", instance=~\"$instance\"}[24h]))",
              "interval": "",
              "legendFormat": "new series over 24h",
              "refId": "B"
@ -2717,7 +2717,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(vm_slow_queries_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_slow_queries_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -2820,7 +2820,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(vm_slow_row_inserts_total{job=\"$job\", instance=\"$instance\"}[5m])) / sum(rate(vm_rows_inserted_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(vm_slow_row_inserts_total{job=~\"$job\", instance=~\"$instance\"}[5m])) / sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -2923,7 +2923,7 @@
          "targets": [
            {
              "exemplar": true,
-              "expr": "sum(increase(vm_metrics_with_dropped_labels_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(increase(vm_metrics_with_dropped_labels_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -3042,7 +3042,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(go_memstats_sys_bytes{job=\"$job\", instance=\"$instance\"}) + sum(vm_cache_size_bytes{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -3050,7 +3050,7 @@
              "refId": "A"
            },
            {
-              "expr": "sum(go_memstats_heap_inuse_bytes{job=\"$job\", instance=\"$instance\"}) + sum(vm_cache_size_bytes{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -3058,7 +3058,7 @@
              "refId": "B"
            },
            {
-              "expr": "sum(go_memstats_stack_inuse_bytes{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(go_memstats_stack_inuse_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -3066,7 +3066,7 @@
              "refId": "C"
            },
            {
-              "expr": "sum(process_resident_memory_bytes{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(process_resident_memory_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -3076,7 +3076,7 @@
            },
            {
              "exemplar": true,
-              "expr": "sum(process_resident_memory_anon_bytes{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(process_resident_memory_anon_bytes{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -3178,7 +3178,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "rate(process_cpu_seconds_total{job=\"$job\", instance=\"$instance\"}[5m])",
+              "expr": "rate(process_cpu_seconds_total{job=~\"$job\", instance=~\"$instance\"}[5m])",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 1,
@ -3285,7 +3285,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(process_open_fds{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(process_open_fds{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
@ -3293,7 +3293,7 @@
              "refId": "A"
            },
            {
-              "expr": "min(process_max_fds{job=\"$job\", instance=\"$instance\"})",
+              "expr": "min(process_max_fds{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "interval": "",
              "intervalFactor": 2,
@ -3401,7 +3401,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(process_io_storage_read_bytes_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -3410,7 +3410,7 @@
              "refId": "A"
            },
            {
-              "expr": "sum(rate(process_io_storage_written_bytes_total{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "hide": false,
              "interval": "",
@ -3513,7 +3513,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(go_goroutines{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(go_goroutines{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "intervalFactor": 2,
              "legendFormat": "gc duration",
@ -3615,7 +3615,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(go_gc_duration_seconds_sum{job=\"$job\", instance=\"$instance\"}[5m]))\n/\nsum(rate(go_gc_duration_seconds_count{job=\"$job\", instance=\"$instance\"}[5m]))",
+              "expr": "sum(rate(go_gc_duration_seconds_sum{job=~\"$job\", instance=~\"$instance\"}[5m]))\n/\nsum(rate(go_gc_duration_seconds_count{job=~\"$job\", instance=~\"$instance\"}[5m]))",
              "format": "time_series",
              "intervalFactor": 2,
              "legendFormat": "avg gc duration",
@ -3715,7 +3715,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(process_num_threads{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(process_num_threads{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "intervalFactor": 2,
              "legendFormat": "threads",
@ -3817,7 +3817,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(vm_tcplistener_conns{job=\"$job\", instance=\"$instance\"})",
+              "expr": "sum(vm_tcplistener_conns{job=~\"$job\", instance=~\"$instance\"})",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -3920,7 +3920,7 @@
          "steppedLine": false,
          "targets": [
            {
-              "expr": "sum(rate(vm_tcplistener_accepts_total{job=\"$job\", instance=\"$instance\"}[$__interval]))",
+              "expr": "sum(rate(vm_tcplistener_accepts_total{job=~\"$job\", instance=~\"$instance\"}[$__interval]))",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
@ -4035,7 +4035,7 @@
        "allValue": null,
        "current": {},
        "datasource": "$ds",
-        "definition": "label_values(vm_app_version{job=\"$job\", instance=\"$instance\"},  version)",
+        "definition": "label_values(vm_app_version{job=~\"$job\", instance=~\"$instance\"},  version)",
        "description": null,
        "error": null,
        "hide": 2,
@ -4045,7 +4045,7 @@
        "name": "version",
        "options": [],
        "query": {
-          "query": "label_values(vm_app_version{job=\"$job\", instance=\"$instance\"},  version)",
+          "query": "label_values(vm_app_version{job=~\"$job\", instance=~\"$instance\"},  version)",
          "refId": "VictoriaMetrics-version-Variable-Query"
        },
        "refresh": 1,
--- a/dashboards/vmagent.json
+++ b/dashboards/vmagent.json
@ -1481,7 +1481,7 @@
      "targets": [
        {
          "exemplar": true,
-          "expr": "sum(rate(vm_log_messages_total{job=\"$job\", instance=~\"$instance\"}[5m])) by (level) ",
+          "expr": "sum(rate(vm_log_messages_total{job=~\"$job\", instance=~\"$instance\"}[5m])) by (level) ",
          "format": "time_series",
          "hide": false,
          "interval": "",
@ -4382,7 +4382,7 @@
        "allValue": ".*",
        "current": {},
        "datasource": "$ds",
-        "definition": "label_values(vmagent_remotewrite_requests_total{job=\"$job\", instance=~\"$instance\"}, url)",
+        "definition": "label_values(vmagent_remotewrite_requests_total{job=~\"$job\", instance=~\"$instance\"}, url)",
        "description": "The remote write URLs",
        "error": null,
        "hide": 0,
@ -4392,7 +4392,7 @@
        "name": "url",
        "options": [],
        "query": {
-          "query": "label_values(vmagent_remotewrite_requests_total{job=\"$job\", instance=~\"$instance\"}, url)",
+          "query": "label_values(vmagent_remotewrite_requests_total{job=~\"$job\", instance=~\"$instance\"}, url)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
--- a/deployment/docker/Makefile
+++ b/deployment/docker/Makefile
@ -2,11 +2,11 @@

 DOCKER_NAMESPACE := victoriametrics

-ROOT_IMAGE ?= alpine:3.14.2
-CERTS_IMAGE := alpine:3.14.2
+ROOT_IMAGE ?= alpine:3.15.0
+CERTS_IMAGE := alpine:3.15.0
 GO_BUILDER_IMAGE := golang:1.17.3-alpine
-BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr : _)
-BASE_IMAGE := local/base:1.1.3-$(shell echo $(ROOT_IMAGE) | tr : _)-$(shell echo $(CERTS_IMAGE) | tr : _)
+BUILDER_IMAGE := local/builder:2.0.0-$(shell echo $(GO_BUILDER_IMAGE) | tr :/ __)
+BASE_IMAGE := local/base:1.1.3-$(shell echo $(ROOT_IMAGE) | tr :/ __)-$(shell echo $(CERTS_IMAGE) | tr :/ __)

 package-base:
 	(docker image ls --format '{{.Repository}}:{{.Tag}}' | grep -q '$(BASE_IMAGE)$$') \
--- a/deployment/docker/alerts.yml
+++ b/deployment/docker/alerts.yml
@ -192,7 +192,7 @@ groups:
        labels:
          severity: warning
        annotations:
-          dashboard: "http://localhost:3000/d/oS7Bi_0Wz?viewPanel=74&var-instance={{ $labels.instance }}"
+          dashboard: "http://localhost:3000/d/wNf0q_kZk?viewPanel=74&var-instance={{ $labels.instance }}"
          summary: "Metrics ingested in ({{ $labels.instance }}) are exceeding labels limit"
          description: "VictoriaMetrics limits the number of labels per each metric with `-maxLabelsPerTimeseries` command-line flag.\n
           This prevents from ingesting metrics with too many labels. Please verify that `-maxLabelsPerTimeseries` is configured
--- a/docs/Articles.md
+++ b/docs/Articles.md
@ -17,6 +17,7 @@ See also [case studies](https://docs.victoriametrics.com/CaseStudies.html).
 * [Sismology: Iguana Solutions’ Monitoring System](https://medium.com/nerd-for-tech/sismology-iguana-solutions-monitoring-system-f46e4170447f)
 * [Prometheus High Availability and Fault Tolerance strategy, long term storage with VictoriaMetrics](https://medium.com/miro-engineering/prometheus-high-availability-and-fault-tolerance-strategy-long-term-storage-with-victoriametrics-82f6f3f0409e)
 * [How we improved our Kubernetes monitoring at Smarkets, and how you could too](https://smarketshq.com/monitoring-kubernetes-clusters-41a4b24c19e3)
+* [Kubernetes and VictoriaMetrics in Mist v4.6](https://mist.io/blog/2021-11-26-kubernetes-and-victoriametrics-in-Mist-v4-6)
 * [Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics](https://www.percona.com/blog/2020/12/01/foiled-by-the-firewall-a-tale-of-transition-from-prometheus-to-victoriametrics/)
 * [Observations on Better Resource Usage with Percona Monitoring and Management v2.12.0](https://www.percona.com/blog/2020/12/23/observations-on-better-resource-usage-with-percona-monitoring-and-management-v2-12-0/)
 * [Better Prometheus rate() function with VictoriaMetrics](https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/)
@ -98,5 +99,6 @@ See also [case studies](https://docs.victoriametrics.com/CaseStudies.html).

 ### Other articles

+* [How ClickHouse inspired us to build a high performance time series database](https://www.youtube.com/watch?v=p9qjb_yoBro). See also [slides](https://docs.google.com/presentation/d/1SdFrwsyR-HMXfbzrY8xfDZH_Dg6E7E5NJ84tQozMn3w/edit?usp=sharing).
 * [Comparing Thanos to VictoriaMetrics cluster](https://faun.pub/comparing-thanos-to-victoriametrics-cluster-b193bea1683)
 * [Evaluation performance and correctness: VictoriaMetrics response](https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87)
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@ -7,6 +7,34 @@ sort: 15
 ## tip


+## [v1.70.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.70.0)
+
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add ability to pass arbitrary query args to `-datasource.url` on a per-group basis via `params` option. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1892).
+* FEATURE: add `now()` function to MetricsQL. This function returns the current timestamp in seconds. See [these docs](https://docs.victoriametrics.com/MetricsQL.html#now).
+* FEATURE: vmauth: allow using optional `name` field in configs. This field is then used as `username` label value for `vmauth_user_requests_total` metric. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1805).
+* FEATURE: vmagent: export `vm_persistentqueue_read_duration_seconds_total` and `vm_persistentqueue_write_duration_seconds_total` metrics, which can be used for detecting persistent queue saturation with `rate(vm_persistentqueue_write_duration_seconds_total) > 0.9` alerting rule.
+* FEATURE: export `vm_filestream_read_duration_seconds_total` and `vm_filestream_write_duration_seconds_total` metrics, which can be used for detecting persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9` alerting rule.
+* FEATURE: export `vm_cache_size_max_bytes` metrics, which show capacity for various caches. These metrics can be used for determining caches reaches its capacity with `vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9` query.
+* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html), [vmrestore](https://docs.victoriametrics.com/vmrestore.html): add `-s3ForcePathStyle` command-line flag, which can be used for making backups to [Aliyun OSS](https://www.aliyun.com/product/oss). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1802).
+* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): improve data migration from OpenTSDB. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1809). Thanks to @johnseekins .
+* FEATURE: suppress `connection reset by peer` errors when remote client resets TCP connection to VictoriaMetrics / vmagent while ingesting the data via InfluxDB line protocol, Graphite protocol or OpenTSDB protocol. This error is expected, so there is no need in logging it.
+* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): store the display type in URL, so it isn't lost when copy-pasting the URL. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1804).
+* FEATURE: vmalert: make `-notifier.url` command-line flag optional. This flag can be omitted if `vmalert` is used solely for recording rules and doesn't evaluate alerting rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1870).
+* FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html), [vmrestore](https://docs.victoriametrics.com/vmrestore.html): export internal metrics at `http://vmbackup:8420/metrics` and `http://vmrestore:8421/metrics` for better visibility of the backup/restore process.
+* FEATURE: allow trailing whitespace after the timestamp when [parsing Graphite plaintext lines](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865).
+* FEATURE: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does. This is needed for improving integration with third-party solutions, which rely on these endpoints. See [tis issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833).
+
+* BUGFIX: vmagent: prevent from scraping duplicate targets if `-promscrape.dropOriginalLabels` command-line flag is set. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1830). Thanks to @guidao for the fix.
+* BUGFIX: vmstorage [enterprise](https://victoriametrics.com/enterprise.html): added missing `vm_tenant_used_tenant_bytes` metric, which shows the approximate per-tenant disk usage. See [these docs](https://docs.victoriametrics.com/PerTenantStatistic.html) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1605).
+* BUGFIX: vmauth: properly take into account the value passed to `-maxIdleConnsPerBackend` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300).
+* BUGFIX: vmagent: fix [reading data from Kafka](https://docs.victoriametrics.com/vmagent.html#reading-metrics-from-kafka).
+* BUGFIX: vmalert: fix [replay mode](https://docs.victoriametrics.com/vmalert.html#rules-backfilling) in enterprise version.
+* BUGFIX: consistently return zero from [deriv()](https://docs.victoriametrics.com/MetricsQL.html#deriv) function applied to a constant time series. Previously it could return small non-zero values in this case.
+* BUGFIX: [vmrestore](https://docs.victoriametrics.com/vmrestore.html): properly resume downloading for partially downloaded big files. Previously such files were re-downloaded from the beginning after the interrupt. Now only the remaining parts of the file are downloaded. This allows saving network bandwidth. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487).
+* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): do not store the last query across vmui page reloads. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1694).
+* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix `Cannot read properties of undefined` error at table view. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1797).
+
+
 ## [v1.69.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.69.0)

 * FEATURE: vmalert: allow groups with empty rules list like Prometheus does. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1742).
--- a/docs/Cluster-VictoriaMetrics.md
+++ b/docs/Cluster-VictoriaMetrics.md
@ -179,7 +179,7 @@ By default the following TCP ports are used:
 It is recommended setting up [vmagent](https://docs.victoriametrics.com/vmagent.html)
 or Prometheus to scrape `/metrics` pages from all the cluster components, so they can be monitored and analyzed
 with [the official Grafana dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11176)
-or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11831).
+or [an alternative dashboard for VictoriaMetrics cluster](https://grafana.com/grafana/dashboards/11831). Graphs on these dashboards contain useful hints - hover the `i` icon at the top left corner of each graph in order to read it.

 It is recommended setting up alerts in [vmalert](https://docs.victoriametrics.com/vmalert.html) or in Prometheus from [this config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/deployment/docker/alerts.yml).

@ -371,7 +371,7 @@ so up to 2 `vmstorage` nodes can be lost without data loss. The minimum number o
 the remaining 3 `vmstorage` nodes could provide the `-replicationFactor=3` for newly ingested data.

 When the replication is enabled, `-dedup.minScrapeInterval=1ms` command-line flag must be passed to `vmselect` nodes.
-Optional `-replicationFactor=N` command-line flag can be passed to `vminsert` for improving query performance when up to `N-1` vmstorage nodes respond slowly and/or temporarily unavailable, since `vmselect` doesn't wait for responses from up to `N-1` `vmstorage` nodes. Sometimes `-replicationFactor` at `vmselect` nodes can result in partial responses. See [this issues](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207) for details.
+Optional `-replicationFactor=N` command-line flag can be passed to `vmselect` for improving query performance when up to `N-1` vmstorage nodes respond slowly and/or temporarily unavailable, since `vmselect` doesn't wait for responses from up to `N-1` `vmstorage` nodes. Sometimes `-replicationFactor` at `vmselect` nodes can result in partial responses. See [this issues](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207) for details.
 The `-dedup.minScrapeInterval=1ms` de-duplicates replicated data during queries. If duplicate data is pushed to VictoriaMetrics from identically configured [vmagent](https://docs.victoriametrics.com/vmagent.html) instances or Prometheus instances, then the `-dedup.minScrapeInterval` must be set to bigger values according to [deduplication docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).

 Note that [replication doesn't save from disaster](https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883),
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@ -205,7 +205,7 @@ if a query covers 1000 metrics with 10K values each, then the remote read API ha
 This is slow and expensive.
 Prometheus remote read API isn't intended for querying foreign data aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details.

-So just query VictoriaMetrics directly via [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
+So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
 or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/#grafana-setup).


--- a/docs/MetricsQL.md
+++ b/docs/MetricsQL.md
@ -324,7 +324,7 @@ See also [implicit query conversions](#implicit-query-conversions).

 #### tlast_over_time

-`tlast_over_time(series_selector[d])` returns the timestamp in seconds for the last raw sample on the given lookbehind window `d` per each time series returned from the given [series_selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). Metric names are stripped from the resulting rollups. See also [last_over_time](#last_over_time).
+`tlast_over_time(series_selector[d])` is an alias for [timestamp](#timestamp).

 #### tmax_over_time

@ -507,6 +507,10 @@ See also [implicit query conversions](#implicit-query-conversions).

 `month(q)` returns the month for every point of every time series returned by `q`. It is expected that `q` returns unix timestamps. The returned values are in the range `[1...12]`, where `1` means January and `12` means December. Metric names are stripped from the resulting series. This function is supported by PromQL.

+#### now
+
+`now()` returns the current timestamp as a floating-point value in seconds. See also [time](#time).
+
 #### pi

 `pi()` returns [Pi number](https://en.wikipedia.org/wiki/Pi). This function is supported by PromQL.
--- a/Show more
+++ b/Show more
				`@ -0,0 +1 @@`
				.uplot,.uplot ,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;position:relative;width:100%;height:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline {display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{height:100%;border-right:1px dashed #607d8b}.u-hz .u-cursor-y,.u-vt .u-cursor-x{width:100%;border-bottom:1px dashed #607d8b}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0 solid;pointer-events:none;will-change:transform;z-index:100;background-clip:padding-box!important}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off{display:none}
				`@ -1 +0,0 @@`
				.uplot,.uplot ,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{text-align:center;font-size:18px;font-weight:700}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;position:relative;width:100%;height:100%}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline {display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>{vertical-align:middle;display:inline-block}.u-legend .u-marker{width:1em;height:1em;margin-right:4px;background-clip:padding-box!important}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{position:absolute;pointer-events:none}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{height:100%;border-right:1px dashed #607d8b}.u-hz .u-cursor-y,.u-vt .u-cursor-x{width:100%;border-bottom:1px dashed #607d8b}.u-cursor-pt{position:absolute;top:0;left:0;border-radius:50%;border:0 solid;pointer-events:none;will-change:transform;z-index:100;background-clip:padding-box!important}.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off{display:none}