- v1.11 doc updates (#5852)

- fix dead links

deployment/docker/vmanomaly/vmanomaly-integration/docker-compose.yml

@@ -73,7 +73,7 @@ services:
     restart: always
   vmanomaly:
     container_name: vmanomaly
-    image: victoriametrics/vmanomaly:v1.10.0
+    image: victoriametrics/vmanomaly:v1.11.0
     depends_on:
       - "victoriametrics"
     ports:

deployment/docker/vmanomaly/vmanomaly-integration/vmanomaly_config.yml

@@ -1,4 +1,6 @@
-scheduler:
+schedulers:
+  periodic:
+    # class: "scheduler.periodic.PeriodicScheduler"
     infer_every: "1m"
     fit_every: "2m"
     fit_window: "3h"

docs/anomaly-detection/CHANGELOG.md

@@ -15,7 +15,18 @@ aliases:
 
 Please find the changelog for VictoriaMetrics Anomaly Detection below.
 
-> **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or later versions for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100).**
+> **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or later versions for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100) or newer.**
+
+## v1.11.0
+Released: 2024-02-22
+- FEATURE: Multi-scheduler support. Now users can use multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), each spec can be run with its own (even multiple) [schedulers](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/).
+  - Introduction of `schedulers` arg in model spec:
+    - It allows each model to be managed by 1 (or more) schedulers, so overall resource usage is optimized and flexibility is preserved. 
+    - Passing an empty list or not specifying this param implies that each model is run in **all** the schedulers, which is a backward-compatible behavior.
+    - Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#schedulers)
+- DEPRECATION: slight refactor of a scheduler config section 
+  - Now schedulers are passed as a mapping of `scheduler_alias: scheduler_spec` under [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) sections. Using old format (< [1.11.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1110)) will produce warnings for now and will be removed in future versions.
+- DEPRECATION: The `--watch` CLI option for config file reloads is deprecated and will be ignored in the future.
 
 ## v1.10.0
 Released: 2024-02-15

docs/anomaly-detection/Overview.md

@@ -138,13 +138,13 @@ optionally preserving labels).
 
 > See [Quickstart](/anomaly-detection/QuickStart.html).
 
-> See [Integration guide: vmanomaly and vmalert](anomaly-detection/guides/guide-vmanomaly-vmalert.html).
+> See [Integration guide: vmanomaly and vmalert](/anomaly-detection/guides/guide-vmanomaly-vmalert.html).
 
 ### Config file
 There are 4 required sections in config file:
 
-* [`scheduler`](/anomaly-detection/components/scheduler.html) - defines how often to run and make inferences, as well as what timerange to use to train the model.
+* [`schedulers`](/anomaly-detection/components/scheduler.html) - defines how often to run and make inferences, as well as what timerange to use to train the model.
-* [`model`](/anomaly-detection/components/models.html) - specific model parameters and configurations,
+* [`models`](/anomaly-detection/components/models.html) - specific model parameters and configurations.
 * [`reader`](/anomaly-detection/components/reader.html) - how to read data and where it is located
 * [`writer`](/anomaly-detection/components/writer.html) - where and how to write the generated output.
 
@@ -153,13 +153,14 @@ There are 4 required sections in config file:
 > For a detailed description, see [config sections](/anomaly-detection/components)
 
 #### Config example
-Here is an example of config file that will run FB Prophet model, that will be retrained every 2 hours on 14 days of previous data. It will generate inference (including `anomaly_score` metric) every 1 minute.
+Here is an example of config file that will run [Facebook's Prophet model](/anomaly-detection/components/models.html#prophet), that will be retrained every 2 hours on 14 days of previous data. It will generate inference results (including `anomaly_score` metric) every 1 minute.
 
 
-You need to put your datasource urls to use it:
+You need to specify your datasource urls to use it:
 
 ```yaml
-scheduler:
+schedulers:
+  periodic:
     infer_every: "1m"
     fit_every: "2h"
     fit_window: "14d"
@@ -298,4 +299,3 @@ groups:
           description: "{{ $labels.instance }} of job {{ $labels.job }} license expires in {{ $value | humanizeDuration }}. 
             Please make sure to update the license before it expires."
 ```
-

docs/anomaly-detection/components/README.md

@@ -17,7 +17,7 @@ This chapter describes different components, that correspond to respective secti
 
 - [Model(s) section](models.html) - Required
 - [Reader section](reader.html) - Required
-- [Scheduler section](scheduler.html) - Required
+- [Scheduler(s) section](scheduler.html) - Required
 - [Writer section](writer.html) - Required
 - [Monitoring section](monitoring.html) -  Optional
 

docs/anomaly-detection/components/models.md

@@ -36,15 +36,17 @@ models:
       # i.e. to assure reproducibility of produced results each time model is fit on the same input
       random_state: 42
     # if there is no explicit `queries` arg, then the model will be run on ALL queries found in reader section
+...
 ```  
 
-Old-style configs (< [1.10.0](/anomaly-detection/changelog#v1100) )
+Old-style configs (< [1.10.0](/anomaly-detection/changelog#v1100))
 
 ```yaml
 model:
     class: "model.zscore.ZscoreModel"
     z_threshold: 2.5
     # no explicit `queries` arg is provided
+...
 ```
 
 will be **implicitly** converted to
@@ -56,6 +58,7 @@ models:
     z_threshold: 2.5
     # queries arg is created and propagated with all query aliases found in `queries` arg of `reader` section
     queries: ["q1", "q2", "q3"]  # i.e., if your `queries` in `reader` section has exactly q1, q2, q3 aliases
+...
 ```
 
 
@@ -88,6 +91,31 @@ models:
     queries: ["q1", "q2", "q3"]  # i.e., if your `queries` in `reader` section has exactly q1, q2, q3 aliases
 ```
 
+### Schedulers
+
+Introduced in [1.11.0](/anomaly-detection/changelog#1110), as a part to support multi-scheduler configs, `schedulers` arg is meant to define [schedulers](/anomaly-detection/components/scheduler) particular model should be attached to.
+
+`schedulers` arg is supported for all [the built-in](/anomaly-detection/components/models/#built-in-models) (as well as for [custom](/anomaly-detection/components/models/#custom-model-guide)) models.
+
+This arg is **backward compatible** - if there is no explicit `schedulers` arg, then the model, defined in a config, will be attached to ALL the schedulers found in scheduler section:
+
+```yaml
+models:
+  model_alias_1:
+    ...
+    # no explicit `schedulers` arg is provided
+```
+
+will be implicitly converted to
+
+```yaml
+models:
+  model_alias_1:
+    ...
+    # queries arg is created and propagated with all query aliases found in `queries` arg of `reader` section
+    schedulers: ["s1", "s2", "s3"]  # i.e., if your `schedulers` section has exactly s1, s2, s3 aliases
+```
+
 ## Model types
 
 There are **2 model types**, supported in `vmanomaly`, resulting in **4 possible combinations**:

docs/anomaly-detection/components/scheduler.md

@@ -15,6 +15,46 @@ aliases:
 Scheduler defines how often to run and make inferences, as well as what timerange to use to train the model.
 Is specified in `scheduler` section of a config for VictoriaMetrics Anomaly Detection.
 
+> **Note: Starting from [v1.11.0](/anomaly-detection/changelog#v1110) scheduler section in config supports multiple schedulers via aliasing. <br>Also, `vmanomaly` expects scheduler section to be named `schedulers`. Using old (flat) format with `scheduler` key is deprecated and will be removed in future versions.**
+
+```yaml
+schedulers:
+  scheduler_periodic_1m:
+    # class: "scheduler.periodic.PeriodicScheduler"
+    infer_every: "1m"
+    fit_every: "2m"
+    fit_window: "3h"
+  scheduler_periodic_5m:
+    # class: "scheduler.periodic.PeriodicScheduler"
+    infer_every: "5m"
+    fit_every: "10m"
+    fit_window: "3h"
+...
+```  
+
+Old-style configs (< [1.11.0](/anomaly-detection/changelog#v1110))
+
+```yaml
+scheduler:
+  # class: "scheduler.periodic.PeriodicScheduler"
+  infer_every: "1m"
+  fit_every: "2m"
+  fit_window: "3h"
+...
+```
+
+will be **implicitly** converted to
+
+```yaml
+schedulers:
+  default_scheduler:  # default scheduler alias, backward compatibility
+    # class: "scheduler.periodic.PeriodicScheduler"
+    infer_every: "1m"
+    fit_every: "2m"
+    fit_window: "3h"
+...
+```
+
 ## Parameters
 
 `class`: str, default=`"scheduler.periodic.PeriodicScheduler"`,

docs/anomaly-detection/guides/guide-vmanomaly-vmalert.md

@@ -32,11 +32,13 @@ aliases:
 
 ## 1. What is vmanomaly?
 
-*VictoriaMetrics Anomaly Detection* ([vmanomaly](https://docs.victoriametrics.com/anomaly-detection/Overview.html)) is a service that continuously scans time series stored in VictoriaMetrics and detects unexpected changes within data patterns in real-time. It does so by utilizing user-configurable machine learning models.
+*VictoriaMetrics Anomaly Detection* ([vmanomaly](/anomaly-detection/Overview.html)) is a service that continuously scans time series stored in VictoriaMetrics and detects unexpected changes within data patterns in real-time. It does so by utilizing user-configurable machine learning models.
 
 All the service parameters are defined in a config file.
 
-> **Note**: Starting from [1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100), each `vmanomaly` configuration file can support more that one model type. To utilize *different models* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/models/) config section for more details.
+> **Note**: Starting from [1.10.0](/anomaly-detection/changelog/#v1100), each `vmanomaly` configuration file can support more that one model type. To utilize *different models* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/components/models/) config section for more details.
+
+> **Note**: Starting from [1.11.0](/anomaly-detection/changelog/#v1110), each `vmanomaly` configuration file can support more that one model type, each attached to one (or more) schedulers. To utilize *different models* with *different schedulers* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/components/models/#schedulers) and [scheduler](/anomaly-detection/components/scheduler/) config sections for more details.
 
 
 **vmanomaly** does the following:
@@ -115,9 +117,9 @@ This query will yield a total of eight time series, each corresponding to a CPU
 **Parameter description**:
 The configuration file for `vmanomaly` comprises 4 essential sections:
 
-1. [`scheduler`](/anomaly-detection/components/scheduler.html) - This section determines the frequency of model inferences and training, including the time range for model training.
+1. [`scheduler`](/anomaly-detection/components/scheduler.html) - This section determines the frequency of model inferences and training, including the time range for model training. Starting from [v1.11.0](/anomaly-detection/changelog/#v1110), multiple individual schedulers are supported for each model type in a single config.
 
-2. [`models`](/anomaly-detection/components/models.html) - Here, you define specific parameters and configurations for the models being used for anomaly detection.
+2. [`models`](/anomaly-detection/components/models.html) - Here, you define specific parameters and configurations for the models being used for anomaly detection. Starting from [v1.10.0](/anomaly-detection/changelog/#v1100), multiple model configurations are supported in a single config.
 
 3. [`reader`](/anomaly-detection/components/reader.html) - This section outlines the methodology for data reading, including the data source location.
 
@@ -127,7 +129,7 @@ The configuration file for `vmanomaly` comprises 4 essential sections:
 
 Detailed parameters in each section:
 
-* `scheduler`
+* `schedulers` ([PeriodicScheduler](/anomaly-detection/components/scheduler/#periodic-scheduler) is used here)
   * `infer_every` - Specifies the frequency at which the trained models perform inferences on new data, essentially determining how often new anomaly score data points are generated. Format examples: 30s, 4m, 2h, 1d (time units: 's' for seconds, 'm' for minutes, 'h' for hours, 'd' for days). This parameter essentially asks, at regular intervals (e.g., every 1 minute), whether the latest data points appear abnormal based on historical data.
   * `fit_every` - Sets the frequency for retraining the models. A higher frequency ensures more updated models but requires more CPU resources. If omitted, models are retrained in each `infer_every` cycle. Format is similar to `infer_every`.
   * `fit_window` - Defines the data interval for training the models. Longer intervals allow for capturing extensive historical behavior and better seasonal pattern detection but may slow down the model's response to permanent metric changes and increase resource consumption. A minimum of two full seasonal cycles is recommended. Example format: 3h for three hours of data.
@@ -147,7 +149,9 @@ Below is an illustrative example of a `vmanomaly_config.yml` configuration file.
 
 
 ``` yaml
-scheduler:
+schedulers:
+  periodic:
+    # class: "scheduler.periodic.PeriodicScheduler"
     infer_every: "1m"
     fit_every: "2m"
     fit_window: "3h"
@@ -160,6 +164,7 @@ models:
 
 reader:
   datasource_url: "http://victoriametrics:8428/"
+  sampling_period: "60s" 
   queries:
     node_cpu_rate: "sum(rate(node_cpu_seconds_total[5m])) by (mode, instance, job)"
 
@@ -203,7 +208,7 @@ groups:
       summary: Anomaly Score exceeded 1.0. `sum(rate(node_cpu_seconds_total))` is showing abnormal behavior. 
 ```
 
-In the query expression `expr`, it's crucial to establish a criterion based on the generated anomaly scores. Typically, an [anomaly score](https://docs.victoriametrics.com/anomaly-detection/faq/#what-is-anomaly-score) ranging from 0.0 to 1.0 indicates that the analyzed value falls within normal behavior. Scores exceeding 1.0 signal increasing confidence from our model that the observed value is anomalous.
+In the query expression `expr`, it's crucial to establish a criterion based on the generated anomaly scores. Typically, an [anomaly score](/anomaly-detection/faq/#what-is-anomaly-score) ranging from 0.0 to 1.0 indicates that the analyzed value falls within normal behavior. Scores exceeding 1.0 signal increasing confidence from our model that the observed value is anomalous.
 
 Selecting an appropriate threshold for the anomaly score depends on your specific requirements and the context of the data. One effective method for determining this threshold is through visual analysis. By plotting the `anomaly_score` metric in conjunction with the predicted 'expected' range, delineated by `yhat_lower` and `yhat_upper`, you can make a more informed decision. Later in this tutorial, we will demonstrate this process with a practical example.
 
@@ -395,7 +400,7 @@ services:
     restart: always
   vmanomaly:
     container_name: vmanomaly
-    image: victoriametrics/vmanomaly:v1.9.2
+    image: victoriametrics/vmanomaly:v1.11.0
     depends_on:
       - "victoriametrics"
     ports: