### Describe Your Changes
- release 1.17.2 updates
- added sections on logging and CLI args to docs
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
This commit is contained in:
Fred Navruzov2024-10-22 12:42:37 +02:00committed byGitHub
Please find the changelog for VictoriaMetrics Anomaly Detection below.
## v1.17.2
Released: 2024-10-22
- IMPROVEMENT: Added `vmanomaly_version_info` (service) and `vmanomaly_ui_version_info` (vmui) gauges to self-monitoring metrics.
- IMPROVEMENT: Added `instance` and `job` labels to [pushed](https://docs.victoriametrics.com/keyconcepts/#push-model) metrics so they have the same labels as vmanomaly metrics that are [pulled](https://docs.victoriametrics.com/keyconcepts/#pull-model)/scraped. Metric labels can be customized via the [`extra_labels` argument](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/?highlight=extra_labels#push-config-parameters). By default job label will be `vmanomaly` and the instance label will be `f'{hostname}:{vmanomaly_port}`. See [monitoring.push](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#push-config-parameters) for examples and details.
- IMPROVEMENT: Added a subsection to [monitoring](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#logs-generated-by-vmanomaly) page with detailed per-component service logs, including reader and writer logs, error handling, metrics updates, and multi-tenancy warnings.
- IMPROVEMENT: Added a new [Command-line arguments](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments) subsection to the [Quickstart guide](https://docs.victoriametrics.com/anomaly-detection/quickstart/), providing details on available options for configuring `vmanomaly`.
## v1.17.1
Released: 2024-10-18
- FIX: Fixed an issue occurred when [Prophet model](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) is trained on *constant* data (data consisting of the same value and no variation across time). The bug prevented the `fit` stage from completing successfully, resulting in the model instance not being stored in the model registry, after automated model cleanup was addded in [v1.17.0](#1170).
- FIX: [Prophet models](https://docs.victoriametrics.com/anomaly-detection/components/models/#prophet) no longer fail to train on *constant* data, data consisting of the same value and no variation across time. The bug prevented the `fit` stage from completing successfully, resulting in the model instance not being stored in the model registry, after automated model cleanup was added in [v1.17.0](#1170).
@ -230,7 +230,7 @@ P.s. `infer` data volume will remain the same for both models, so it does not af
If you're dealing with a large query in the `queries` argument of [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) (especially when running [within a scheduler using a long](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/?highlight=fit_window#periodic-scheduler) `fit_window`), you may encounter issues such as query timeouts (due to the `search.maxQueryDuration` server limit) or rejections (if the `search.maxPointsPerTimeseries` server limit is exceeded).
We recommend upgrading to [v1.17.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1171), which introduced the `max_points_per_query` argument (both global and [query-specific](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters)) for the [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This argument overrides how `search.maxPointsPerTimeseries` flag handling (introduced in [v1.14.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1141)) is used in `vmanomaly` for splitting long `fit_window` queries into smaller sub-intervals. This helps users avoid hitting the `search.maxQueryDuration` limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead.
We recommend upgrading to [v1.17.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1171), which introduced the `max_points_per_query` argument (both global and [query-specific](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters)) for the [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). This argument overrides how `search.maxPointsPerTimeseries` flag handling (introduced in [v1.14.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1141)) is used in `vmanomaly` for splitting long `fit_window` queries into smaller sub-intervals. This helps users avoid hitting the `search.maxQueryDuration` limit for individual queries by distributing initial query across multiple subquery requests with minimal overhead.
By splitting long `fit_window` queries into smaller sub-intervals, this helps avoid hitting the `search.maxQueryDuration` limit, distributing the load across multiple subquery requests with minimal overhead. To resolve the issue, reduce `max_points_per_query` to a value lower than `search.maxPointsPerTimeseries` until the problem is gone:
@ -25,6 +25,30 @@ The following options are available:
> **Note**: Starting from [v1.16.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1160), a similar optimization is available for data read from VictoriaMetrics TSDB. See instructions [here](https://docs.victoriametrics.com/anomaly-detection/faq/#on-disk-mode).
### Command-line arguments
The `vmanomaly` service supports several command-line arguments to configure its behavior, including options for licensing, logging levels, and more. These arguments can be passed when starting the service via Docker or any other setup. Below is the list of available options:
```shellhelp
VictoriaMetrics Anomaly Detection Service
positional arguments:
config YAML config file. Multiple files will override each other's top level values (aka shallow merge), so multiple configs can be combined.
options:
-h show this help message and exit
--license STRING License key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.
--licenseFile PATH Path to file with license key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.
--license.forceOffline
Whether to force offline verification for VictoriaMetrics Enterprise license key, which has been passed either via -license or via -licenseFile command-line flag.
The issued license key must support offline verification feature. Contact info@victoriametrics.com if you need offline license verification.
--loggerLevel {FATAL,WARNING,ERROR,DEBUG,INFO}
Minimum level to log. Possible values: DEBUG, INFO, WARNING, ERROR, FATAL.
```
You can specify these options when running `vmanomaly` to fine-tune logging levels or handle licensing configurations, as per your requirements.
### Docker
> To run `vmanomaly`, you need to have VictoriaMetrics Enterprise license. You can get a trial license key [**here**](https://victoriametrics.com/products/enterprise/trial/).
@ -34,13 +58,13 @@ Below are the steps to get `vmanomaly` up and running inside a Docker container:
1. Pull Docker image:
```sh
docker pull victoriametrics/vmanomaly:v1.17.1
docker pull victoriametrics/vmanomaly:v1.17.2
```
2. (Optional step) tag the `vmanomaly` Docker image:
```sh
docker image tag victoriametrics/vmanomaly:v1.17.1 vmanomaly
docker image tag victoriametrics/vmanomaly:v1.17.2 vmanomaly
```
3. Start the `vmanomaly` Docker container with a *license file*, use the command below.
docker run -it -v $YOUR_LICENSE_FILE_PATH:/license \
-v $YOUR_CONFIG_FILE_PATH:/config.yml \
vmanomaly /config.yml \
--licenseFile=/license
--licenseFile=/license \
--loggerLevel=INFO
```
In case you found `PermissionError: [Errno 13] Permission denied:` in `vmanomaly` logs, set user/user group to 1000 in the run command above / in a docker-compose file:
@ -64,7 +89,8 @@ docker run -it --user 1000:1000 \
-v $YOUR_LICENSE_FILE_PATH:/license \
-v $YOUR_CONFIG_FILE_PATH:/config.yml \
vmanomaly /config.yml \
--licenseFile=/license
--licenseFile=/license \
--loggerLevel=INFO
```
```yaml
@ -72,13 +98,14 @@ docker run -it --user 1000:1000 \
@ -226,6 +226,13 @@ For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_
## Metrics generated by vmanomaly
- [Startup metrics](#startup-metrics)
- [Reader metrics](#reader-behaviour-metrics)
- [Model metrics](#models-behaviour-metrics)
- [Writer metrics](#writer-behaviour-metrics)
### Startup metrics
<tableclass="params">
<thead>
<tr>
@ -243,10 +250,120 @@ For detailed guidance on configuring mTLS parameters such as `verify_tls`, `tls_
<td>Gauge</td>
<td>vmanomaly start time in UNIX time</td>
</tr>
<tr>
<td>
`vmanomaly_version_info`
</td>
<td>Gauge</td>
<td>vmanomaly version information, contained in `version` label. Added in [v1.17.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1172)</td>
</tr>
<tr>
<td>
`vmanomaly_ui_version_info`
</td>
<td>Gauge</td>
<td>vmanomaly UI version information, contained in `version` label. Added in [v1.17.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1172)</td>
</tr>
</tbody>
</table>
### Models Behaviour Metrics
[Back to metric sections](#metrics-generated-by-vmanomaly)
### Reader behaviour metrics
Label names [description](#labelnames)
> **Note**: additional labels (`scheduler_alias`, `preset`) were added to writer and reader metrics in [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170) to improve consistency across the components. Also, metrics `vmanomaly_reader_request_duration_seconds` and `vmanomaly_reader_response_parsing_seconds` changed their type to `Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170)).
<tableclass="params">
<thead>
<tr>
<th>Metric</th>
<th>Type</th>
<th>Description</th>
<th>Labelnames</th>
</tr>
</thead>
<tbody>
<tr>
<td>
`vmanomaly_reader_request_duration_seconds`
</td>
<td>`Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))</td>
<td>The total time (in seconds) taken by queries to VictoriaMetrics `url` for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`url`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_responses` (named `vmanomaly_reader_response_count` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))
</td>
<td>`Counter`</td>
<td>The count of responses received from VictoriaMetrics `url` for the `query_key` query, categorized by `code`, within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>The total number of bytes received in responses for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_response_parsing_seconds`
</td>
<td>`Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))</td>
<td>The total time (in seconds) taken for data parsing at each `step` (json, dataframe) for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`step`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_timeseries_received`
</td>
<td>`Counter`</td>
<td>The total number of timeseries received from VictoriaMetrics for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_datapoints_received`
</td>
<td>`Counter`</td>
<td>The total number of datapoints received from VictoriaMetrics for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
</tbody>
</table>
[Back to metric sections](#metrics-generated-by-vmanomaly)
### Models behaviour metrics
Label names [description](#labelnames)
> **Note**: There is a new label key `model_alias` introduced in multi-model support [v1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100). This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.
[Back to metric sections](#metrics-generated-by-vmanomaly)
### Writer behaviour metrics
Label names [description](#labelnames)
> **Note**: additional labels (`scheduler_alias`, `preset`) were added to writer and reader metrics in [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170) to improve consistency across the components. Also, metrics `vmanomaly_writer_request_duration_seconds` and `vmanomaly_writer_request_serialize_seconds` changed their type to `Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170)).
<td>`Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))</td>
<td>The total time (in seconds) taken by queries to VictoriaMetrics `url` for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`url`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_responses` (named `vmanomaly_reader_response_count` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))
</td>
<td>`Counter`</td>
<td>The count of responses received from VictoriaMetrics `url` for the `query_key` query, categorized by `code`, within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>The total number of bytes received in responses for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_response_parsing_seconds`
</td>
<td>`Histogram` (was `Summary` prior to [v1.17.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1170))</td>
<td>The total time (in seconds) taken for data parsing at each `step` (json, dataframe) for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`step`, `query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_timeseries_received`
</td>
<td>`Counter`</td>
<td>The total number of timeseries received from VictoriaMetrics for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
<tr>
<td>
`vmanomaly_reader_datapoints_received`
</td>
<td>`Counter`</td>
<td>The total number of datapoints received from VictoriaMetrics for the `query_key` query within the specified scheduler `scheduler_alias`, in the `vmanomaly` service running in `preset` mode.</td>
<td>
`query_key`, `scheduler_alias`, `preset`
</td>
</tr>
</tbody>
</table>
[Back to metric sections](#metrics-generated-by-vmanomaly)
* `code` - response status code or `connection_error`, `timeout`.
* `step` - json or dataframe reading step.
[Back to metric sections](#metrics-generated-by-vmanomaly)
## Logs generated by vmanomaly
The `vmanomaly` service logs operations, errors, and performance for its components (service, reader, writer), alongside [self-monitoring metrics](#metrics-generated-by-vmanomaly) updates. Below is a description of key logs for each component and the related metrics affected.
`{{X}}` indicates a placeholder in the log message templates described below, which will be replaced with the appropriate entity during logging.
> **Note**: Applicable to version [v1.17.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1171) or newer.
> **Note**: By default, `vmanomaly` uses the `INFO` logging level. You can change this by specifying the `--loggerLevel` argument. See command-line arguments [here](https://docs.victoriametrics.com/anomaly-detection/quickstart/#command-line-arguments).
- [Startup logs](#startup-logs)
- [Reader logs](#reader-logs)
- [Service logs](#service-logs)
- [Writer logs](#writer-logs)
### Startup logs
The `vmanomaly` service logs important information during the startup process. This includes checking for the license, validating configurations, and setting up schedulers, readers, and writers. Below are key logs that are generated during startup, which can help troubleshoot issues with the service's initial configuration or license validation.
---
**License check**. If no license key or file is provided, the service will fail to start and log an error message. If a license file is provided but cannot be read, the service logs a failure. Log messages:
```text
Please provide a license code using --license or --licenseFile arg, or as VM_LICENSE_FILE env. See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.
```
```text
failed to read file {{args.license_file}}: {{error_message}}
```
---
**Config validation**. If the service's configuration fails to load or does not meet validation requirements, an error message is logged and the service will exit. If the configuration is loaded successfully, a message confirming the successful load is logged. Log messages:
```text
Config validation failed, please fix these errors: {{error_details}}
```
```text
Config has been loaded successfully.
```
---
**Model and data directory setup**. The service checks the environment variables `VMANOMALY_MODEL_DUMPS_DIR` and `VMANOMALY_DATA_DUMPS_DIR` to determine where to store models and data. If these variables are not set, models and data will be stored in memory. Please find the [on-disk mode details here](https://docs.victoriametrics.com/anomaly-detection/faq/#on-disk-mode). Log messages:
```text
Using ENV MODEL_DUMP_DIR=`{{model_dump_dir}}` to store anomaly detection models.
```
```text
ENV MODEL_DUMP_DIR is not set. Models will be kept in RAM between consecutive `fit` calls.
```
```text
Using ENV DATA_DUMP_DIR=`{{data_dump_dir}}` to store anomaly detection data.
```
```text
ENV DATA_DUMP_DIR is not set. Models' training data will be stored in RAM.
```
---
**Scheduler and service initialization**. After configuration is successfully loaded, the service initializes [schedulers](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) and services for each defined `scheduler_alias`. If there are issues with a specific scheduler (e.g., no models or queries found to attach to a scheduler), a warning is logged. When schedulers are initialized, the service logs a list of active schedulers. Log messages:
```text
Scheduler {{scheduler_alias}} wrapped and initialized with {{N}} model spec(s).
```
```text
No model spec(s) found for scheduler `{{scheduler_alias}}`, skipping setting it up.
```
```text
Active schedulers: {{list_of_schedulers}}.
```
[Back to logging sections](#logs-generated-by-vmanomaly)
---
### Reader logs
The `reader` component logs events during the process of querying VictoriaMetrics and retrieving the data necessary for anomaly detection. This includes making HTTP requests, handling SSL, parsing responses, and processing data into formats like DataFrames. The logs help to troubleshoot issues such as connection problems, timeout errors, or misconfigured queries.
---
**Starting a healthcheck request**. When the `reader` component initializes, it checks whether the VictoriaMetrics endpoint is accessible by sending a request for `_vmanomaly_healthcheck`. Log messages:
```text
[Scheduler {{scheduler_alias}}] Max points per timeseries set as: {{vm_max_datapoints_per_ts}}
**No data found (False)**. Based on [`query_from_last_seen_timestamp`](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=query_from_last_seen_timestamp#config-parameters) VmReader flag. A `warning` log is generated when no data is found in the requested range. This could indicate that the query was misconfigured or that no new data exists for the time period requested. Log message format:
```text
[Scheduler {{scheduler_alias}}] No data between {{start_s}} and {{end_s}} for query "{{query_key}}"
```
---
**No unseen data found (True)**. Based on [`query_from_last_seen_timestamp`](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=query_from_last_seen_timestamp#config-parameters) VmReader flag. A `warning` log is generated when no new data is returned (i.e., all data has already been seen in a previous inference step(s)). This helps in identifying situations where data for inference has already been processed. Based on VmReader's `adjust` flag. Log messages:
```text
[Scheduler {{scheduler_alias}}] No unseen data between {{start_s}} and {{end_s}} for query "{{query_key}}"
```
---
**Connection or timeout errors**. When the reader fails to retrieve data due to connection or timeout errors, a `warning` log is generated. These errors could result from network issues, incorrect query endpoints, or VictoriaMetrics being temporarily unavailable. Log message format:
```text
[Scheduler {{scheduler_alias}}] Error querying {{query_key}} for {{url}}: {{error_message}}
```
---
**Max datapoints warning**. If the requested query range (defined by `fit_every` or `infer_every` [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1) args) exceeds the maximum number of datapoints allowed by VictoriaMetrics, a `warning` log is generated, and the request is split into multiple intervals (option available since [v1.14.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1141)). This ensures that the request does not violate VictoriaMetrics’ constraints. Log messages:
```text
[Scheduler {{scheduler_alias}}] Query "{{query_key}}" from {{start_s}} to {{end_s}} with step {{step}} may exceed max datapoints per timeseries and will be split...
```
---
**Multi-tenancy warnings**. If the reader detects any issues related to missing or misconfigured multi-tenancy labels (supported since [v1.16.2](https://docs.victoriametrics.com/anomaly-detection/changelog/index.html#v1162)), a `warning` log is generated to indicate the issue. See additional details [here](https://docs.victoriametrics.com/anomaly-detection/components/writer/#multitenancy-support). Log message format:
```text
The label vm_account_id was not found in the label set of {{query_key}}, but tenant_id='multitenant' is set in reader configuration...
```
---
**Metrics updated in read operations**. During successful query execution process, the following reader [self-monitoring metrics](#reader-behaviour-metrics) are updated:
- `vmanomaly_reader_request_duration_seconds`: Records the time (in seconds) taken to complete the query request.
- `vmanomaly_reader_responses`: Tracks the number of response codes received from VictoriaMetrics.
- `vmanomaly_reader_received_bytes`: Counts the number of bytes received in the response.
- `vmanomaly_reader_response_parsing_seconds`: Records the time spent parsing the response into different formats (e.g., JSON or DataFrame).
- `vmanomaly_reader_timeseries_received`: Tracks how many timeseries were retrieved in the query result.
- `vmanomaly_reader_datapoints_received`: Counts the number of datapoints retrieved in the query result.
---
**Metrics skipped in case of failures**. If an error occurs (connection or timeout), `vmanomaly_reader_received_bytes`, `vmanomaly_reader_timeseries_received`, and `vmanomaly_reader_datapoints_received` are not incremented because no valid data was received.
[Back to logging sections](#logs-generated-by-vmanomaly)
### Service logs
The `model` component (wrapped in service) logs operations during the fitting and inference stages for each model spec attached to particular [scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/) `scheduler_alias`. These logs inform about skipped runs, connection or timeout issues, invalid data points, and successful or failed model operations.
---
**Skipped runs**. When there are insufficient valid data points to fit or infer using a model, the run is skipped and a `warning` log is generated. This can occur when the query returns no new data or when the data contains invalid values (e.g., `NaN`, `INF`). The skipped run is also reflected in the `vmanomaly_model_runs_skipped` metric. Log messages:
When there are insufficient valid data points (at least 1 for [online models](https://docs.victoriametrics.com/anomaly-detection/components/models#online-models) and 2 for [offline models](https://docs.victoriametrics.com/anomaly-detection/components/models#offline-models))
```text
[Scheduler {{scheduler_alias}}] Skipping run for stage 'fit' for model '{{model_alias}}' (query_key: {{query_key}}): Not enough valid data to fit: {{valid_values_cnt}}
```
When all the received timestamps during an `infer` call have already been processed, meaning the [`anomaly_score`](https://docs.victoriametrics.com/anomaly-detection/faq/index.html#what-is-anomaly-score) has already been produced for those points
```text
[Scheduler {{scheduler_alias}}] Skipping run for stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): No unseen data to infer on.
```
When the model fails to produce any valid or finite outputs (such as [`anomaly_score`](https://docs.victoriametrics.com/anomaly-detection/faq/index.html#what-is-anomaly-score))
```text
[Scheduler {{scheduler_alias}}] Skipping run for stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): No (valid) datapoints produced.
```
---
**Errors during model execution**. If the model fails to fit or infer data due to internal service errors or model spec misconfigurations, an `error` log is generated and the error is also reflected in the `vmanomaly_model_run_errors` metric. This can occur during both `fit` and `infer` stages. Log messages:
```text
[Scheduler {{scheduler_alias}}] Error during stage 'fit' for model '{{model_alias}}' (query_key: {{query_key}}): {{error_message}}
```
```text
[Scheduler {{scheduler_alias}}] Error during stage 'infer' for model '{{model_alias}}' (query_key: {{query_key}}): {{error_message}}
```
---
**Model instance created during inference**. In cases where an [online model](https://docs.victoriametrics.com/anomaly-detection/components/models#online-models) instance is created during the inference stage (without a prior fit, a feature introduced in [v1.15.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1152)), a `debug` log is produced. This helps track models that are created dynamically based on incoming data. Log messages:
```text
[Scheduler {{scheduler_alias}}] Model instance '{{model_alias}}' created for '{{query_key}}' during inference.
```
---
**Successful model runs**. When a model successfully fits, logs track the number of valid datapoints processed and the time taken for the operation. These logs are accompanied by updates to [self-monitoring metrics](#models-behaviour-metrics) like `vmanomaly_model_runs`, `vmanomaly_model_run_duration_seconds`, `vmanomaly_model_datapoints_accepted`, and `vmanomaly_model_datapoints_produced`. Log messages:
For [non-rolling models](https://docs.victoriametrics.com/anomaly-detection/components/models/#non-rolling-models)
```text
[Scheduler {{scheduler_alias}}] Fitting on {{valid_values_cnt}}/{{total_values_cnt}} valid datapoints for "{{query_key}}" using model "{{model_alias}}".
```
```text
[Scheduler {{scheduler_alias}}] Model '{{model_alias}}' fit completed in {{model_run_duration}} seconds for {{query_key}}.
```
For [rolling models](https://docs.victoriametrics.com/anomaly-detection/components/models/#rolling-models) (combined stage)
```text
[Scheduler {{scheduler_alias}}] Fit-Infer on {{datapoint_count}} points for "{{query_key}}" using model "{{model_alias}}".
```
---
**Metrics updated in model runs**. During successful fit or infer operations, the following [self-monitoring metrics](#models-behaviour-metrics) are updated for each run:
- `vmanomaly_model_runs`: Tracks how many times the model ran (`fit`, `infer`, or `fit_infer`) for a specific `query_key`.
- `vmanomaly_model_run_duration_seconds`: Records the total time (in seconds) for the model invocation, based on the results of the `query_key`.
- `vmanomaly_model_datapoints_accepted`: The number of valid datapoints processed by the model during the run.
- `vmanomaly_model_datapoints_produced`: The number of datapoints generated by the model during inference.
- `vmanomaly_models_active`: Tracks the number of models currently **available for infer** for a specific `query_key`.
---
**Metrics skipped in case of failures**. If a model run fails due to an error or if no valid data is available, the metrics such as `vmanomaly_model_datapoints_accepted`, `vmanomaly_model_datapoints_produced`, and `vmanomaly_model_run_duration_seconds` are not updated.
---
[Back to logging sections](#logs-generated-by-vmanomaly)
### Writer logs
The `writer` component logs events during the process of sending produced data (like `anomaly_score` [metrics](https://docs.victoriametrics.com/anomaly-detection/faq/index.html#what-is-anomaly-score)) to VictoriaMetrics. This includes data preparation, serialization, and network requests to VictoriaMetrics endpoints. The logs can help identify issues in data transmission, such as connection errors, invalid data points, and track the performance of write requests.
---
**Starting a write request**. A `debug` level log is produced when the `writer` component starts the process of writing data to VictoriaMetrics. It includes details like the number of datapoints, bytes of payload, and the query being written. This is useful for tracking the payload size and performance at the start of the request. Log messages:
```text
[Scheduler {{scheduler_alias}}] POST {{url}} with {{N}} datapoints, {{M}} bytes of payload, for {{query_key}}
```
---
**No valid data points**. A `warning` log is generated if there are no valid datapoints to write (i.e., all are `NaN` or unsupported like `INF`). This indicates that the writer will not send any data to VictoriaMetrics. Log messages:
```text
[Scheduler {{scheduler_alias}}] No valid datapoints to save for metric: {{query_key}}
```
---
**Connection, timeout, or I/O errors**. When the writer fails to send data due to connection, timeout, or I/O errors, an `error` log is generated. These errors often arise from network problems, incorrect URLs, or VictoriaMetrics being unavailable. The log includes details of the failed request and the reason for the failure. Log messages:
[Scheduler {{scheduler_alias}}] Cannot write {{N}} points for {{query_key}}: timeout for {{url}} {{error_message}}
```
```text
[Scheduler {{scheduler_alias}}] Cannot write {{N}} points for {{query_key}}: I/O error for {{url}} {{error_message}}
```
---
**Multi-tenancy warnings**. If the `tenant_id` is set to `multitenant` but the `vm_account_id` label is missing from the query result, or vice versa, a `warning` log is produced (supported since [v1.16.2](https://docs.victoriametrics.com/anomaly-detection/changelog/index.html#v1162)). This helps in debugging label set issues that may occur due to the multi-tenant configuration - see [this section for details](https://docs.victoriametrics.com/anomaly-detection/components/writer/#multitenancy-support). Log messages:
```text
The label vm_account_id was not found in the label set of {{query_key}}, but tenant_id='multitenant' is set in writer...
```
```text
The label set for the metric {{query_key}} contains multi-tenancy labels, but the write endpoint is configured for single-tenant mode (tenant_id != 'multitenant')...
```
---
**Metrics updated in write operations**. During the successful write process of *non-empty data*, the following [self-monitoring metrics](#writer-behaviour-metrics) are updated:
- `vmanomaly_writer_request_duration_seconds`: Records the time (in seconds) taken to complete the write request.
- `vmanomaly_writer_sent_bytes`: Tracks the number of bytes sent in the request.
- `vmanomaly_writer_responses`: Captures the HTTP response code returned by VictoriaMetrics. In case of connection, timeout, or I/O errors, a specific error code (`connection_error`, `timeout`, or `io_error`) is recorded instead.
- `vmanomaly_writer_request_serialize_seconds`: Records the time taken for data serialization.
- `vmanomaly_writer_datapoints_sent`: Counts the number of valid datapoints that were successfully sent.
- `vmanomaly_writer_timeseries_sent`: Tracks the number of timeseries sent to VictoriaMetrics.
**Metrics skipped in case of failures**. If an error occurs (connection, timeout, or I/O error), only `vmanomaly_writer_request_duration_seconds` is updated with appropriate error code.
[Back to logging sections](#logs-generated-by-vmanomaly)
Fred Navruzov