VictoriaMetrics/docs/anomaly-detection/components/monitoring.md

title	weight	menu	aliases
Monitoring	5	docs parent weight identifier vmanomaly-components 5 vmanomaly-monitoring	./monitoring.html

There are 2 models to monitor VictoriaMetrics Anomaly Detection behavior - push and pull. Parameters for each of them should be specified in the config file, monitoring section.

Note

: there was an enhancement of self-monitoring metrics for consistency across the components (v.1.17.0). Documentation was updated accordingly. Key changes included:

• Converting several self-monitoring metrics from Summary to Histogram to enable quantile calculation. This addresses the limitation of the prometheus_client's Summary implementation, which does not support quantiles. The change ensures metrics are more informative for performance analysis. Affected metrics are:
  • vmanomaly_reader_request_duration_seconds (VmReader)
  • vmanomaly_reader_response_parsing_seconds (VmReader)
  • vmanomaly_writer_request_duration_seconds (VmWriter)
  • vmanomaly_writer_request_serialize_seconds (VmWriter)
• Adding a query_key label to the vmanomaly_reader_response_parsing_seconds metric to provide finer granularity in tracking the performance of individual queries. This metric has also been switched from Summary to Histogram to align with the other metrics and support quantile calculations.
• Adding preset and scheduler_alias keys to VmReader and VmWriter metrics for consistency in multi-scheduler setups.
• Renaming Counters vmanomaly_reader_response_count to vmanomaly_reader_responses and vmanomaly_writer_response_count to vmanomaly_writer_responses.

Pull Model Config parameters

Parameter	Default	Description
addr	"0.0.0.0"	Server IP Address
port	8080	Port

Push Config parameters

Parameter	Default	Description
url		Link where to push metrics to. Example: "http://localhost:8480/"
tenant_id		Tenant ID for cluster version. Example: "0:0"
health_path	"health"	Deprecated since v1.8.0. Absolute, to override /health path
user		BasicAuth username
password		BasicAuth password
bearer_token	token	Token is passed in the standard format with header: Authorization: bearer {token}. Available since v1.15.9
bearer_token_file	path_to_file	Path to a file, which contains token, that is passed in the standard format with header: Authorization: bearer {token}. Available since v1.15.9
verify_tls	false	Verify TLS certificate. If False, it will not verify the TLS certificate. If True, it will verify the certificate using the system's CA store. If a path to a CA bundle file (like ca.crt), it will verify the certificate using the provided CA bundle.
tls_cert_file	path/to/cert.crt	Path to a file with the client certificate, i.e. client.crt. Available since v1.16.3.
tls_key_file	path/to/key.crt	Path to a file with the client certificate key, i.e. client.key. Available since v1.16.3.
timeout	"5s"	Stop waiting for a response after a given number of seconds.
extra_labels		Section for custom labels specified by user.

Monitoring section config example

monitoring:
  pull: # Enable /metrics endpoint.
    addr: "0.0.0.0"
    port: 8080
  push:
    url: "http://localhost:8480/"
    tenant_id: "0:0" # For cluster version only
    user: "USERNAME"
    password: "PASSWORD"
    verify_tls: False
    timeout: "5s"
    extra_labels:
      job: "vmanomaly-push"
      test: "test-1"

mTLS protection

Starting from v1.16.3, vmanomaly components such as VmWriter support mTLS to ensure secure communication with VictoriaMetrics Enterprise, configured with mTLS.

For detailed guidance on configuring mTLS parameters such as verify_tls, tls_cert_file, and tls_key_file, please refer to the mTLS protection section in the Reader documentation. The configuration principles apply consistently across all these vmanomaly components.

Metrics generated by vmanomaly

Metric	Type	Description
vmanomaly_start_time_seconds	Gauge	vmanomaly start time in UNIX time

Models Behaviour Metrics

Label names description

Note

: There is a new label key model_alias introduced in multi-model support v1.10.0. This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.

Note

: as a part of self-monitoring metrics enchancement (v.1.17.0), new metrics, like vmanomaly_model_run_errors, was added. Some of them changed the type (Summary -> Histogram), like vmanomaly_model_run_duration_seconds.

Metric	Type	Description	Labelnames
vmanomaly_model_runs	Counter	How many successful stage (fit, infer, fit_infer) runs occurred for models of class model_alias based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_run_duration_seconds	Histogram (was Summary prior to v1.17.0)	The total time (in seconds) taken by model invocations during the stage (fit, infer, fit_infer), based on the results of the query_key query, for models of class model_alias, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_datapoints_accepted	Counter	The number of datapoints accepted (excluding NaN or Inf values) by models of class model_alias from the results of the query_key query during the stage (infer, fit_infer), within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_datapoints_produced	Counter	The number of datapoints generated by models of class model_alias during the stage (infer, fit_infer) based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_models_active	Gauge	The number of model instances of class model_alias currently available for inference for the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, model_alias, scheduler_alias, preset
vmanomaly_model_runs_skipped	Counter	The number of times model runs (of class model_alias) were skipped in expected situations (e.g., no data for fitting/inference, or no new data to infer on) during the stage (fit, infer, fit_infer), based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset
vmanomaly_model_run_errors	Counter	The number of times model runs (of class model_alias) failed due to internal service errors during the stage (fit, infer, fit_infer), based on results from the query_key query, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	stage, query_key, model_alias, scheduler_alias, preset

Writer Behaviour Metrics

Label names description

Note

: additional labels (scheduler_alias, preset) were added to writer and reader metrics in v1.17.0 to improve consistency across the components. Also, metrics vmanomaly_writer_request_duration_seconds and vmanomaly_writer_request_serialize_seconds changed their type to Histogram (was Summary prior to v1.17.0).

Metric	Type	Description	Labelnames
vmanomaly_writer_request_duration_seconds	Histogram (was Summary prior to v1.17.0)	The total time (in seconds) taken by write requests to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_responses (named vmanomaly_reader_response_count prior to v1.17.0)	Counter	The count of response codes received from VictoriaMetrics url for the query_key query, categorized by code, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, code, query_key, scheduler_alias, preset
vmanomaly_writer_sent_bytes	Counter	The total number of bytes sent to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_writer_request_serialize_seconds	Histogram (was Summary prior to v1.17.0)	The total time (in seconds) taken for serializing data for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset
vmanomaly_writer_datapoints_sent	Counter	The total number of datapoints sent to VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset
vmanomaly_writer_timeseries_sent	Counter	The total number of timeseries sent to VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset

Reader Behaviour Metrics

Label names description

Metric	Type	Description	Labelnames
vmanomaly_reader_request_duration_seconds	Histogram (was Summary prior to v1.17.0)	The total time (in seconds) taken by queries to VictoriaMetrics url for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, scheduler_alias, preset
vmanomaly_reader_responses (named vmanomaly_reader_response_count prior to v1.17.0)	Counter	The count of responses received from VictoriaMetrics url for the query_key query, categorized by code, within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	url, query_key, code, scheduler_alias, preset
vmanomaly_reader_received_bytes	Counter	The total number of bytes received in responses for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset
vmanomaly_reader_response_parsing_seconds	Histogram (was Summary prior to v1.17.0)	The total time (in seconds) taken for data parsing at each step (json, dataframe) for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	step, query_key, scheduler_alias, preset
vmanomaly_reader_timeseries_received	Counter	The total number of timeseries received from VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset
vmanomaly_reader_datapoints_received	Counter	The total number of datapoints received from VictoriaMetrics for the query_key query within the specified scheduler scheduler_alias, in the vmanomaly service running in preset mode.	query_key, scheduler_alias, preset

Labelnames

• stage - stage of model - 'fit', 'infer' or 'fit_infer' for models that do it simultaneously, see model types.
• query_key - query alias from reader config section.
• model_alias - model alias from models config section. Introduced in v1.10.0.
• scheduler_alias - scheduler alias from schedulers config section. Introduced in v1.11.0.
• preset - preset alias for preset mode of vmanomaly. Introduced in v1.12.0.
• url - writer or reader url endpoint.
• code - response status code or connection_error, timeout.
• step - json or dataframe reading step.