mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
Merge branch 'master' into issue-5801
# Conflicts: # docs/CHANGELOG.md
This commit is contained in:
Alexander Marshalov
commit
8c17a880e1
30 changed files with 555 additions and 192 deletions
34
README.md
34
README.md
|
|
@ -18,24 +18,27 @@ VictoriaMetrics is available in [binary releases](https://github.com/VictoriaMet
|
|||
[Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/), [Snap packages](https://snapcraft.io/victoriametrics)
|
||||
and [source code](https://github.com/VictoriaMetrics/VictoriaMetrics).
|
||||
|
||||
The cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
Documentation for the cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
|
||||
Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.html) of VictoriaMetrics and follow the
|
||||
[quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
|
||||
|
||||
There is also a user-friendly database for logs - [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/).
|
||||
|
||||
If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://slack.victoriametrics.com/).
|
||||
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically, and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically,
|
||||
and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details.
|
||||
[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) -
|
||||
see [these docs](https://docs.victoriametrics.com/LTS-releases.md).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services.
|
||||
We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details.
|
||||
|
||||
## Prominent features
|
||||
|
||||
|
|
@ -102,6 +105,23 @@ VictoriaMetrics has the following prominent features:
|
|||
|
||||
See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/CaseStudies.html) and [various Articles about VictoriaMetrics](https://docs.victoriametrics.com/Articles.html).
|
||||
|
||||
## Components
|
||||
|
||||
VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](https://docs.victoriametrics.com/):
|
||||
|
||||
- [vmagent](https://docs.victoriametrics.com/vmagent/) - lightweight agent for receiving metrics via [pull-based](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format)
|
||||
and [push-based](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible
|
||||
remote storage systems such as VictoriaMetrics.
|
||||
- [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules.
|
||||
- [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) - a tool for validating alerting and recording rules.
|
||||
- [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products.
|
||||
- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - auhtorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting cababilities.
|
||||
- [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics.
|
||||
- [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) -
|
||||
tools for creating backups and restoring from backups for VictoriaMetrics data.
|
||||
- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
- [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) - user-friendly cost-efficient database for logs.
|
||||
|
||||
## Operation
|
||||
|
||||
### Install
|
||||
|
|
|
|||
|
|
@ -2,13 +2,17 @@
|
|||
|
||||
## Supported Versions
|
||||
|
||||
The following versions of VictoriaMetrics receive regular security fixes:
|
||||
|
||||
| Version | Supported |
|
||||
|---------|--------------------|
|
||||
| [latest release](https://docs.victoriametrics.com/CHANGELOG.html) | :white_check_mark: |
|
||||
| v1.97.x LTS release | :white_check_mark: |
|
||||
| v1.93.x LTS release | :white_check_mark: |
|
||||
| v1.97.x [LTS line](https://docs.victoriametrics.com/LTS-releases.html) | :white_check_mark: |
|
||||
| v1.93.x [LTS line](https://docs.victoriametrics.com/LTS-releases.html) | :white_check_mark: |
|
||||
| other releases | :x: |
|
||||
|
||||
See [this page](https://victoriametrics.com/security/) for more details.
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
Please report any security issues to security@victoriametrics.com
|
||||
|
|
|
|||
|
|
@ -208,6 +208,11 @@ const (
|
|||
influxMeasurementFieldSeparator = "influx-measurement-field-separator"
|
||||
influxSkipDatabaseLabel = "influx-skip-database-label"
|
||||
influxPrometheusMode = "influx-prometheus-mode"
|
||||
influxCertFile = "influx-cert-file"
|
||||
influxKeyFile = "influx-key-file"
|
||||
influxCAFile = "influx-CA-file"
|
||||
influxServerName = "influx-server-name"
|
||||
influxInsecureSkipVerify = "influx-insecure-skip-verify"
|
||||
)
|
||||
|
||||
var (
|
||||
|
|
@ -272,7 +277,28 @@ var (
|
|||
},
|
||||
&cli.BoolFlag{
|
||||
Name: influxPrometheusMode,
|
||||
Usage: "Wether to restore the original timeseries name previously written from Prometheus to InfluxDB v1 via remote_write.",
|
||||
Usage: "Whether to restore the original timeseries name previously written from Prometheus to InfluxDB v1 via remote_write.",
|
||||
Value: false,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: influxCertFile,
|
||||
Usage: "Optional path to client-side TLS certificate file to use when connecting to -influx-addr",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: influxKeyFile,
|
||||
Usage: "Optional path to client-side TLS key to use when connecting to -influx-addr",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: influxCAFile,
|
||||
Usage: "Optional path to TLS CA file to use for verifying connections to -influx-addr. By default, system CA is used",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: influxServerName,
|
||||
Usage: "Optional TLS server name to use for connections to -influx-addr. By default, the server name from -influx-addr is used",
|
||||
},
|
||||
&cli.BoolFlag{
|
||||
Name: influxInsecureSkipVerify,
|
||||
Usage: "Whether to skip tls verification when connecting to -influx-addr",
|
||||
Value: false,
|
||||
},
|
||||
}
|
||||
|
|
@ -496,6 +522,10 @@ const (
|
|||
remoteReadPassword = "remote-read-password"
|
||||
remoteReadHTTPTimeout = "remote-read-http-timeout"
|
||||
remoteReadHeaders = "remote-read-headers"
|
||||
remoteReadCertFile = "remote-read-cert-file"
|
||||
remoteReadKeyFile = "remote-read-key-file"
|
||||
remoteReadCAFile = "remote-read-CA-file"
|
||||
remoteReadServerName = "remote-read-server-name"
|
||||
remoteReadInsecureSkipVerify = "remote-read-insecure-skip-verify"
|
||||
remoteReadDisablePathAppend = "remote-read-disable-path-append"
|
||||
)
|
||||
|
|
@ -574,6 +604,22 @@ var (
|
|||
"For example, --remote-read-headers='My-Auth:foobar' would send 'My-Auth: foobar' HTTP header with every request to the corresponding remote source storage. \n" +
|
||||
"Multiple headers must be delimited by '^^': --remote-read-headers='header1:value1^^header2:value2'",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: remoteReadCertFile,
|
||||
Usage: "Optional path to client-side TLS certificate file to use when connecting to -remote-read-src-addr",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: remoteReadKeyFile,
|
||||
Usage: "Optional path to client-side TLS key to use when connecting to -remote-read-src-addr",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: remoteReadCAFile,
|
||||
Usage: "Optional path to TLS CA file to use for verifying connections to -remote-read-src-addr. By default, system CA is used",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: remoteReadServerName,
|
||||
Usage: "Optional TLS server name to use for connections to remoteReadSrcAddr. By default, the server name from -remote-read-src-addr is used",
|
||||
},
|
||||
&cli.BoolFlag{
|
||||
Name: remoteReadInsecureSkipVerify,
|
||||
Usage: "Whether to skip TLS certificate verification when connecting to the remote read address",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
package influx
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
|
|
@ -33,7 +34,8 @@ type Config struct {
|
|||
Retention string
|
||||
ChunkSize int
|
||||
|
||||
Filter Filter
|
||||
Filter Filter
|
||||
TLSConfig *tls.Config
|
||||
}
|
||||
|
||||
// Filter contains configuration for filtering
|
||||
|
|
@ -86,10 +88,10 @@ type LabelPair struct {
|
|||
// configured with passed Config
|
||||
func NewClient(cfg Config) (*Client, error) {
|
||||
c := influx.HTTPConfig{
|
||||
Addr: cfg.Addr,
|
||||
Username: cfg.Username,
|
||||
Password: cfg.Password,
|
||||
InsecureSkipVerify: true,
|
||||
Addr: cfg.Addr,
|
||||
Username: cfg.Username,
|
||||
Password: cfg.Password,
|
||||
TLSConfig: cfg.TLSConfig,
|
||||
}
|
||||
hc, err := influx.NewHTTPClient(c)
|
||||
if err != nil {
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ import (
|
|||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/prometheus"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/common"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/native/stream"
|
||||
)
|
||||
|
|
@ -84,6 +85,18 @@ func main() {
|
|||
Action: func(c *cli.Context) error {
|
||||
fmt.Println("InfluxDB import mode")
|
||||
|
||||
// create TLS config
|
||||
certFile := c.String(influxCertFile)
|
||||
keyFile := c.String(influxKeyFile)
|
||||
caFile := c.String(influxCAFile)
|
||||
serverName := c.String(influxServerName)
|
||||
insecureSkipVerify := c.Bool(influxInsecureSkipVerify)
|
||||
|
||||
tc, err := httputils.TLSConfig(certFile, caFile, keyFile, serverName, insecureSkipVerify)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create TLS Config: %s", err)
|
||||
}
|
||||
|
||||
iCfg := influx.Config{
|
||||
Addr: c.String(influxAddr),
|
||||
Username: c.String(influxUser),
|
||||
|
|
@ -96,7 +109,9 @@ func main() {
|
|||
TimeEnd: c.String(influxFilterTimeEnd),
|
||||
},
|
||||
ChunkSize: c.Int(influxChunkSize),
|
||||
TLSConfig: tc,
|
||||
}
|
||||
|
||||
influxClient, err := influx.NewClient(iCfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create influx client: %s", err)
|
||||
|
|
@ -134,6 +149,10 @@ func main() {
|
|||
Headers: c.String(remoteReadHeaders),
|
||||
LabelName: c.String(remoteReadFilterLabel),
|
||||
LabelValue: c.String(remoteReadFilterLabelValue),
|
||||
CertFile: c.String(remoteReadCertFile),
|
||||
KeyFile: c.String(remoteReadKeyFile),
|
||||
CAFile: c.String(remoteReadCAFile),
|
||||
ServerName: c.String(remoteReadServerName),
|
||||
InsecureSkipVerify: c.Bool(remoteReadInsecureSkipVerify),
|
||||
DisablePathAppend: c.Bool(remoteReadDisablePathAppend),
|
||||
})
|
||||
|
|
|
|||
|
|
@ -11,9 +11,9 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/utils"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/httputils"
|
||||
"github.com/gogo/protobuf/proto"
|
||||
"github.com/golang/snappy"
|
||||
"github.com/prometheus/prometheus/prompb"
|
||||
|
|
@ -64,6 +64,13 @@ type Config struct {
|
|||
// LabelName, LabelValue stands for label=~value pair used for read requests.
|
||||
// Is optional.
|
||||
LabelName, LabelValue string
|
||||
|
||||
// Optional cert file, key file, CA file and server name for client side TLS configuration
|
||||
CertFile string
|
||||
KeyFile string
|
||||
CAFile string
|
||||
ServerName string
|
||||
|
||||
// TLSSkipVerify defines whether to skip TLS certificate verification when connecting to the remote read address.
|
||||
InsecureSkipVerify bool
|
||||
}
|
||||
|
|
@ -103,10 +110,15 @@ func NewClient(cfg Config) (*Client, error) {
|
|||
}
|
||||
}
|
||||
|
||||
tr, err := httputils.Transport(cfg.Addr, cfg.CertFile, cfg.KeyFile, cfg.CAFile, cfg.ServerName, cfg.InsecureSkipVerify)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create transport: %s", err)
|
||||
}
|
||||
|
||||
c := &Client{
|
||||
c: &http.Client{
|
||||
Timeout: cfg.Timeout,
|
||||
Transport: utils.Transport(cfg.Addr, cfg.InsecureSkipVerify),
|
||||
Transport: tr,
|
||||
},
|
||||
addr: strings.TrimSuffix(cfg.Addr, "/"),
|
||||
disablePathAppend: cfg.DisablePathAppend,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
package utils
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
|
@ -13,8 +13,7 @@ const (
|
|||
maxTimeMsecs = int64(1<<63-1) / 1e6
|
||||
)
|
||||
|
||||
// GetTime returns time from the given string.
|
||||
func GetTime(s string) (time.Time, error) {
|
||||
func parseTime(s string) (time.Time, error) {
|
||||
secs, err := promutils.ParseTime(s)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("cannot parse %s: %w", s, err)
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
package utils
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
|
@ -165,7 +165,7 @@ func TestGetTime(t *testing.T) {
|
|||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := GetTime(tt.s)
|
||||
got, err := parseTime(tt.s)
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("ParseTime() error = %v, wantErr %v", err, tt.wantErr)
|
||||
return
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
package utils
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Transport creates http.Transport object based on provided URL.
|
||||
// Returns Transport with TLS configuration if URL contains `https` prefix
|
||||
func Transport(URL string, insecureSkipVerify bool) *http.Transport {
|
||||
t := http.DefaultTransport.(*http.Transport).Clone()
|
||||
if !strings.HasPrefix(URL, "https") {
|
||||
return t
|
||||
}
|
||||
t.TLSClientConfig = TLSConfig(insecureSkipVerify)
|
||||
return t
|
||||
}
|
||||
|
||||
// TLSConfig creates tls.Config object from provided arguments
|
||||
func TLSConfig(insecureSkipVerify bool) *tls.Config {
|
||||
return &tls.Config{
|
||||
InsecureSkipVerify: insecureSkipVerify,
|
||||
}
|
||||
}
|
||||
|
|
@ -16,7 +16,6 @@ import (
|
|||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/limiter"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/native"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/stepper"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/utils"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl/vm"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/searchutils"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
|
|
@ -54,14 +53,14 @@ func (p *vmNativeProcessor) run(ctx context.Context) error {
|
|||
startTime: time.Now(),
|
||||
}
|
||||
|
||||
start, err := utils.GetTime(p.filter.TimeStart)
|
||||
start, err := parseTime(p.filter.TimeStart)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to parse %s, provided: %s, error: %w", vmNativeFilterTimeStart, p.filter.TimeStart, err)
|
||||
}
|
||||
|
||||
end := time.Now().In(start.Location())
|
||||
if p.filter.TimeEnd != "" {
|
||||
end, err = utils.GetTime(p.filter.TimeEnd)
|
||||
end, err = parseTime(p.filter.TimeEnd)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to parse %s, provided: %s, error: %w", vmNativeFilterTimeEnd, p.filter.TimeEnd, err)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6210,7 +6210,7 @@
|
|||
"type": "prometheus",
|
||||
"uid": "$ds"
|
||||
},
|
||||
"description": "Shows network usage by vmstorage services.\n* Writes show traffic sent to vmselects.\n* Reads show traffic received from vminserts.",
|
||||
"description": "Shows network usage by vmstorage services.\n* Writes show traffic sent to clients.\n* Reads show traffic received from clients.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
|
|
@ -6310,10 +6310,10 @@
|
|||
"uid": "$ds"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) by(name) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from {{name}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -6323,11 +6323,11 @@
|
|||
"uid": "$ds"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(vm_tcplistener_written_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"expr": "sum(rate(vm_tcplistener_written_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) by(name) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to {{name}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -7420,7 +7420,7 @@
|
|||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_select\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from client",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -7434,7 +7434,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to client",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -7550,7 +7550,7 @@
|
|||
"expr": "sum(rate(vm_tcpdialer_read_bytes_total{job=~\"$job_select\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from vmstorage",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -7564,7 +7564,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to vmstorage",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -8442,7 +8442,7 @@
|
|||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_insert\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from client",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -8456,7 +8456,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to client",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -8572,7 +8572,7 @@
|
|||
"expr": "sum(rate(vm_tcpdialer_read_bytes_total{job=~\"$job_insert\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from vmstorage",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -8586,7 +8586,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to vmstorage",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6211,7 +6211,7 @@
|
|||
"type": "victoriametrics-datasource",
|
||||
"uid": "$ds"
|
||||
},
|
||||
"description": "Shows network usage by vmstorage services.\n* Writes show traffic sent to vmselects.\n* Reads show traffic received from vminserts.",
|
||||
"description": "Shows network usage by vmstorage services.\n* Writes show traffic sent to clients.\n* Reads show traffic received from clients.",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
|
|
@ -6311,10 +6311,10 @@
|
|||
"uid": "$ds"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) by(name) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from {{name}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -6324,11 +6324,11 @@
|
|||
"uid": "$ds"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(vm_tcplistener_written_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"expr": "sum(rate(vm_tcplistener_written_bytes_total{job=~\"$job_storage\", instance=~\"$instance\"}[$__rate_interval])) by(name) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to {{name}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -7421,7 +7421,7 @@
|
|||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_select\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from client",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -7435,7 +7435,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to client",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -7551,7 +7551,7 @@
|
|||
"expr": "sum(rate(vm_tcpdialer_read_bytes_total{job=~\"$job_select\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from vmstorage",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -7565,7 +7565,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to vmstorage",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -8443,7 +8443,7 @@
|
|||
"expr": "sum(rate(vm_tcplistener_read_bytes_total{job=~\"$job_insert\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from client",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -8457,7 +8457,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to client",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
@ -8573,7 +8573,7 @@
|
|||
"expr": "sum(rate(vm_tcpdialer_read_bytes_total{job=~\"$job_insert\", instance=~\"$instance\"}[$__rate_interval])) * 8 > 0",
|
||||
"format": "time_series",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "read",
|
||||
"legendFormat": "read from vmstorage",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
|
|
@ -8587,7 +8587,7 @@
|
|||
"format": "time_series",
|
||||
"hide": false,
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "write ",
|
||||
"legendFormat": "write to vmstorage",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ services:
|
|||
restart: always
|
||||
vmanomaly:
|
||||
container_name: vmanomaly
|
||||
image: victoriametrics/vmanomaly:v1.9.2
|
||||
image: victoriametrics/vmanomaly:v1.10.0
|
||||
depends_on:
|
||||
- "victoriametrics"
|
||||
ports:
|
||||
|
|
|
|||
|
|
@ -3,10 +3,11 @@ scheduler:
|
|||
fit_every: "2m"
|
||||
fit_window: "3h"
|
||||
|
||||
model:
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
models:
|
||||
prophet:
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
|
||||
reader:
|
||||
datasource_url: "http://victoriametrics:8428/"
|
||||
|
|
|
|||
|
|
@ -23,12 +23,19 @@ The following `tip` changes can be tested by building VictoriaMetrics components
|
|||
|
||||
Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our
|
||||
[sandbox](https://play-grafana.victoriametrics.com/d/oS7Bi_0Wz_vm/victoriametrics-cluster-vm).
|
||||
The sandbox cluster installation is running under the constant load generated by
|
||||
[prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) and used for testing latest releases.
|
||||
The sandbox cluster installation runs under the constant load generated by
|
||||
[prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) and is used for testing latest releases.
|
||||
|
||||
See also [LTS releases](https://docs.victoriametrics.com/LTS-releases.html).
|
||||
|
||||
## tip
|
||||
|
||||
* FEATURE: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmstorage` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): expose `vm_last_partition_parts` [metrics](https://docs.victoriametrics.com/#monitoring), which show the number of [parts in the latest partition](https://docs.victoriametrics.com/#storage). These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition.
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): support client-side TLS configuration for [InfluxDB](https://docs.victoriametrics.com/vmctl/#migrating-data-from-influxdb-1x). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5783).
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): support client-side TLS configuration for [Remote Read protocol](https://docs.victoriametrics.com/vmctl/#migrating-data-by-remote-read-protocol). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5748). Thanks to @khushijain21 for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5798).
|
||||
|
||||
* BUGFIX: fix the misleading error `0ms is out of allowed range [0 ...` when passing `step=0` to [/api/v1/query](https://docs.victoriametrics.com/keyconcepts/#instant-query)
|
||||
or [/api/v1/query_range](https://docs.victoriametrics.com/keyconcepts/#range-query). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5795).
|
||||
* BUGFIX: [Single-node VictoriaMetrics](https://docs.victoriametrics.com/) and `vmselect` in [VictoriaMetrics cluster](https://docs.victoriametrics.com/cluster-victoriametrics/): fixed floating-point error when parsing time in RFC3339 format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5801) for details.
|
||||
|
||||
## [v1.98.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.98.0)
|
||||
|
|
|
|||
27
docs/LTS-releases.md
Normal file
27
docs/LTS-releases.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
sort: 300
|
||||
weight: 300
|
||||
title: Long-term support releases
|
||||
menu:
|
||||
docs:
|
||||
parent: 'victoriametrics'
|
||||
weight: 300
|
||||
aliases:
|
||||
- /LTS-releases.html
|
||||
---
|
||||
|
||||
# Long-term support releases
|
||||
|
||||
[Enterprise version of VictoriaMetrics](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (aka LTS releases).
|
||||
Every LTS line receives bugfixes and [security fixes](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/SECURITY.md) for 12 months after
|
||||
the initial release. New LTS lines are published every 6 months, so the latest two LTS lines are supported at any given moment. This gives up to 6 months
|
||||
for the migration to new LTS lines for [VictoriaMetrics Enterprise](https://docs.victoriametrics.com/enterprise/) users.
|
||||
|
||||
All the bugfixes and security fixes, which are included in LTS releases, are also available in [the latest release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest),
|
||||
so non-enterprise users are advised to regularly [upgrade](https://docs.victoriametrics.com/#how-to-upgrade-victoriametrics) VictoriaMetrics products
|
||||
to [the latest available releases](https://docs.victoriametrics.com/CHANGELOG.html).
|
||||
|
||||
## Currently supported LTS release lines
|
||||
|
||||
- 1.97.x - the latest one is [v1.97.2 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.2)
|
||||
- 1.93.x - the latest one is [v1.93.12 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.12)
|
||||
|
|
@ -21,24 +21,27 @@ VictoriaMetrics is available in [binary releases](https://github.com/VictoriaMet
|
|||
[Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/), [Snap packages](https://snapcraft.io/victoriametrics)
|
||||
and [source code](https://github.com/VictoriaMetrics/VictoriaMetrics).
|
||||
|
||||
The cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
Documentation for the cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
|
||||
Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.html) of VictoriaMetrics and follow the
|
||||
[quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
|
||||
|
||||
There is also a user-friendly database for logs - [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/).
|
||||
|
||||
If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://slack.victoriametrics.com/).
|
||||
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically, and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically,
|
||||
and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details.
|
||||
[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) -
|
||||
see [these docs](https://docs.victoriametrics.com/LTS-releases.md).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services.
|
||||
We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details.
|
||||
|
||||
## Prominent features
|
||||
|
||||
|
|
@ -105,6 +108,23 @@ VictoriaMetrics has the following prominent features:
|
|||
|
||||
See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/CaseStudies.html) and [various Articles about VictoriaMetrics](https://docs.victoriametrics.com/Articles.html).
|
||||
|
||||
## Components
|
||||
|
||||
VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](https://docs.victoriametrics.com/):
|
||||
|
||||
- [vmagent](https://docs.victoriametrics.com/vmagent/) - lightweight agent for receiving metrics via [pull-based](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format)
|
||||
and [push-based](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible
|
||||
remote storage systems such as VictoriaMetrics.
|
||||
- [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules.
|
||||
- [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) - a tool for validating alerting and recording rules.
|
||||
- [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products.
|
||||
- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - auhtorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting cababilities.
|
||||
- [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics.
|
||||
- [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) -
|
||||
tools for creating backups and restoring from backups for VictoriaMetrics data.
|
||||
- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
- [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) - user-friendly cost-efficient database for logs.
|
||||
|
||||
## Operation
|
||||
|
||||
### Install
|
||||
|
|
|
|||
|
|
@ -29,24 +29,27 @@ VictoriaMetrics is available in [binary releases](https://github.com/VictoriaMet
|
|||
[Docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/), [Snap packages](https://snapcraft.io/victoriametrics)
|
||||
and [source code](https://github.com/VictoriaMetrics/VictoriaMetrics).
|
||||
|
||||
The cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
Documentation for the cluster version of VictoriaMetrics is available [here](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
|
||||
Learn more about [key concepts](https://docs.victoriametrics.com/keyConcepts.html) of VictoriaMetrics and follow the
|
||||
[quick start guide](https://docs.victoriametrics.com/Quick-Start.html) for a better experience.
|
||||
|
||||
There is also a user-friendly database for logs - [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/).
|
||||
|
||||
If you have questions about VictoriaMetrics, then feel free asking them in the [VictoriaMetrics community Slack chat](https://slack.victoriametrics.com/).
|
||||
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
[Contact us](mailto:info@victoriametrics.com) if you need enterprise support for VictoriaMetrics.
|
||||
See [features available in enterprise package](https://docs.victoriametrics.com/enterprise.html).
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
Enterprise binaries can be downloaded and evaluated for free
|
||||
from [the releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest).
|
||||
You can also [request a free trial license](https://victoriametrics.com/products/enterprise/trial/).
|
||||
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically, and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
VictoriaMetrics is developed at a fast pace, so it is recommended to check the [CHANGELOG](https://docs.victoriametrics.com/CHANGELOG.html) periodically,
|
||||
and to perform [regular upgrades](#how-to-upgrade-victoriametrics).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See our [Security page](https://victoriametrics.com/security/) for more details.
|
||||
[VictoriaMetrics enterprise](https://docs.victoriametrics.com/enterprise/) provides long-term support lines of releases (LTS releases) -
|
||||
see [these docs](https://docs.victoriametrics.com/LTS-releases.md).
|
||||
|
||||
VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services.
|
||||
We apply strict security measures in everything we do. See [Security page](https://victoriametrics.com/security/) for more details.
|
||||
|
||||
## Prominent features
|
||||
|
||||
|
|
@ -113,6 +116,23 @@ VictoriaMetrics has the following prominent features:
|
|||
|
||||
See [case studies for VictoriaMetrics](https://docs.victoriametrics.com/CaseStudies.html) and [various Articles about VictoriaMetrics](https://docs.victoriametrics.com/Articles.html).
|
||||
|
||||
## Components
|
||||
|
||||
VictoriaMetrics ecosystem contains the following components additionally to [single-node VictoriaMetrics](https://docs.victoriametrics.com/):
|
||||
|
||||
- [vmagent](https://docs.victoriametrics.com/vmagent/) - lightweight agent for receiving metrics via [pull-based](https://docs.victoriametrics.com/vmagent/#how-to-collect-metrics-in-prometheus-format)
|
||||
and [push-based](https://docs.victoriametrics.com/vmagent/#how-to-push-data-to-vmagent) protocols, transforming and sending them to the configured Prometheus-compatible
|
||||
remote storage systems such as VictoriaMetrics.
|
||||
- [vmalert](https://docs.victoriametrics.com/vmalert/) - a service for processing Prometheus-compatible alerting and recording rules.
|
||||
- [vmalert-tool](https://docs.victoriametrics.com/vmalert-tool/) - a tool for validating alerting and recording rules.
|
||||
- [vmauth](https://docs.victoriametrics.com/vmauth/) - authorization proxy and load balancer optimized for VictoriaMetrics products.
|
||||
- [vmgateway](https://docs.victoriametrics.com/vmgateway/) - auhtorization proxy with per-[tenant](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy) rate limiting cababilities.
|
||||
- [vmctl](https://docs.victoriametrics.com/vmctl/) - a tool for migrating and copying data between different storage systems for metrics.
|
||||
- [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/) and [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/) -
|
||||
tools for creating backups and restoring from backups for VictoriaMetrics data.
|
||||
- `vminsert`, `vmselect` and `vmstorage` - components of [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html).
|
||||
- [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) - user-friendly cost-efficient database for logs.
|
||||
|
||||
## Operation
|
||||
|
||||
### Install
|
||||
|
|
|
|||
|
|
@ -17,6 +17,23 @@ Please find the changelog for VictoriaMetrics Anomaly Detection below.
|
|||
|
||||
> **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or later versions for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100).**
|
||||
|
||||
## v1.10.0
|
||||
Released: 2024-02-15
|
||||
- FEATURE: Multi-model support. Now users can specify multiple [model specs](https://docs.victoriametrics.com/anomaly-detection/components/models/) in a single config (via aliasing), as well as to reference what [queries from VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=queries#config-parameters) it should be run on.
|
||||
- Introduction of `queries` arg in model spec:
|
||||
- It allows the model to be executed only on a particular query subset from `reader` section.
|
||||
- Passing an empty list or not specifying this param implies that each model is run on results from **all** queries, which is a backward-compatible behavior.
|
||||
- Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/#queries)
|
||||
|
||||
- DEPRECATION: slight refactor of a model config section
|
||||
- Now models are passed as a mapping of `model_alias: model_spec` under [model](https://docs.victoriametrics.com/anomaly-detection/components/models/) sections. Using old format (<= [1.9.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v192)) will produce warnings for now and will be removed in future versions.
|
||||
- Please find more details in docs on [Model section](https://docs.victoriametrics.com/anomaly-detection/components/models/)
|
||||
- IMPROVEMENT: now logs from [`monitoring.pull`](https://docs.victoriametrics.com/anomaly-detection/components/monitoring/#monitoring-section-config-example) GET requests to `/metrics` endpoint are shown only in DEBUG mode
|
||||
- IMPROVEMENT: labelset for multivariate models is deduplicated and cleaned, resulting in better UX
|
||||
|
||||
> **Note**: These updates support more flexible setup and effective resource management in service, as now it's not longer needed to spawn several instances of `vmanomaly` to split queries/models context across.
|
||||
|
||||
|
||||
## v1.9.2
|
||||
Released: 2024-01-29
|
||||
- BUGFIX: now multivariate models (like [`IsolationForestMultivariateModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/#isolation-foresthttpsenwikipediaorgwikiisolation_forest-multivariate)) are properly handled throughout fit/infer phases.
|
||||
|
|
|
|||
|
|
@ -162,10 +162,11 @@ scheduler:
|
|||
fit_every: "2h"
|
||||
fit_window: "14d"
|
||||
|
||||
model:
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
models:
|
||||
prophet: # or use a model alias of your choice here
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
|
||||
reader:
|
||||
datasource_url: [YOUR_DATASOURCE_URL] #Example: "http://victoriametrics:8428/"
|
||||
|
|
@ -217,7 +218,7 @@ This will expose metrics at `http://0.0.0.0:8080/metrics` page.
|
|||
To use *vmanomaly* you need to pull docker image:
|
||||
|
||||
```sh
|
||||
docker pull victoriametrics/vmanomaly:v1.9.2
|
||||
docker pull victoriametrics/vmanomaly:v1.10.0
|
||||
```
|
||||
|
||||
> Note: please check what is latest release in [CHANGELOG](/anomaly-detection/CHANGELOG.html)
|
||||
|
|
@ -227,7 +228,7 @@ docker pull victoriametrics/vmanomaly:v1.9.2
|
|||
You can put a tag on it for your convinience:
|
||||
|
||||
```sh
|
||||
docker image tag victoriametrics/vmanomaly:v1.9.2 vmanomaly
|
||||
docker image tag victoriametrics/vmanomaly:v1.10.0 vmanomaly
|
||||
```
|
||||
Here is an example of how to run *vmanomaly* docker container with [license file](#licensing):
|
||||
|
||||
|
|
|
|||
|
|
@ -16,10 +16,149 @@ aliases:
|
|||
|
||||
# Models
|
||||
|
||||
This section describes `Model` component of VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](/anomaly-detection/overview.html)) and the guide of how to define a respective section of a config to launch the service.
|
||||
This section describes `Models` component of VictoriaMetrics Anomaly Detection (or simply [`vmanomaly`](/anomaly-detection/overview.html)) and the guide of how to define a respective section of a config to launch the service.
|
||||
vmanomaly includes various [built-in models](#built-in-models) and you can integrate your custom model with vmanomaly see [custom model](#custom-model-guide)
|
||||
|
||||
|
||||
> **Note: Starting from [v1.10.0](/anomaly-detection/changelog#v1100) model section in config supports multiple models via aliasing. <br>Also, `vmanomaly` expects model section to be named `models`. Using old (flat) format with `model` key is deprecated and will be removed in future versions. Having `model` and `models` sections simultaneously in a config will result in only `models` being used:**
|
||||
|
||||
```yaml
|
||||
models:
|
||||
model_univariate_1:
|
||||
class: "model.zscore.ZscoreModel"
|
||||
z_threshold: 2.5
|
||||
queries: ["query_alias2"] # referencing queries defined in `reader` section
|
||||
model_multivariate_1:
|
||||
class: "model.isolation_forest.IsolationForestMultivariateModel"
|
||||
contamination: "auto"
|
||||
args:
|
||||
n_estimators: 100
|
||||
# i.e. to assure reproducibility of produced results each time model is fit on the same input
|
||||
random_state: 42
|
||||
# if there is no explicit `queries` arg, then the model will be run on ALL queries found in reader section
|
||||
```
|
||||
|
||||
Old-style configs (< [1.10.0](/anomaly-detection/changelog#v1100) )
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.zscore.ZscoreModel"
|
||||
z_threshold: 2.5
|
||||
# no explicit `queries` arg is provided
|
||||
```
|
||||
|
||||
will be **implicitly** converted to
|
||||
|
||||
```yaml
|
||||
models:
|
||||
default_model: # default model alias, backward compatibility
|
||||
class: "model.zscore.ZscoreModel"
|
||||
z_threshold: 2.5
|
||||
# queries arg is created and propagated with all query aliases found in `queries` arg of `reader` section
|
||||
queries: ["q1", "q2", "q3"] # i.e., if your `queries` in `reader` section has exactly q1, q2, q3 aliases
|
||||
```
|
||||
|
||||
|
||||
## Common args
|
||||
|
||||
From [1.10.0](/anomaly-detection/changelog#1100), **common args**, supported by *every model (and model type)* were introduced.
|
||||
|
||||
### Queries
|
||||
|
||||
Introduced in [1.10.0](/anomaly-detection/changelog#1100), as a part to support multi-model configs, `queries` arg is meant to define [queries from VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=queries#config-parameters) particular model should be run on (meaning, all the series returned by each of these queries will be used in such model for fitting and inferencing).
|
||||
|
||||
`queries` arg is supported for all [the built-in](/anomaly-detection/components/models/#built-in-models) (as well as for [custom](/anomaly-detection/components/models/#custom-model-guide)) models.
|
||||
|
||||
This arg is **backward compatible** - if there is no explicit `queries` arg, then the model, defined in a config, will be run on ALL queries found in reader section:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
model_alias_1:
|
||||
...
|
||||
# no explicit `queries` arg is provided
|
||||
```
|
||||
|
||||
will be implicitly converted to
|
||||
|
||||
```yaml
|
||||
models:
|
||||
model_alias_1:
|
||||
...
|
||||
# queries arg is created and propagated with all query aliases found in `queries` arg of `reader` section
|
||||
queries: ["q1", "q2", "q3"] # i.e., if your `queries` in `reader` section has exactly q1, q2, q3 aliases
|
||||
```
|
||||
|
||||
## Model types
|
||||
|
||||
There are **2 model types**, supported in `vmanomaly`, resulting in **4 possible combinations**:
|
||||
|
||||
- [Univariate models](#univariate-models)
|
||||
- [Multivariate models](#multivariate-models)
|
||||
|
||||
Each of these models can be
|
||||
- [rolling](#rolling-models)
|
||||
- [non-rolling](#non-rolling-models)
|
||||
|
||||
### Univariate Models
|
||||
|
||||
For a univariate type, **one separate model** is fit/used for inference per **each time series**, defined in its [queries](#queries) arg.
|
||||
|
||||
For example, if you have some **univariate** model, defined to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be 3*5=15 models created in total. Each such model produce **individual [output](#vmanomaly-output)** for each of time series.
|
||||
|
||||
If during an inference, you got a series having **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step.
|
||||
|
||||
**Implications:** Univariate models are a go-to default, when your queries returns **changing** amount of **individual** time series of **different** magnitude, [trend](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#trend) or [seasonality](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-1/#seasonality), so you won't be mixing incompatible data with different behavior within a single fit model (context isolation).
|
||||
|
||||
**Examples:** [Prophet](#prophet), [Holt-Winters](#holt-winters)
|
||||
|
||||
<!-- TODO: add schema -->
|
||||
|
||||
### Multivariate Models
|
||||
|
||||
For a multivariate type, **one shared model** is fit/used for inference on **all time series** simultaneously, defined in its [queries](#queries) arg.
|
||||
|
||||
For example, if you have some **multivariate** model to use 3 [MetricQL queries](https://docs.victoriametrics.com/metricsql/), each returning 5 time series, there will be one shared model created in total. Once fit, this model will expect **exactly 15 time series with exact same labelsets as an input**. This model will produce **one shared [output](#vmanomaly-output)**.
|
||||
|
||||
If during an inference, you got a **different amount of series** or some series having a **new labelset** (not present in any of fitted models), the inference will be skipped until you get a model, trained particularly for such labelset during forthcoming re-fit step.
|
||||
|
||||
**Implications:** Multivariate models are a go-to default, when your queries returns **fixed** amount of **individual** time series (say, some aggregations), to be used for adding cross-series (and cross-query) context, useful for catching [collective anomalies](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/index.html#collective-anomalies) or [novelties](https://victoriametrics.com/blog/victoriametrics-anomaly-detection-handbook-chapter-2/index.html#novelties) (expanded to multi-input scenario). For example, you may set it up for anomaly detection of CPU usage in different modes (`idle`, `user`, `system`, etc.) and use its cross-dependencies to detect **unseen (in fit data)** behavior.
|
||||
|
||||
**Examples:** [IsolationForest](#isolation-forest-multivariate)
|
||||
|
||||
<!-- TODO: add schema -->
|
||||
|
||||
### Rolling Models
|
||||
|
||||
A rolling model is a model that, once trained, **cannot be (naturally) used to make inference on data, not seen during its fit phase**.
|
||||
|
||||
An instance of rolling model is **simultaneously fit and used for inference** during its `infer` method call.
|
||||
|
||||
As a result, such model instances are **not stored** between consecutive re-fit calls (defined by `fit_every` [arg](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler) in `PeriodicScheduler`), leading to **lower RAM** consumption.
|
||||
|
||||
Such models put **more pressure** on your reader's source, i.e. if your model should be fit on large amount of data (say, 14 days with 1-minute resolution) and at the same time you have **frequent inference** (say, once per minute) on new chunks of data - that's because such models require (fit + infer) window of data to be fit first to be used later in each inference call.
|
||||
|
||||
> **Note**: Rolling models require `fit_every` to be set equal to `infer_every` in your [PeriodicScheduler](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler).
|
||||
|
||||
**Examples:** [RollingQuantile](#rolling-quantile)
|
||||
|
||||
<!-- TODO: add schema -->
|
||||
|
||||
### Non-Rolling Models
|
||||
|
||||
Everything that is not classified as [rolling](#rolling-models).
|
||||
|
||||
Produced models can be explicitly used to **infer on data, not seen during its fit phase**, thus, it **doesn't require re-fit procedure**.
|
||||
|
||||
Such models put **less pressure** on your reader's source, i.e. if you fit on large amount of data (say, 14 days with 1-minute resolution) but do it occasionally (say, once per day), at the same time you have **frequent inference**(say, once per minute) on new chunks of data
|
||||
|
||||
> **Note**: However, it's still highly recommended, to keep your model up-to-date with tendencies found in your data as it evolves in time.
|
||||
|
||||
Produced model instances are **stored in-memory** between consecutive re-fit calls (defined by `fit_every` [arg](/anomaly-detection/components/scheduler/?highlight=fit_every#periodic-scheduler) in `PeriodicScheduler`), leading to **higher RAM** consumption.
|
||||
|
||||
**Examples:** [Prophet](#prophet)
|
||||
|
||||
<!-- TODO: add schema -->
|
||||
|
||||
## Built-in Models
|
||||
|
||||
### Overview
|
||||
|
|
@ -68,18 +207,19 @@ Depending on chosen `seasonality` parameter FB Prophet can return additional met
|
|||
*Config Example*
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.prophet.ProphetModel"
|
||||
seasonalities:
|
||||
- name: 'hourly'
|
||||
period: 0.04166666666
|
||||
fourier_order: 30
|
||||
# Inner model args (key-value pairs) accepted by
|
||||
# https://facebook.github.io/prophet/docs/quick_start.html#python-api
|
||||
args:
|
||||
# See https://facebook.github.io/prophet/docs/uncertainty_intervals.html
|
||||
interval_width: 0.98
|
||||
country_holidays: 'US'
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.prophet.ProphetModel"
|
||||
seasonalities:
|
||||
- name: 'hourly'
|
||||
period: 0.04166666666
|
||||
fourier_order: 30
|
||||
# Inner model args (key-value pairs) accepted by
|
||||
# https://facebook.github.io/prophet/docs/quick_start.html#python-api
|
||||
args:
|
||||
# See https://facebook.github.io/prophet/docs/uncertainty_intervals.html
|
||||
interval_width: 0.98
|
||||
country_holidays: 'US'
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -95,9 +235,10 @@ Resulting metrics of the model are described [here](#vmanomaly-output)
|
|||
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.zscore.ZscoreModel"
|
||||
z_threshold: 2.5
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.zscore.ZscoreModel"
|
||||
z_threshold: 2.5
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -131,14 +272,15 @@ Used to compute "seasonal_periods" param for the model (e.g. '1D' or '1W').
|
|||
*Config Example*
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.holtwinters.HoltWinters"
|
||||
seasonality: '1d'
|
||||
frequency: '1h'
|
||||
# Inner model args (key-value pairs) accepted by statsmodels.tsa.holtwinters.ExponentialSmoothing
|
||||
args:
|
||||
seasonal: 'add'
|
||||
initialization_method: 'estimated'
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.holtwinters.HoltWinters"
|
||||
seasonality: '1d'
|
||||
frequency: '1h'
|
||||
# Inner model args (key-value pairs) accepted by statsmodels.tsa.holtwinters.ExponentialSmoothing
|
||||
args:
|
||||
seasonal: 'add'
|
||||
initialization_method: 'estimated'
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -156,9 +298,10 @@ The MAD model is a robust method for anomaly detection that is *less sensitive*
|
|||
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.mad.MADModel"
|
||||
threshold: 2.5
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.mad.MADModel"
|
||||
threshold: 2.5
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -175,13 +318,13 @@ Resulting metrics of the model are described [here](#vmanomaly-output).
|
|||
*Config Example*
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.rolling_quantile.RollingQuantileModel"
|
||||
quantile: 0.9
|
||||
window_steps: 96
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.rolling_quantile.RollingQuantileModel"
|
||||
quantile: 0.9
|
||||
window_steps: 96
|
||||
```
|
||||
|
||||
|
||||
Resulting metrics of the model are described [here](#vmanomaly-output).
|
||||
|
||||
### [Seasonal Trend Decomposition](https://en.wikipedia.org/wiki/Seasonal_adjustment)
|
||||
|
|
@ -198,9 +341,10 @@ Here we use Seasonal Decompose implementation from `statsmodels` [library](https
|
|||
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.std.StdModel"
|
||||
period: 2
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.std.StdModel"
|
||||
period: 2
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -233,15 +377,16 @@ Here we use ARIMA implementation from `statsmodels` [library](https://www.statsm
|
|||
*Config Example*
|
||||
|
||||
```yaml
|
||||
model:
|
||||
class: "model.arima.ArimaModel"
|
||||
# ARIMA's (p,d,q) order
|
||||
order: [1, 1, 0]
|
||||
z_threshold: 2.7
|
||||
resample_freq: '1m'
|
||||
# Inner model args (key-value pairs) accepted by statsmodels.tsa.arima.model.ARIMA
|
||||
args:
|
||||
trend: 'c'
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
class: "model.arima.ArimaModel"
|
||||
# ARIMA's (p,d,q) order
|
||||
order: [1, 1, 0]
|
||||
z_threshold: 2.7
|
||||
resample_freq: '1m'
|
||||
# Inner model args (key-value pairs) accepted by statsmodels.tsa.arima.model.ARIMA
|
||||
args:
|
||||
trend: 'c'
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -264,14 +409,15 @@ Here we use Isolation Forest implementation from `scikit-learn` [library](https:
|
|||
|
||||
|
||||
```yaml
|
||||
model:
|
||||
# To use univariate model, substitute class argument with "model.isolation_forest.IsolationForestModel".
|
||||
class: "model.isolation_forest.IsolationForestMultivariateModel"
|
||||
contamination: "auto"
|
||||
args:
|
||||
n_estimators: 100
|
||||
# i.e. to assure reproducibility of produced results each time model is fit on the same input
|
||||
random_state: 42
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
# To use univariate model, substitute class argument with "model.isolation_forest.IsolationForestModel".
|
||||
class: "model.isolation_forest.IsolationForestMultivariateModel"
|
||||
contamination: "auto"
|
||||
args:
|
||||
n_estimators: 100
|
||||
# i.e. to assure reproducibility of produced results each time model is fit on the same input
|
||||
random_state: 42
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -318,6 +464,8 @@ Here in this guide, we will
|
|||
|
||||
### 1. Custom model
|
||||
|
||||
> **Note**: By default, each custom model is created as [**univariate**](#univariate-models) / [**non-rolling**](#non-rolling-models) model. If you want to override this behavior, define models inherited from `RollingModel` (to get a rolling model), or having `is_multivariate` class arg set to `True` (please refer to the code example below).
|
||||
|
||||
We'll create `custom_model.py` file with `CustomModel` class that will inherit from vmanomaly `Model` base class.
|
||||
In the `CustomModel` class there should be three required methods - `__init__`, `fit` and `infer`:
|
||||
* `__init__` method should initiate parameters for the model.
|
||||
|
|
@ -327,7 +475,7 @@ In the `CustomModel` class there should be three required methods - `__init__`,
|
|||
super().__init__(**kwargs)
|
||||
```
|
||||
to initialize the base class each model derives from
|
||||
* `fit` method should contain the model training process.
|
||||
* `fit` method should contain the model training process. Please be aware that for `RollingModel` defining `fit` method is not needed, as the whole fit/infer process should be defined completely in `infer` method.
|
||||
* `infer` should return Pandas.DataFrame object with model's inferences.
|
||||
|
||||
For the sake of simplicity, the model in this example will return one of two values of `anomaly_score` - 0 or 1 depending on input parameter `percentage`.
|
||||
|
|
@ -340,6 +488,7 @@ import scipy.stats as st
|
|||
import logging
|
||||
|
||||
from model.model import Model
|
||||
# from model.model import RollingModel # inherit from it for your model to be of rolling type
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
@ -348,6 +497,10 @@ class CustomModel(Model):
|
|||
Custom model implementation.
|
||||
"""
|
||||
|
||||
# by default, each `Model` will be created as a univariate one
|
||||
# uncomment line below for it to be of multivariate type
|
||||
# is_multivariate = True
|
||||
|
||||
def __init__(self, percentage: float = 0.95, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.percentage = percentage
|
||||
|
|
@ -362,7 +515,6 @@ class CustomModel(Model):
|
|||
if self._std == 0.0:
|
||||
self._std = 1 / 65536
|
||||
|
||||
|
||||
def infer(self, df: pd.DataFrame) -> np.array:
|
||||
# Inference process:
|
||||
y = df['y']
|
||||
|
|
@ -373,7 +525,6 @@ class CustomModel(Model):
|
|||
df_pred['anomaly_score'] = df_pred['anomaly_score'].astype('int32', errors='ignore')
|
||||
|
||||
return df_pred
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
|
@ -381,8 +532,8 @@ class CustomModel(Model):
|
|||
### 2. Configuration file
|
||||
|
||||
Next, we need to create `config.yaml` file with VM Anomaly Detection configuration and model input parameters.
|
||||
In the config file `model` section we need to put our model class `model.custom.CustomModel` and all parameters used in `__init__` method.
|
||||
You can find out more about configuration parameters in vmanomaly docs.
|
||||
In the config file `models` section we need to put our model class `model.custom.CustomModel` and all parameters used in `__init__` method.
|
||||
You can find out more about configuration parameters in [vmanomaly config docs](/anomaly-detection/components/).
|
||||
|
||||
|
||||
```yaml
|
||||
|
|
@ -391,11 +542,12 @@ scheduler:
|
|||
fit_every: "1m"
|
||||
fit_window: "1d"
|
||||
|
||||
model:
|
||||
# note: every custom model should implement this exact path, specified in `class` field
|
||||
class: "model.model.CustomModel"
|
||||
# custom model params are defined here
|
||||
percentage: 0.9
|
||||
models:
|
||||
your_desired_alias_for_a_model:
|
||||
# note: every custom model should implement this exact path, specified in `class` field
|
||||
class: "model.model.CustomModel"
|
||||
# custom model params are defined here
|
||||
percentage: 0.9
|
||||
|
||||
reader:
|
||||
datasource_url: "http://localhost:8428/"
|
||||
|
|
@ -426,15 +578,13 @@ monitoring:
|
|||
### 3. Running custom model
|
||||
Let's pull the docker image for vmanomaly:
|
||||
|
||||
|
||||
```sh
|
||||
docker pull us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial:latest
|
||||
docker pull victoriametrics/vmanomaly:latest
|
||||
```
|
||||
|
||||
|
||||
Now we can run the docker container putting as volumes both config and model file:
|
||||
|
||||
**Note**: place the model file to `/model/custom.py` path when copying
|
||||
> **Note**: place the model file to `/model/custom.py` path when copying
|
||||
|
||||
```sh
|
||||
docker run -it \
|
||||
|
|
@ -442,11 +592,10 @@ docker run -it \
|
|||
-v [YOUR_LICENSE_FILE_PATH]:/license.txt \
|
||||
-v $(PWD)/custom_model.py:/vmanomaly/src/model/custom.py \
|
||||
-v $(PWD)/custom.yaml:/config.yaml \
|
||||
us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial:latest /config.yaml \
|
||||
victoriametrics/vmanomaly:latest /config.yaml \
|
||||
--license-file=/license.txt
|
||||
```
|
||||
|
||||
|
||||
Please find more detailed instructions (license, etc.) [here](/anomaly-detection/overview.html#run-vmanomaly-docker-container)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -128,6 +128,8 @@ monitoring:
|
|||
### Models Behaviour Metrics
|
||||
Label names [description](#labelnames)
|
||||
|
||||
> **Note**: There is a new label key `model_alias` introduced in multi-model support [v1.10.0](/anomaly-detection/changelog/#v1100). This label key adjustment was made to preserve unique label set production during writing produced metrics back to VictoriaMetrics.
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
|
|
@ -142,25 +144,25 @@ Label names [description](#labelnames)
|
|||
<td><code>vmanomaly_model_runs</code></td>
|
||||
<td>Counter</td>
|
||||
<td>How many times models ran (per model)</td>
|
||||
<td><code>stage, query_key</code></td>
|
||||
<td><code>stage, query_key, model_alias</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>vmanomaly_model_run_duration_seconds</code></td>
|
||||
<td>Summary</td>
|
||||
<td>How much time (in seconds) model invocations took</td>
|
||||
<td><code>stage, query_key</code></td>
|
||||
<td><code>stage, query_key, model_alias</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>vmanomaly_model_datapoints_accepted</code></td>
|
||||
<td>Counter</td>
|
||||
<td>How many datapoints did models accept</td>
|
||||
<td><code>stage, query_key</code></td>
|
||||
<td><code>stage, query_key, model_alias</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>vmanomaly_model_datapoints_produced</code></td>
|
||||
<td>Counter</td>
|
||||
<td>How many datapoints were generated by models</td>
|
||||
<td><code>stage, query_key</code></td>
|
||||
<td><code>stage, query_key, model_alias</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>vmanomaly_models_active</code></td>
|
||||
|
|
@ -172,7 +174,7 @@ Label names [description](#labelnames)
|
|||
<td><code>vmanomaly_model_runs_skipped</code></td>
|
||||
<td>Counter</td>
|
||||
<td>How many times a run was skipped (per model)</td>
|
||||
<td><code>stage, query_key</code></td>
|
||||
<td><code>stage, query_key, model_alias</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
|
@ -286,6 +288,8 @@ Label names [description](#labelnames)
|
|||
|
||||
<code>query_key</code> - query alias from [`reader`](/anomaly-detection/components/reader.html) config section.
|
||||
|
||||
<code>model_alias</code> - model alias from [`models`](/anomaly-detection/components/models.html) config section. **Introduced in [v1.10.0](/anomaly-detection/changelog/#v1100).**
|
||||
|
||||
<code>url</code> - writer or reader url endpoint.
|
||||
|
||||
<code>code</code> - response status code or `connection_error`, `timeout`.
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ aliases:
|
|||
|
||||
All the service parameters are defined in a config file.
|
||||
|
||||
> **Note**: As of the time of writing, in the [1.9.2](https://docs.victoriametrics.com/anomaly-detection/changelog/#v192) release and earlier versions, each `vmanomaly` configuration file is limited to supporting only one model type. To utilize *different models* on your data, it is necessary to run multiple instances of the `vmanomaly` process. Each instance should operate with its own configuration file, differing in the `model` section.
|
||||
> **Note**: Starting from [1.10.0](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1100), each `vmanomaly` configuration file can support more that one model type. To utilize *different models* on your data, it is no longer necessary to run multiple instances of the `vmanomaly` process. Please refer to [model](/anomaly-detection/models/) config section for more details.
|
||||
|
||||
|
||||
**vmanomaly** does the following:
|
||||
|
|
@ -117,7 +117,7 @@ The configuration file for `vmanomaly` comprises 4 essential sections:
|
|||
|
||||
1. [`scheduler`](/anomaly-detection/components/scheduler.html) - This section determines the frequency of model inferences and training, including the time range for model training.
|
||||
|
||||
2. [`model`](/anomaly-detection/components/models.html) - Here, you define specific parameters and configurations for the model being used for anomaly detection.
|
||||
2. [`models`](/anomaly-detection/components/models.html) - Here, you define specific parameters and configurations for the models being used for anomaly detection.
|
||||
|
||||
3. [`reader`](/anomaly-detection/components/reader.html) - This section outlines the methodology for data reading, including the data source location.
|
||||
|
||||
|
|
@ -132,7 +132,7 @@ Detailed parameters in each section:
|
|||
* `fit_every` - Sets the frequency for retraining the models. A higher frequency ensures more updated models but requires more CPU resources. If omitted, models are retrained in each `infer_every` cycle. Format is similar to `infer_every`.
|
||||
* `fit_window` - Defines the data interval for training the models. Longer intervals allow for capturing extensive historical behavior and better seasonal pattern detection but may slow down the model's response to permanent metric changes and increase resource consumption. A minimum of two full seasonal cycles is recommended. Example format: 3h for three hours of data.
|
||||
|
||||
* `model`
|
||||
* `models`
|
||||
* `class` - Specifies the model to be used. Options include custom models ([guide here](/anomaly-detection/components/models.html#custom-model-guide)) or a selection from [built-in models](/anomaly-detection/components/models.html#built-in-models), such as the [Facebook Prophet](/anomaly-detection/components/models.html#prophet) (`model.prophet.ProphetModel`).
|
||||
* `args` - Model-specific parameters, formatted as a YAML dictionary in the `key: value` structure. Parameters available in [FB Prophet](https://facebook.github.io/prophet/docs/quick_start.html) can be used as an example.
|
||||
|
||||
|
|
@ -152,10 +152,11 @@ scheduler:
|
|||
fit_every: "2m"
|
||||
fit_window: "3h"
|
||||
|
||||
model:
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
models:
|
||||
prophet:
|
||||
class: "model.prophet.ProphetModel"
|
||||
args:
|
||||
interval_width: 0.98
|
||||
|
||||
reader:
|
||||
datasource_url: "http://victoriametrics:8428/"
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ See [these docs](#running-victoriametrics-enterprise) for details on how to run
|
|||
VictoriaMetrics Enterprise includes [all the features of the community edition](https://docs.victoriametrics.com/#prominent-features),
|
||||
plus the following additional features:
|
||||
|
||||
- Stable releases with long-term support, which contains important bugfixes. See, for example, [v1.97.x LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.1).
|
||||
- Stable releases with long-term support, which contains important bugfixes and security fixes. See [these docs](https://docs.victoriametrics.com/LTS-releases.html).
|
||||
- First-class consulting and technical support provided by the core VictoriaMetrics dev team.
|
||||
- [Monitoring of monitoring](https://victoriametrics.com/products/mom/) - this feature allows forecasting
|
||||
and preventing possible issues in VictoriaMetrics setups.
|
||||
|
|
|
|||
|
|
@ -280,7 +280,11 @@ foo_field2{tag1="value1", tag2="value2"} 40
|
|||
|
||||
### Configuration
|
||||
|
||||
The configuration flags should contain self-explanatory descriptions.
|
||||
Run the following command to get all configuration options:
|
||||
```sh
|
||||
./vmctl influx --help
|
||||
```
|
||||
|
||||
|
||||
### Filtering
|
||||
|
||||
|
|
@ -421,7 +425,11 @@ So no data changes will be applied.
|
|||
|
||||
### Configuration
|
||||
|
||||
The configuration flags should contain self-explanatory descriptions.
|
||||
Run the following command to get all configuration options:
|
||||
```sh
|
||||
./vmctl prometheus --help
|
||||
```
|
||||
|
||||
|
||||
### Filtering
|
||||
|
||||
|
|
|
|||
2
go.mod
2
go.mod
|
|
@ -8,7 +8,7 @@ require (
|
|||
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.0
|
||||
github.com/VictoriaMetrics/easyproto v0.1.4
|
||||
github.com/VictoriaMetrics/fastcache v1.12.2
|
||||
github.com/VictoriaMetrics/metrics v1.31.0
|
||||
github.com/VictoriaMetrics/metrics v1.32.0
|
||||
github.com/VictoriaMetrics/metricsql v0.72.1
|
||||
github.com/aws/aws-sdk-go-v2 v1.25.0
|
||||
github.com/aws/aws-sdk-go-v2/config v1.27.0
|
||||
|
|
|
|||
4
go.sum
4
go.sum
|
|
@ -69,8 +69,8 @@ github.com/VictoriaMetrics/easyproto v0.1.4/go.mod h1:QlGlzaJnDfFd8Lk6Ci/fuLxfTo
|
|||
github.com/VictoriaMetrics/fastcache v1.12.2 h1:N0y9ASrJ0F6h0QaC3o6uJb3NIZ9VKLjCM7NQbSmF7WI=
|
||||
github.com/VictoriaMetrics/fastcache v1.12.2/go.mod h1:AmC+Nzz1+3G2eCPapF6UcsnkThDcMsQicp4xDukwJYI=
|
||||
github.com/VictoriaMetrics/metrics v1.24.0/go.mod h1:eFT25kvsTidQFHb6U0oa0rTrDRdz4xTYjpL8+UPohys=
|
||||
github.com/VictoriaMetrics/metrics v1.31.0 h1:X6+nBvAP0UB+GjR0Ht9hhQ3pjL1AN4b8dt9zFfzTsUo=
|
||||
github.com/VictoriaMetrics/metrics v1.31.0/go.mod h1:r7hveu6xMdUACXvB8TYdAj8WEsKzWB0EkpJN+RDtOf8=
|
||||
github.com/VictoriaMetrics/metrics v1.32.0 h1:r9JK2zndYv0TIxFXLEHwhQqRdnu8/O3cwJiCBX4vJCM=
|
||||
github.com/VictoriaMetrics/metrics v1.32.0/go.mod h1:r7hveu6xMdUACXvB8TYdAj8WEsKzWB0EkpJN+RDtOf8=
|
||||
github.com/VictoriaMetrics/metricsql v0.72.1 h1:fLIHgzezXgD4NjY5ksF4lRkHILW88uI5Lz0Q+N2ucnY=
|
||||
github.com/VictoriaMetrics/metricsql v0.72.1/go.mod h1:k4UaP/+CjuZslIjd+kCigNG9TQmUqh5v0TP/nMEy90I=
|
||||
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ func GetDuration(r *http.Request, argKey string, defaultValue int64) (int64, err
|
|||
}
|
||||
msecs := int64(secs * 1e3)
|
||||
if msecs <= 0 || msecs > maxDurationMsecs {
|
||||
return 0, fmt.Errorf("%q=%dms is out of allowed range [%d ... %d]", argKey, msecs, 0, int64(maxDurationMsecs))
|
||||
return 0, fmt.Errorf("%s=%dms is out of allowed range [%dms ... %dms]", argKey, msecs, 1, int64(maxDurationMsecs))
|
||||
}
|
||||
return msecs, nil
|
||||
}
|
||||
|
|
|
|||
34
vendor/github.com/VictoriaMetrics/metrics/gauge.go
generated
vendored
34
vendor/github.com/VictoriaMetrics/metrics/gauge.go
generated
vendored
|
|
@ -17,7 +17,7 @@ import (
|
|||
// - foo{bar="baz",aaa="b"}
|
||||
//
|
||||
// f must be safe for concurrent calls.
|
||||
// if f is nil, then it is expected that the gauge value is changed via Gauge.Set() call.
|
||||
// if f is nil, then it is expected that the gauge value is changed via Set(), Inc(), Dec() and Add() calls.
|
||||
//
|
||||
// The returned gauge is safe to use from concurrent goroutines.
|
||||
//
|
||||
|
|
@ -55,6 +55,38 @@ func (g *Gauge) Set(v float64) {
|
|||
atomic.StoreUint64(&g.valueBits, n)
|
||||
}
|
||||
|
||||
// Inc increments g by 1.
|
||||
//
|
||||
// The g must be created with nil callback in order to be able to call this function.
|
||||
func (g *Gauge) Inc() {
|
||||
g.Add(1)
|
||||
}
|
||||
|
||||
// Dec decrements g by 1.
|
||||
//
|
||||
// The g must be created with nil callback in order to be able to call this function.
|
||||
func (g *Gauge) Dec() {
|
||||
g.Add(-1)
|
||||
}
|
||||
|
||||
// Add adds fAdd to g. fAdd may be positive and negative.
|
||||
//
|
||||
// The g must be created with nil callback in order to be able to call this function.
|
||||
func (g *Gauge) Add(fAdd float64) {
|
||||
if g.f != nil {
|
||||
panic(fmt.Errorf("cannot call Set on gauge created with non-nil callback"))
|
||||
}
|
||||
for {
|
||||
n := atomic.LoadUint64(&g.valueBits)
|
||||
f := math.Float64frombits(n)
|
||||
fNew := f + fAdd
|
||||
nNew := math.Float64bits(fNew)
|
||||
if atomic.CompareAndSwapUint64(&g.valueBits, n, nNew) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (g *Gauge) marshalTo(prefix string, w io.Writer) {
|
||||
v := g.Get()
|
||||
if float64(int64(v)) == v {
|
||||
|
|
|
|||
2
vendor/modules.txt
vendored
2
vendor/modules.txt
vendored
|
|
@ -99,7 +99,7 @@ github.com/VictoriaMetrics/easyproto
|
|||
# github.com/VictoriaMetrics/fastcache v1.12.2
|
||||
## explicit; go 1.13
|
||||
github.com/VictoriaMetrics/fastcache
|
||||
# github.com/VictoriaMetrics/metrics v1.31.0
|
||||
# github.com/VictoriaMetrics/metrics v1.32.0
|
||||
## explicit; go 1.17
|
||||
github.com/VictoriaMetrics/metrics
|
||||
# github.com/VictoriaMetrics/metricsql v0.72.1
|
||||
|
|
|
|||
Loading…
Reference in a new issue