github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

Browse source
This commit is contained in:
Aliaksandr Valialkin 2020-11-19 18:56:02 +02:00
commit fb2d4e56ce
17 changed files with 430 additions and 172 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

144
README.md
View file

@ -13,8 +13,10 @@
VictoriaMetrics is fast, cost-effective and scalable monitoring solution and time series database.
It is available in [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and
in [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). Just download VictoriaMetrics and see [how to start it](#how-to-start-victoriametrics).
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/), [Snap package](https://snapcraft.io/victoriametrics)
and in [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). Just download VictoriaMetrics and see [how to start it](#how-to-start-victoriametrics).
If you use Ubuntu, then just run `snap install victoriametrics` in order to install and run it.
Then read [Prometheus setup](#prometheus-setup) and [Grafana setup](#grafana-setup) docs.
Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
@ -28,7 +30,7 @@ See [features available for enterprise customers](https://victoriametrics.com/en
Click on a link in order to read the corresponding case study
* [Adidas](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#adidas)
* [adidas](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#adidas)
* [CERN](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#cern)
* [COLOPL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#colopl)
* [Zerodha](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#zerodha)
@ -161,7 +163,7 @@ Click on a link in order to read the corresponding case study
* [We kindly ask](#we-kindly-ask)
### How to start VictoriaMetrics
## How to start VictoriaMetrics
Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags.
@ -178,7 +180,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying-
It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics.
#### Environment variables
### Environment variables
Each flag value can be set via environment variables according to these rules:
@ -188,7 +190,7 @@ Each flag value can be set via environment variables according to these rules:
* It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`
### Prometheus setup
## Prometheus setup
Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
in order to send data to VictoriaMetrics. Add the following lines
@ -244,7 +246,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics
which can be used as faster and less resource-hungry alternative to Prometheus in certain cases.
### Grafana setup
## Grafana setup
Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url:
@ -259,7 +261,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL
which is used by Grafana.
### How to upgrade VictoriaMetrics
## How to upgrade VictoriaMetrics
It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise.
@ -277,7 +279,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
### How to apply new config to VictoriaMetrics
## How to apply new config to VictoriaMetrics
VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
@ -289,7 +291,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).
Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
@ -314,7 +316,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](#
See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus.
### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
Use `http://<victoriametric-addr>:8428` url instead of InfluxDB url in agents' configs.
For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB:
@ -376,7 +378,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default](
while VictoriaMetrics stores them with *milliseconds* precision.
### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`:
@ -408,7 +410,7 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
```
### Querying Graphite data
## Querying Graphite data
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
@ -417,13 +419,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t
* Tags can be explored via [Graphite tags API](#graphite-tags-api-usage)
* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
### How to send data from OpenTSDB-compatible agents
## How to send data from OpenTSDB-compatible agents
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
#### Sending data via `telnet put` protocol
### Sending data via `telnet put` protocol
Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance,
the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`:
@ -453,7 +455,7 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]}
```
#### Sending OpenTSDB data via HTTP `/api/put` requests
### Sending OpenTSDB data via HTTP `/api/put` requests
Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance,
the following command enables OpenTSDB HTTP server on port `4242`:
@ -491,7 +493,7 @@ The `/api/v1/export` endpoint should return the following response:
```
### Prometheus querying API usage
## Prometheus querying API usage
VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/):
@ -507,7 +509,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
#### Prometheus querying API enhancements
### Prometheus querying API enhancements
Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
@ -529,7 +531,7 @@ Additionally VictoriaMetrics provides the following handlers:
* `/api/v1/status/active_queries` - it returns a list of currently running queries.
### Graphite API usage
## Graphite API usage
VictoriaMetrics supports the following Graphite APIs:
@ -537,7 +539,7 @@ VictoriaMetrics supports the following Graphite APIs:
* Tags API - see [these docs](#graphite-tags-api-usage).
#### Graphite Metrics API usage
### Graphite Metrics API usage
VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
@ -551,55 +553,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a
that start with `node_`. By default `delimiter=.`.
#### Graphite Tags API usage
### Graphite Tags API usage
VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html):
* [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
* [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
* [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/{tag_name}](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
* [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
### How to build from sources
## How to build from sources
We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics
from sources. Building from sources is reasonable when developing additional features specific
to your needs or when testing bugfixes.
#### Development build
### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make victoria-metrics` from the root folder of the repository.
It builds `victoria-metrics` binary and puts it into the `bin` folder.
#### Production build
### Production build
1. [Install docker](https://docs.docker.com/install/).
2. Run `make victoria-metrics-prod` from the root folder of the repository.
It builds `victoria-metrics-prod` binary and puts it into the `bin` folder.
#### ARM build
### ARM build
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
#### Development ARM build
### Development ARM build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository.
It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder.
#### Production ARM build
### Production ARM build
1. [Install docker](https://docs.docker.com/install/).
2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository.
It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder.
#### Pure Go build (CGO_ENABLED=0)
### Pure Go build (CGO_ENABLED=0)
`Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
This is an experimental mode, which may result in a lower compression ratio and slower decompression performance.
@ -609,7 +611,7 @@ Use it with caution!
2. Run `make victoria-metrics-pure` from the root folder of the repository.
It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
#### Building docker images
### Building docker images
Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -623,17 +625,20 @@ For example, the following command builds the image on top of [scratch](https://
ROOT_IMAGE=scratch make package-victoria-metrics
```
### Start with docker-compose
## Start with docker-compose
[Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml)
helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command.
More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker).
### Setting up service
## Setting up service
Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics).
### How to work with snapshots
## How to work with snapshots
VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
for all the data stored under `-storageDataPath` directory.
@ -662,7 +667,7 @@ Steps for restoring from a snapshot:
to the directory pointed by `-storageDataPath`.
3. Start VictoriaMetrics.
### How to delete time series
## How to delete time series
Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
where `<timeseries_selector_for_delete>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
@ -694,7 +699,7 @@ It isn't recommended using delete API for the following cases, since it brings n
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
### Forced merge
## Forced merge
VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions.
@ -709,7 +714,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http
when new data is ingested into it.
### How to export time series
## How to export time series
VictoriaMetrics provides the following handlers for exporting data:
@ -719,7 +724,7 @@ VictoriaMetrics provides the following handlers for exporting data:
* `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details.
#### How to export data in native format
### How to export data in native format
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
@ -740,7 +745,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
#### How to export data in JSON line format
### How to export data in JSON line format
Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances,
since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space.
@ -775,7 +780,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search
Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format).
#### How to export CSV data
### How to export CSV data
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/csv?format=<format>&match=<timeseries_selector_for_export>`,
where:
@ -799,7 +804,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data).
### How to import time series data
## How to import time series data
Time series data can be imported via any supported ingestion protocol:
@ -816,7 +821,7 @@ Time series data can be imported via any supported ingestion protocol:
* `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
#### How to import data in native format
### How to import data in native format
The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`.
Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format):
@ -845,7 +850,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"`
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import data in JSON line format
### How to import data in JSON line format
Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
@ -873,7 +878,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import CSV data
### How to import CSV data
Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:
@ -926,7 +931,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import data in Prometheus exposition format
### How to import data in Prometheus exposition format
VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics:
@ -961,7 +966,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
### Relabeling
## Relabeling
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
@ -985,7 +990,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules:
See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling).
### Federation
## Federation
VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/)
at `http://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>`.
@ -996,7 +1001,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series.
For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation
with scrape intervals exceeding `5m`.
### Capacity planning
## Capacity planning
A rough estimation of the required resources for ingestion path:
@ -1036,7 +1041,8 @@ The required resources for query path:
* Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually
require negligible network bandwidth.
### High availability
## High availability
* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones).
* Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag:
@ -1077,7 +1083,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair
with the enabled de-duplication. See [this section](#deduplication) for details.
### Deduplication
## Deduplication
VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
@ -1088,7 +1094,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst
`external_labels` section in their configs, so they write data to the same time series.
### Retention
## Retention
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
that the data will be stored for 3 months and then deleted.
@ -1103,7 +1109,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
### Multiple retentions
## Multiple retentions
Just start multiple VictoriaMetrics instances with distinct values for the following flags:
@ -1116,7 +1122,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi
The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md).
### Downsampling
## Downsampling
There is no downsampling support at the moment, but:
@ -1132,11 +1138,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d
For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave
only a single data point out of 20 initial data points per each 5m interval.
### Multi-tenancy
## Multi-tenancy
Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead.
### Scalability and cluster version
## Scalability and cluster version
Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU.
This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions
@ -1146,7 +1154,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version]
horizontally scalable long-term remote storage for really large Prometheus deployments.
[Contact us](mailto:info@victoriametrics.com) for paid support.
### Alerting
## Alerting
It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting.
@ -1157,7 +1166,7 @@ Additionally, alerting can be set up with the following tools:
* With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/).
### Security
## Security
Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet.
Consider setting the following command-line flags:
@ -1177,7 +1186,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut
or similar auth proxy.
### Tuning
## Tuning
* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags,
which are automatically adjusted for the available CPU and RAM resources.
@ -1193,7 +1202,7 @@ or similar auth proxy.
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
```
### Monitoring
## Monitoring
VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page.
These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md)
@ -1223,7 +1232,7 @@ The most interesting metrics are:
VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page.
### Troubleshooting
## Troubleshooting
* It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need
of tweaking these flag values arises.
@ -1236,7 +1245,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
* VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
This may lead to the following "issues":
* Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to persistent storage
by requesting `/internal/force_flush` http handler.
by requesting `/internal/force_flush` http handler. This handler is mostly needed for testing and debugging purposes.
* The last few seconds of inserted data may be lost on unclean shutdown (i.e. OOM, `kill -9` or hardware reset).
See [this article for technical details](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
@ -1291,7 +1300,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
* VictoriaMetrics ignores `NaN` values during data ingestion.
### Backfilling
## Backfilling
VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data).
Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data.
@ -1307,7 +1316,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in
for data with timestamps close to the current time.
### Data updates
## Data updates
VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
for the same time series with identical timestamps. While is possible substituting old time series with new time series via
@ -1315,7 +1324,7 @@ for the same time series with identical timestamps. While is possible substituti
should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.
### Replication
## Replication
Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead.
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details.
@ -1325,14 +1334,14 @@ Storage-level replication may be offloaded to durable persistent storage such as
See also [high availability docs](#high-availability) and [backup docs](#backups).
### Backups
## Backups
VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md)
and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools.
We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details.
### Profiling
## Profiling
VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -1352,6 +1361,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
## Integrations
* [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts).
@ -1363,6 +1373,8 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
See [this example](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml).
* [Ansible role for installing single-node VictoriaMetrics](https://github.com/dreamteam-gg/ansible-victoriametrics-role).
* [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role).
* [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics).
## Third-party contributions
@ -1371,10 +1383,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
* [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details.
## Contacts
Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com).
## Community and contributions
Feel free asking any questions regarding VictoriaMetrics:

7
app/victoria-metrics/Makefile
View file

@ -108,3 +108,10 @@ victoria-metrics-package-deb-rpm-all: \
victoria-metrics-package-deb-arm64 \
victoria-metrics-package-rpm \
victoria-metrics-package-rpm-arm64
### Packaging as snap
victoria-metrics-package-snap:
which snapcraft || snap install snapcraft
which multipass || snap install multipass
snapcraft

4
app/vmselect/promql/exec.go
View file

@ -33,8 +33,8 @@ func Exec(ec *EvalConfig, q string, isFirstPointOnly bool) ([]netstorage.Result,
defer func() {
d := time.Since(startTime)
if d >= *logSlowQueryDuration {
logger.Warnf("slow query according to -search.logSlowQueryDuration=%s: duration=%.3f seconds, start=%d, end=%d, step=%d, query=%q",
*logSlowQueryDuration, d.Seconds(), ec.Start/1000, ec.End/1000, ec.Step/1000, q)
logger.Warnf("slow query according to -search.logSlowQueryDuration=%s: remoteAddr=%s, duration=%.3f seconds, start=%d, end=%d, step=%d, query=%q",
*logSlowQueryDuration, ec.QuotedRemoteAddr, d.Seconds(), ec.Start/1000, ec.End/1000, ec.Step/1000, q)
slowQueries.Inc()
}
}()

11
docs/CHANGELOG.md
View file

@ -2,6 +2,17 @@
# tip
* FEATURE: added [Snap package for single-node VictoriaMetrics](https://snapcraft.io/victoriametrics). This simplifies installation under Ubuntu to a single command:
```bash
snap install victoriametrics
```
* FEATURE: vminsert: export `vm_rpc_vmstorage_is_reachable` metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes.
* FEATURE: add `-loggerWarnsPerSecondLimit` command-line flag for rate limiting of WARN messages in logs. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905
* FEATURE: apply `loggerErrorsPerSecondLimit` and `-loggerWarnsPerSecondLimit` rate limit per caller. I.e. log messages are suppressed if the same caller logs the same message
at the rate exceeding the given limit. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905#issuecomment-729395855
* FEATURE: add remoteAddr to slow query log in order to simplify identifying the client that sends slow queries to VictoriaMetrics.
Slow query logging is controlled with `-search.logSlowQueryDuration` command-line flag.
# [v1.47.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.47.0)

4
docs/CaseStudies.md
View file

@ -6,7 +6,7 @@ and feel free asking for references, reviews and additional case studies from re
See also [articles about VictoriaMetrics from our users](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Articles#third-party-articles-and-slides).
* [Adidas](#adidas)
* [adidas](#adidas)
* [CERN](#cern)
* [COLOPL](#colopl)
* [Zerodha](#zerodha)
@ -20,7 +20,7 @@ See also [articles about VictoriaMetrics from our users](https://github.com/Vict
* [Idealo.de](#idealode)
## Adidas
## adidas
See [slides](https://promcon.io/2019-munich/slides/remote-write-storage-wars.pdf) and [video](https://youtu.be/OsH6gPdxR4s)
from [Remote Write Storage Wars](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) talk at [PromCon 2019](https://promcon.io/2019-munich/).

2
docs/Cluster-VictoriaMetrics.md
View file

@ -285,7 +285,7 @@ Each instance type - `vminsert`, `vmselect` and `vmstorage` - can run on the mos
#### vmstorage
* The recommended total number of vCPU cores for all the `vmstorage` instances can be calculated from the ingestion rate: `vCPUs = ingestion_rate / 150K`.
* The recommended total amount of RAM for all the `vmstorage` instances can be calculated from the number of active time series: `RAM = active_time_series * 1KB`.
* The recommended total amount of RAM for all the `vmstorage` instances can be calculated from the number of active time series: `RAM = 2 * active_time_series * 1KB`.
Time series is active if it received at least a single data point during the last hour or if it has been queried during the last hour.
The required RAM per each `vmstorage` should be multiplied by `-replicationFactor` if [replication](#replication-and-data-safety) is enabled.
Additional RAM can be required for query processing.

4
docs/FAQ.md
View file

@ -90,7 +90,7 @@ The main differences between Cortex and VictoriaMetrics:
- Cortex may lose up to 12 hours of recent data on Ingestor failure - see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss).
VictoriaMetrics may lose only a few seconds of recent data, which isn't synced to persistent storage yet.
See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
- Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from Adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies).
- Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol - InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-import-time-series-data) for details.
@ -111,7 +111,7 @@ The main differences between Cortex and VictoriaMetrics:
which is much easier to setup and operate than Thanos components.
- Thanos may be harder to setup and operate comparing to VictoriaMetrics, since it has more moving parts, which can be connected with less reliable networks.
See [this article for details](https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683).
- Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from Adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
- Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol - InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-import-time-series-data) for details.

6
docs/Quick-Start.md
View file

@ -1,10 +1,12 @@
# Quick Start
1. Download the latest VictoriaMetrics release from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
1. If you run Ubuntu, then just run `snap install victoriametrics` command in order to install and start VictoriaMetrics, then read [these docs](https://snapcraft.io/victoriametrics).
Otherwise download the latest VictoriaMetrics release from [releases page](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
from [Docker hub](https://hub.docker.com/r/victoriametrics/victoria-metrics/)
or [build it from sources](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics#how-to-build-from-sources).
2. Run the binary or Docker image with the desired command-line flags. Pass `-help` in order to see description for all the available flags
2. This step isn't needed if you run VictoriaMetrics via `snap install victoriametrics` as described above.
Otherwise run the binary or Docker image with the desired command-line flags. Pass `-help` in order to see description for all the available flags
and their default values. Default flag values should fit the majoirty of cases. The minimum required flags to configure are:
* `-storageDataPath` - path to directory where VictoriaMetrics stores all the data.

144
docs/Single-server-VictoriaMetrics.md
View file

@ -13,8 +13,10 @@
VictoriaMetrics is fast, cost-effective and scalable monitoring solution and time series database.
It is available in [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases),
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) and
in [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). Just download VictoriaMetrics and see [how to start it](#how-to-start-victoriametrics).
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/), [Snap package](https://snapcraft.io/victoriametrics)
and in [source code](https://github.com/VictoriaMetrics/VictoriaMetrics). Just download VictoriaMetrics and see [how to start it](#how-to-start-victoriametrics).
If you use Ubuntu, then just run `snap install victoriametrics` in order to install and run it.
Then read [Prometheus setup](#prometheus-setup) and [Grafana setup](#grafana-setup) docs.
Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
@ -28,7 +30,7 @@ See [features available for enterprise customers](https://victoriametrics.com/en
Click on a link in order to read the corresponding case study
* [Adidas](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#adidas)
* [adidas](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#adidas)
* [CERN](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#cern)
* [COLOPL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#colopl)
* [Zerodha](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies#zerodha)
@ -161,7 +163,7 @@ Click on a link in order to read the corresponding case study
* [We kindly ask](#we-kindly-ask)
### How to start VictoriaMetrics
## How to start VictoriaMetrics
Start VictoriaMetrics [executable](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
or [docker image](https://hub.docker.com/r/victoriametrics/victoria-metrics/) with the desired command-line flags.
@ -178,7 +180,7 @@ VictoriaMetrics accepts [Prometheus querying API requests](#prometheus-querying-
It is recommended setting up [monitoring](#monitoring) for VictoriaMetrics.
#### Environment variables
### Environment variables
Each flag value can be set via environment variables according to these rules:
@ -188,7 +190,7 @@ Each flag value can be set via environment variables according to these rules:
* It is possible setting prefix for environment vars with `-envflag.prefix`. For instance, if `-envflag.prefix=VM_`, then env vars must be prepended with `VM_`
### Prometheus setup
## Prometheus setup
Prometheus must be configured with [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
in order to send data to VictoriaMetrics. Add the following lines
@ -244,7 +246,7 @@ Take a look also at [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics
which can be used as faster and less resource-hungry alternative to Prometheus in certain cases.
### Grafana setup
## Grafana setup
Create [Prometheus datasource](http://docs.grafana.org/features/datasources/prometheus/) in Grafana with the following url:
@ -259,7 +261,7 @@ or [MetricsQL](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL
which is used by Grafana.
### How to upgrade VictoriaMetrics
## How to upgrade VictoriaMetrics
It is safe upgrading VictoriaMetrics to new versions unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
say otherwise. It is safe skipping multiple versions during the upgrade unless [release notes](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) say otherwise.
@ -277,7 +279,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
### How to apply new config to VictoriaMetrics
## How to apply new config to VictoriaMetrics
VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:
@ -289,7 +291,7 @@ Prometheus doesn't drop data during VictoriaMetrics restart.
See [this article](https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/) for details.
### How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).
Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
@ -314,7 +316,7 @@ VictoriaMetrics also supports [importing data in Prometheus exposition format](#
See also [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md), which can be used as drop-in replacement for Prometheus.
### How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
## How to send data from InfluxDB-compatible agents such as [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)
Use `http://<victoriametric-addr>:8428` url instead of InfluxDB url in agents' configs.
For instance, put the following lines into `Telegraf` config, so it sends data to VictoriaMetrics instead of InfluxDB:
@ -376,7 +378,7 @@ Note that Influx line protocol expects [timestamps in *nanoseconds* by default](
while VictoriaMetrics stores them with *milliseconds* precision.
### How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
## How to send data from Graphite-compatible agents such as [StatsD](https://github.com/etsy/statsd)
Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command line flag. For instance,
the following command will enable Graphite receiver in VictoriaMetrics on TCP and UDP port `2003`:
@ -408,7 +410,7 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
```
### Querying Graphite data
## Querying Graphite data
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
@ -417,13 +419,13 @@ Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via t
* Tags can be explored via [Graphite tags API](#graphite-tags-api-usage)
* [go-graphite/carbonapi](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml)
### How to send data from OpenTSDB-compatible agents
## How to send data from OpenTSDB-compatible agents
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
#### Sending data via `telnet put` protocol
### Sending data via `telnet put` protocol
Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command line flag. For instance,
the following command enables OpenTSDB receiver in VictoriaMetrics on TCP and UDP port `4242`:
@ -453,7 +455,7 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277292000]}
```
#### Sending OpenTSDB data via HTTP `/api/put` requests
### Sending OpenTSDB data via HTTP `/api/put` requests
Enable HTTP server for OpenTSDB `/api/put` requests by setting `-opentsdbHTTPListenAddr` command line flag. For instance,
the following command enables OpenTSDB HTTP server on port `4242`:
@ -491,7 +493,7 @@ The `/api/v1/export` endpoint should return the following response:
```
### Prometheus querying API usage
## Prometheus querying API usage
VictoriaMetrics supports the following handlers from [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/):
@ -507,7 +509,7 @@ VictoriaMetrics supports the following handlers from [Prometheus querying API](h
These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.
#### Prometheus querying API enhancements
### Prometheus querying API enhancements
Additionally to unix timestamps and [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) VictoriaMetrics accepts relative times in `time`, `start` and `end` query args.
For example, the following query would return data for the last 30 minutes: `/api/v1/query_range?start=-30m&query=...`.
@ -529,7 +531,7 @@ Additionally VictoriaMetrics provides the following handlers:
* `/api/v1/status/active_queries` - it returns a list of currently running queries.
### Graphite API usage
## Graphite API usage
VictoriaMetrics supports the following Graphite APIs:
@ -537,7 +539,7 @@ VictoriaMetrics supports the following Graphite APIs:
* Tags API - see [these docs](#graphite-tags-api-usage).
#### Graphite Metrics API usage
### Graphite Metrics API usage
VictoriaMetrics supports the following handlers from [Graphite Metrics API](https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api):
@ -551,55 +553,55 @@ VictoriaMetrics accepts the following additional query args at `/metrics/find` a
that start with `node_`. By default `delimiter=.`.
#### Graphite Tags API usage
### Graphite Tags API usage
VictoriaMetrics supports the following handlers from [Graphite Tags API](https://graphite.readthedocs.io/en/stable/tags.html):
* [/tags/tagSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
* [/tags/tagMultiSeries](https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb)
* [/tags](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/tag_name](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/{tag_name}](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/findSeries](https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags)
* [/tags/autoComplete/tags](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
* [/tags/autoComplete/values](https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support)
### How to build from sources
## How to build from sources
We recommend using either [binary releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) or
[docker images](https://hub.docker.com/r/victoriametrics/victoria-metrics/) instead of building VictoriaMetrics
from sources. Building from sources is reasonable when developing additional features specific
to your needs or when testing bugfixes.
#### Development build
### Development build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make victoria-metrics` from the root folder of the repository.
It builds `victoria-metrics` binary and puts it into the `bin` folder.
#### Production build
### Production build
1. [Install docker](https://docs.docker.com/install/).
2. Run `make victoria-metrics-prod` from the root folder of the repository.
It builds `victoria-metrics-prod` binary and puts it into the `bin` folder.
#### ARM build
### ARM build
ARM build may run on Raspberry Pi or on [energy-efficient ARM servers](https://blog.cloudflare.com/arm-takes-wing/).
#### Development ARM build
### Development ARM build
1. [Install Go](https://golang.org/doc/install). The minimum supported version is Go 1.13.
2. Run `make victoria-metrics-arm` or `make victoria-metrics-arm64` from the root folder of the repository.
It builds `victoria-metrics-arm` or `victoria-metrics-arm64` binary respectively and puts it into the `bin` folder.
#### Production ARM build
### Production ARM build
1. [Install docker](https://docs.docker.com/install/).
2. Run `make victoria-metrics-arm-prod` or `make victoria-metrics-arm64-prod` from the root folder of the repository.
It builds `victoria-metrics-arm-prod` or `victoria-metrics-arm64-prod` binary respectively and puts it into the `bin` folder.
#### Pure Go build (CGO_ENABLED=0)
### Pure Go build (CGO_ENABLED=0)
`Pure Go` mode builds only Go code without [cgo](https://golang.org/cmd/cgo/) dependencies.
This is an experimental mode, which may result in a lower compression ratio and slower decompression performance.
@ -609,7 +611,7 @@ Use it with caution!
2. Run `make victoria-metrics-pure` from the root folder of the repository.
It builds `victoria-metrics-pure` binary and puts it into the `bin` folder.
#### Building docker images
### Building docker images
Run `make package-victoria-metrics`. It builds `victoriametrics/victoria-metrics:<PKG_TAG>` docker image locally.
`<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository.
@ -623,17 +625,20 @@ For example, the following command builds the image on top of [scratch](https://
ROOT_IMAGE=scratch make package-victoria-metrics
```
### Start with docker-compose
## Start with docker-compose
[Docker-compose](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml)
helps to spin up VictoriaMetrics, [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) and Grafana with one command.
More details may be found [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#folder-contains-basic-images-and-tools-for-building-and-running-victoria-metrics-in-docker).
### Setting up service
## Setting up service
Read [these instructions](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/43) on how to set up VictoriaMetrics as a service in your OS.
There is also [snap package for Ubuntu](https://snapcraft.io/victoriametrics).
### How to work with snapshots
## How to work with snapshots
VictoriaMetrics can create [instant snapshots](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
for all the data stored under `-storageDataPath` directory.
@ -662,7 +667,7 @@ Steps for restoring from a snapshot:
to the directory pointed by `-storageDataPath`.
3. Start VictoriaMetrics.
### How to delete time series
## How to delete time series
Send a request to `http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>`,
where `<timeseries_selector_for_delete>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
@ -694,7 +699,7 @@ It isn't recommended using delete API for the following cases, since it brings n
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
### Forced merge
## Forced merge
VictoriaMetrics performs [data compactions in background](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282)
in order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions.
@ -709,7 +714,7 @@ since VictoriaMetrics automatically performs [optimal merges in background](http
when new data is ingested into it.
### How to export time series
## How to export time series
VictoriaMetrics provides the following handlers for exporting data:
@ -719,7 +724,7 @@ VictoriaMetrics provides the following handlers for exporting data:
* `/api/v1/export/csv` for exporting data in CSV. See [these docs](#how-to-export-csv-data) for details.
#### How to export data in native format
### How to export data in native format
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>`,
where `<timeseries_selector_for_export>` may contain any [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors)
@ -740,7 +745,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
The exported data can be imported to VictoriaMetrics via [/api/v1/import/native](#how-to-import-data-in-native-format).
#### How to export data in JSON line format
### How to export data in JSON line format
Consider [exporting data in native format](#how-to-export-data-in-native-format) if big amounts of data must be migrated between VictoriaMetrics instances,
since exporting in native format usually consumes lower amounts of CPU and memory resources, while the resulting exported data occupies lower amounts of disk space.
@ -775,7 +780,7 @@ The maximum duration for each request to `/api/v1/export` is limited by `-search
Exported data can be imported via POST'ing it to [/api/v1/import](#how-to-import-data-in-json-line-format).
#### How to export CSV data
### How to export CSV data
Send a request to `http://<victoriametrics-addr>:8428/api/v1/export/csv?format=<format>&match=<timeseries_selector_for_export>`,
where:
@ -799,7 +804,7 @@ unix timestamp in seconds or [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) val
The exported CSV data can be imported to VictoriaMetrics via [/api/v1/import/csv](#how-to-import-csv-data).
### How to import time series data
## How to import time series data
Time series data can be imported via any supported ingestion protocol:
@ -816,7 +821,7 @@ Time series data can be imported via any supported ingestion protocol:
* `/api/v1/import/prometheus` for importing data in Prometheus exposition format. See [these docs](#how-to-import-data-in-prometheus-exposition-format) for details.
#### How to import data in native format
### How to import data in native format
The most efficient protocol for importing data into VictoriaMetrics is `/api/v1/import/native`.
Example for importing data obtained via [/api/v1/export/native](#how-to-export-data-in-native-format):
@ -845,7 +850,7 @@ For example, `/api/v1/import/native?extra_label=foo=bar` would add `"foo":"bar"`
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import data in JSON line format
### How to import data in JSON line format
Example for importing data obtained via [/api/v1/export](#how-to-export-data-in-json-line-format):
@ -873,7 +878,7 @@ For example, `/api/v1/import?extra_label=foo=bar` would add `"foo":"bar"` label
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import CSV data
### How to import CSV data
Arbitrary CSV data can be imported via `/api/v1/import/csv`. The CSV data is imported according to the provided `format` query arg.
The `format` query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:
@ -926,7 +931,7 @@ For example, `/api/v1/import/csv?extra_label=foo=bar` would add `"foo":"bar"` la
Note that it could be required to flush response cache after importing historical data. See [these docs](#backfilling) for detail.
#### How to import data in Prometheus exposition format
### How to import data in Prometheus exposition format
VictoriaMetrics accepts data in [Prometheus exposition format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
via `/api/v1/import/prometheus` path. For example, the following line imports a single line in Prometheus exposition format into VictoriaMetrics:
@ -961,7 +966,7 @@ VictoriaMetrics also may scrape Prometheus targets - see [these docs](#how-to-sc
### Relabeling
## Relabeling
VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if `-relabelConfig` command-line flag points
to a file containing a list of [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) entries.
@ -985,7 +990,7 @@ VictoriaMetrics provides the following extra actions for relabeling rules:
See also [relabeling in vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling).
### Federation
## Federation
VictoriaMetrics exports [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/)
at `http://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>`.
@ -996,7 +1001,7 @@ on the interval `[now - max_lookback ... now]` is scraped for each time series.
For instance, `/federate?match[]=up&max_lookback=1h` would return last points on the `[now - 1h ... now]` interval. This may be useful for time series federation
with scrape intervals exceeding `5m`.
### Capacity planning
## Capacity planning
A rough estimation of the required resources for ingestion path:
@ -1036,7 +1041,8 @@ The required resources for query path:
* Network usage: depends on the frequency and the type of incoming requests. Typical Grafana dashboards usually
require negligible network bandwidth.
### High availability
## High availability
* Install multiple VictoriaMetrics instances in distinct datacenters (availability zones).
* Pass addresses of these instances to [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md) via `-remoteWrite.url` command-line flag:
@ -1077,7 +1083,7 @@ Another option is to write data simultaneously from Prometheus HA pair to a pair
with the enabled de-duplication. See [this section](#deduplication) for details.
### Deduplication
## Deduplication
VictoriaMetrics de-duplicates data points if `-dedup.minScrapeInterval` command-line flag
is set to positive duration. For example, `-dedup.minScrapeInterval=60s` would de-duplicate data points
@ -1088,7 +1094,7 @@ write data to the same VictoriaMetrics instance. Note that these Prometheus inst
`external_labels` section in their configs, so they write data to the same time series.
### Retention
## Retention
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
that the data will be stored for 3 months and then deleted.
@ -1103,7 +1109,7 @@ VictoriaMetrics supports retention smaller than 1 month. For example, `-retentio
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
### Multiple retentions
## Multiple retentions
Just start multiple VictoriaMetrics instances with distinct values for the following flags:
@ -1116,7 +1122,7 @@ so it could route requests from particular user to VictoriaMetrics with the desi
The same scheme could be implemented for multiple tenants in [VictoriaMetrics cluster](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md).
### Downsampling
## Downsampling
There is no downsampling support at the moment, but:
@ -1132,11 +1138,13 @@ It is possible to (ab)use [-dedup.minScrapeInterval](#deduplication) for basic d
For instance, if interval between the ingested data points is 15s, then `-dedup.minScrapeInterval=5m` will leave
only a single data point out of 20 initial data points per each 5m interval.
### Multi-tenancy
## Multi-tenancy
Single-node VictoriaMetrics doesn't support multi-tenancy. Use [cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster) instead.
### Scalability and cluster version
## Scalability and cluster version
Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU.
This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutions
@ -1146,7 +1154,8 @@ So try single-node VictoriaMetrics at first and then [switch to cluster version]
horizontally scalable long-term remote storage for really large Prometheus deployments.
[Contact us](mailto:info@victoriametrics.com) for paid support.
### Alerting
## Alerting
It is recommended using [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md) for alerting.
@ -1157,7 +1166,7 @@ Additionally, alerting can be set up with the following tools:
* With Grafana - see [the corresponding docs](https://grafana.com/docs/alerting/rules/).
### Security
## Security
Do not forget protecting sensitive endpoints in VictoriaMetrics when exposing it to untrusted networks such as the internet.
Consider setting the following command-line flags:
@ -1177,7 +1186,7 @@ Prefer authorizing all the incoming requests from untrusted networks with [vmaut
or similar auth proxy.
### Tuning
## Tuning
* There is no need for VictoriaMetrics tuning since it uses reasonable defaults for command-line flags,
which are automatically adjusted for the available CPU and RAM resources.
@ -1193,7 +1202,7 @@ or similar auth proxy.
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge
```
### Monitoring
## Monitoring
VictoriaMetrics exports internal metrics in Prometheus format at `/metrics` page.
These metrics may be collected by [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md)
@ -1223,7 +1232,7 @@ The most interesting metrics are:
VictoriaMetrics also exposes currently running queries with their execution times at `/api/v1/status/active_queries` page.
### Troubleshooting
## Troubleshooting
* It is recommended to use default command-line flag values (i.e. don't set them explicitly) until the need
of tweaking these flag values arises.
@ -1236,7 +1245,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
* VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.
This may lead to the following "issues":
* Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to persistent storage
by requesting `/internal/force_flush` http handler.
by requesting `/internal/force_flush` http handler. This handler is mostly needed for testing and debugging purposes.
* The last few seconds of inserted data may be lost on unclean shutdown (i.e. OOM, `kill -9` or hardware reset).
See [this article for technical details](https://valyala.medium.com/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
@ -1291,7 +1300,7 @@ VictoriaMetrics also exposes currently running queries with their execution time
* VictoriaMetrics ignores `NaN` values during data ingestion.
### Backfilling
## Backfilling
VictoriaMetrics accepts historical data in arbitrary order of time via [any supported ingestion method](#how-to-import-time-series-data).
Make sure that configured `-retentionPeriod` covers timestamps for the backfilled data.
@ -1307,7 +1316,7 @@ Yet another solution is to increase `-search.cacheTimestampOffset` flag value in
for data with timestamps close to the current time.
### Data updates
## Data updates
VictoriaMetrics doesn't support updating already existing sample values to new ones. It stores all the ingested data points
for the same time series with identical timestamps. While is possible substituting old time series with new time series via
@ -1315,7 +1324,7 @@ for the same time series with identical timestamps. While is possible substituti
should be used only for one-off updates. It shouldn't be used for frequent updates because of non-zero overhead related to data removal.
### Replication
## Replication
Single-node VictoriaMetrics doesn't support application-level replication. Use cluster version instead.
See [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cluster/README.md#replication-and-data-safety) for details.
@ -1325,14 +1334,14 @@ Storage-level replication may be offloaded to durable persistent storage such as
See also [high availability docs](#high-availability) and [backup docs](#backups).
### Backups
## Backups
VictoriaMetrics supports backups via [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md)
and [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md) tools.
We also provide provide `vmbackuper` tool for paid enterprise subscribers - see [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/466) for details.
### Profiling
## Profiling
VictoriaMetrics provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -1352,6 +1361,7 @@ The command for collecting CPU profile waits for 30 seconds before returning.
The collected profiles may be analyzed with [go tool pprof](https://github.com/google/pprof).
## Integrations
* [Helm charts for single-node and cluster versions of VictoriaMetrics](https://github.com/VictoriaMetrics/helm-charts).
@ -1363,6 +1373,8 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
See [this example](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml).
* [Ansible role for installing single-node VictoriaMetrics](https://github.com/dreamteam-gg/ansible-victoriametrics-role).
* [Ansible role for installing cluster VictoriaMetrics](https://github.com/Slapper/ansible-victoriametrics-cluster-role).
* [Snap package for VictoriaMetrics](https://snapcraft.io/victoriametrics).
## Third-party contributions
@ -1371,10 +1383,12 @@ The collected profiles may be analyzed with [go tool pprof](https://github.com/g
* [Prometheus -> VictoriaMetrics exporter #2](https://github.com/AnchorFree/tsdb-remote-write)
* [Prometheus Oauth proxy](https://gitlab.com/optima_public/prometheus_oauth_proxy) - see [this article](https://medium.com/@richard.holly/powerful-saas-solution-for-detection-metrics-c67b9208d362) for details.
## Contacts
Contact us with any questions regarding VictoriaMetrics at [info@victoriametrics.com](mailto:info@victoriametrics.com).
## Community and contributions
Feel free asking any questions regarding VictoriaMetrics:

91
lib/logger/logger.go
View file

@ -10,7 +10,6 @@ import (
"runtime"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
@ -25,6 +24,8 @@ var (
errorsPerSecondLimit = flag.Int("loggerErrorsPerSecondLimit", 10, "Per-second limit on the number of ERROR messages. If more than the given number of errors "+
"are emitted per second, then the remaining errors are suppressed. Zero value disables the rate limit")
warnsPerSecondLimit = flag.Int("loggerWarnsPerSecondLimit", 10, "Per-second limit on the number of WARN messages. If more than the given number of warns "+
"are emitted per second, then the remaining warns are suppressed. Zero value disables the rate limit")
)
// Init initializes the logger.
@ -36,7 +37,7 @@ func Init() {
setLoggerOutput()
validateLoggerLevel()
validateLoggerFormat()
go errorsLoggedCleaner()
go logLimiterCleaner()
logAllFlags()
}
@ -125,14 +126,60 @@ func logLevelSkipframes(skipframes int, level, format string, args ...interface{
logMessage(level, msg, 3+skipframes)
}
func errorsLoggedCleaner() {
func logLimiterCleaner() {
for {
time.Sleep(time.Second)
atomic.StoreUint64(&errorsLogged, 0)
logLimiter.reset()
}
}
var errorsLogged uint64
var logLimiter = newLogLimit()
func newLogLimit() *logLimit {
return &logLimit{
m: make(map[string]uint64),
}
}
type logLimit struct {
mu sync.Mutex
m map[string]uint64
}
func (ll *logLimit) reset() {
ll.mu.Lock()
ll.m = make(map[string]uint64, len(ll.m))
ll.mu.Unlock()
}
// needSuppress checks if the number of calls for the given location exceeds the given limit.
//
// When the number of calls equals limit, log message prefix returned.
func (ll *logLimit) needSuppress(location string, limit uint64) (bool, string) {
// fast path
var msg string
if limit == 0 {
return false, msg
}
ll.mu.Lock()
defer ll.mu.Unlock()
if n, ok := ll.m[location]; ok {
if n >= limit {
switch n {
// report only once
case limit:
msg = fmt.Sprintf("suppressing log message with rate limit=%d: ", limit)
default:
return true, msg
}
}
ll.m[location] = n + 1
} else {
ll.m[location] = 1
}
return false, msg
}
type logWriter struct {
}
@ -143,13 +190,6 @@ func (lw *logWriter) Write(p []byte) (int, error) {
}
func logMessage(level, msg string, skipframes int) {
// rate limit ERROR log messages
if level == "ERROR" {
if n := atomic.AddUint64(&errorsLogged, 1); *errorsPerSecondLimit > 0 && n > uint64(*errorsPerSecondLimit) {
return
}
}
timestamp := ""
if !*disableTimestamps {
timestamp = time.Now().UTC().Format("2006-01-02T15:04:05.000Z")
@ -164,23 +204,39 @@ func logMessage(level, msg string, skipframes int) {
// Strip /VictoriaMetrics/ prefix
file = file[n+len("/VictoriaMetrics/"):]
}
location := fmt.Sprintf("%s:%d", file, line)
// rate limit ERROR and WARN log messages with given limit.
if level == "ERROR" || level == "WARN" {
limit := uint64(*errorsPerSecondLimit)
if level == "WARN" {
limit = uint64(*warnsPerSecondLimit)
}
ok, suppressMessage := logLimiter.needSuppress(location, limit)
if ok {
return
}
if len(suppressMessage) > 0 {
msg = suppressMessage + msg
}
}
for len(msg) > 0 && msg[len(msg)-1] == '\n' {
msg = msg[:len(msg)-1]
}
var logMsg string
switch *loggerFormat {
case "json":
caller := fmt.Sprintf("%s:%d", file, line)
if *disableTimestamps {
logMsg = fmt.Sprintf(`{"level":%q,"caller":%q,"msg":%q}`+"\n", levelLowercase, caller, msg)
logMsg = fmt.Sprintf(`{"level":%q,"caller":%q,"msg":%q}`+"\n", levelLowercase, location, msg)
} else {
logMsg = fmt.Sprintf(`{"ts":%q,"level":%q,"caller":%q,"msg":%q}`+"\n", timestamp, levelLowercase, caller, msg)
logMsg = fmt.Sprintf(`{"ts":%q,"level":%q,"caller":%q,"msg":%q}`+"\n", timestamp, levelLowercase, location, msg)
}
default:
if *disableTimestamps {
logMsg = fmt.Sprintf("%s\t%s:%d\t%s\n", levelLowercase, file, line, msg)
logMsg = fmt.Sprintf("%s\t%s\t%s\n", levelLowercase, location, msg)
} else {
logMsg = fmt.Sprintf("%s\t%s\t%s:%d\t%s\n", timestamp, levelLowercase, file, line, msg)
logMsg = fmt.Sprintf("%s\t%s\t%s\t%s\n", timestamp, levelLowercase, location, msg)
}
}
@ -190,7 +246,6 @@ func logMessage(level, msg string, skipframes int) {
mu.Unlock()
// Increment vm_log_messages_total
location := fmt.Sprintf("%s:%d", file, line)
counterName := fmt.Sprintf(`vm_log_messages_total{app_version=%q, level=%q, location=%q}`, buildinfo.Version, levelLowercase, location)
metrics.GetOrCreateCounter(counterName).Inc()

5
ports/OpenBSD/VictoriaMetrics/Makefile
View file

@ -1,4 +1,4 @@
# $OpenBSD$
# $OpenBSD: Makefile,v $
COMMENT = fast, cost-effective and scalable time series database
@ -21,17 +21,18 @@ USE_GMAKE = Yes
MODULES = lang/go
MODGO_GOPATH = ${MODGO_WORKSPACE}
SUBST_VARS = VARBASE
do-build:
cd ${WRKSRC} && GOOS=openbsd ${MAKE_ENV} ${MAKE_PROGRAM} victoria-metrics-pure
cd ${WRKSRC} && GOOS=openbsd ${MAKE_ENV} ${MAKE_PROGRAM} vmbackup
cd ${WRKSRC} && GOOS=openbsd ${MAKE_ENV} ${MAKE_PROGRAM} vmrestore
do-install:
${INSTALL_PROGRAM} ./pkg/vmlogger.pl ${PREFIX}/bin/vmetricslogger.pl
${INSTALL_PROGRAM} ${WRKSRC}/bin/victoria-metrics-pure ${PREFIX}/bin/vmetrics
${INSTALL_PROGRAM} ${WRKSRC}/bin/vmbackup ${PREFIX}/bin/vmetricsbackup
${INSTALL_PROGRAM} ${WRKSRC}/bin/vmrestore ${PREFIX}/bin/vmetricsrestore
${INSTALL_DATA_DIR} ${PREFIX}/share/doc/vmetrics/
${INSTALL_DATA} ${WRKSRC}/README.md ${PREFIX}/share/doc/vmetrics/
${INSTALL_DATA} ${WRKSRC}/LICENSE ${PREFIX}/share/doc/vmetrics/

1
ports/OpenBSD/VictoriaMetrics/pkg/PLIST
View file

@ -5,6 +5,7 @@
@rcscript ${RCDIR}/vmetrics
@bin bin/vmetrics
@bin bin/vmetricsbackup
@bin bin/vmetricsrestore
@bin bin/vmetricslogger.pl
share/doc/vmetrics/
share/doc/vmetrics/Articles.md

6
ports/OpenBSD/VictoriaMetrics/pkg/vmetrics.rc
View file

@ -8,12 +8,14 @@ daemon_user=_vmetrics
. /etc/rc.d/rc.subr
pexp="${daemon}.*"
pexp="${daemon} -loggerDisableTimestamps ${daemon_flags}.*"
rc_bg=YES
rc_reload=NO
me=`basename $0`
rc_start() {
${rcexec} "${daemon} -loggerDisableTimestamps ${daemon_flags} < /dev/null 2>&1 | ${TRUEPREFIX}/bin/vmetricslogger.pl"
${rcexec} "${daemon} -loggerDisableTimestamps ${daemon_flags} < /dev/null 2>&1 | ${TRUEPREFIX}/bin/vmetricslogger.pl $me"
}
rc_cmd $1

26
ports/OpenBSD/VictoriaMetrics/pkg/vmlogger.pl
View file

@ -1,18 +1,22 @@
#!/usr/bin/perl
use Sys::Syslog qw(:standard :macros);
openlog("victoria-metrics", "pid", "daemon");
openlog($ARGV[0], "pid", "daemon");
while (my $l = <>) {
my @d = split /\t/, $l;
# go level : "INFO", "WARN", "ERROR", "FATAL", "PANIC":
my $lvl = $d[0];
$lvl = LOG_EMERG if ($lvl eq 'panic');
$lvl = 'crit' if ($lvl eq 'fatal');
$lvl = 'err' if ($lvl eq 'error');
$lvl = 'warning' if ($lvl eq 'warn');
chomp $d[2];
syslog( $lvl, $d[2] );
my %lvl_map = (
'panic' => LOG_EMERG,
'fatal' => LOG_CRIT,
'error' => LOG_ERR,
'warn' => LOG_WARNING,
'info' => LOG_INFO,
);
while (my $l = <STDIN>) {
my ($lvl, undef, $message) = split /\t/, $_, 3;
next unless $message;
$lvl = $lvl_map{ lc $lvl } || LOG_WARNING;
chomp $message;
syslog( $lvl, $message );
}
closelog();

57
snap/local/README.md Normal file
View file

@ -0,0 +1,57 @@
### Snap integration
https://snapcraft.io/
snap link: https://snapcraft.io/victoriametrics
#### develop
Install snapcraft and multipass:
```text
snap install snapcraft
snap install multipass
```
build victoria-metrics prod binary and run snapcraft ```snapcraft --debug```.
It produces snap package with current git version - `victoriametrics_v1.46.0+git1.1bebd021a-dirty_all.snap`.
You can install it with command: `snap install victoriametrics_v1.46.0+git1.1bebd021a-dirty_all.snap --dangerous`
package and publish:
```text
make victoria-metrics-package-snap
snap login
snap_file=$(ls | grep -e '_multi.snap$')
snapcraft upload --release=latest/edge "$snap_file"
rm "$snap_file"
```
#### usage
installation and configuration:
```text
# install
snap install victoriametrics
# logs
snap logs victoriametrics
# restart
snap restart victoriametrics
```
Configuration management:
Prometheus scrape config can be edited with your favorite editor, its located at
```text
vi /var/snap/victoriametrics/current/etc/victoriametrics-scrape-config.yaml
```
after changes, you can trigger config reread with `curl localhost:8248/-/reload`.
Configuration tuning is possible with editing extra_flags:
```text
echo 'FLAGS="-selfScrapeInterval=10s -search.logSlowQueryDuration=20s"' > /var/snap/victoriametrics/current/extra_flags
snap restart victoriametrics
```
Data folder located at `/var/snap/victoriametrics/current/var/lib/victoriametrics/`

7
snap/local/victoriametrics-wrapper Executable file
View file

@ -0,0 +1,7 @@
#!/bin/bash
test -e $SNAP_DATA/extra_flags || echo 'FLAGS=""' > $SNAP_DATA/extra_flags
# get args
. $SNAP_DATA/extra_flags
$SNAP/bin/victoria-metrics -storageDataPath /var/lib/victoriametrics --promscrape.config /etc/victoriametrics-scrape-config.yaml $FLAGS

83
snap/snapcraft.yaml Normal file
View file

@ -0,0 +1,83 @@
name: victoriametrics
base: core18
version: git
icon: logo.png
summary: VictoriaMetrics is fast, cost-effective and scalable time-series database.
description: |
* VictoriaMetrics can be used as long-term storage for Prometheus or for vmagent.
See [these docs](#prometheus-setup) for details.
* Supports Prometheus querying API, so it can be used as Prometheus drop-in replacement in Grafana.
VictoriaMetrics implements MetricsQL, https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL query language, which is inspired by PromQL.
* Supports global query view. Multiple Prometheus instances may write data into VictoriaMetrics. Later this data may be used in a single query.
* High performance and good scalability for both inserts and selects.
* High data compression, so up to 70x more data points may be crammed into limited storage comparing to TimescaleDB.
* Optimized for storage with high-latency IO and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc).
* A single-node VictoriaMetrics may substitute moderately sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB.
* Easy operation:
* VictoriaMetrics consists of a single without external dependencies.
* All the configuration is done via explicit command-line flags with reasonable defaults.
* All the data is stored in a single directory pointed by `-storageDataPath` flag.
* Easy and fast backups from
to S3 or GCS with [vmbackup] / [vmrestore].
* Storage is protected from corruption on unclean shutdown (i.e. OOM, hardware reset or `kill -9`) thanks to the storage architecture.
* Supports metrics' scraping, ingestion and backfilling via the following protocols:
* [Metrics from Prometheus exporters]
such as node_exporter.
* Prometheus remote write API(
* InfluxDB line protocol over HTTP, TCP and UDP.
* Graphite plaintext protocol
if `-graphiteListenAddr` is set.
* OpenTSDB put message, if `-opentsdbListenAddr` is set.
* HTTP OpenTSDB /api/put requests, if `-opentsdbHTTPListenAddr` is set.
* JSON line format.
* Native binary format.
* Prometheus exposition format.
* Arbitrary CSV data.
* Supports metrics' relabeling. See [these docs](#relabeling) for details.
* Ideally works with big amounts of time series data from Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
* Has open source cluster version (https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
** Configuration management**
* Prometheus scrape config can be edited with your favorite editor, its located at
`vi /var/snap/victoriametrics/current/etc/victoriametrics-scrape-config.yaml`
after changes, you can trigger config reread with `curl localhost:8248/-/reload`.
* Configuration tuning is possible with editing extra_flags:
`echo 'FLAGS="-selfScrapeInterval=10s -search.logSlowQueryDuration=20s"' > /var/snap/victoriametrics/current/extra_flags && snap restart victoriametrics`
* Data folder located at `/var/snap/victoriametrics/current/var/lib/victoriametrics/`.
grade: stable # must be 'stable' to release into candidate/stable channels
confinement: strict # use 'strict' once you have the right plugs and slots
parts:
build:
plugin: go
go-channel: 1.15/stable
go-importpath: github.com/VictoriaMetrics/VictoriaMetrics
source: .
source-type: local
build-packages: [build-essential, make]
override-build: |
make app-local APP_NAME=victoria-metrics
mkdir -p $SNAPCRAFT_PART_INSTALL/bin
cp -p $SNAPCRAFT_PART_BUILD/bin/victoria-metrics $SNAPCRAFT_PART_INSTALL/bin/
cp -p $SNAPCRAFT_PART_SRC/snap/local/victoriametrics-wrapper $SNAPCRAFT_PART_INSTALL/bin/
apps:
victoriametrics:
command: bin/victoriametrics-wrapper
daemon: simple
restart-condition: always
restart-delay: 10s
plugs: [network,network-bind]
plugs:
network:
network-bind:
layout:
/var/lib/victoriametrics:
bind: $SNAP_DATA/var/lib/victoriametrics
/etc/victoriametrics-scrape-config.yaml:
bind-file: $SNAP_DATA/etc/victoriametrics-scrape-config.yaml
architectures:
- build-on: ['arm64', 'amd64']
run-on: ['arm64','amd64']