docs: add vmagent and alloy data ingestion docs (#6678)

Adds Prometheus Grafana Alloy and vmagent to the data ingestion
protocols. Grafana Agent was not added since it has been deprecated in
favor of alloy

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
This commit is contained in:
Mathias Palmersheim 2024-08-06 08:29:42 -05:00 committed by GitHub
parent f283126084
commit a46d554f74
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 368 additions and 30 deletions

View file

@ -0,0 +1,222 @@
---
title: Grafana Alloy
weight: 3
sort: 3
menu:
docs:
identifier: "alloy"
parent: "data-ingestion"
weight: 3
aliases:
- /data-ingestion/Grafana-Alloy.html
- /data-ingestion/grafana-alloy.html
- /data-ingestion/Grafana-Agent.html
- /data-ingestion/grafana-agent.html
---
[Grafana Alloy](https://grafana.com/docs/alloy/latest/) supports sending data via the Prometheus remote write protocol and OpenTelemetry Protocol (OTLP).
Collecting metrics and forwarding them to VictoriaMetrics using Prometheus scraping and remote writing is more straightforward, but using OpenTelemetry enables more complex processing operations to occur before sending data to VictoriaMetrics.
The Alloy configuration file can be found in the following location depending on your platform:
- Linux: `/etc/alloy/config.alloy`
- Windows: `%ProgramFiles%\GrafanaLabs\Alloy\config.alloy`
- MacOS: `$(brew --prefix)/etc/alloy/config.alloy`
To configure Grafana Alloy to push collected metrics to VictoriaMetrics via Prometheus remote write protocol,
update the `prometheus.remote_write` endpoint:
```Alloy
prometheus.remote_write "victoriametrics" {
endpoint {
url = "https://<victoriametrics-addr>/prometheus/api/v1/write"
}
}
```
For pushing data to VictoriaMetrics cluster the `url` should point to vminsert and include
the [tenantID](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format):
```sh
https://<vminsert-addr>/insert/<tenant_id>/prometheus/api/v1/write
```
> Note: read more about [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
> or [multitenancy via labels](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels).
After the configuration has been updated, Alloy must be reloaded or restarted for the change to be applied:
- Linux: `sudo systemctl reload alloy.service`
- Windows: `Restart-Service Alloy`, this can also be done from the GUI using task manager
- MacOS: `brew services restart alloy`
- Helm chart: changing the `alloy.configMap` in the Alloy [helm values](https://raw.githubusercontent.com/grafana/alloy/main/operations/helm/charts/alloy/values.yaml)
## remote write
In the example below we will be using the node exporter component built into Alloy to generate metrics,
but any Prometheus scrape target can forward data to VictoriaMetrics.
Metrics are forwarded from the scrape target to VictoriaMetrics by creating a `prometheus.remote_write` component
and configuring the `promethues.scrape` component to forward metrics to the `prometheus.remote_write` component.
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [prometheus.remote_write.victoriametrics.receiver]
}
prometheus.remote_write "victoriametrics" {
endpoint {
url = "https://<victoriametrics-addr>/prometheus/api/v1/write"
}
}
```
## remote write with basic authentication
This is the same as the previous configuration but adds the `basic_auth` parameters:
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [prometheus.remote_write.victoriametrics.receiver]
}
prometheus.remote_write "victoriametrics" {
endpoint {
url = "https://<victoriametrics-addr>/prometheus/api/v1/write"
basic_auth {
username = "<victoriametrics_user>"
password = "<victoriametrics_password>"
}
}
}
```
## remote write with bearer authentication
This is the same as the first config but adds the `bearer_token` parameter:
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [prometheus.remote_write.victoriametrics.receiver]
}
prometheus.remote_write "victoriametrics" {
endpoint {
url = "https://<victoriametrics-addr>/prometheus/api/v1/write"
bearer_token = "<token>"
}
}
```
## OpenTelemetry
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [otelcol.receiver.prometheus.victoriametrics.receiver]
}
otelcol.receiver.prometheus "victoriametrics" {
output {
metrics = [otelcol.processor.batch.batch.input]
}
}
otelcol.processor.batch "batch" {
output {
metrics = [otelcol.exporter.otlphttp.victoriametrics.input]
}
}
otelcol.exporter.otlphttp "victoriametrics" {
client {
endpoint = "http://<victoriametrics-addr>:<victoriametrics_port>/opentelemetry"
}
}
```
## OpenTelemetry with Basic Authentication
This is the same configuration without authentication but contains the `otelcol.auth.basic` block
and references it in `otelcol.exporter.otlphttp`:
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [otelcol.receiver.prometheus.victoriametrics.receiver]
}
otelcol.auth.basic "otel_auth" {
username = "<user>"
password = "<password>"
}
otelcol.receiver.prometheus "victoriametrics" {
output {
metrics = [otelcol.processor.batch.batch.input]
}
}
otelcol.processor.batch "batch" {
output {
metrics = [otelcol.exporter.otlphttp.victoriametrics.input]
}
}
otelcol.exporter.otlphttp "victoriametrics" {
client {
endpoint = "https://<victoriametrics-addr:<victoriametrics_port>/opentelemetry"
auth = otelcol.auth.basic.otel_auth.handler
}
}
```
## OpenTelemetry with Bearer Authentication
This is the same as the basic authentication configuration but swaps the `otelcol.auth.basic` for `otelcol.auth.bearer`:
```Alloy
prometheus.exporter.unix "nodeexporter" {}
prometheus.scrape "nodeexporter" {
targets = prometheus.exporter.unix.nodeexporter.targets
forward_to = [otelcol.receiver.prometheus.victoriametrics.receiver]
}
otelcol.auth.bearer "otel_auth" {
token = "<token>"
}
otelcol.receiver.prometheus "victoriametrics" {
output {
metrics = [otelcol.processor.batch.batch.input]
}
}
otelcol.processor.batch "batch" {
output {
metrics = [otelcol.exporter.otlphttp.victoriametrics.input]
}
}
otelcol.exporter.otlphttp "victoriametrics" {
client {
endpoint = "https://<victoriametrics-addr>:<victoriametrics_port>/opentelemetry"
auth = otelcol.auth.bearer.otel_auth.handler
}
}
```
## References
- [Grafana Alloy Helm Chart](https://github.com/grafana/alloy/tree/main/operations/helm)
- [Grafana Alloy Documenation](https://grafana.com/docs/alloy/latest)

View file

@ -0,0 +1,22 @@
---
title: Prometheus
weight: 1
sort: 1
menu:
docs:
identifier: "prometheus"
parent: "data-ingestion"
weight: 1
aliases:
- /data-ingestion/prometheus.html
- /data-ingestion/Prometheus.html
---
VictoriaMetrics supports ingesting data from Prometheus via the Prometheus remote write protocol.
The documentation for pushing data from Prometheus to VictoriaMetrics is located [here](https://docs.victoriametrics.com/#prometheus-setup)
## References
- [Prometheus configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)
- [Prometheus Helm chart values](https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml)
- [Prometheus Feature Flags](https://prometheus.io/docs/prometheus/latest/feature_flags/#prometheus-agent)

View file

@ -1,13 +1,12 @@
---
title: Proxmox
weight: 1
sort: 1
weight: 6
sort: 6
menu:
docs:
identifier: "proxmox"
parent: "data-ingestion"
weight: 1
# sort: 1
weight: 6
aliases:
- /data-ingestion/proxmox.html
- /data-ingestion/Proxmox.html
@ -28,8 +27,8 @@ If want help Sending your data to VictoriaMetrics Cloud check out [our blog](htt
- Server: the hostname or IP of your VictoriaMetrics Instance
- Port: This will vary depending how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html)
- Protocol: use HTTPS if you have TLS/SSL configured otherwise use HTTP
- Organization: leave empty since it doesn't get used
- Bucket: leave empty since it doesn't get used
- Organization: leave it empty since it doesn't get used
- Bucket: leave it empty since it doesn't get used
- Token: your token from vmauth or leave blank if you don't have authentication enabled
- If you need to ignore TLS/SSL errors check the advanced box and uncheck the verify certificate box
4. Click the `Create` button
@ -50,10 +49,10 @@ You should see 1 time series per node in your PVE cluster.
3. Set the parameters as follows:
- Name: VictoriaMetrics (can be set to any string)
- URL: http(s)://<ip_or_host>:<port>
- set the URL to https if you have TLS enabled and http if you do not
- Port: This will vary depending how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html)
- Organization: leave empty since it doesn't get used
- Bucket: leave empty since it doesn't get used
- set the URL to HTTPS if you have TLS enabled and HTTP if you do not
- Port: This will vary depending on how you are sending data to VictoriaMetrics, but the defaults for all components are listed in the [data ingestion documentation](https://docs.victoriametrics.com/data-ingestion.html)
- Organization: leave it empty since it doesn't get used
- Bucket: leave it empty since it doesn't get used
- Token: your token from vmauth or leave blank if you don't have authentication enabled
4. Click the `Create` button

View file

@ -6,14 +6,22 @@ If you are unsure what port number to use when pushing data to VictoriaMetrics s
- VictoriaMetrics Single: 8428
- vmagent: 8429
- vmauth: 8427
- vminsert: 8482
- vminsert: 8480
In the rest of the documentation we will assume you have configured your push endpoint to use TLS/SSL on port 443 so the urls in the rest of the documentation will look like `https://<victoriametrics_url>` instead of `http://<victoriametrics_url>:8428` for VictoriaMetrics single.
## Documented Collectors/Agents
* [Telegraf]({{< relref "Telegraf.md" >}})
* [Vector]({{< relref "Vector.md" >}})
- [Telegraf](https://docs.victoriametrics.com/data-ingestion/telegraf/)
- [Vector](https://docs.victoriametrics.com/data-ingestion/vector/)
- [vmagent](https://docs.victoriametrics.com/data-ingestion/vmagent)
- [Grafana Alloy](https://docs.victoriametrics.com/data-ingestion/alloy/)
- [Prometheus](https://docs.victoriametrics.com/data-ingestion/prometheus/)
## Supported Platforms
* [Proxmox Virtual Environment and Proxmox Backup Server]({{< relref "Proxmox.md" >}})
- [Proxmox Virtual Environment and Proxmox Backup Server](https://docs.victoriametrics.com/data-ingestion/proxmox/)

View file

@ -1,13 +1,12 @@
---
title: Telegraf
weight: 1
sort: 1
weight: 5
sort: 5
menu:
docs:
identifier: "telegraf"
parent: "data-ingestion"
weight: 1
# sort: 1
weight: 5
aliases:
- /data-ingestion/telegraf.html
- /data-ingestion/Telegraf.html
@ -22,10 +21,10 @@ victoriametrics_user="telegraf"
victoriametrics_password="password"
victoriametrics_token="my_token"
```
and they can be referenced in a Telegraf configuration file by prepending the variable name with `$` ex. `$victoriametrics_url` will be translated to `https://metrics.example.com` if it referenced in a Telegraf configuration using the values from `/etc/default/telegraf` in the values seen above.
Otherwise please replace the variables below to fit your setup.
and they can be referenced in a Telegraf configuration file by prepending the variable name with `$` ex. `$victoriametrics_url` will be translated to `https://metrics.example.com` if it is referenced in a Telegraf configuration using the values from `/etc/default/telegraf` in the values seen above.
Otherwise, please replace the variables below to fit your setup.
If you want to mimic this behavior on windows please read [Influx Data's blog on storing variables in the registry](https://www.influxdata.com/blog/using-telegraf-on-windows/)
If you want to mimic this behavior on Windows please read [Influx Data's blog on storing variables in the registry](https://www.influxdata.com/blog/using-telegraf-on-windows/)
## Minimum Configuration with no Authentication
```toml
@ -67,7 +66,7 @@ This is the same as the minimum configuration but adds the authorization header
```
## Route certain metrics
If you only want to route certain metrics to VictoriaMetrics use the `namepass` option with a comma separated listed of the measurements you wish to send to VictoriaMetrics.
If you only want to route certain metrics to VictoriaMetrics use the `namepass` option with a comma separated list of the measurements you wish to send to VictoriaMetrics.
```
[[outputs.influxdb]]
@ -100,5 +99,5 @@ This is not recommended since it can allow sending metrics to a compromised site
# References
- [Install Telegraf](https://docs.influxdata.com/telegraf/v1/install/)
- [InfluxDBv1 output for Telegraf](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/influxdb)
- [Storing Telegraf variables in the windows registry](https://www.influxdata.com/blog/using-telegraf-on-windows/)
- [Storing Telegraf variables in the Windows registry](https://www.influxdata.com/blog/using-telegraf-on-windows/)
- [Telegraf variables](https://docs.influxdata.com/telegraf/v1/configuration/#example-telegraf-environment-variables)

View file

@ -1,19 +1,18 @@
---
title: Vector
weight: 1
sort: 1
weight: 4
sort: 4
menu:
docs:
identifier: "Vector"
parent: "data-ingestion"
weight: 1
# sort: 1
weight: 4
aliases:
- /data-ingestion/Vector.html
- /data-ingestion/vector.html
---
To Send data to Vector you need to configure with a Prometheus remote write sink and forward metrics to that sink from at least 1 source.
You will need to replace the values in `<>` with your to match your setup.
You will need to replace the values in `<>` to match your setup.
## Minimum Config
```yaml
@ -117,4 +116,4 @@ sinks:
# References
- [Vector documentation](https://vector.dev/docs/)
- [VictoriaLogs documentation for using vector]({{< ref "/victorialogs/data-ingestion/vector" >}})
- [VictoriaLogs documentation for using vector](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector)

View file

@ -0,0 +1,89 @@
---
title: vmagent
weight: 2
sort: 2
menu:
docs:
identifier: "vmagent"
parent: "data-ingestion"
weight: 2
aliases:
- /data-ingestion/vmagent.html
- /data-ingestion/VMAgent.html
---
vmagent can receive data via the same protocols as VictoriaMetrics Single or Cluster versions,
as well as scrape Prometheus endpoints. In other words,
it supports both [Push](https://docs.victoriametrics.com/keyconcepts/#push-model) and [Pull](https://docs.victoriametrics.com/keyconcepts/#pull-model) models.
This section of the documentation only covers forwarding data from vmagent to another destination.
For extra information about vmagent as well as quickstart guide please refer to the [vmagent documentation](https://docs.victoriametrics.com/vmagent/).
To configure vmagent to push metrics to VictoriaMetrics via Prometheus remote write protocol,
configure the `-remoteWrite.url` cmd-line flag:
```sh
/path/to/vmagent -remoteWrite.url=https://<victoriametrics_url>/api/v1/write
```
For pushing data to VictoriaMetrics cluster the `-remoteWrite.url` should point to vminsert and include
the [tenantID](https://docs.victoriametrics.com/cluster-victoriametrics/#url-format):
```sh
/path/to/vmagent -remoteWrite.url=https://<vminsert-addr>/insert/<tenant_id>/prometheus/api/v1/write
```
> Note: read more about [multitenancy](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy)
> or [multitenancy via labels](https://docs.victoriametrics.com/cluster-victoriametrics/#multitenancy-via-labels).
Please note, `-remoteWrite.url` cmd-line flag can be specified multiple times with different values. In this case,
vmagent will [replicate](https://docs.victoriametrics.com/vmagent/#replication-and-high-availability) data to each
specified destination. In addition, it is possible to configure [metrics sharding](https://docs.victoriametrics.com/vmagent/#sharding-among-remote-storages)
across `-remoteWrite.url` destinations.
## Remote write with basic authentication
This requires setting the `-remoteWrite.basicAuth.username` and `-remoteWrite.basicAuth.password` command line flags:
```sh
/path/to/vmagent -remoteWrite.url=https://<victoriametrics_url>/api/v1/write \
-remoteWrite.basicAuth.username=<username> \
-remoteWrite.basicAuth.password=<password>
```
## Remote write with bearer Authentication
```sh
/path/to/vmagent -remoteWrite.url=https://<victoriametrics_url>/api/v1/write \
-remoteWrite.bearerToken=<token>
```
The token can be placed in a file and accessed via the `-remoteWrite.bearerTokenFile` command line argument.
The file needs to be readable by the user vmagent is running as. The token file should only contain the token as seen below:
```
<token>
```
The command to run vmagent with a token file will be the following:
```sh
/path/to/vmagent -remoteWrite.url=https://<victoriametrics_url>/api/v1/write \
-remoteWrite.bearerTokenFile=/path/to/tokenfile
```
## Ignore TLS/SSL errors
If you are using self-signed certificates you can either ignore certificates issues using
the `-remoteWrite.tlsInsecureSkipVerify`, which is a security risk, or use `-remoteWrite.tlsCAFile` to point
to a file containing the self-signed CA certificate:
```sh
/path/to/vmagent -remoteWrite.url=https://<victoriametrics_url>/api/v1/write \
-remoteWrite.tlsInsecureSkipVerify
```
## References
- [vmagent docs](https://docs.victoriametrics.com/vmagent/)
- [vmagent commandline flags](https://docs.victoriametrics.com/vmagent/#advanced-usage)