github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba

Browse source 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
This commit is contained in:
Aliaksandr Valialkin 2022-08-04 22:26:38 +03:00
parent 6e5ac32fba
commit 83a4abda3f
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
11 changed files with 215 additions and 363 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
README.md
View file

@ -310,27 +310,12 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. Currently the following [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) types are supported: VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
* [static_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config)
* [file_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config)
* [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
* [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config)
* [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config)
* [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config)
* [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config)
* [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config)
* [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config)
* [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config)
* [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config)
* [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config)
* [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config)
* [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config)
File a [feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need support for other `*_sd_config` types.
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values.
See [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs.html).
VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format). VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format).
See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus. See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus.

27
app/vmagent/README.md
View file

@ -155,33 +155,8 @@ Use `-remoteWrite.*` command-line flag instead for configuring remote write sett
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values.
The following scrape types in [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) section are supported: See [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs.html).
* `static_configs` is for scraping statically defined targets. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config) for details.
* `file_sd_configs` is for scraping targets defined in external files (aka file-based service discovery). See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config) for details.
* `kubernetes_sd_configs` is for discovering and scraping Kubernetes (K8S) targets. See [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) for details.
* `ec2_sd_configs` is for discovering and scraping Amazon EC2 targets. See [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config) for details. `vmagent` doesn't support the `profile` config param yet.
* `gce_sd_configs` is for discovering and scraping Google Compute Engine (GCE) targets. See [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config) for details. `vmagent` provides the following additional functionality for `gce_sd_config`:
* if `project` arg is missing then `vmagent` uses the project for the instance where it runs;
* if `zone` arg is missing then `vmagent` uses the zone for the instance where it runs;
* if `zone` arg equals to `"*"`, then `vmagent` discovers all the zones for the given project;
* `zone` may contain a list of zones, i.e. `zone: [us-east1-a, us-east1-b]`.
* `azure_sd_configs` - is for scraping the targets registered in Azure Cloud.
See [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config) for details.
* `consul_sd_configs` is for discovering and scraping targets registered in Consul. See [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config) for details.
* `dns_sd_configs` is for discovering and scraping targets from DNS records (SRV, A and AAAA). See [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config) for details.
* `openstack_sd_configs` is for discovering and scraping OpenStack targets. See [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config) for details. [OpenStack identity API v3](https://docs.openstack.org/api-ref/identity/v3/) is supported only.
* `docker_sd_configs` is for discovering and scraping Docker targets. See [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config) for details.
* `dockerswarm_sd_configs` is for discovering and scraping Docker Swarm targets. See [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config) for details.
* `eureka_sd_configs` is for discovering and scraping targets registered in [Netflix Eureka](https://github.com/Netflix/eureka). See [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config) for details.
* `digitalocean_sd_configs` is for discovering and scraping targerts registered in [DigitalOcean](https://www.digitalocean.com/). See [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config) for details.
* `http_sd_configs` is for discovering and scraping targerts provided by external http-based service discovery. See [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) for details.
Note that `vmagent` doesn't support `refresh_interval` option for these scrape configs. Use the corresponding `-promscrape.*CheckInterval`
command-line flag instead. For example, `-promscrape.consulSDCheckInterval=60s` sets `refresh_interval` for all the `consul_sd_configs`
entries to 60s. Run `vmagent -help` in order to see default values for the `-promscrape.*CheckInterval` flags.
Please file feature requests to [our issue tracker](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need other service discovery mechanisms to be supported by `vmagent`.
## scrape_config enhancements ## scrape_config enhancements

1
docs/CHANGELOG.md
View file

@ -21,6 +21,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
* FEATURE: add ability to push internal metrics (e.g. metrics exposed at `/metrics` page) to the configured remote storage from all the VictoriaMetrics components. See [these docs](https://docs.victoriametrics.com/#push-metrics). * FEATURE: add ability to push internal metrics (e.g. metrics exposed at `/metrics` page) to the configured remote storage from all the VictoriaMetrics components. See [these docs](https://docs.victoriametrics.com/#push-metrics).
* FEATURE: improve performance for heavy queries over big number of time series on systems with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896). Thanks to @zqyzyq for [the idea](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b596ac3745314fcc170a14e3ded062971cf7ced2). * FEATURE: improve performance for heavy queries over big number of time series on systems with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896). Thanks to @zqyzyq for [the idea](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/b596ac3745314fcc170a14e3ded062971cf7ced2).
* FEATURE: improve performance for registering new time series in `indexdb` by up to 50%. Thanks to @ahfuzhang for [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249). * FEATURE: improve performance for registering new time series in `indexdb` by up to 50%. Thanks to @ahfuzhang for [the issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add service discovery for [Yandex Cloud](https://cloud.yandex.com/en/). See [these docs](https://docs.victoriametrics.com/sd_configs.html#yandexcloud-sd-configs) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): set `up` metric to `0` for partial scrapes in [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus. * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): set `up` metric to `0` for partial scrapes in [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode). Previously the `up` metric was set to `1` when at least a single metric has been scraped before the error. This aligns the behaviour of `vmselect` with Prometheus.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): restart all the scrape jobs during [config reload](https://docs.victoriametrics.com/vmagent.html#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884). * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): restart all the scrape jobs during [config reload](https://docs.victoriametrics.com/vmagent.html#configuration-update) after `global` section is changed inside `-promscrape.config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884).

24
docs/README.md
View file

@ -310,28 +310,12 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. Currently the following [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) types are supported: VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
* [static_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config)
* [file_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config)
* [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
* [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config)
* [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config)
* [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config)
* [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config)
* [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config)
* [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config)
* [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config)
* [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config)
* [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config)
* [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config)
* [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config)
* [yandexcloud_sd_config](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/sd_configs.md#yandex_cloud_service_discovery_config)
File a [feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need support for other `*_sd_config` types.
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values.
See [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs.html).
VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format). VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format).
See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus. See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus.
@ -2118,6 +2102,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay
-promscrape.suppressScrapeErrorsDelay duration -promscrape.suppressScrapeErrorsDelay duration
The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors
-promscrape.yandexcloudSDCheckInterval duration
Interval for checking for changes in Yandex Cloud. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. (default 30s)
-pushmetrics.extraLabel array -pushmetrics.extraLabel array
Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags. Supports an array of values separated by comma or specified via multiple flags.

23
docs/Single-server-VictoriaMetrics.md
View file

@ -314,27 +314,12 @@ Prometheus doesn't drop data during VictoriaMetrics restart. See [this article](
## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter) ## How to scrape Prometheus exporters such as [node-exporter](https://github.com/prometheus/node_exporter)
VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets. Currently the following [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) types are supported: VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in `prometheus.yml` config file according to [the specification](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). Just set `-promscrape.config` command-line flag to the path to `prometheus.yml` config - and VictoriaMetrics should start scraping the configured targets.
* [static_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config)
* [file_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config)
* [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
* [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config)
* [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config)
* [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config)
* [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config)
* [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config)
* [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config)
* [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config)
* [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config)
* [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config)
* [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config)
* [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config)
File a [feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need support for other `*_sd_config` types.
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders, which are substituted by the corresponding `ENV_VAR` environment variable values.
See [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs.html).
VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format). VictoriaMetrics also supports [importing data in Prometheus exposition format](#how-to-import-data-in-prometheus-exposition-format).
See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus. See also [vmagent](https://docs.victoriametrics.com/vmagent.html), which can be used as drop-in replacement for Prometheus.
@ -2121,6 +2106,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay Whether to suppress scrape errors logging. The last error for each target is always available at '/targets' page even if scrape errors logging is suppressed. See also -promscrape.suppressScrapeErrorsDelay
-promscrape.suppressScrapeErrorsDelay duration -promscrape.suppressScrapeErrorsDelay duration
The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors The delay for suppressing repeated scrape errors logging per each scrape targets. This may be used for reducing the number of log lines related to scrape errors. See also -promscrape.suppressScrapeErrors
-promscrape.yandexcloudSDCheckInterval duration
Interval for checking for changes in Yandex Cloud. This works only if yandexcloud_sd_configs is configured in '-promscrape.config' file. (default 30s)
-pushmetrics.extraLabel array -pushmetrics.extraLabel array
Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url Optional labels to add to metrics pushed to -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to -pushmetrics.url
Supports an array of values separated by comma or specified via multiple flags. Supports an array of values separated by comma or specified via multiple flags.

34
docs/sd_configs.md
View file

@ -1,4 +1,36 @@
## Yandex Cloud Service Discovery Configs # Prometheus service discovery
[vmagent](https://docs.victoriametrics.com/vmagent.html) and [single-node VictoriaMetrics](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter) supports the following Prometheus-compatible service discovery options for Prometheus-compatible scrape targets in the file pointed by `-promscrape.config` command-line flag.
* `azure_sd_configs` - is for scraping the targets registered in Azure Cloud.
See [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config) for details.
* `consul_sd_configs` is for discovering and scraping targets registered in Consul. See [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config) for details.
* `digitalocean_sd_configs` is for discovering and scraping targerts registered in [DigitalOcean](https://www.digitalocean.com/). See [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config) for details.
* `dns_sd_configs` is for discovering and scraping targets from DNS records (SRV, A and AAAA). See [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config) for details.
* `docker_sd_configs` is for discovering and scraping Docker targets. See [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config) for details.
* `dockerswarm_sd_configs` is for discovering and scraping Docker Swarm targets. See [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config) for details.
* `ec2_sd_configs` is for discovering and scraping Amazon EC2 targets. See [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config) for details. `vmagent` doesn't support the `profile` config param yet.
* `eureka_sd_configs` is for discovering and scraping targets registered in [Netflix Eureka](https://github.com/Netflix/eureka). See [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config) for details.
* `file_sd_configs` is for scraping targets defined in external files (aka file-based service discovery). See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config) for details.
* `gce_sd_configs` is for discovering and scraping Google Compute Engine (GCE) targets. See [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config) for details. `vmagent` provides the following additional functionality for `gce_sd_config`:
* if `project` arg is missing then `vmagent` uses the project for the instance where it runs;
* if `zone` arg is missing then `vmagent` uses the zone for the instance where it runs;
* if `zone` arg equals to `"*"`, then `vmagent` discovers all the zones for the given project;
* `zone` may contain a list of zones, i.e. `zone: [us-east1-a, us-east1-b]`.
* `http_sd_configs` is for discovering and scraping targerts provided by external http-based service discovery. See [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) for details.
* `kubernetes_sd_configs` is for discovering and scraping Kubernetes (K8S) targets. See [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) for details.
* `openstack_sd_configs` is for discovering and scraping OpenStack targets. See [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config) for details. [OpenStack identity API v3](https://docs.openstack.org/api-ref/identity/v3/) is supported only.
* `static_configs` is for scraping statically defined targets. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config) for details.
* `yandexcloud_sd_configs` is for discoverying and scraping [Yandex Cloud](https://cloud.yandex.com/en/) targets. See [these docs](#yandexcloud-sd-configs) for details.
Note that the `refresh_interval` option isn't supported for these scrape configs. Use the corresponding `-promscrape.*CheckInterval`
command-line flag instead. For example, `-promscrape.consulSDCheckInterval=60s` sets `refresh_interval` for all the `consul_sd_configs`
entries to 60s. Run `vmagent -help` or `victoria-metrics -help` in order to see default values for the `-promscrape.*CheckInterval` flags.
Please file feature requests to [our issue tracker](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need other service discovery mechanisms to be supported by VictoriaMetrics and `vmagent`.
## yandexcloud_sd_configs
Yandex Cloud SD configurations allow retrieving scrape targets from accessible folders. Yandex Cloud SD configurations allow retrieving scrape targets from accessible folders.

28
docs/vmagent.md
View file

@ -159,34 +159,8 @@ Use `-remoteWrite.*` command-line flag instead for configuring remote write sett
The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values. The file pointed by `-promscrape.config` may contain `%{ENV_VAR}` placeholders which are substituted by the corresponding `ENV_VAR` environment variable values.
The following scrape types in [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) section are supported: See [the list of supported service discovery types for Prometheus scrape targets](https://docs.victoriametrics.com/sd_configs.html).
* `static_configs` is for scraping statically defined targets. See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config) for details.
* `file_sd_configs` is for scraping targets defined in external files (aka file-based service discovery). See [these docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config) for details.
* `kubernetes_sd_configs` is for discovering and scraping Kubernetes (K8S) targets. See [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) for details.
* `ec2_sd_configs` is for discovering and scraping Amazon EC2 targets. See [ec2_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config) for details. `vmagent` doesn't support the `profile` config param yet.
* `gce_sd_configs` is for discovering and scraping Google Compute Engine (GCE) targets. See [gce_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config) for details. `vmagent` provides the following additional functionality for `gce_sd_config`:
* if `project` arg is missing then `vmagent` uses the project for the instance where it runs;
* if `zone` arg is missing then `vmagent` uses the zone for the instance where it runs;
* if `zone` arg equals to `"*"`, then `vmagent` discovers all the zones for the given project;
* `zone` may contain a list of zones, i.e. `zone: [us-east1-a, us-east1-b]`.
* `azure_sd_configs` - is for scraping the targets registered in Azure Cloud.
See [azure_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config) for details.
* `consul_sd_configs` is for discovering and scraping targets registered in Consul. See [consul_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config) for details.
* `dns_sd_configs` is for discovering and scraping targets from DNS records (SRV, A and AAAA). See [dns_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config) for details.
* `openstack_sd_configs` is for discovering and scraping OpenStack targets. See [openstack_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#openstack_sd_config) for details. [OpenStack identity API v3](https://docs.openstack.org/api-ref/identity/v3/) is supported only.
* `docker_sd_configs` is for discovering and scraping Docker targets. See [docker_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#docker_sd_config) for details.
* `dockerswarm_sd_configs` is for discovering and scraping Docker Swarm targets. See [dockerswarm_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config) for details.
* `eureka_sd_configs` is for discovering and scraping targets registered in [Netflix Eureka](https://github.com/Netflix/eureka). See [eureka_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#eureka_sd_config) for details.
* `digitalocean_sd_configs` is for discovering and scraping targerts registered in [DigitalOcean](https://www.digitalocean.com/). See [digitalocean_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#digitalocean_sd_config) for details.
* `http_sd_configs` is for discovering and scraping targerts provided by external http-based service discovery. See [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) for details.
* `yandexcloud_sd_configs` is for discovering and scraping targets registered in [Yandex Cloud](https://cloud.yandex.ru/). See [yandexcloud_sd_configs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/sd_configs.md#yandex_cloud_service_discovery_configs) for details.
Note that `vmagent` doesn't support `refresh_interval` option for these scrape configs. Use the corresponding `-promscrape.*CheckInterval`
command-line flag instead. For example, `-promscrape.consulSDCheckInterval=60s` sets `refresh_interval` for all the `consul_sd_configs`
entries to 60s. Run `vmagent -help` in order to see default values for the `-promscrape.*CheckInterval` flags.
Please file feature requests to [our issue tracker](https://github.com/VictoriaMetrics/VictoriaMetrics/issues) if you need other service discovery mechanisms to be supported by `vmagent`.
## scrape_config enhancements ## scrape_config enhancements

236
lib/promscrape/discovery/yandexcloud/api.go
View file

@ -7,7 +7,6 @@ import (
"io/ioutil" "io/ioutil"
"net/http" "net/http"
"net/url" "net/url"
"path"
"sync" "sync"
"time" "time"
@ -16,12 +15,6 @@ import (
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils" "github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
) )
const (
defaultInstanceCredsEndpoint = "http://169.254.169.254/latest/meta-data/iam/security-credentials/default"
defaultAPIEndpoint = "https://api.cloud.yandex.net"
defaultAPIVersion = "v1"
)
var configMap = discoveryutils.NewConfigMap() var configMap = discoveryutils.NewConfigMap()
type apiCredentials struct { type apiCredentials struct {
@ -35,19 +28,14 @@ type yandexPassportOAuth struct {
YandexPassportOAuthToken string `json:"yandexPassportOauthToken"` YandexPassportOAuthToken string `json:"yandexPassportOauthToken"`
} }
// iamToken Yandex Cloud IAM token response
// https://cloud.yandex.com/en-ru/docs/iam/operations/iam-token/create
type iamToken struct {
IAMToken string `json:"iamToken"`
ExpiresAt time.Time `json:"expiresAt"`
}
type apiConfig struct { type apiConfig struct {
client *http.Client client *http.Client
tokenLock sync.Mutex
creds *apiCredentials
yandexPassportOAuth *yandexPassportOAuth yandexPassportOAuth *yandexPassportOAuth
serviceEndpoints map[string]*url.URL serviceEndpoints map[string]string
// credsLock protects the refresh of creds
credsLock sync.Mutex
creds *apiCredentials
} }
func getAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) { func getAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
@ -59,12 +47,8 @@ func getAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
} }
func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) { func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
cfg := &apiConfig{ transport := &http.Transport{
client: &http.Client{ MaxIdleConnsPerHost: 100,
Transport: &http.Transport{
MaxIdleConnsPerHost: 100,
},
},
} }
if sdc.TLSConfig != nil { if sdc.TLSConfig != nil {
opts := &promauth.Options{ opts := &promauth.Options{
@ -73,47 +57,49 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
} }
ac, err := opts.NewConfig() ac, err := opts.NewConfig()
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot initialize TLS config: %w", err)
}
cfg.client.Transport = &http.Transport{
TLSClientConfig: ac.NewTLSConfig(),
MaxIdleConnsPerHost: 100,
} }
transport.TLSClientConfig = ac.NewTLSConfig()
} }
cfg := &apiConfig{
if err := cfg.getEndpoints(sdc.APIEndpoint); err != nil { client: &http.Client{
return nil, err Transport: transport,
},
} }
apiEndpoint := sdc.APIEndpoint
if apiEndpoint == "" {
apiEndpoint = "https://api.cloud.yandex.net"
}
serviceEndpoints, err := cfg.getServiceEndpoints(apiEndpoint)
if err != nil {
return nil, fmt.Errorf("cannot obtain endpoints for yandex services: %w", err)
}
cfg.serviceEndpoints = serviceEndpoints
if sdc.YandexPassportOAuthToken != nil { if sdc.YandexPassportOAuthToken != nil {
logger.Infof("Using yandex passport OAuth token") logger.Infof("yandexcloud_sd: using yandex passport OAuth token")
cfg.yandexPassportOAuth = &yandexPassportOAuth{ cfg.yandexPassportOAuth = &yandexPassportOAuth{
YandexPassportOAuthToken: sdc.YandexPassportOAuthToken.String(), YandexPassportOAuthToken: sdc.YandexPassportOAuthToken.String(),
} }
} }
return cfg, nil return cfg, nil
} }
// getFreshAPICredentials checks token lifetime and update if needed // getFreshAPICredentials checks token lifetime and update if needed
func (cfg *apiConfig) getFreshAPICredentials() (*apiCredentials, error) { func (cfg *apiConfig) getFreshAPICredentials() (*apiCredentials, error) {
cfg.tokenLock.Lock() cfg.credsLock.Lock()
defer cfg.tokenLock.Unlock() defer cfg.credsLock.Unlock()
if cfg.creds != nil && time.Until(cfg.creds.Expiration) > 10*time.Second { if cfg.creds != nil && time.Until(cfg.creds.Expiration) > 10*time.Second {
// Credentials aren't expired yet. // Credentials aren't expired yet.
return cfg.creds, nil return cfg.creds, nil
} }
// Refresh credentials.
newCreds, err := getCreds(cfg) newCreds, err := getCreds(cfg)
if err != nil { if err != nil {
return nil, fmt.Errorf("cannot refresh service account api token: %w", err) return nil, fmt.Errorf("cannot refresh service account api token: %w", err)
} }
cfg.creds = newCreds cfg.creds = newCreds
logger.Infof("yandexcloud_sd: successfully refreshed service account api token; expiration: %.3f seconds", time.Until(newCreds.Expiration).Seconds())
logger.Infof("successfully refreshed service account api token; expiration: %.3f seconds", time.Until(newCreds.Expiration).Seconds())
return newCreds, nil return newCreds, nil
} }
@ -122,12 +108,10 @@ func getCreds(cfg *apiConfig) (*apiCredentials, error) {
if cfg.yandexPassportOAuth == nil { if cfg.yandexPassportOAuth == nil {
return getInstanceCreds(cfg) return getInstanceCreds(cfg)
} }
it, err := getIAMToken(cfg) it, err := getIAMToken(cfg)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot get IAM token: %w", err)
} }
return &apiCredentials{ return &apiCredentials{
Token: it.IAMToken, Token: it.IAMToken,
Expiration: it.ExpiresAt, Expiration: it.ExpiresAt,
@ -135,104 +119,118 @@ func getCreds(cfg *apiConfig) (*apiCredentials, error) {
} }
// getInstanceCreds gets Yandex Cloud IAM token using instance Service Account // getInstanceCreds gets Yandex Cloud IAM token using instance Service Account
// https://cloud.yandex.com/en-ru/docs/compute/operations/vm-connect/auth-inside-vm //
// See https://cloud.yandex.com/en-ru/docs/compute/operations/vm-connect/auth-inside-vm
func getInstanceCreds(cfg *apiConfig) (*apiCredentials, error) { func getInstanceCreds(cfg *apiConfig) (*apiCredentials, error) {
resp, err := cfg.client.Get(defaultInstanceCredsEndpoint) endpoint := "http://169.254.169.254/latest/meta-data/iam/security-credentials/default"
resp, err := cfg.client.Get(endpoint)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed query security credentials api, url: %s, err: %w", defaultInstanceCredsEndpoint, err) return nil, fmt.Errorf("cannot read instance creds from %s: %w", endpoint, err)
} }
r, err := ioutil.ReadAll(resp.Body) data, err := readResponseBody(resp, endpoint)
_ = resp.Body.Close()
if err != nil { if err != nil {
return nil, fmt.Errorf("cannot read response from %q: %w", defaultInstanceCredsEndpoint, err) return nil, err
} }
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("auth failed, bad status code: %d, want: 200", resp.StatusCode)
}
var ac apiCredentials var ac apiCredentials
if err := json.Unmarshal(r, &ac); err != nil { if err := json.Unmarshal(data, &ac); err != nil {
return nil, fmt.Errorf("cannot parse auth credentials response: %w", err) return nil, fmt.Errorf("cannot parse auth credentials response from %s: %w", endpoint, err)
} }
return &ac, nil return &ac, nil
} }
// getIAMToken gets Yandex Cloud IAM token using OAuth: // getIAMToken gets Yandex Cloud IAM token using OAuth
// https://cloud.yandex.com/en-ru/docs/iam/operations/iam-token/create //
// See https://cloud.yandex.com/en-ru/docs/iam/operations/iam-token/create
func getIAMToken(cfg *apiConfig) (*iamToken, error) { func getIAMToken(cfg *apiConfig) (*iamToken, error) {
iamURL := *cfg.serviceEndpoints["iam"] iamURL := cfg.serviceEndpoints["iam"] + "/iam/v1/tokens"
iamURL.Path = path.Join(iamURL.Path, "iam", defaultAPIVersion, "tokens")
passport, err := json.Marshal(cfg.yandexPassportOAuth) passport, err := json.Marshal(cfg.yandexPassportOAuth)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed marshall yandex passport OAuth token, err: %w", err) logger.Panicf("BUG: cannot marshal yandex passport OAuth token: %s", err)
} }
body := bytes.NewBuffer(passport)
resp, err := cfg.client.Post(iamURL.String(), "application/json", bytes.NewBuffer(passport)) resp, err := cfg.client.Post(iamURL, "application/json", body)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed query yandex cloud iam api, url: %s, err: %w", iamURL.String(), err) logger.Panicf("BUG: cannot create request to yandex cloud iam api %q: %s", iamURL, err)
} }
data, err := readResponseBody(resp, iamURL)
r, err := ioutil.ReadAll(resp.Body)
_ = resp.Body.Close()
if err != nil { if err != nil {
return nil, fmt.Errorf("cannot read response from %q: %w", iamURL.String(), err) return nil, err
} }
var it iamToken
if resp.StatusCode != http.StatusOK { if err := json.Unmarshal(data, &it); err != nil {
return nil, fmt.Errorf("auth failed, bad status code: %d, want: 200", resp.StatusCode) return nil, fmt.Errorf("cannot parse iam token: %w; data: %s", err, data)
} }
it := iamToken{}
if err := json.Unmarshal(r, &it); err != nil {
return nil, fmt.Errorf("cannot parse auth credentials response: %w", err)
}
return &it, nil return &it, nil
} }
// getEndpoints makes services endpoints map: // iamToken represents Yandex Cloud IAM token response
// https://cloud.yandex.com/en-ru/docs/api-design-guide/concepts/endpoints //
func (cfg *apiConfig) getEndpoints(apiEndpoint string) error { // See https://cloud.yandex.com/en-ru/docs/iam/operations/iam-token/create
if apiEndpoint == "" { type iamToken struct {
apiEndpoint = defaultAPIEndpoint IAMToken string `json:"iamToken"`
} ExpiresAt time.Time `json:"expiresAt"`
}
// getServiceEndpoints returns services endpoints map
//
// See https://cloud.yandex.com/en-ru/docs/api-design-guide/concepts/endpoints
func (cfg *apiConfig) getServiceEndpoints(apiEndpoint string) (map[string]string, error) {
apiEndpointURL, err := url.Parse(apiEndpoint) apiEndpointURL, err := url.Parse(apiEndpoint)
if err != nil { if err != nil {
return fmt.Errorf("cannot parse api_endpoint: %s as url, err: %w", apiEndpoint, err) return nil, fmt.Errorf("cannot parse api_endpoint %q: %w", apiEndpoint, err)
} }
scheme := apiEndpointURL.Scheme
apiEndpointURL.Path = path.Join(apiEndpointURL.Path, "endpoints") if scheme == "" {
return nil, fmt.Errorf("missing scheme in api_endpoint %q", apiEndpoint)
resp, err := cfg.client.Get(apiEndpointURL.String()) }
if apiEndpointURL.Host == "" {
return nil, fmt.Errorf("missing host in api_endpoint %q", apiEndpoint)
}
endpointsURL := apiEndpoint + "/endpoints"
resp, err := cfg.client.Get(endpointsURL)
if err != nil { if err != nil {
return fmt.Errorf("failed query endpoints, url: %s, err: %w", apiEndpointURL.String(), err) return nil, fmt.Errorf("cannot query %q: %w", endpointsURL, err)
} }
r, err := ioutil.ReadAll(resp.Body) data, err := readResponseBody(resp, endpointsURL)
_ = resp.Body.Close()
if err != nil { if err != nil {
return fmt.Errorf("cannot read response from %q: %w", apiEndpointURL.String(), err) return nil, err
} }
if resp.StatusCode != http.StatusOK { var eps endpoints
return fmt.Errorf("auth failed, bad status code: %d, want: 200", resp.StatusCode) if err := json.Unmarshal(data, &eps); err != nil {
return nil, fmt.Errorf("cannot parse API endpoints list: %w; data=%s", err, data)
} }
m := make(map[string]string, len(eps.Endpoints))
for _, endpoint := range eps.Endpoints {
m[endpoint.ID] = scheme + "://" + endpoint.Address
}
return m, nil
}
endpoints, err := parseEndpoints(r) type endpoint struct {
ID string `json:"id"`
Address string `json:"address"`
}
type endpoints struct {
Endpoints []endpoint `json:"endpoints"`
}
// getAPIResponse calls Yandex Cloud apiURL and returns response body.
func getAPIResponse(apiURL string, cfg *apiConfig) ([]byte, error) {
creds, err := cfg.getFreshAPICredentials()
if err != nil { if err != nil {
return err return nil, err
} }
req, err := http.NewRequest("GET", apiURL, nil)
cfg.serviceEndpoints = make(map[string]*url.URL, len(endpoints.Endpoints)) if err != nil {
for _, endpoint := range endpoints.Endpoints { logger.Panicf("BUG: cannot create new request for yandex cloud api url %s: %s", apiURL, err)
cfg.serviceEndpoints[endpoint.ID] = &url.URL{
Scheme: apiEndpointURL.Scheme,
Host: endpoint.Address,
}
} }
req.Header.Set("Authorization", "Bearer "+creds.Token)
return nil resp, err := cfg.client.Do(req)
if err != nil {
return nil, fmt.Errorf("cannot query yandex cloud api url %s: %w", apiURL, err)
}
return readResponseBody(resp, apiURL)
} }
// readResponseBody reads body from http.Response. // readResponseBody reads body from http.Response.
@ -243,30 +241,8 @@ func readResponseBody(resp *http.Response, apiURL string) ([]byte, error) {
return nil, fmt.Errorf("cannot read response from %q: %w", apiURL, err) return nil, fmt.Errorf("cannot read response from %q: %w", apiURL, err)
} }
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("unexpected status code for %q; got %d; want %d; response body: %q", return nil, fmt.Errorf("unexpected status code for %q; got %d; want %d; response body: %s",
apiURL, resp.StatusCode, http.StatusOK, data) apiURL, resp.StatusCode, http.StatusOK, data)
} }
return data, nil return data, nil
} }
// getAPIResponse calls Yandex Cloud apiURL and returns response body.
func getAPIResponse(apiURL string, cfg *apiConfig) ([]byte, error) {
creds, err := cfg.getFreshAPICredentials()
if err != nil {
return nil, err
}
req, err := http.NewRequest("GET", apiURL, nil)
if err != nil {
return nil, fmt.Errorf("cannot create new request for yandex cloud api url %s: %w", apiURL, err)
}
req.Header.Set("Authorization", "Bearer "+creds.Token)
resp, err := cfg.client.Do(req)
if err != nil {
return nil, fmt.Errorf("cannot query yandex cloud api url %s: %w", apiURL, err)
}
return readResponseBody(resp, apiURL)
}

19
lib/promscrape/discovery/yandexcloud/instance.go
View file

@ -27,11 +27,10 @@ func getInstancesLabels(cfg *apiConfig) ([]map[string]string, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
instances = append(instances, inst...) instances = append(instances, inst...)
} }
logger.Infof("Collected %d instances", len(instances)) logger.Infof("yandexcloud_sd: collected %d instances", len(instances))
return addInstanceLabels(instances), nil return addInstanceLabels(instances), nil
} }
@ -55,20 +54,20 @@ func addInstanceLabels(instances []instance) []map[string]string {
m["__meta_yandexcloud_instance_label_"+discoveryutils.SanitizeLabelName(k)] = v m["__meta_yandexcloud_instance_label_"+discoveryutils.SanitizeLabelName(k)] = v
} }
for _, netInterface := range server.NetworkInterfaces { for _, ni := range server.NetworkInterfaces {
privateIPLabel := fmt.Sprintf("__meta_yandexcloud_instance_private_ip_%s", netInterface.Index) privateIPLabel := fmt.Sprintf("__meta_yandexcloud_instance_private_ip_%s", ni.Index)
m[privateIPLabel] = netInterface.PrimaryV4Address.Address m[privateIPLabel] = ni.PrimaryV4Address.Address
if len(netInterface.PrimaryV4Address.OneToOneNat.Address) > 0 { if len(ni.PrimaryV4Address.OneToOneNat.Address) > 0 {
publicIPLabel := fmt.Sprintf("__meta_yandexcloud_instance_public_ip_%s", netInterface.Index) publicIPLabel := fmt.Sprintf("__meta_yandexcloud_instance_public_ip_%s", ni.Index)
m[publicIPLabel] = netInterface.PrimaryV4Address.OneToOneNat.Address m[publicIPLabel] = ni.PrimaryV4Address.OneToOneNat.Address
} }
for j, dnsRecord := range netInterface.PrimaryV4Address.DNSRecords { for j, dnsRecord := range ni.PrimaryV4Address.DNSRecords {
dnsRecordLabel := fmt.Sprintf("__meta_yandexcloud_instance_private_dns_%d", j) dnsRecordLabel := fmt.Sprintf("__meta_yandexcloud_instance_private_dns_%d", j)
m[dnsRecordLabel] = dnsRecord.FQDN m[dnsRecordLabel] = dnsRecord.FQDN
} }
for j, dnsRecord := range netInterface.PrimaryV4Address.OneToOneNat.DNSRecords { for j, dnsRecord := range ni.PrimaryV4Address.OneToOneNat.DNSRecords {
dnsRecordLabel := fmt.Sprintf("__meta_yandexcloud_instance_public_dns_%d", j) dnsRecordLabel := fmt.Sprintf("__meta_yandexcloud_instance_public_dns_%d", j)
m[dnsRecordLabel] = dnsRecord.FQDN m[dnsRecordLabel] = dnsRecord.FQDN
} }

24
lib/promscrape/discovery/yandexcloud/resources.go
View file

@ -2,34 +2,10 @@ package yandexcloud
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"time" "time"
) )
type endpoint struct {
ID string `json:"id"`
Address string `json:"address"`
}
type endpoints struct {
Endpoints []endpoint `json:"endpoints"`
}
// See https://cloud.yandex.com/en-ru/docs/api-design-guide/concepts/endpoints
func parseEndpoints(data []byte) (*endpoints, error) {
var endpointsResponse endpoints
if err := json.Unmarshal(data, &endpointsResponse); err != nil {
return nil, fmt.Errorf("cannot parse endpoints list: %w", err)
}
if endpointsResponse.Endpoints == nil {
return nil, errors.New("yandex cloud API endpoints list is empty")
}
return &endpointsResponse, nil
}
type organization struct { type organization struct {
Name string `json:"name"` Name string `json:"name"`
ID string `json:"id"` ID string `json:"id"`

139
lib/promscrape/discovery/yandexcloud/yandexcloud.go
View file

@ -3,7 +3,7 @@ package yandexcloud
import ( import (
"flag" "flag"
"fmt" "fmt"
"path" "net/url"
"time" "time"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth" "github.com/VictoriaMetrics/VictoriaMetrics/lib/promauth"
@ -36,142 +36,103 @@ func (sdc *SDConfig) GetLabels(baseDir string) ([]map[string]string, error) {
} }
func (cfg *apiConfig) getInstances(folderID string) ([]instance, error) { func (cfg *apiConfig) getInstances(folderID string) ([]instance, error) {
computeURL := *cfg.serviceEndpoints["compute"] instancesURL := cfg.serviceEndpoints["compute"] + "/compute/v1/instances"
computeURL.Path = path.Join(computeURL.Path, "compute", defaultAPIVersion, "instances") instancesURL += "?folderId=" + url.QueryEscape(folderID)
q := computeURL.Query()
q.Set("folderId", folderID)
computeURL.RawQuery = q.Encode()
nextLink := computeURL.String()
instances := make([]instance, 0) var instances []instance
nextLink := instancesURL
for { for {
resp, err := getAPIResponse(nextLink, cfg) data, err := getAPIResponse(nextLink, cfg)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot get instances: %w", err)
} }
instancesPage, err := parseInstancesPage(resp) ip, err := parseInstancesPage(data)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot parse instances response from %q: %w; response body: %s", nextLink, err, data)
} }
instances = append(instances, instancesPage.Instances...) instances = append(instances, ip.Instances...)
if len(instancesPage.NextPageToken) == 0 { if len(ip.NextPageToken) == 0 {
return instances, nil return instances, nil
} }
nextLink = instancesURL + "&pageToken=" + url.QueryEscape(ip.NextPageToken)
q.Set("pageToken", instancesPage.NextPageToken)
computeURL.RawQuery = q.Encode()
nextLink = computeURL.String()
} }
} }
func (cfg *apiConfig) getFolders(clouds []cloud) ([]folder, error) { func (cfg *apiConfig) getFolders(clouds []cloud) ([]folder, error) {
rmURL := *cfg.serviceEndpoints["resource-manager"] foldersURL := cfg.serviceEndpoints["resource-manager"] + "/resource-manager/v1/folders"
rmURL.Path = path.Join(rmURL.Path, "resource-manager", defaultAPIVersion, "folders") var folders []folder
q := rmURL.Query()
folders := make([]folder, 0)
for _, cl := range clouds { for _, cl := range clouds {
q.Set("cloudId", cl.ID) cloudURL := foldersURL + "?cloudId=" + url.QueryEscape(cl.ID)
rmURL.RawQuery = q.Encode() nextLink := cloudURL
nextLink := rmURL.String()
for { for {
resp, err := getAPIResponse(nextLink, cfg) data, err := getAPIResponse(nextLink, cfg)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot get folders: %w", err)
} }
fp, err := parseFoldersPage(data)
foldersPage, err := parseFoldersPage(resp)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot parse folders response from %q: %w; response body: %s", nextLink, err, data)
} }
folders = append(folders, fp.Folders...)
folders = append(folders, foldersPage.Folders...) if len(fp.NextPageToken) == 0 {
if len(foldersPage.NextPageToken) == 0 {
break break
} }
nextLink = cloudURL + "&pageToken=" + url.QueryEscape(fp.NextPageToken)
q.Set("pageToken", foldersPage.NextPageToken)
rmURL.RawQuery = q.Encode()
nextLink = rmURL.String()
} }
} }
return folders, nil return folders, nil
} }
func (cfg *apiConfig) getClouds(organizations []organization) ([]cloud, error) { func (cfg *apiConfig) getClouds(orgs []organization) ([]cloud, error) {
rmURL := *cfg.serviceEndpoints["resource-manager"] cloudsURL := cfg.serviceEndpoints["resource-manager"] + "/resource-manager/v1/clouds"
rmURL.Path = path.Join(rmURL.Path, "resource-manager", defaultAPIVersion, "clouds") if len(orgs) == 0 {
q := rmURL.Query() orgs = append(orgs, organization{
if len(organizations) == 0 {
organizations = append(organizations, organization{
ID: "", ID: "",
}) })
} }
var clouds []cloud
clouds := make([]cloud, 0) for _, org := range orgs {
for _, org := range organizations { orgURL := cloudsURL
if org.ID != "" { if org.ID != "" {
q.Set("organizationId", org.ID) orgURL += "?organizationId=" + url.QueryEscape(org.ID)
rmURL.RawQuery = q.Encode()
} }
nextLink := orgURL
nextLink := rmURL.String()
for { for {
resp, err := getAPIResponse(nextLink, cfg) data, err := getAPIResponse(nextLink, cfg)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot get clouds: %w", err)
} }
cp, err := parseCloudsPage(data)
cloudsPage, err := parseCloudsPage(resp)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot parse clouds response from %q: %w; response body: %s", nextLink, err, data)
} }
clouds = append(clouds, cp.Clouds...)
clouds = append(clouds, cloudsPage.Clouds...) if len(cp.NextPageToken) == 0 {
if len(cloudsPage.NextPageToken) == 0 {
break break
} }
nextLink = orgURL + "&pageToken=" + url.QueryEscape(cp.NextPageToken)
q.Set("pageToken", cloudsPage.NextPageToken)
rmURL.RawQuery = q.Encode()
nextLink = rmURL.String()
} }
} }
return clouds, nil return clouds, nil
} }
func (cfg *apiConfig) getOrganizations() ([]organization, error) { func (cfg *apiConfig) getOrganizations() ([]organization, error) {
omURL := *cfg.serviceEndpoints["organization-manager"] orgsURL := cfg.serviceEndpoints["organization-manager"] + "/organization-manager/v1/organizations"
omURL.Path = path.Join(omURL.Path, "organization-manager", defaultAPIVersion, "organizations") var orgs []organization
q := omURL.Query() nextLink := orgsURL
nextLink := omURL.String()
organizations := make([]organization, 0)
for { for {
resp, err := getAPIResponse(nextLink, cfg) data, err := getAPIResponse(nextLink, cfg)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot get organizations: %w", err)
} }
op, err := parseOrganizationsPage(data)
organizationsPage, err := parseOrganizationsPage(resp)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("cannot parse organizations response from %q: %w; response body: %s", nextLink, err, data)
} }
orgs = append(orgs, op.Organizations...)
organizations = append(organizations, organizationsPage.Organizations...) if len(op.NextPageToken) == 0 {
return orgs, nil
if len(organizationsPage.NextPageToken) == 0 {
return organizations, nil
} }
nextLink = orgsURL + "&pageToken=" + url.QueryEscape(op.NextPageToken)
q.Set("pageToken", organizationsPage.NextPageToken)
omURL.RawQuery = q.Encode()
nextLink = omURL.String()
} }
} }