github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
8080 commits 150 branches 748 tags 347 MiB
Commit graph

279 commits

Author SHA1 Message Date
Karan Sharma
004a24c950
sd/nomad: panic in nomad watcher because of nil map (#3784) 
properly initialize url.Values
 2023-02-08 08:37:02 -08:00
Aliaksandr Valialkin
855d560789
lib/promscrape/discovery/azure: add __meta_azure_machine_size label in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/11650
 2023-01-27 17:07:57 -08:00
Aliaksandr Valialkin
134f7622d6
lib/promscrape/discovery/kubernetes: add support for __meta_kubernetes_pod_container_id 
See https://github.com/prometheus/prometheus/issues/11843
and https://github.com/prometheus/prometheus/pull/11844
 2023-01-27 16:33:57 -08:00
Zakhar Bessarab
40d524edb8
discovery/{consul,nomad}: fix cancelling serviceWatcher in-flight requests (#3658) 
* lib/promscrape/discovery/{consul,nomad}: fix background service update watches not canceling requests on serviceWatcher stop

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/{consul,nomad}: fix closing serviseWatcher during scrape job restart

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-17 21:47:51 -08:00
Aliaksandr Valialkin
2e018aebf3
lib/promscrape/discovery: missing changes after b4ad3a3b4c 2023-01-11 23:03:14 -08:00
Aliaksandr Valialkin
434f22f871
lib/promscrape: follow-up for 8537533beb 
- Add a comment describing the purpose of the `role` field inside `apiConfig` struct
- Revert changes at lib/promscrape/discovery/dockerswarm/dockerswarm.go ,
  since they reduce code readability. E.g. the reader needs to look up the named string constants
  in order to get their values.
 2023-01-11 22:56:48 -08:00
Zakhar Bessarab
ae5b85966a
lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects (#3632) 
* lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-11 22:56:40 -08:00
Aliaksandr Valialkin
ab318660cd
lib/promscrape/discovery/gce: follow-up for b2ccdaaa2f 
- Use promutils.Labels.GetLabels() instead of comparing promutils.Labels.Labels to nil.
  This make the code more consistent with other places.

- Mention the release where the issue has been introduced at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624
 2023-01-10 13:51:57 -08:00
Zakhar Bessarab
02f5c16433
lib/promscrape/discovery/gce: fix crash in case instance does not have any labels set (#3625) 
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-10 13:51:35 -08:00
Aliaksandr Valialkin
43a4dcdaf8
lib/promscrape/discovery/nomad: sync nomad_sd_configs fields with the Prometheus implementation 
See the list of configs supported by Prometheus at f88a0a7d83/discovery/nomad/nomad.go (L76-L84)

- Removed "token" option. In can be set either via NOMAD_TOKEN env var or via `bearer_token` config option.
- Removed "scheme" option. It is automatically detected depending on whether the `tls_config` is set.
- Removed "services" and "tags" options, since they aren't supported by Prometheus.
- Added "region" option. If it is missing, then the region is read from NOMAD_REGION env var.
  If this var is empty, then it is set to "global" in the same way as Nomad client does.
  See 865ee8d37c/api/api.go (L297)
  and 865ee8d37c/api/api.go (L555-L556)
- If the "server" option is missing, then it is read from NOMAD_ADDR in the same way
  as Nomad client does - see 865ee8d37c/api/api.go (L294-L296)

This is a follow-up for 8aee209c53

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-09 21:30:19 -08:00
Roman Khavronenko
ca5136a0ee
lib/promscrape: remove datacenter field from nomad_sd_config (#3612) 
Looks like `datacenter` field isn't part of `/v1/services` API.
See https://developer.hashicorp.com/nomad/api-docs/services#list-services
and https://developer.hashicorp.com/nomad/api-docs/services#read-service

Related issues:
https://github.com/traefik/traefik/issues/9109
https://github.com/prometheus/prometheus/issues/11776

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-09 21:24:46 -08:00
Aliaksandr Valialkin
ec7a3b79ab
lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 21:53:08 -08:00
Aliaksandr Valialkin
54410bf51b
lib/promscrape: follow-up after bced9fb978 
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-05 21:23:21 -08:00
Zakhar Bessarab
de5aad2cde
lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568) 2023-01-05 21:23:15 -08:00
Aliaksandr Valialkin
750d309f63
lib/promscrape/discovery/nomad: follow-up after 48f371a46c 
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 18:09:23 -08:00
Karan Sharma
8f42c5a024
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549) 
Add nomad_sd_config support for service discovery
 2023-01-05 18:07:02 -08:00
Zakhar Bessarab
99f9b02283
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586) 
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin
c2fc996e01
lib/promscrape/discovery/azure: typo fix 2022-12-21 21:25:25 -08:00
Aliaksandr Valialkin
9330da3195
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname> 
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
 2022-12-19 13:02:56 -08:00
Aliaksandr Valialkin
04abd5e113
docs/CHANGELOG.md: document the bugfix at 05b42601c3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
 2022-12-08 18:35:47 -08:00
Zakhar Bessarab
c939a8e8a2
lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403) 
* lib/promscrape/discovery/azure: remove API server from URL returned by azure

* lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer
 2022-12-08 18:35:46 -08:00
Aliaksandr Valialkin
be6da5053f
lib/promscrape: optimize service discovery speed 
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
 2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin
2524d94fe1
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels 2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin
c33bcae457
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones 
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
 2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin
abf7e4e72f
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/11482
 2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin
ba739c8052
lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page 2022-10-28 22:15:30 +03:00
Nikolay
ea0596d9d8
lib/promscrape/discovery/kubernetes: correctly wrap error (#3250) 
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin
f0a748a3aa
lib/promscrape/discovery/azure: remove unneeded conversion to string 2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin
c628f5b6eb
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does 
See https://github.com/prometheus/prometheus/pull/11326
 2022-09-30 20:49:08 +03:00
Aliaksandr Valialkin
592612b63f
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error 
This should improve debugging issues with Kubernetes API server
 2022-09-13 19:37:40 +03:00
Aliaksandr Valialkin
86394b4179
lib/promscrape: optimize discoveryutils.SanitizeLabelName() 
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
 2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll 
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
 2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
c27bd63f6c
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html 2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin
1a00c9ef03
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11034
 2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin
2fb63dda83
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11002
 2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin
2b58bd9876
lib/promscrape/discovery/dns: add support for resolving MX records 
See https://github.com/prometheus/prometheus/pull/10099
 2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin
77bd4e37cc
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress 
See 7e65ad3e43
and 7e1111ff14
 2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin
ecbe1ddf1b
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs 
This option was ignored since d289ecded1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
 2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin
80ecfcf759
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels 
See https://github.com/prometheus/prometheus/issues/9187
 2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin
85b04732ed
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default 
This aligns with Prometheus behaviour.

See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
 2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin
17290a4598
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f 2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin
8ddad31eef
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
 2022-08-04 22:28:21 +03:00
Igor Tiunov
0ba86fe87e
YC service discovery (#2923) 
* YC service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386

* Fixed linter suggestions

* fixed golint errors
 2022-08-04 22:28:20 +03:00
Nikolay
c007b129cb
lib/promscrape: adds azure service discovery (#2743) 
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-13 23:45:43 +03:00
Aliaksandr Valialkin
4d03ac90fc
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
 2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin
c4cc45d7f8
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/10759
 2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin
ecc11dc32d
lib/promauth: refactor NewConfig in order to improve maintainability 
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options

This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
 2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin
3ae6300497
lib/promauth: add ability to send additional http headers in requests to scrape targets 
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
 2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin
b6e3c12811
lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string 
This improves code readability and maintainability a bit, since the format string
is passed as string literal into fmt.Errorf.
 2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin
3dbb19d624
lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534 
- make more clear error logs
- simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct
 2022-06-06 14:41:28 +03:00
Nikolay
72e43ef2fe
lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 (#2672) 2022-06-04 01:11:23 +03:00
hadesy
28d4624f60
promscrape/discovery: support kubeconfig (#2533) 2022-06-04 01:11:23 +03:00
Roman Khavronenko
7406665fc3
lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) 
* lib/promscrape/discovery/kubernetes: properly updates discovered scrape works
previously, added or updated scrapeworks may override previuosly
discovered.
it happens because swosByKey may contain small subset of kubernetes
objects with it's labels.
It happens for objectsUpdated and objectsAdded maps, which include only changed elements

* Properly calculate vm_promscrape_discovery_kubernetes_scrape_works

Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-21 01:17:21 +03:00
Boris Petersen
3a8b4fab97
Add ability to sign requests for all AWS services (#2604) 
This adds the ability to utilize sigv4 signing for all AWS services not
just "aps". When the newly introduced property "service" is not set it
will default to "aps".

Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
 2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin
0d0561ca8c
lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future 
Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.
 2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin
810dd74fb9
lib/promscrape: properly implement ScrapeConfig.clone() 
Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields -
their contents was replaced with `<secret>` value.

This led to inability to use passwords and secrets in `-promscrape.config` file.
The bug has been introduced in v1.77.0 in the commit 67b10896d2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551
 2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin
9d40bb7137
lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section 2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin
2ce1d09135
lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call 
Previously filters wheren't passed to this call after the commit 0e09fdb8b0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
 2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin
873f55bac5
lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query 
The filters shouldn't be passed to DescribeAvailabilityZones API call.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287

Related commits:
0e09fdb8b0
d289ecded1
 2022-05-05 10:29:47 +03:00
Nikolay
7e58cba6cf
{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458) 
* {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite
moves aws related code into separate lib from lib/promscrape
it allows to write data from vmagent to the AWS managed prometheus (cortex)

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287

* Apply suggestions from code review

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-04 20:28:37 +03:00
Aliaksandr Valialkin
aa82987d70
lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing 
This reflects the logic used in Prometheus.

See https://github.com/prometheus/prometheus/pull/10080
 2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin
c2b13e6a04
lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects 
Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste
 2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin
a89e31b304
lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does 
See https://github.com/prometheus/prometheus/issues/9510
and https://github.com/prometheus/prometheus/pull/10080
 2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin
cc6eae6992
lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems 
Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.
 2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin
fea9d1e6ee
lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240

This is a follow-up for 2341bd48d7
 2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin
e9f08b1e6a
lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects 
There is high chance that ScrapeWork objects won't be generated because of relabeling
 2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin
909a3ee0e4
lib/promscrape: follow-up after 91e290a8ff 2022-04-20 16:12:26 +03:00
Nikolay
429848a67d
lib/promscrape: reduce latency for k8s GetLabels (#2454) 
replaces internStringMap with sync.Map - it greatly reduces lock contention
concurently reload scrape work for api watcher - each object labels added by dedicated CPU

changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70
 2022-04-20 16:12:25 +03:00
Aliaksandr Valialkin
d0bac8e224
all: typo fix: Kuberntes -> Kubernetes 2022-04-20 10:51:41 +03:00
Roman Khavronenko
5a4b16794d
Consul SD - update services on the watcher's start (#2202) 
* lib/discovery/consul: update services on the watcher's start

Previously, watcher's start was only initing goroutines for discovery
but not waiting for the first iteration to end. It means first Consul
discovery wasn't returning discovered targets until the next iteration.

The change makes the watcher's start blocking until we get first discovery
iteration done and all registries updated.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: remove workarounds for consul SD

Now when consul SD lib properly updates services
on the first start, we don't need workarounds in vmalert.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/discovery/consul: update after review

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-21 15:33:33 +02:00
Nikolay
c11d0949c8
fixes all_tenants query option usage for openstack service discovery (#2184) 
explicit use configuration parametr instead of conditional add
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182
 2022-02-14 13:13:03 +02:00
artifactori
943fc056ca
Show gce sdconfig zone on vmagent:8429/config (#2178) 
* vmagent: add test for marshalling gce sdconfig with ZoneYAML

* vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig
 2022-02-12 00:48:38 +02:00
Aliaksandr Valialkin
727b29d4a3
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpointslice_{label,annotation}* labels to be consistent with other role values for Kubernetes service discovery 2022-02-11 14:56:10 +02:00
Nikolay
265938a385
fixes service discovery for kubernetes (#2173) 
* fixes service discovery for kubernetes
now it must take in account all pods that belong to the discovered endpoint and endpointslice
adds simple test for endpoints
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134

* wip

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin
dd91759f1f
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_node_provider_id label for discovered Kubernetes nodes in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/9603
 2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin
bc18368c15
lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace 
See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881
 2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin
de8299f465
lib/promscrape/discovery/dockerswarm: follow up after 68a117a25a 
- Document the bugfix at docs/CHANGELOG.md
- Set __address__ field after copying commonLabels to the resulting map of discovered labels.
  This makes sure that the correct __address__ label is used.
 2022-01-11 09:22:03 +02:00
Alexander Shtuchkin
45a92e6ce1
Fix for #2038: Make correct __address__ value for dockerswarm promscrape (#2041) 2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin
847004fa77
app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764
 2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin
ad445a06cd
lib/promscrape: properly show proxy_url option value at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755
 2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin
ea69eef375
lib/promscrape/discovery/kubernetes: log a warning if role: endpoints discovers more than 1000 targets per a single endpoint 
In this case `role: endpointslice` must be used instead.

See the following references:

* https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity
* https://github.com/kubernetes/kubernetes/pull/99975
* https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398
 2021-10-19 13:22:28 +03:00
Aliaksandr Valialkin
3e9beb0f8d
lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name 
This is a follow-up for 31b42b30b6
 2021-10-15 12:27:31 +03:00
Nikolay
9be5689b3f
changes auth validation for openstack (#1663) 
* changes auth validation for openstack
must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-29 00:33:38 +03:00
Nikolay
1ab2f844a2 makes filters optional for ec2 api requests (#1627) 
filters can be applied only for DescribeInstances requests, like prometheus does.
related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
 2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin
b684624f67 lib/promscrape/discovery/docker: support host networking mode 
See https://github.com/prometheus/prometheus/issues/9116
 2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin
6ed9f10da5 lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery 
See https://github.com/prometheus/prometheus/issues/8902
 2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin
146c14d879 lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator 
This is a follow up commit after 31b42b30b6
 2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin
ca61d7c82b lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus 
See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)
 2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin
327034b54f lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices 
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
 2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin
4a2d7aec7f lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config 2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin
77bb9e1656 lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 
See https://github.com/prometheus/prometheus/pull/8978
 2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin
336a2aa2e0 lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does 2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin
6fc3696260 lib/promscrape/discovery/consul: use case-insensitive comparison for service names 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
 2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin
2edfea8c36 lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID 2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
4461e20e7d lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 13:22:16 +03:00
Lu Jiajing
12b4cbb68f Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 13:22:16 +03:00
Nikolay
501429c3ff adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin
f10fa0d1d7 lib/promscrape/discovery/consul: properly pass namespace to Consul watcher 
Follow-up for 58a2989fe7
 2021-06-22 17:43:20 +03:00