github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
4333 commits 150 branches 748 tags 347 MiB
Commit graph

176 commits

Author SHA1 Message Date
Roman Khavronenko
7406665fc3
lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615) 
* lib/promscrape/discovery/kubernetes: properly updates discovered scrape works
previously, added or updated scrapeworks may override previuosly
discovered.
it happens because swosByKey may contain small subset of kubernetes
objects with it's labels.
It happens for objectsUpdated and objectsAdded maps, which include only changed elements

* Properly calculate vm_promscrape_discovery_kubernetes_scrape_works

Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-21 01:17:21 +03:00
Boris Petersen
3a8b4fab97
Add ability to sign requests for all AWS services (#2604) 
This adds the ability to utilize sigv4 signing for all AWS services not
just "aps". When the newly introduced property "service" is not set it
will default to "aps".

Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
 2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin
0d0561ca8c
lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future 
Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.
 2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin
810dd74fb9
lib/promscrape: properly implement ScrapeConfig.clone() 
Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields -
their contents was replaced with `<secret>` value.

This led to inability to use passwords and secrets in `-promscrape.config` file.
The bug has been introduced in v1.77.0 in the commit 67b10896d2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551
 2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin
9d40bb7137
lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section 2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin
2ce1d09135
lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call 
Previously filters wheren't passed to this call after the commit 0e09fdb8b0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
 2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin
873f55bac5
lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query 
The filters shouldn't be passed to DescribeAvailabilityZones API call.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287

Related commits:
0e09fdb8b0
d289ecded1
 2022-05-05 10:29:47 +03:00
Nikolay
7e58cba6cf
{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458) 
* {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite
moves aws related code into separate lib from lib/promscrape
it allows to write data from vmagent to the AWS managed prometheus (cortex)

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287

* Apply suggestions from code review

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-05-04 20:28:37 +03:00
Aliaksandr Valialkin
aa82987d70
lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing 
This reflects the logic used in Prometheus.

See https://github.com/prometheus/prometheus/pull/10080
 2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin
c2b13e6a04
lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects 
Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste
 2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin
a89e31b304
lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does 
See https://github.com/prometheus/prometheus/issues/9510
and https://github.com/prometheus/prometheus/pull/10080
 2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin
cc6eae6992
lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems 
Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.
 2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin
fea9d1e6ee
lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240

This is a follow-up for 2341bd48d7
 2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin
e9f08b1e6a
lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects 
There is high chance that ScrapeWork objects won't be generated because of relabeling
 2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin
909a3ee0e4
lib/promscrape: follow-up after 91e290a8ff 2022-04-20 16:12:26 +03:00
Nikolay
429848a67d
lib/promscrape: reduce latency for k8s GetLabels (#2454) 
replaces internStringMap with sync.Map - it greatly reduces lock contention
concurently reload scrape work for api watcher - each object labels added by dedicated CPU

changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70
 2022-04-20 16:12:25 +03:00
Aliaksandr Valialkin
d0bac8e224
all: typo fix: Kuberntes -> Kubernetes 2022-04-20 10:51:41 +03:00
Roman Khavronenko
5a4b16794d
Consul SD - update services on the watcher's start (#2202) 
* lib/discovery/consul: update services on the watcher's start

Previously, watcher's start was only initing goroutines for discovery
but not waiting for the first iteration to end. It means first Consul
discovery wasn't returning discovered targets until the next iteration.

The change makes the watcher's start blocking until we get first discovery
iteration done and all registries updated.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: remove workarounds for consul SD

Now when consul SD lib properly updates services
on the first start, we don't need workarounds in vmalert.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/discovery/consul: update after review

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-21 15:33:33 +02:00
Nikolay
c11d0949c8
fixes all_tenants query option usage for openstack service discovery (#2184) 
explicit use configuration parametr instead of conditional add
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182
 2022-02-14 13:13:03 +02:00
artifactori
943fc056ca
Show gce sdconfig zone on vmagent:8429/config (#2178) 
* vmagent: add test for marshalling gce sdconfig with ZoneYAML

* vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig
 2022-02-12 00:48:38 +02:00
Aliaksandr Valialkin
727b29d4a3
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpointslice_{label,annotation}* labels to be consistent with other role values for Kubernetes service discovery 2022-02-11 14:56:10 +02:00
Nikolay
265938a385
fixes service discovery for kubernetes (#2173) 
* fixes service discovery for kubernetes
now it must take in account all pods that belong to the discovered endpoint and endpointslice
adds simple test for endpoints
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134

* wip

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin
dd91759f1f
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_node_provider_id label for discovered Kubernetes nodes in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/9603
 2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin
bc18368c15
lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace 
See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881
 2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin
de8299f465
lib/promscrape/discovery/dockerswarm: follow up after 68a117a25a 
- Document the bugfix at docs/CHANGELOG.md
- Set __address__ field after copying commonLabels to the resulting map of discovered labels.
  This makes sure that the correct __address__ label is used.
 2022-01-11 09:22:03 +02:00
Alexander Shtuchkin
45a92e6ce1
Fix for #2038: Make correct __address__ value for dockerswarm promscrape (#2041) 2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin
847004fa77
app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764
 2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin
ad445a06cd
lib/promscrape: properly show proxy_url option value at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755
 2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin
ea69eef375
lib/promscrape/discovery/kubernetes: log a warning if role: endpoints discovers more than 1000 targets per a single endpoint 
In this case `role: endpointslice` must be used instead.

See the following references:

* https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity
* https://github.com/kubernetes/kubernetes/pull/99975
* https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398
 2021-10-19 13:22:28 +03:00
Aliaksandr Valialkin
3e9beb0f8d
lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name 
This is a follow-up for 31b42b30b6
 2021-10-15 12:27:31 +03:00
Nikolay
9be5689b3f
changes auth validation for openstack (#1663) 
* changes auth validation for openstack
must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-29 00:33:38 +03:00
Nikolay
1ab2f844a2 makes filters optional for ec2 api requests (#1627) 
filters can be applied only for DescribeInstances requests, like prometheus does.
related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
 2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin
b684624f67 lib/promscrape/discovery/docker: support host networking mode 
See https://github.com/prometheus/prometheus/issues/9116
 2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin
6ed9f10da5 lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery 
See https://github.com/prometheus/prometheus/issues/8902
 2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin
146c14d879 lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator 
This is a follow up commit after 31b42b30b6
 2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin
ca61d7c82b lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus 
See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)
 2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin
327034b54f lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices 
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
 2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin
4a2d7aec7f lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config 2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin
77bb9e1656 lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 
See https://github.com/prometheus/prometheus/pull/8978
 2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin
336a2aa2e0 lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does 2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin
6fc3696260 lib/promscrape/discovery/consul: use case-insensitive comparison for service names 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
 2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin
2edfea8c36 lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID 2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
4461e20e7d lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 13:22:16 +03:00
Lu Jiajing
12b4cbb68f Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 13:22:16 +03:00
Nikolay
501429c3ff adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin
f10fa0d1d7 lib/promscrape/discovery/consul: properly pass namespace to Consul watcher 
Follow-up for 58a2989fe7
 2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin
4adf6c9766 lib/promscrape/discovery/http: follow up after e307bbb29a 2021-06-22 13:42:10 +03:00
Nikolay
e03a3d3a36 adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin
3ab3902f17 lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does 2021-06-22 13:18:51 +03:00