Aliaksandr Valialkin
aa37e6b438
lib/promscrape: retry http requests if the server returns 429 status code
...
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin
c27bd63f6c
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html
2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin
1a00c9ef03
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11034
2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin
2fb63dda83
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11002
2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin
2b58bd9876
lib/promscrape/discovery/dns: add support for resolving MX records
...
See https://github.com/prometheus/prometheus/pull/10099
2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin
310779d8b5
lib/promscrape: follow-up after 2c553d5a2f
...
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
2022-08-08 14:49:16 +03:00
Fury
59fdb4cb72
add support to scrape multi tenant metrics ( #2950 )
...
* add support to scrape multi tenant metrics
* add support to scrape multi tenant metrics
Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
2022-08-08 14:49:15 +03:00
Aliaksandr Valialkin
77bd4e37cc
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress
...
See 7e65ad3e437e1111ff14
2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin
ecbe1ddf1b
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs
...
This option was ignored since d289ecded1https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin
80ecfcf759
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels
...
See https://github.com/prometheus/prometheus/issues/9187
2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin
85b04732ed
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default
...
This aligns with Prometheus behaviour.
See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin
17290a4598
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f
2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin
8ddad31eef
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
2022-08-04 22:28:21 +03:00
Igor Tiunov
0ba86fe87e
YC service discovery ( #2923 )
...
* YC service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
* Fixed linter suggestions
* fixed golint errors
2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin
1b5799f894
lib/promscrape: set up=0 for partially failed scrape in stream parsing mode
...
This behaviour aligns with Prometheus behavior
2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin
2d1366353c
lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
2022-07-18 17:15:42 +03:00
Nikolay
c007b129cb
lib/promscrape: adds azure service discovery ( #2743 )
...
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364
* makes linter happy
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* wip
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:45:43 +03:00
Aliaksandr Valialkin
5794886662
lib/promscrape: properly set Host header when sending requests via http proxy
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin
4d03ac90fc
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin
c4cc45d7f8
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does
...
See https://github.com/prometheus/prometheus/pull/10759
2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin
f9303e494c
lib/promscrape: fix a test after c66f676f3b
2022-07-06 13:25:17 +03:00
Aliaksandr Valialkin
498c6d6e72
lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target
...
See https://github.com/VictoriaMetrics/operator/issues/497
2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin
ecc11dc32d
lib/promauth: refactor NewConfig in order to improve maintainability
...
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
2022-07-04 14:31:43 +03:00
ttyv
00956e585d
lib/promscrape: fix vmagent tickerCh reload behaviour ( #2786 )
...
Co-authored-by: Dmitriy <dab@ttyv.ru>
2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin
71b0dfdefa
lib/promscrape: always send stale markers with the real scrape timestamp
...
This guarantees that query won't return data just after the series is disappeared.
2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin
3ae6300497
lib/promauth: add ability to send additional http headers in requests to scrape targets
...
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin
fe2269b999
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package
...
This package already has the same name, so there is no need in explicit name
2022-06-21 20:24:28 +03:00
Roman Khavronenko
2b5e1dee91
vmagent: update SD duration histogram metric if SD is active ( #2677 )
...
The change updates histogram for registering SD update duration
only SD is considered as `active`. SD is active if at least
one scraper for this SD has started.
This change supposed to reduce metrics cardinality produced
by duration histogram which gets updated even if SD isn't configured.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-07 15:53:06 +03:00
Aliaksandr Valialkin
b6e3c12811
lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string
...
This improves code readability and maintainability a bit, since the format string
is passed as string literal into fmt.Errorf.
2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin
68b6ddfb14
all: follow-up after 8edb390e21
...
- Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb
- Add /service-discovery page for API compatibility with Prometheus
- Properly load bootstrap.min.css from /prometheus/targets
- Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there
2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin
3dbb19d624
lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534
...
- make more clear error logs
- simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct
2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin
6c2fb9d8c4
lib/promscrape: add -promscrape.cluster.name command-line flag
...
This flag is used for proper data de-duplication when the same target is scraped
from multiple vmagent clusters.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679
2022-06-04 01:11:23 +03:00
Dmytro Kozlov
ce8aade80e
lib/promscrape: adds service discovery visualization for /targets page( #2675 )
...
* lib/promscrape: updated template
* lib/promscrape: fixed click on unhealthy and all btns
* app/vmselect: jquery scripts into static folder
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2022-06-04 01:11:23 +03:00
Nikolay
72e43ef2fe
lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911 ( #2672 )
2022-06-04 01:11:23 +03:00
hadesy
28d4624f60
promscrape/discovery: support kubeconfig ( #2533 )
2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin
afced37c0b
all: add initial support for query tracing
...
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin
386f6110ec
lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum
...
In this case there is no need in converting int64 to int
2022-06-01 01:43:25 +03:00
Dmytro Kozlov
cd1fa2e4cd
issue-2594: use embedded for static files ( #2650 )
...
embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries.
Co-authored-by: f41gh7 <nik@victoriametrics.com>
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594
2022-05-31 12:42:48 +03:00
Dmytro Kozlov
6add79143b
removed redundant return (fixed linter) ( #2647 )
...
* removed redundant return
* updated lint package version
2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin
f149d56ac2
lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag
...
This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575
The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 .
Thanks to @jelmd .
2022-05-25 23:00:30 +03:00
Roman Khavronenko
7406665fc3
lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery ( #2615 )
...
* lib/promscrape/discovery/kubernetes: properly updates discovered scrape works
previously, added or updated scrapeworks may override previuosly
discovered.
it happens because swosByKey may contain small subset of kubernetes
objects with it's labels.
It happens for objectsUpdated and objectsAdded maps, which include only changed elements
* Properly calculate vm_promscrape_discovery_kubernetes_scrape_works
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-21 01:17:21 +03:00
Boris Petersen
3a8b4fab97
Add ability to sign requests for all AWS services ( #2604 )
...
This adds the ability to utilize sigv4 signing for all AWS services not
just "aps". When the newly introduced property "service" is not set it
will default to "aps".
Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin
116c0b8f2e
docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag
2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin
0d0561ca8c
lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future
...
Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.
2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin
810dd74fb9
lib/promscrape: properly implement ScrapeConfig.clone()
...
Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields -
their contents was replaced with `<secret>` value.
This led to inability to use passwords and secrets in `-promscrape.config` file.
The bug has been introduced in v1.77.0 in the commit 67b10896d2https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551
2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin
af0da45d3e
lib/promscrape: rename promscrape_stale_samples_created_total metric to vm_promscrape_stale_samples_created_total, so its name is consistent with the rest of vm_promscrape_ metrics
2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin
9d40bb7137
lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config via az_filters section
2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin
2ce1d09135
lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call
...
Previously filters wheren't passed to this call after the commit 0e09fdb8b0https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin
873f55bac5
lib/awsapi: pass filtersQueryString arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query
...
The filters shouldn't be passed to DescribeAvailabilityZones API call.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
Related commits:
0e09fdb8b0d289ecded1
2022-05-05 10:29:47 +03:00
Nikolay
7e58cba6cf
{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite ( #2458 )
...
* {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite
moves aws related code into separate lib from lib/promscrape
it allows to write data from vmagent to the AWS managed prometheus (cortex)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
* Apply suggestions from code review
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-04 20:28:37 +03:00
