Aliaksandr Valialkin
7bf2cbad32
lib/promscrape: reduce default value for -promscrape.discovery.concurrency
from 500 to 100
...
This should reduce load on Kubernetes API server and Consul when big number of targets are discovered
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
2020-06-20 17:53:42 +03:00
Aliaksandr Valialkin
6ff821c70d
lib/promscrape/discovery/ec2: expose __meta_ec2_ami
like the next Prometheus release will do
...
See b5d61fb66c
for details
2020-06-20 17:45:23 +03:00
Aliaksandr Valialkin
5820c0ffb7
lib/promrelabel: allows regex capture groups in target_label like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569
2020-06-19 02:21:24 +03:00
Aliaksandr Valialkin
c602284a99
lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response
2020-05-24 14:41:14 +03:00
Aliaksandr Valialkin
f9847352b4
app/vmagent: add -dryRun
option for checking all the configs mentioned in command-line flags without running vmagent
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin
d1a9d8aa1c
lib/promscrape: add -promscrape.config.dryRun
flag for checking -promscrape.config
for errors or unsupported options
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 14:55:11 +03:00
Aliaksandr Valialkin
cf87b810b7
lib/promscrape: add -promscrape.discovery.concurrency
and -promscrape.discovery.concurrentWaitTime
flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes
2020-05-19 17:35:53 +03:00
Aliaksandr Valialkin
0afd48d2ee
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
0a134ace63
app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin
8300cc17af
app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set
2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin
68928bf3df
lib/promscrape/discovery/gce: discover per-zone instances for gce_sd_config
in parallel. This should reduce discovery latency
2020-05-06 15:00:09 +03:00
Aliaksandr Valialkin
3f52a97f9b
lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs
2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin
364789c24c
lib/promscrape: properly connect to TCP6 addresses if -enableTCP6
is set
2020-05-06 00:01:57 +03:00
Aliaksandr Valialkin
7b5ef63384
lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP
2020-05-05 10:54:09 +03:00
Aliaksandr Valialkin
4fa817be10
lib/promscrape: allow explicitly setting empty token via token: ""
in consul_sd_config
2020-05-05 07:50:15 +03:00
Aliaksandr Valialkin
40c3ffb359
lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin
218e566647
lib/promscrape: move common code for discovery api config map handling into discoveryutils
2020-05-04 20:51:01 +03:00
Aliaksandr Valialkin
6310b20e72
lib/promscrape/discovery/kubernetes/: unify apiConfig creation
2020-05-04 20:50:49 +03:00
Aliaksandr Valialkin
66b0ae79a5
lib/promscrape: remove debug line left after the commit e4aac6ea40
2020-05-03 17:15:32 +03:00
Aliaksandr Valialkin
69004a5f67
lib/promscrape: fix tests after the commit 658a8742ac
...
The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does.
See https://www.robustperception.io/life-of-a-label for details.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453
2020-05-03 16:56:15 +03:00
DexterZhang
658a8742ac
fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] ( #454 )
2020-05-03 16:51:03 +03:00
Aliaksandr Valialkin
e4aac6ea40
lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels
...
This should make consistent intervals between data points for scrape targets across reloads.
Previously these intervals were random.
2020-05-03 14:30:21 +03:00
Aliaksandr Valialkin
d9f1b4d6a3
lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after 3b234d82e5
2020-05-03 14:28:31 +03:00
Aliaksandr Valialkin
3b234d82e5
lib/promscrape: reload only modified scrapers on config changes
...
This should improve scrape stability when big number of targets are scraped and these targets are frequently changed.
Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests:
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460
2020-05-03 12:45:40 +03:00
Aliaksandr Valialkin
5ec036439d
lib/promscrape: set 30 seconds timeout for discovery api requests
...
Previously such requests could hang for long time. This could make debugging harder.
2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
43c39dc36c
vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp
...
The upstream fasthttp may contain issues like 996610f021
,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
521df0e2fc
lib/promscrape: handle connection reset when targets responds with http redirect
2020-04-28 02:13:02 +03:00
肖贝贝
2b16c188e8
fix: vmagent not follow 301/302 redirect bug ( #445 )
...
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
2020-04-28 01:29:37 +03:00
Aliaksandr Valialkin
95942f1ac6
lib/promscrape/discovery/gce: make golangci-lint happy
2020-04-27 19:28:10 +03:00
Aliaksandr Valialkin
b768bc9a6a
lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs
2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin
de59703a16
lib/promscrape/discovery/gce: properly set filter
query arg in api url
2020-04-27 16:01:17 +03:00
Aliaksandr Valialkin
0224071ebe
lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config
2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin
6954d0edb7
lib/promscrape/discovery/gce: allow empty zone
arg in gce_sd_config
- in this case zones for the given project are automatically discovered
2020-04-26 14:34:11 +03:00
Aliaksandr Valialkin
7b8008e0bd
lib/promscrape/discovery/gce: make golint happy by ignoring resp.Body.Close() result
2020-04-24 18:13:09 +03:00
Aliaksandr Valialkin
9ef5935552
lib/promscrape: initial implementation for gce_sd_configs
aga Prometheus-compatible service discovery for Google Compute Engine
2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin
24461153bf
lib/promscrape: query /api/v1/namespaces/*
for the configured namespaces in kubernetes_sd_config
...
This should fix authroization issues described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
2020-04-24 14:33:50 +03:00
Aliaksandr Valialkin
00e897119f
lib/promscrape: add -promscrape.configCheckInterval
command-line flag for automating config checking
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/431
2020-04-23 23:41:08 +03:00
Aliaksandr Valialkin
a9a7a7175e
lib/promscrape: access Config entries by reference, so they can be compared by addresses
2020-04-23 14:38:20 +03:00
Aliaksandr Valialkin
1c5d14a2eb
lib/promscrape: move KubernetesSDConfig to lib/promscrape/discovery/kubernetes
2020-04-23 11:34:22 +03:00
Aliaksandr Valialkin
a714568374
lib/promscrape/discovery/kubernetes: hide role switch logic behind GetLabels function
2020-04-22 22:16:11 +03:00
Aliaksandr Valialkin
5454b518a6
lib/promscrape/discovery/kubernetes: reuse a client for empty api_server
inside different jobs
2020-04-20 17:07:11 +03:00
Aliaksandr Valialkin
43375df923
lib/promscrape/discovery/kubernetes: update stale comments
2020-04-17 14:06:20 +03:00
Aliaksandr Valialkin
5d1537a395
lib/promscrape: suppress scrape errors if -promscrape.suppressScrapeErrors
flag is set
2020-04-16 23:41:30 +03:00
Aliaksandr Valialkin
600490131f
lib/promscrape: print all the labels for the target on error message for failed scrape
...
This should improve debuggability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/420
2020-04-16 23:35:05 +03:00
Aliaksandr Valialkin
bd4c6d21dd
lib/promscrape: retry target scraping when the target closes previously established keep-alive connection to it
...
This should fix the following error:
the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
2020-04-16 23:25:29 +03:00
Aliaksandr Valialkin
426a0567c4
lib/promscrape: code cleanup in runScraper func
2020-04-15 11:36:24 +03:00
Aliaksandr Valialkin
067c7afebc
lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets
page
...
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin
ac35635b71
lib/promscrape/discovery/kubernetes: remove only unused client for API server during cleaning
2020-04-14 14:19:21 +03:00
Aliaksandr Valialkin
78863d7066
lib/promscrape: add promrelabel.GetLabelValueByName helper function
2020-04-14 14:12:01 +03:00
Aliaksandr Valialkin
c64f003cfb
lib/promscrape: mention job name in error messages when target cannot be scraped
...
This should improve debuggability
2020-04-14 13:33:13 +03:00
Aliaksandr Valialkin
4718a5d951
lib/promscrape: reset ScrapeWork.ID in tests
2020-04-14 13:31:31 +03:00
Aliaksandr Valialkin
257521a634
lib/promscrape: properly expose statuses for targets with duplicate scrape urls at /targets
page
...
Previously targets with duplicate scrape urls were merged into a single line on the page.
Now each target with duplicate scrape url is displayed on a separate line.
2020-04-14 13:10:01 +03:00
Aliaksandr Valialkin
6a75c95194
lib/promscrape: remove labels starting with __meta_
after applying relabel_configs
as Prometheus does
...
This should reduce CPU load during scraping when target discovery generates
big number of `__meta_*` labels (for instance, k8s discovery).
See https://www.robustperception.io/life-of-a-label for details.
2020-04-14 12:23:22 +03:00
Aliaksandr Valialkin
01d7d799dc
lib/promscrape: rename 'scrape_config->scrape_limit' to 'scrape_config->sample_limit'
...
`scrape_config` block from Prometheus config contains `sample_limit` field,
while in `vmagent` this field was mistakenly named as `scrape_limit`.
2020-04-14 11:59:57 +03:00
Aliaksandr Valialkin
2e4e202c2b
lib/promscrape: add initial support for kubernetes_sd_config
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334
2020-04-13 21:03:28 +03:00
Aliaksandr Valialkin
2814b1490f
lib/promscrape: add -promscrape.config.strictParse
flag for detecting errors in -promscrape.config
file
2020-04-13 13:15:44 +03:00
Aliaksandr Valialkin
90b4a6dd12
lib/promscrape: extract common auth code to lib/promauth
2020-04-13 12:59:10 +03:00
Aliaksandr Valialkin
3e55c7e069
lib/promscrape: reduce timestamp jitter when scraping targets
...
This should improve compression for timestamps
2020-04-01 16:11:35 +03:00
Aliaksandr Valialkin
499594f421
lib/promscrape: allow overriding external_labels as Prometheus does
...
Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say:
> In communication with external systems, they are always applied only
> when a time series does not have a given label yet and are ignored otherwise.
Though this may result in consistency chaos when scrape targets override `external_labels`,
let's stick with Prometheus behavior for the sake of backwards compatibility.
There is last resort in vmagent with `-remoteWrite.label`, which consistently
sets the configured labels to all the metrics before sending them to remote storage.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366
2020-03-12 20:24:42 +02:00
Aliaksandr Valialkin
d4beb17ebe
lib/promscrape: remove possible races when registering and de-registering scrape workers for /targets
page
2020-03-11 16:30:21 +02:00
Aliaksandr Valialkin
cdf70b7944
lib/promscrape: consistently update /targets
page after SIGHUP
2020-03-11 03:20:03 +02:00
Aliaksandr Valialkin
49d7cb1a3f
all: fix golangci-lint
issues
2020-03-10 19:41:46 +02:00
Aliaksandr Valialkin
7c432da788
lib/promscrape: do not retry idempotent requests when scraping targets
...
This should prevent from the following unexpected side-effects of idempotent request retries:
- increased actual timeout when scraping the target comparing to the configured scrape_timeout
- increased load on the target
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357
2020-03-09 13:31:52 +02:00
Aliaksandr Valialkin
986dba5ab3
app/vmagent: do not allow non-supported fields in -remoteWrite.relabelConfig
and file_sd_configs
...
This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363
2020-03-06 20:19:13 +02:00
Aliaksandr Valialkin
9a944fd169
lib/promscrape: consistency renaming: stopCh -> globalStopCh
2020-03-03 20:08:08 +02:00
Aliaksandr Valialkin
b63e4464f4
lib/promscrape: properly reload new configs on SIGHUP
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/335
2020-02-26 13:54:00 +02:00
Aliaksandr Valialkin
6739c2749d
lib/promscrape: go fmt
2020-02-25 20:56:44 +02:00
Aliaksandr Valialkin
7a33da8fea
lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331
2020-02-25 20:49:50 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00