Aliaksandr Valialkin
6608705652
app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules
...
Previously only the lower part of 64-bit hash was used for calculating the offset.
This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash
for calculating the offset.
2021-10-27 19:59:16 +03:00
Aliaksandr Valialkin
4821adfd95
lib/promscrape: properly show proxy_url
option value at /config
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755
2021-10-26 21:23:54 +03:00
Aliaksandr Valialkin
7fa15f7f86
lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set
...
The response body isn't used if -promscrape.noStaleMarkers is set after the commit 2876137c92
,
so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694
2021-10-22 16:44:44 +03:00
Aliaksandr Valialkin
6106d4069d
lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed
...
This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels.
See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets
This is a follow-up after 9882cda8b9
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728
2021-10-22 13:54:30 +03:00
Aliaksandr Valialkin
2876137c92
lib/promscrape: reduce memory usage if -promscrape.noStaleMarkers
command-line flag is passed
...
Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled.
This should reduce memory usage when the scraped targets return large responses.
2021-10-22 13:10:29 +03:00
Aliaksandr Valialkin
3408a05d12
lib/promscrape/discovery/kubernetes: log a warning if role: endpoints
discovers more than 1000 targets per a single endpoint
...
In this case `role: endpointslice` must be used instead.
See the following references:
* https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity
* https://github.com/kubernetes/kubernetes/pull/99975
* https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398
2021-10-19 13:20:40 +03:00
Nikolay
cbcc622786
changes job source for /target api ( #1723 )
...
use jobNameOriginal instead of relabeled as prometheus does
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707
2021-10-19 08:49:36 +03:00
Aliaksandr Valialkin
c37f285466
lib/promscrape: set honor_timestamps: true
by default if this option isnt set explicitly in scrape configs
...
This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
2021-10-16 20:49:08 +03:00
Aliaksandr Valialkin
c055bc478c
lib/promscrape: expose promscrape_series_limit_max_series
and promscrape_series_limit_current_series
metrics per each scrape target with the enabled unique series limiter
2021-10-16 18:47:13 +03:00
Aliaksandr Valialkin
06b0982d6b
lib/promscrape: always initialize http client for stream parsing mode
...
Stream parsing mode can be automatically enabled when scraping targets with big response bodies
exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.
2021-10-16 13:18:23 +03:00
Aliaksandr Valialkin
32793adbd9
lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse
...
This should reduce memory usage when scraping targets with big response bodies.
2021-10-16 13:00:30 +03:00
Aliaksandr Valialkin
9866dd95c1
lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice
...
This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode
at the cost of the increased memory usage, since now the potentially big response is kept
in the lastScrape byte slice per each scrapeWork.
In practice the memory usage increase shouldn't be big, since the response size
is usually much smaller than the parsed metrics from this response after the relabeling,
which usually adds a big pile of target-specific labels per each metric.
2021-10-15 15:39:23 +03:00
Aliaksandr Valialkin
f6d33596ff
lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name
...
This is a follow-up for 31b42b30b6
2021-10-15 12:27:12 +03:00
Aliaksandr Valialkin
bbd34fa15e
lib/promscrape: add -promscrape.minResponseSizeForStreamParse
command-line option for automatic switching to stream parsing mode when scraping targets with big responses
...
This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes.
This is common case in Kubernetes monitoring.
2021-10-14 12:29:35 +03:00
Aliaksandr Valialkin
1a7287c408
lib/promscrape: return error if sample_limit
or series_limit
options are set when stream parsing mode is enabled
2021-10-14 12:11:23 +03:00
Aliaksandr Valialkin
e3c8304deb
lib/promscrape: add ability to show the original labels for discovered targets at /targets page
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698
2021-10-13 15:59:58 +03:00
Roman Khavronenko
c0a932a55f
lib/promscrape: make errcheck happy ( #1703 )
2021-10-13 14:57:30 +03:00
Aliaksandr Valialkin
9882cda8b9
lib/promscrape: shard targets among cluster nodes after relabeling is applied
...
This guarantees that targets with the same set of labels go to the same vmagent node.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495
2021-10-12 17:06:00 +03:00
Aliaksandr Valialkin
5a58c041c2
app/vmagent: expose -promscrape.config contents at /config page as Prometheus does
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695
2021-10-12 16:25:37 +03:00
Aliaksandr Valialkin
873aac584e
lib/promscrape: use Prometheus format for target labels at /targets
page
...
This should simplify copy-pasting the labels to/from PromQL / MetricsQL
2021-10-11 12:41:37 +03:00
Ziqi Zhao
402c995d6d
fix some typos ( #1678 )
...
Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>
2021-10-06 14:43:10 +03:00
Aliaksandr Valialkin
6ee66fb6b1
lib/promscrape: reduce memory allocations in mergeLabels() after 48e3e6c8df
2021-09-30 16:56:12 +03:00
Nikolay
3d17112a7e
changes auth validation for openstack ( #1663 )
...
* changes auth validation for openstack
must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-09-29 00:28:49 +03:00
Aliaksandr Valialkin
a0313c046b
lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total
metric for counting of the failed scrapes due to the exceeded response size
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639
2021-09-23 14:47:54 +03:00
Nikolay
0e09fdb8b0
makes filters optional for ec2 api requests ( #1627 )
...
filters can be applied only for DescribeInstances requests, like prometheus does.
related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
2021-09-17 18:00:37 +03:00
Aliaksandr Valialkin
fb6ed0ce19
lib/promscrape/discovery/docker: support host networking mode
...
See https://github.com/prometheus/prometheus/issues/9116
2021-09-13 13:30:16 +03:00
Aliaksandr Valialkin
6295861acd
lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery
2021-09-13 13:03:42 +03:00
Aliaksandr Valialkin
728c4c3841
lib/promscrape: generate scrape_timeout_seconds
metric per each scrape target in the same way as Prometheus 2.30 does
...
See https://github.com/prometheus/prometheus/pull/9247
2021-09-12 15:20:44 +03:00
Aliaksandr Valialkin
0b4eb0fa7d
lib/promscrape: make fmt
2021-09-12 13:34:15 +03:00
Aliaksandr Valialkin
48e3e6c8df
lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling
...
See https://github.com/prometheus/prometheus/pull/8911
2021-09-12 13:33:41 +03:00
Aliaksandr Valialkin
f3e89754a9
lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added
metric
...
Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics.
The main idea is to perform the calculations on scrape_series_added and series_limit
only if the set of metrics exposed by the target has been changed.
Scrape targets rarely change the set of exposed metrics,
so this optimization should reduce CPU usage in general case.
2021-09-12 12:53:14 +03:00
Aliaksandr Valialkin
9286107e82
lib/promscrape: send stale markers for disappeared metrics like Prometheus does
2021-09-11 10:51:04 +03:00
Aliaksandr Valialkin
4aeb8db83f
lib/promscrape: add ability to set series_limit
and stream_parse
options via relabeling
...
This allows managing these options on a per-target basis.
Typical use case: to manage these options for pods via Kubernetes annotations.
2021-09-09 18:49:39 +03:00
Aliaksandr Valialkin
468f941f7e
lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric
2021-09-09 17:37:37 +03:00
Aliaksandr Valialkin
086b5d0cf1
lib/promscrape: add scrape_
prefix to job
and target
labels exported by promscrape_series_limit_rows_dropped_total
metric
...
This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.
2021-09-09 17:29:21 +03:00
Aliaksandr Valialkin
f77dde837a
lib/promscrape: add the ability to limit the number of unique series per each scrape target
...
The number of series per target can be limited with the following options:
* Global limit with `-promscrape.maxSeriesPerTarget` command-line option.
* Per-target limit with `max_series: N` option in `scrape_config` section.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561
2021-09-01 16:03:59 +03:00
Aliaksandr Valialkin
5c63d69454
lib/promscrape/discovery/kubernetes: return back support role: endpointslices
, since it is used by VictoriaMetrics operator
...
This is a follow up commit after 31b42b30b6
2021-08-29 12:37:03 +03:00
Aliaksandr Valialkin
31b42b30b6
lib/promscrape/discovery/kubernetes: rename role: endpointslices
to role: endpointslice
to be consistent with Prometheus
...
See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)
2021-08-29 11:23:08 +03:00
Aliaksandr Valialkin
2e001db4de
lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress
and role: endpointslices
...
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122
The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
2021-08-29 11:16:59 +03:00
Aliaksandr Valialkin
10f960fa0c
lib/promscrape: add ability to load scrape configs from multiple files
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
2021-08-26 08:51:16 +03:00
Aliaksandr Valialkin
c27ee35c5c
lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config
2021-08-25 13:05:49 +03:00
Aliaksandr Valialkin
d5622b32e2
lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets
...
Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.
This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
2021-08-21 21:17:26 +03:00
Aliaksandr Valialkin
f46a73dcdd
lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape
...
This will make timestamps for stale markers more consistent for timestamps for other samples
2021-08-19 14:18:05 +03:00
Aliaksandr Valialkin
c09446a9aa
lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does
2021-08-18 21:59:03 +03:00
Aliaksandr Valialkin
cdc372bb98
app/vmselect: add -search.noStaleMarkers
command-line flag for disabling stale markers handling in queries
...
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
2021-08-18 13:59:02 +03:00
Aliaksandr Valialkin
226143f31b
lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag
...
This option can be useful when vmagent consumes too much additional memory
for staleness markers functionality and when staleness markers aren't needed.
2021-08-18 13:43:21 +03:00
Aliaksandr Valialkin
03c959f1df
lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets
...
This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
2021-08-17 00:55:51 +03:00
Aliaksandr Valialkin
a0e18f06eb
lib/promscrape: restore red highlighting for DOWN targets at /targets page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461
2021-08-15 16:03:57 +03:00
Aliaksandr Valialkin
4401464c22
all: add support for Prometheus staleness markers
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2021-08-13 12:10:17 +03:00
Aliaksandr Valialkin
d826352688
app/vmagent: follow-up after fe445f753b
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
2021-08-05 09:52:32 +03:00
Omar Ghader
46e27d60a6
feature: Add multitenant for vmagent ( #1505 )
...
* feature: Add multitenant for vmagent
* Minor fix
* Fix rcs index out of range
* Minor fix
* Fix multi Init
* Fix multi Init
* Fix multi Init
* Add default multi
* Adjust naming
* Add TenantInserted metrics
* Add TenantInserted metrics
* fix: remove unused metrics for vmagent
* fix: remove unused metrics for vmagent
Co-authored-by: mghader <marc.ghader@ubisoft.com>
Co-authored-by: Sebastian YEPES <syepes@gmail.com>
2021-08-05 09:52:31 +03:00
Aliaksandr Valialkin
50663ba41f
lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29
...
See https://github.com/prometheus/prometheus/pull/8978
2021-08-03 16:11:49 +03:00
Aliaksandr Valialkin
3cad8b4564
lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id
label as Prometheus 2.29 does
2021-08-03 16:11:49 +03:00
Aliaksandr Valialkin
1318736ad1
lib/promscrape: add missing whitespace at /targets page before up
word
2021-07-26 12:22:59 +03:00
Aliaksandr Valialkin
7088f17494
lib/promscrape/discovery/consul: use case-insensitive comparison for service names
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
2021-07-02 14:49:27 +03:00
Aliaksandr Valialkin
158c50c0ee
lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does
...
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
2021-07-01 15:27:47 +03:00
Nikolay
ae485c2bfd
fixes /targets button style ( #1423 )
...
* fixes /targets button style
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422
* updates boostrap version
2021-07-01 11:48:07 +03:00
Aliaksandr Valialkin
9ce211a514
lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID
2021-06-29 13:12:28 +03:00
Aliaksandr Valialkin
1b0501a09e
lib/promscrape: typo fix in /targets
output
...
The typo has been introduced in fb72a2133f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
2021-06-28 21:26:37 +03:00
Aliaksandr Valialkin
cb5453953f
lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common
...
This is a follow up after c85a5b7fcb
2021-06-25 13:20:20 +03:00
Aliaksandr Valialkin
a69045e440
lib/promscrape: consistently sort service discovery routines
...
This should simplify further maintenance of the code
2021-06-25 12:10:46 +03:00
Lu Jiajing
c85a5b7fcb
Support Docker ServiceDiscovery ( #1402 )
...
* add docker discovery
* add test
* add labels test and add scrape work
* remove TODO
* refactor to merge apiConfig and sdConfig
* apply suggestion
2021-06-25 11:42:47 +03:00
Nikolay
434e33da9b
adds missing MustStop call to do and http sd ( #1404 )
2021-06-25 11:39:18 +03:00
Aliaksandr Valialkin
e8a5bb92b7
lib/promscrape/discovery/consul: properly pass namespace to Consul watcher
...
Follow-up for 58a2989fe7
2021-06-22 17:42:41 +03:00
Aliaksandr Valialkin
ac54f34f9e
lib/promscrape/discovery/http: follow up after e307bbb29a
2021-06-22 13:40:33 +03:00
Aliaksandr Valialkin
755040a171
lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does
2021-06-22 13:34:02 +03:00
Nikolay
e307bbb29a
adds http_sd ( #1399 )
...
* adds http_sd
* adds X-Prometheus-Refresh-Interval-Seconds header
* Update lib/promscrape/discovery/http/api.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-06-22 13:33:37 +03:00
Nikolay
58a2989fe7
adds consul enterprise namespace support ( #1400 )
...
* adds consul enterprise namespace support
* Update lib/promscrape/discovery/consul/consul.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-06-22 12:49:44 +03:00
Aliaksandr Valialkin
fb72a2133f
lib/promscrape: show jobs with empty scrape targets on /targets page
2021-06-18 10:53:52 +03:00
Nikolay
6c434b260e
fixes DO service discovery labels ( #1389 )
...
adds test for digitalocean sd
2021-06-17 15:12:20 +03:00
Aliaksandr Valialkin
36d55bff66
lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377
2021-06-14 14:04:00 +03:00
Nikolay
729c4eeb9c
adds digital ocean sd ( #1376 )
...
* adds digital ocean sd config
* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367
* typo fix
2021-06-14 13:15:04 +03:00
Aliaksandr Valialkin
06b8e7d148
lib/promscrape: increase the duration for reading the full response in stream parsing mode
...
Increase the duration from 10x to 30x of the configured `scrape_interval'.
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
2021-06-14 12:28:09 +03:00
Aliaksandr Valialkin
78f83dc5ad
app/{vmagent,vminsert}: follow-up after 2fe045e2a4
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
2021-06-04 20:27:58 +03:00
jelmd
2fe045e2a4
new feature: debug relabeling ( #1344 )
...
* new feature: relabel logging
Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.
* vminsert wants relabel logging, too.
2021-06-04 17:50:23 +03:00
Hason Chan
6f19bb23a1
fix eureka_sd_configs HTTPClientConfig incorrect parsing ( #1350 )
2021-06-04 11:47:17 +03:00
Aliaksandr Valialkin
a52a20659a
lib/promscrape: fix tests after f0c21b6300
2021-05-28 01:32:50 +03:00
Aliaksandr Valialkin
60341722d5
docs: document f0c21b6300
2021-05-27 15:03:30 +03:00
faceair
f0c21b6300
lib/promscrape: apply body size & sample limit to stream parse ( #1331 )
...
* lib/promscrape: apply body size limit to stream parse
Signed-off-by: faceair <git@faceair.me>
* lib/promscrape: apply sample limit to stream parse
Signed-off-by: faceair <git@faceair.me>
2021-05-27 14:52:44 +03:00
Nikolay
5b8176c68e
basic OAuth2 support for remoteWrite and scrape targets ( #1316 )
...
* adds OAuth2 support for remoteWrite and scrapping
* adds tests
changes init
2021-05-22 16:20:18 +03:00
Aliaksandr Valialkin
c54bb73867
all: do not skip SIGHUP signal during service initialization
...
This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-21 16:34:06 +03:00
Aliaksandr Valialkin
22585531ad
lib/promscrape/discovery/kubernetes: make golangci-lint
happy by removing empty branches
2021-05-20 12:00:29 +03:00
Aliaksandr Valialkin
eb8093ca6b
lib/promscrape/discovery/kubernetes: reload objects on object parse error
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-18 23:25:48 +03:00
Aliaksandr Valialkin
0cf1fe19e6
lib/promscrape/discovery/kubernetes: simplify the reload logic for urlWatcher.objectsByKey
2021-05-18 15:40:46 +03:00
Aliaksandr Valialkin
bfb61de606
lib/promscrape/discovery/kubernetes: properly update vm_promscrape_discovery_kubernetes_scrape_works metric
...
Previously it wasn't descreased during config update.
2021-05-18 15:33:45 +03:00
Aliaksandr Valialkin
3166994244
lib/promscrape/discovery/kubernetes: log errors and stop service discovery when unexpected updates are received from Kubernetes API server
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-18 15:10:48 +03:00
Aliaksandr Valialkin
2eed410466
lib/promscrape/discovery/kubernetes: key ScrapeWork objects by urlWatcher instead of namespace
...
This makes the code less fragile if urlWatcher would depend on additional to namepsace properties.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-17 23:47:08 +03:00
Aliaksandr Valialkin
733706e6c6
lib/promscrape: reload auth tokens from files every second
...
Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart.
Now there is no need in vmagent restart.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297
2021-05-14 20:00:08 +03:00
Aliaksandr Valialkin
10a47af631
app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport
...
By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300
2021-05-14 18:12:24 +03:00
Aliaksandr Valialkin
fc3519fa26
lib/promscrape: limit scrape_timeout
by scrape_interval
like Prometheus does
2021-05-13 16:09:45 +03:00
匠心零度
b4f5be8bd8
fix vagent imbalance problem ( #1292 )
...
/path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ...
/path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ...
/path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ...
Co-authored-by: lirenzuo <lirenzuo@shein.com>
2021-05-13 11:14:51 +03:00
Aliaksandr Valialkin
e6fda03e8f
vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289
2021-05-13 10:44:14 +03:00
Aliaksandr Valialkin
f6a641de62
lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services
...
This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289
2021-05-13 10:38:50 +03:00
Aliaksandr Valialkin
cfd6aa28e1
lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices scrape targets every 5 seconds, since they may depend on changed service and pod objects
...
This should make endpoints and endpointslices scrape targets eventually consistent with the maximum delay of 5 seconds after the related service or pod object changes.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-12 14:10:34 +03:00
Aliaksandr Valialkin
904bbffc7f
lib/promscrape/discovery/kubernetes: start watchers for pods and services before starting watchers for endpoints
...
This should eliminate possible race when an update on endpoints depends on pods and/or services, which are missing in the cache yet.
This could result in missing targets based on endpoints or endpointslices.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-05 12:23:50 +03:00
Aliaksandr Valialkin
2ab1266593
lib/promscrape/discovery/kubernetes: remove a mutex at urlWatcher - use groupWatcher mutex for accessing all the urlWatcher children
...
This simplifies the code a bit and reduces the probability of improper mutex handling and deadlocks.
2021-04-29 10:14:26 +03:00
Nikolay
4e5a88114a
vmagent kubernetes_sd tests ( #1253 )
...
* first part of tests for kubernetes sd
* makes linter happy
* added more test cases
* adds pub/sub for tests
2021-04-29 10:10:24 +03:00
Aliaksandr Valialkin
2ec7d8b384
lib/promscrape/discovery/kubernetes: fix a deadlock introduced in eddba29664
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
Thanks to @f41gh7 for providing the initial idea for deadlock fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1248
2021-04-27 14:57:51 +03:00
Aliaksandr Valialkin
908e35affd
lib/promscrape: apply scrape_timeout
on receiving the first response byte for stream_parse: true
scrape targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1017#issuecomment-767235047
2021-04-23 21:53:35 +03:00
Aliaksandr Valialkin
eddba29664
lib/promscrape/discovery/kubernetes: refresh role: endpoints
targets on service object removal as Prometheus does
...
This is a follow-up for ae37cfd528
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-04-23 20:26:57 +03:00
Aliaksandr Valialkin
ae37cfd528
lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices targets on service object update like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-04-23 20:11:40 +03:00
Aliaksandr Valialkin
6bc52fe41a
all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com
2021-04-20 20:16:17 +03:00
Aliaksandr Valialkin
3f0bcbe067
lib/promscrape: create a single swosFunc
per scrape_config
2021-04-08 09:31:48 +03:00
Aliaksandr Valialkin
5a0938d807
lib/promscrape: do not spend CPU time on constructing scrapeWork key if clustering is disabled
2021-04-07 21:54:22 +03:00
Aliaksandr Valialkin
59f9960992
lib/promscrape/discovery: remove superflouos check in registerPendingAPIWatchers
...
The check `_, ok := uw.aws[aw]; !ok` isn't needed, since aw cannot exist in uw.aws
because of the check inside subscribeAPIWatcher
2021-04-07 13:07:39 +03:00
Aliaksandr Valialkin
3ec6639bbb
lib/promscrape/discovery/kubernetes: register pending apiWatchers in uw.aws
2021-04-06 11:12:13 +03:00
Aliaksandr Valialkin
5f593b0ed3
lib/promscrape/discovery/kubernetes: remove superflouos mustStart and mustStop functions
2021-04-05 22:44:12 +03:00
Lu Jiajing
b59164cf33
fix access to nil *url.URL ( #1180 )
...
* fix access to nil *url.URL
Signed-off-by: Megrez Lu <lujiajing1126@gmail.com>
* Update lib/promscrape/discovery/kubernetes/api_watcher.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-04-05 22:25:31 +03:00
Aliaksandr Valialkin
b46194472f
lib/promscrape/discovery/kubernetes: reduce CPU time spent on registering big number of Kubernetes objects shared among big number of scrape jobs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182
2021-04-05 22:04:30 +03:00
Aliaksandr Valialkin
a51d0ec6ec
lib/promscrape/discovery/kubernetes: load objects missing in local cache from api seriver in getObjectByRole()
...
This should fix possible race for `role: endpoints` and `role: endpointslices` service discovery,
when the referred `pod` and `service` objects aren't propagated to urlWatcher cache yet.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182#issuecomment-813353359 for details.
2021-04-05 20:31:17 +03:00
Aliaksandr Valialkin
f010d773d6
lib/promscrape/discovery/kubernetes: synchronously load Kubernetes objects on first access
...
Remove async registration of apiWatchers, since it breaks discovering `role: endpoints` and `role: endpointslices` targets,
which depend on pod and service objects.
There is no need in reloading `endpoints` and `endpointslices` targets if the referenced `pod` or `service` objects change,
since in this case the corresponding `endpoints` and `endpointslices` objects should also change because they contain
ResourceVersion of the referenced `pod` or `service` objects, which is modified on object update.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1182
2021-04-05 14:20:12 +03:00
Aliaksandr Valialkin
6742839fd6
lib/promscrape: pass X-Prometheus-Scrape-Timeout-Seconds
header to scrape targets as Prometheus does
2021-04-05 12:15:24 +03:00
Aliaksandr Valialkin
500e625e8c
lib/promscrape: properly send full url in GET
request via simple HTTP proxy
...
This is a follow-up for a0ae0f86666a75ec57b45eab2429da7ab4a7b250
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179
2021-04-04 01:20:06 +03:00
Aliaksandr Valialkin
5153410ced
lib/promscrape: support for simple HTTP proxies without CONNECT
method support such as https://github.com/prometheus-community/PushProx
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179
2021-04-04 00:40:40 +03:00
Aliaksandr Valialkin
4c56b1a6dd
lib/promscrape: add tests for authorization
config, which has been added in df148f48b7
2021-04-03 22:13:22 +03:00
Aliaksandr Valialkin
df148f48b7
lib/promscrape: add support for authorization
config in -promscrape.config
as Prometheus 2.26 does
...
See https://github.com/prometheus/prometheus/pull/8512
2021-04-02 21:17:45 +03:00
Aliaksandr Valialkin
7f9c68cdcb
lib/promscrape: add follow_redirect
option to scrape_configs
section like Prometheus does
...
See https://github.com/prometheus/prometheus/pull/8546
2021-04-02 19:56:40 +03:00
Aliaksandr Valialkin
5b08e6fb16
lib/promscrape/discovery/kubernetes: properly track objects with the same names in multiple namespaces
...
This is a follow-up for 12e4785fe8
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170
2021-04-02 14:45:32 +03:00
Aliaksandr Valialkin
12e4785fe8
lib/promscrape/discovery/kubernetes: properly discover targets in multiple namespaces
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170
2021-04-02 14:28:30 +03:00
Nikolay
fdb8995642
Adds aws ECS credentials support ( #1175 )
2021-04-02 11:56:40 +03:00
Aliaksandr Valialkin
f39c84b21f
lib/promscrape/discovery/kubernetes: typo fix in error message
2021-03-26 12:46:14 +02:00
Aliaksandr Valialkin
9761ffd161
lib/promscrape/discovery/kubernetes: properly handle too old resource version
error message from Kubernetes watch API
2021-03-26 12:28:10 +02:00
Aliaksandr Valialkin
d4aadba9fa
app/vmagent: add -promscrape.consul.waitTime
command-line flag for configuring Consul service discovery wait time
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144
2021-03-23 19:33:25 +02:00
Nikolay
29f9ef9b7f
changes consul_service label value ( #1143 )
...
according to prometheus discovery.
It should mitigate issue with case sensetive services
https://github.com/hashicorp/consul/issues/5707
2021-03-23 15:35:01 +02:00
Aliaksandr Valialkin
828669e4e1
all: make golint
happy
2021-03-17 00:49:28 +02:00
Aliaksandr Valialkin
f104f3eb2a
all: make golangci-lint happy after the commit 6378205415
2021-03-17 00:24:40 +02:00
Aliaksandr Valialkin
6378205415
lib/netutil: enable IPv6 UDP listening if -enableTCP6
command-line flag is passed to VictoriaMetrics
...
This is a follow-up for 18cfc4be7b
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
2021-03-17 00:16:17 +02:00
Aliaksandr Valialkin
85a95bf60c
all: various fixes in command-line flag descriptions
2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin
c14dafce43
lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics
...
Do not cache too big byte buffers and too big writeRequestCtx objects,
since it is cheaper to re-create them instead of wasting RAM for their caching.
This reverts 7f6f350ee1
2021-03-15 11:45:39 +02:00
Aliaksandr Valialkin
7f6f350ee1
lib/promscrape: return back the logic for flushing big buffers to storage from the commit 3fd8653b40
...
This should reduce memory usage when vmagent scrapes targets with big number of metrics and `-promscrape.streamParse` isn't enabled
2021-03-14 22:26:00 +02:00
Aliaksandr Valialkin
b88806ecbf
lib/promscrape/discovery/kubernetes: do not start object watcher until initial objects are loaded
2021-03-14 21:55:00 +02:00
Aliaksandr Valialkin
83edbb7cab
lib/promscrape: retry service discovery in a few seconds if it starts returning 0 targets
...
This should reduce recovery time from temporary issues during service discovery
2021-03-14 21:53:23 +02:00
Aliaksandr Valialkin
bf15d6a6a2
lib/promscrape: remove duplicate target
word in error message
2021-03-14 21:52:02 +02:00
Aliaksandr Valialkin
d409898515
lib/promscrape/discovery/kubernetes: further optimize kubernetes service discovery for the case with many scrape jobs
...
Do not re-calculate labels per each scrape job - reuse them instead for scrape jobs with identical Kubernetes role
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
2021-03-14 21:14:53 +02:00
Aliaksandr Valialkin
7a16e8e3a2
lib/promscrape/discovery: fixes after 133b288681
...
- Removed a deadlock in addAPIWatcher
- Do not create unused ScrapeWork objects
- Do not spend CPU resources on creating objectByKey map in addAPIWatcher
This work is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1125
2021-03-13 15:18:51 +02:00
Aliaksandr Valialkin
def014eb75
lib/promscrape/discovery/kubernetes: remove debug lines left after the commit 133b288681
2021-03-12 11:22:33 +02:00
Aliaksandr Valialkin
a6a71ef861
lib/promscrape: add ability to configure proxy options via proxy_tls_config
, proxy_basic_auth
, proxy_bearer_token
and proxy_bearer_token_file
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116
2021-03-12 03:36:19 +02:00
Aliaksandr Valialkin
133b288681
lib/promscrape/discovery/kubernetes: use a single watcher per apiURL
...
Previously multiple scrape jobs could create multiple watchers for the same apiURL. Now only a single watcher is used.
This should reduce load on Kubernetes API server when many scrape job configs use Kubernetes service discovery.
2021-03-11 16:43:04 +02:00
Aliaksandr Valialkin
bebcb8130c
lib/promscrape/discovery/kubernetes: localize Bookmark parsing code
...
This is a follow-up for e772d1c920
2021-03-11 13:08:08 +02:00
Aliaksandr Valialkin
e772d1c920
lib/promscrape/discovery/kubernetes: reduce load on Kubernetes API server by using watch bookmarks
...
This allows continuing object watch from the last bookbark instead of reloading all the objects
on watch errors or timeouts.
See https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks
2021-03-10 15:06:35 +02:00
Aliaksandr Valialkin
36fd007247
lib/proxy: set missing ServerName in TLS config for proxy_url
.
...
While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.
2021-03-09 18:58:18 +02:00
Nikolay
ad34f42467
Changes tlsConfig init for proxy connections ( #1121 )
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116
2021-03-09 18:51:00 +02:00
Aliaksandr Valialkin
3fd8653b40
lib/promscrape: apply sample_limit
after metric relabeling is applied as Prometheus does
...
See the description for `sample_limit` option from Prometheus docs:
Per-scrape limit on number of scraped samples that will be accepted.
If more than this number of samples are present after metric relabeling
the entire scrape will be treated as failed. 0 means no limit.
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
2021-03-09 15:47:18 +02:00
Aliaksandr Valialkin
4b18a4f026
lib/promscrape/discovery/kubernetes: remove too verbose logs about starting and stopping the watchers
...
Log the number of objects loaded per each watch url
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
2021-03-09 15:05:14 +02:00
Aliaksandr Valialkin
14a399dd06
lib/promscrape: add scrape_offset
option to scrape_config
...
This option can be used for specifying the particular offset per each scrape interval for target scraping
2021-03-08 12:03:33 +02:00
Aliaksandr Valialkin
ab4f090c63
lib/promscrape/discovery/kubernetes: reduce memory usage further when big number of scrape jobs are configured for the same kubernetes_sd_config
role
...
Serialize reloading per-role objects, so they don't occupy too much memory when objects for many scrape jobs are simultaneously refreshed.
Do not reload per-role objects if they were already refreshed by concurrent goroutines. This should reduce load on Kubernetes API server
when big number of scrape jobs are configured for the same Kubernetes role.
This is a follow-up for 17b87725ed
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
2021-03-07 19:51:03 +02:00
Aliaksandr Valialkin
17b87725ed
lib/promscrape/discovery/kubernetes: reduce memory usage when Kubernetes service discovery is configured on a big number of scrape jobs
...
Previously vmagent was creating a separate Kubernetes object cache per each scrape job.
This could result in increased memory usage when monitoring a Kubernetes cluster with big number of objects (pods / nodes / services, etc.)
as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113
Now it uses a shared map of scrape objects across multiple scrape jobs.
2021-03-05 17:29:55 +02:00
Aliaksandr Valialkin
c9a25e931b
lib/promscrape/discovery/kubernetes: move apiWatcher code to a separate file
2021-03-05 12:36:05 +02:00
Aliaksandr Valialkin
444fca89f8
lib/promscrape: make cluster membership calculations consistent across 32-bit and 64-bit architectures
2021-03-05 09:06:17 +02:00
Aliaksandr Valialkin
423cd981fb
lib/promscrape: add -promscrape.cluster.replicationFactor
command-line flag for replicating scrape targets among vmagent
instances in the cluster
2021-03-04 10:20:15 +02:00