github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
4992 commits 150 branches 748 tags 347 MiB
Commit graph

138 commits

Author SHA1 Message Date
Aliaksandr Valialkin
4998402004
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics 
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
 2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil 
lib/bytesutil is more appropriate place for InternString() function
 2022-09-30 07:44:35 +03:00
Bryce Lampe
74f8e12e87
Support "HTTP" and "HTTPS" schemes (#3019) 
* Support "HTTP" and "HTTPS" schemes

* Update lib/promscrape/config.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName() 
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
 2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
46d7792b72
lib/promscrape: follow-up after 2c553d5a2f 
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
 2022-08-08 14:46:26 +03:00
Fury
2c553d5a2f
add support to scrape multi tenant metrics (#2950) 
* add support to scrape multi tenant metrics

* add support to scrape multi tenant metrics

Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
 2022-08-08 14:10:18 +03:00
Igor Tiunov
6e5ac32fba
YC service discovery (#2923) 
* YC service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386

* Fixed linter suggestions

* fixed golint errors
 2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin
0fd86e2364
lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
 2022-07-18 17:15:07 +03:00
Nikolay
7301aa678c
lib/promscrape: adds azure service discovery (#2743) 
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-13 23:43:18 +03:00
Aliaksandr Valialkin
f97355d9fb
lib/promscrape: properly set Host header when sending requests via http proxy 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
 2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin
e6ed92529b
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This package already has the same name, so there is no need in explicit name
 2022-06-21 20:23:32 +03:00
Aliaksandr Valialkin
0922ed2b7e
lib/promscrape: add -promscrape.cluster.name command-line flag 
This flag is used for proper data de-duplication when the same target is scraped
from multiple vmagent clusters.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679
 2022-06-04 00:37:01 +03:00
Aliaksandr Valialkin
d2567ccdd6
lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum 
In this case there is no need in converting int64 to int
 2022-06-01 01:42:34 +03:00
Aliaksandr Valialkin
a0727ab1b1
docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor command-line flag 2022-05-12 18:50:29 +03:00
Aliaksandr Valialkin
123aa4c79e
lib/promscrape: properly implement ScrapeConfig.clone() 
Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields -
their contents was replaced with `<secret>` value.

This led to inability to use passwords and secrets in `-promscrape.config` file.
The bug has been introduced in v1.77.0 in the commit 67b10896d2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551
 2022-05-07 00:05:40 +03:00
Aliaksandr Valialkin
67b10896d2
lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated 
This is a follow-up after 26b78ad707
 2022-04-22 13:19:43 +03:00
Aliaksandr Valialkin
c75d0095f5
lib/promscrape: remove possible data race when cleaning up internStringsMap 2022-04-20 18:40:53 +03:00
Aliaksandr Valialkin
82e34984dd
lib/promscrape: zero out labels after duplicate removal inside mergeLabels() 2022-04-20 18:35:33 +03:00
Aliaksandr Valialkin
2341bd48d7
lib/promscrape: follow-up after 91e290a8ff 2022-04-20 16:11:37 +03:00
Nikolay
91e290a8ff
lib/promscrape: reduce latency for k8s GetLabels (#2454) 
replaces internStringMap with sync.Map - it greatly reduces lock contention
concurently reload scrape work for api watcher - each object labels added by dedicated CPU

changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70
 2022-04-20 16:09:40 +03:00
Aliaksandr Valialkin
3d0549c982
lib/promscrape: optimize getScrapeWork() function 
Reduce the number of memory allocations in this function. This improves its performance by up to 50%.
This should improve service discovery speed when big number of potential targets with big number of meta-labels
are generated by service discovery.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270
 2022-04-20 15:37:00 +03:00
Nikolay
26b78ad707
lib/promscrape: adds job restart method (#2455) 
* lib/promscrape: adds job restart method
it must restart only ScrapeConfig with changed content
this change greatly reduce time, that needed for job restart
and it should decrease possible data loss when config frequently changed at kubernetes based deployments

Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-04-16 20:28:46 +03:00
Aliaksandr Valialkin
ebaa1c7ad5
lib/promscrape: follow-up after baa1c24b36 2022-04-16 14:25:54 +03:00
Nikolay
baa1c24b36
lib/promscrape: removes omitempty for ScrapeConfig (#2457) 
This change fixes incorrect marshalling for ScrapeConfig
it affects http endpoint and ScrapeConfig checksum.

With omitempty, custom Marshaller is not called if field is not a pointer.

Previously this issue happened at vmalert
 2022-04-16 13:22:11 +03:00
Aliaksandr Valialkin
f3d4671bb6
lib/promscrape: follow-up after 7e79adfb55 2022-04-12 12:36:17 +03:00
Nikolay
7e79adfb55
lib/promscrape: allows to use k8s pod name as clusterMemberNum (#2436) 
* lib/promscrape: allows to use k8s pod name as clusterMemberNum
it must improve user expirience and simplify clustering scrapers.
it must allow to use vmagent cluster with distroless images
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-04-12 12:24:11 +03:00
Corporte Gadfly
ad6bdd78d0
match fileSDCheckInterval with prometheus file_sd_config default (#2188) 2022-02-15 12:04:26 +02:00
Roman Khavronenko
e3adcbec6e
lib/promscrape: support prometheus-like duration in scrape configs (#2169) 
* lib/promscrape: support prometheus-like duration in scrape configs

The change allows to specify duration values like `1d`, `1w`
for fields `scrape_interval`, `scrape_timeout`, etc.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/blockcache: make linter happy

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/promscrape: support prometheus-like duration in scrape configs

* add support for extra fields `scrape_align_interval` and `scrape_offset`;
* support Prometheus duration parsing for `__scrape_interval__`
and `__scrape_duration__` labels;

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

* wip

* docs/CHANGELOG.md: document the feature

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-11 16:17:00 +02:00
Aliaksandr Valialkin
9bb60ab00f
lib/promscrape: set -promscrape.config.strictParse to true by default 
This allows detecting long-living silent errors in -promscrape.config
 2022-02-08 15:41:43 +02:00
Aliaksandr Valialkin
06642d97f5
app: allow specifying http and https urls in the following command-line flags 
* -promscrape.config
* -relabelConfig
* -remoteWrite.relabelConfig
* -remoteWrite.urlRelabelConfig
 2021-12-03 00:10:02 +02:00
Aliaksandr Valialkin
fd596945e7
lib/promscrape: improve logging for scrape_config_files parse errors 
Log the actual file path, which led to the parse error.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789
 2021-11-08 13:34:12 +02:00
Aliaksandr Valialkin
cbfc7b7c92
app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764
 2021-11-05 14:41:16 +02:00
Aliaksandr Valialkin
4821adfd95
lib/promscrape: properly show proxy_url option value at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755
 2021-10-26 21:23:54 +03:00
Aliaksandr Valialkin
6106d4069d
lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed 
This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels.
See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets

This is a follow-up after 9882cda8b9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728
 2021-10-22 13:54:30 +03:00
Aliaksandr Valialkin
c37f285466
lib/promscrape: set honor_timestamps: true by default if this option isnt set explicitly in scrape configs 
This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
 2021-10-16 20:49:08 +03:00
Aliaksandr Valialkin
1a7287c408
lib/promscrape: return error if sample_limit or series_limit options are set when stream parsing mode is enabled 2021-10-14 12:11:23 +03:00
Aliaksandr Valialkin
9882cda8b9
lib/promscrape: shard targets among cluster nodes after relabeling is applied 
This guarantees that targets with the same set of labels go to the same vmagent node.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495
 2021-10-12 17:06:00 +03:00
Aliaksandr Valialkin
5a58c041c2
app/vmagent: expose -promscrape.config contents at /config page as Prometheus does 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695
 2021-10-12 16:25:37 +03:00
Aliaksandr Valialkin
6ee66fb6b1
lib/promscrape: reduce memory allocations in mergeLabels() after 48e3e6c8df 2021-09-30 16:56:12 +03:00
Aliaksandr Valialkin
48e3e6c8df lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling 
See https://github.com/prometheus/prometheus/pull/8911
 2021-09-12 13:33:41 +03:00
Aliaksandr Valialkin
4aeb8db83f lib/promscrape: add ability to set series_limit and stream_parse options via relabeling 
This allows managing these options on a per-target basis.

Typical use case: to manage these options for pods via Kubernetes annotations.
 2021-09-09 18:49:39 +03:00
Aliaksandr Valialkin
f77dde837a lib/promscrape: add the ability to limit the number of unique series per each scrape target 
The number of series per target can be limited with the following options:

* Global limit with `-promscrape.maxSeriesPerTarget` command-line option.
* Per-target limit with `max_series: N` option in `scrape_config` section.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561
 2021-09-01 16:03:59 +03:00
Aliaksandr Valialkin
10f960fa0c lib/promscrape: add ability to load scrape configs from multiple files 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
 2021-08-26 08:51:16 +03:00
Aliaksandr Valialkin
cb5453953f lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:20:20 +03:00
Aliaksandr Valialkin
a69045e440 lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 12:10:46 +03:00
Lu Jiajing
c85a5b7fcb
Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 11:42:47 +03:00
Nikolay
434e33da9b
adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:39:18 +03:00
Nikolay
e307bbb29a
adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:33:37 +03:00
Aliaksandr Valialkin
fb72a2133f lib/promscrape: show jobs with empty scrape targets on /targets page 2021-06-18 10:53:52 +03:00
Nikolay
729c4eeb9c
adds digital ocean sd (#1376) 
* adds digital ocean sd config

* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367

* typo fix
 2021-06-14 13:15:04 +03:00