github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
6459 commits 150 branches 748 tags 347 MiB
Commit graph

594 commits

Author SHA1 Message Date
Zakhar Bessarab
322d96bfe5
discovery/{consul,nomad}: fix cancelling serviceWatcher in-flight requests (#3658) 
* lib/promscrape/discovery/{consul,nomad}: fix background service update watches not canceling requests on serviceWatcher stop

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/{consul,nomad}: fix closing serviseWatcher during scrape job restart

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-17 21:47:11 -08:00
Aliaksandr Valialkin
289af65071
lib/promscrape: properly apply series limit 
Fix the following issues:

- Series limit wasn't applied when staleness tracking was disabled.
- Series limit didn't prevent from sending staleness markers for new series exceeding the limit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660

Thanks to @hagen1778 for the initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3665
 2023-01-17 10:14:49 -08:00
Nikolay
20f28eb9d6
/lib/promscrape: use correct err logger for scrape unmarshalling (#3645) 
/lib/promscrape: use correct err logger for scrape unmarshalling
It correctly suppresses scrape errors and adds correct context for err msg
 2023-01-12 17:40:42 +01:00
Aliaksandr Valialkin
e2498af530
lib/promscrape: log the number of unsuccessful scrapes during the last -promscrape.suppressScrapeErrorsDelay 
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3413
Thanks to @jelmd for the pull request.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575
 2023-01-12 01:09:32 -08:00
Aliaksandr Valialkin
ec23ab6bc2
lib/promscrape/discovery: missing changes after b4ad3a3b4c 2023-01-11 23:02:45 -08:00
Aliaksandr Valialkin
b4ad3a3b4c
lib/promscrape: follow-up for 8537533beb 
- Add a comment describing the purpose of the `role` field inside `apiConfig` struct
- Revert changes at lib/promscrape/discovery/dockerswarm/dockerswarm.go ,
  since they reduce code readability. E.g. the reader needs to look up the named string constants
  in order to get their values.
 2023-01-11 22:54:18 -08:00
Zakhar Bessarab
8537533beb
lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects (#3632) 
* lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-11 22:50:34 -08:00
Aliaksandr Valialkin
cbe62f23ba
lib/promscrape/discovery/gce: follow-up for b2ccdaaa2f 
- Use promutils.Labels.GetLabels() instead of comparing promutils.Labels.Labels to nil.
  This make the code more consistent with other places.

- Mention the release where the issue has been introduced at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624
 2023-01-10 13:51:03 -08:00
Zakhar Bessarab
b2ccdaaa2f
lib/promscrape/discovery/gce: fix crash in case instance does not have any labels set (#3625) 
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-10 11:07:11 +01:00
Aliaksandr Valialkin
e5eca54951
lib/promscrape/discovery/nomad: sync nomad_sd_configs fields with the Prometheus implementation 
See the list of configs supported by Prometheus at f88a0a7d83/discovery/nomad/nomad.go (L76-L84)

- Removed "token" option. In can be set either via NOMAD_TOKEN env var or via `bearer_token` config option.
- Removed "scheme" option. It is automatically detected depending on whether the `tls_config` is set.
- Removed "services" and "tags" options, since they aren't supported by Prometheus.
- Added "region" option. If it is missing, then the region is read from NOMAD_REGION env var.
  If this var is empty, then it is set to "global" in the same way as Nomad client does.
  See 865ee8d37c/api/api.go (L297)
  and 865ee8d37c/api/api.go (L555-L556)
- If the "server" option is missing, then it is read from NOMAD_ADDR in the same way
  as Nomad client does - see 865ee8d37c/api/api.go (L294-L296)

This is a follow-up for 8aee209c53

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-09 21:14:48 -08:00
Roman Khavronenko
8aee209c53
lib/promscrape: remove datacenter field from nomad_sd_config (#3612) 
Looks like `datacenter` field isn't part of `/v1/services` API.
See https://developer.hashicorp.com/nomad/api-docs/services#list-services
and https://developer.hashicorp.com/nomad/api-docs/services#read-service

Related issues:
https://github.com/traefik/traefik/issues/9109
https://github.com/prometheus/prometheus/issues/11776

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2023-01-09 09:07:40 +01:00
Aliaksandr Valialkin
28f8dc41b0
lib/promscrape/discoveryutils: cleanup after 5df9fddaf2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-07 01:26:54 -08:00
Zakhar Bessarab
5df9fddaf2
lib/promscrape/discoveryutils: use correct timeout for blocking requests (#3609) 
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-07 01:13:03 -08:00
Aliaksandr Valialkin
ad5bfe3089
lib/promscrape: reduce the number of concurrently executed processScrapedData calls from 2x of the number of CPUs to the number of CPUs 
This should reduce the maximum memory usage for processScrapedData() function by 2x.
The only part, which can be IO-bound in the processScrapedData() is pushData() call,
when it buffers data to persistent queue if the remote storage cannot keep up
with the data ingestion speed. In this case it is OK if the scrape pace will be limited.
 2023-01-07 00:14:30 -08:00
Aliaksandr Valialkin
986a05e18d
lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples 
This should reduce memory usage when scraping big number of targets,
since this limits the summary memory usage during concurrent parsing and relabeling
by the number of available CPU cores.
 2023-01-06 22:59:17 -08:00
Aliaksandr Valialkin
463b957e54
lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 21:52:33 -08:00
Aliaksandr Valialkin
f392913d00
lib/promscrape: follow-up after bced9fb978 
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
 2023-01-05 21:13:06 -08:00
Zakhar Bessarab
bced9fb978
lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568) 2023-01-05 19:34:47 -08:00
Aliaksandr Valialkin
9f348cf8a1
lib/promscrape/discovery/nomad: follow-up after 48f371a46c 
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
 2023-01-05 18:07:20 -08:00
Aliaksandr Valialkin
1a28f0e5b3
lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 
This allows copy-n-pasting the url to another browser window and seeing the same result.

The limit in 1000 chars is selected in order to prevent from potential issues with systems
which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string

If the limit is exceeded, then query args are sent via POST method and aren't visible in the url.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 16:48:04 -08:00
Karan Sharma
48f371a46c
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549) 
Add nomad_sd_config support for service discovery
 2023-01-05 23:03:58 +01:00
Zakhar Bessarab
185cdcd813
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586) 
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2023-01-05 03:34:25 -08:00
Aliaksandr Valialkin
0dea3b71da
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2023-01-05 03:26:49 -08:00
Aliaksandr Valialkin
a1076abcbf
lib/promscrape: follow-up for a7e29c38bc 
- Document the bugfix at docs/CHANGELOG.md
- Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
 2023-01-05 02:52:08 -08:00
Zakhar Bessarab
a7e29c38bc
lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595) 
* lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/targetstatus: address review comment

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
 2023-01-05 02:39:31 -08:00
Aliaksandr Valialkin
5bca3a5be2
app/vmselect: remove dependency on lib/promscrape from app/vmselect 2023-01-03 23:28:27 -08:00
Aliaksandr Valialkin
add2c4bf07
lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..)) 2023-01-03 22:16:22 -08:00
Aliaksandr Valialkin
c4229a1bba
lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize 
This is a follow-up for 7ad9fff7e5
 2022-12-28 14:42:11 -08:00
Clément Nussbaumer
7ad9fff7e5
fix(promscrape): check MaxScrapeSize after gzip decompression (#3550) 2022-12-28 12:19:41 -08:00
Aliaksandr Valialkin
0076422350
lib/promscrape/discovery/azure: typo fix 2022-12-21 21:25:16 -08:00
Aliaksandr Valialkin
0bf3ae9559
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname> 
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
 2022-12-19 13:08:11 -08:00
Aliaksandr Valialkin
65f8fc527f
lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page 2022-12-16 17:02:41 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures 
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
 2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
a30ae502ef
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2022-12-10 12:44:45 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging 
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
 2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
2406c0dcfd
docs/CHANGELOG.md: document the bugfix at 05b42601c3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
 2022-12-08 18:35:28 -08:00
Zakhar Bessarab
05b42601c3
lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403) 
* lib/promscrape/discovery/azure: remove API server from URL returned by azure

* lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer
 2022-12-08 18:29:10 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
 2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
14660d4df5
all: typo fix: the the -> the 2022-12-03 21:53:01 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed 
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
 2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
295c84df66
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels 2022-11-29 20:27:48 -08:00
Aliaksandr Valialkin
654e94f420
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
 2022-11-28 18:37:09 -08:00
Roman Khavronenko
03d88bc066
vmagent: expose metrics for tracking config state (#3375) 
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-11-22 00:38:43 +02:00
Aliaksandr Valialkin
95f0266558
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones 
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
 2022-11-21 22:32:05 +02:00
Aliaksandr Valialkin
5955d23232
lib/promscrape: add a benchmark for internLabelStrings() 2022-11-16 23:02:49 +02:00
Aliaksandr Valialkin
86bce7f5f9
lib/promscrape: add more cases to TestAddRowToTimeseries 
This is a follow-up for 16fdd2af8a
 2022-11-09 16:13:56 +02:00
Jeremy PLANCKEEL
16fdd2af8a
test(golang): add test to function addRowToTimeseries (#3282) 
Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>
 2022-11-09 15:41:26 +02:00
Aliaksandr Valialkin
7fa5d043f5
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/11482
 2022-11-07 15:25:53 +02:00
Aliaksandr Valialkin
76e8888272
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set. 
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.

While at it, properly handle some corner cases:

- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278

Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
 2022-10-28 22:14:26 +03:00
Aliaksandr Valialkin
454baf84d6
lib/promscrape/discovery/kubernetes: do not print an empty kubeconfig_file option in yaml at /config page 2022-10-28 22:14:25 +03:00
Aliaksandr Valialkin
518c340ae3
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax 
This is a follow-up for 02096e06d0
 2022-10-26 14:49:33 +03:00
Nikolay
1059c4d84a
lib/promscrape/discovery/kubernetes: correctly wrap error (#3250) 
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-10-18 20:37:42 +03:00
Aliaksandr Valialkin
069401a304
all: log error when environment variables referred from -promscrape.config are missing 
This should prevent from using incorrect config files
 2022-10-18 10:47:16 +03:00
Aliaksandr Valialkin
50f5eae0e0
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places 
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.

This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
 2022-10-09 14:51:16 +03:00
Aliaksandr Valialkin
5269b1ad77
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis 
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
 2022-10-07 23:36:14 +03:00
Aliaksandr Valialkin
f9df0cae16
lib/promscrape: allow specifying full target url in __address__ label 
Previously the `__address__` label could contain only `host:port` part of the target url,
while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__`
labels. Now it is possible to set the full url in `__address__` label.

This makes valid the following scrape config, which is frequently used by novice users:

scrape_configs:
- job_name: foo
  static_configs:
  - targets:
    - http://host1/metrics1
    - https://host2/metrics2
 2022-10-07 22:43:04 +03:00
Aliaksandr Valialkin
4998402004
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics 
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
 2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin
f86070169d
lib/promscrape/discovery/azure: remove unneeded conversion to string 2022-10-01 16:04:37 +03:00
Aliaksandr Valialkin
c54e14cdec
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does 
See https://github.com/prometheus/prometheus/pull/11326
 2022-09-30 20:48:32 +03:00
Aliaksandr Valialkin
17289ff481
lib/regexutil: cache MatchString results for unoptimized regexps 
This increases relabeling performance by 3x for unoptimized regexs
 2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil 
lib/bytesutil is more appropriate place for InternString() function
 2022-09-30 07:44:35 +03:00
Aliaksandr Valialkin
6a32a64073
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed 2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin
56ce7ce85b
lib/promscrape: typo fix after 74c00a8762 2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin
74c00a8762
lib/promscrape: read response body into memory in stream parsing mode before parsing it 
This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
 2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin
ccad651a61
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error 
This should improve debugging issues with Kubernetes API server
 2022-09-13 19:36:55 +03:00
Bryce Lampe
74f8e12e87
Support "HTTP" and "HTTPS" schemes (#3019) 
* Support "HTTP" and "HTTPS" schemes

* Update lib/promscrape/config.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin
30b8d91727
lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache 
This is a follow-up for c06e7a142c
 2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName() 
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
 2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
1f89278d88
all: subsitute ioutil.ReadAll with io.ReadAll 
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
 2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
7d26414b2e
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit 
The following metrics are generated:

- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped

These metrics simplify alerting on targets, which expose too many time series

See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
 2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin
bb68ab99fa
lib/promscrape: retry http requests if the server returns 429 status code 
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
 2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin
c3f8481011
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html 2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin
95d36da358
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11034
 2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin
c4fcd9f1c5
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does 
See https://github.com/prometheus/prometheus/pull/11002
 2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin
511805d88d
lib/promscrape/discovery/dns: add support for resolving MX records 
See https://github.com/prometheus/prometheus/pull/10099
 2022-08-15 00:32:34 +03:00
Aliaksandr Valialkin
46d7792b72
lib/promscrape: follow-up after 2c553d5a2f 
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
 2022-08-08 14:46:26 +03:00
Fury
2c553d5a2f
add support to scrape multi tenant metrics (#2950) 
* add support to scrape multi tenant metrics

* add support to scrape multi tenant metrics

Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
 2022-08-08 14:10:18 +03:00
Aliaksandr Valialkin
0ef29ceb14
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress 
See 7e65ad3e43
and 7e1111ff14
 2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin
f2816ef031
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs 
This option was ignored since d289ecded1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
 2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin
3e8890e71b
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels 
See https://github.com/prometheus/prometheus/issues/9187
 2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin
68de1f4e4a
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default 
This aligns with Prometheus behaviour.

See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
 2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin
02de848c88
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f 2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin
83a4abda3f
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
 2022-08-04 22:26:43 +03:00
Igor Tiunov
6e5ac32fba
YC service discovery (#2923) 
* YC service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386

* Fixed linter suggestions

* fixed golint errors
 2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin
4c2f9a1a2e
lib/promscrape: set up=0 for partially failed scrape in stream parsing mode 
This behaviour aligns with Prometheus behavior
 2022-07-22 13:29:44 +03:00
Aliaksandr Valialkin
0fd86e2364
lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
 2022-07-18 17:15:07 +03:00
Nikolay
7301aa678c
lib/promscrape: adds azure service discovery (#2743) 
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-07-13 23:43:18 +03:00
Aliaksandr Valialkin
f97355d9fb
lib/promscrape: properly set Host header when sending requests via http proxy 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
 2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin
01f55bc66b
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
 2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin
b186b63e07
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/10759
 2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin
e6ba2af7a1
lib/promscrape: fix a test after c66f676f3b 2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin
c66f676f3b
lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target 
See https://github.com/VictoriaMetrics/operator/issues/497
 2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin
855436efd2
lib/promauth: refactor NewConfig in order to improve maintainability 
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options

This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
 2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin
e40b40afe6
Revert "lib/promscrape, vmagent: fix path to files (#2801)" 
This reverts commit 0a8e35835c.

Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005
 2022-06-30 18:23:56 +03:00
Dmytro Kozlov
0a8e35835c
lib/promscrape, vmagent: fix path to files (#2801) 
vmagent: respect `-pathPrefix` flag for static files and links
 2022-06-30 16:22:54 +02:00
ttyv
bdf9f4669a
lib/promscrape: fix vmagent tickerCh reload behaviour (#2786) 
Co-authored-by: Dmitriy <dab@ttyv.ru>
 2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin
52eadb729e
lib/promscrape: always send stale markers with the real scrape timestamp 
This guarantees that query won't return data just after the series is disappeared.
 2022-06-23 11:34:18 +03:00
Aliaksandr Valialkin
1c4f67c5d2
lib/promauth: add ability to send additional http headers in requests to scrape targets 
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
 2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin
e6ed92529b
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 
This package already has the same name, so there is no need in explicit name
 2022-06-21 20:23:32 +03:00