github-mirrors/VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

Author	SHA1	Message	Date
Oleksandr Redko	0e1c395609	app,lib: fix typos in comments (#3804 )	2023-02-13 09:32:35 -08:00
Aliaksandr Valialkin	e6616c74a2	lib/promscrape/discovery/openstack: use port 80 for the discovered target by default if it isnt specified in the config	2023-02-11 14:42:09 -08:00
Zakhar Bessarab	bbf663bd04	lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3791 ) * lib/promscrape: fix cancelling in-flight scrape requests during configuration reload when using `streamParse` mode (see #3747) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/CHANGELOG.md --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-09 11:18:36 -08:00
Karan Sharma	004a24c950	sd/nomad: panic in nomad watcher because of nil map (#3784 ) properly initialize url.Values	2023-02-08 08:37:02 -08:00
Aliaksandr Valialkin	ac695f36bb	lib/promscrape: add a comment explaining the logic behind adding `exported_` perfix to metric names This is a follow-up for `7b87fac8e7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3557 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2023-02-01 12:02:05 -08:00
Dmytro Kozlov	3c1e455805	lib/promscrape: fix `honor_labels` behavior (#3739 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-02-01 12:02:04 -08:00
Aliaksandr Valialkin	855d560789	lib/promscrape/discovery/azure: add __meta_azure_machine_size label in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/11650	2023-01-27 17:07:57 -08:00
Aliaksandr Valialkin	134f7622d6	lib/promscrape/discovery/kubernetes: add support for __meta_kubernetes_pod_container_id See https://github.com/prometheus/prometheus/issues/11843 and https://github.com/prometheus/prometheus/pull/11844	2023-01-27 16:33:57 -08:00
Aliaksandr Valialkin	71a170d404	lib/promscrape: follow-up for `393876e52a` - Document the change in docs/CHANGELOG.md - Reduce memory usage when sending stale markers even more by parsing the response in stream parsing mode - Update the TestSendStaleSeries Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675	2023-01-23 21:56:18 -08:00
Roman Khavronenko	8e2a8a6ae2	lib/promscrape: limit number of sent stale series at once (#3686 ) Stale series are sent when there is a difference between current and previous scrapes. Those series which disappeared in the current scrape are marked as stale and sent to the remote storage. Sending stale series requires memory allocation and in case when too many series disappear in the same it could result in noticeable memory spike. For example, re-deploy of a big fleet of service can result into excessive memory usage for vmagent, because all the series with old pod name will be marked as stale and sent to the remote write storage. This change limits the number of stale series which can be sent at once, so memory usage remains steady. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-23 21:56:17 -08:00
Aliaksandr Valialkin	95d4db0506	lib/promscrape: properly log the actual response size after `c4229a1bba`	2023-01-23 21:13:06 -08:00
Aliaksandr Valialkin	a844b97942	lib/promscrape: follow-up for `d79f1b106c` - Document the fix at docs/CHANGELOG.md - Limit the concurrency for sendStaleMarkers() function in order to limit its memory usage when big number of targets disappear and staleness markers are sent for all the metrics exposed by these targets. - Make sure that the writeRequestCtx is returned to the pool when there is no need to send staleness markers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668	2023-01-17 23:13:08 -08:00
lzfhust	5ac0f18ca8	using writeRequestCtxPool when delete kubernetes clusters from kubernetes_sd_configs (#3669 )	2023-01-17 23:12:59 -08:00
Zakhar Bessarab	40d524edb8	discovery/{consul,nomad}: fix cancelling serviceWatcher in-flight requests (#3658 ) * lib/promscrape/discovery/{consul,nomad}: fix background service update watches not canceling requests on serviceWatcher stop Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/discovery/{consul,nomad}: fix closing serviseWatcher during scrape job restart Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * wip Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-17 21:47:51 -08:00
Aliaksandr Valialkin	c33728befb	lib/promscrape: properly apply series limit Fix the following issues: - Series limit wasn't applied when staleness tracking was disabled. - Series limit didn't prevent from sending staleness markers for new series exceeding the limit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660 Thanks to @hagen1778 for the initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3665	2023-01-17 10:30:16 -08:00
Nikolay	43d1f2d0c4	/lib/promscrape: use correct err logger for scrape unmarshalling (#3645 ) /lib/promscrape: use correct err logger for scrape unmarshalling It correctly suppresses scrape errors and adds correct context for err msg	2023-01-12 09:00:06 -08:00
Aliaksandr Valialkin	a819e30ddf	lib/promscrape: log the number of unsuccessful scrapes during the last -promscrape.suppressScrapeErrorsDelay This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3413 Thanks to @jelmd for the pull request. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575	2023-01-12 01:12:22 -08:00
Aliaksandr Valialkin	2e018aebf3	lib/promscrape/discovery: missing changes after `b4ad3a3b4c`	2023-01-11 23:03:14 -08:00
Aliaksandr Valialkin	434f22f871	lib/promscrape: follow-up for `8537533beb` - Add a comment describing the purpose of the `role` field inside `apiConfig` struct - Revert changes at lib/promscrape/discovery/dockerswarm/dockerswarm.go , since they reduce code readability. E.g. the reader needs to look up the named string constants in order to get their values.	2023-01-11 22:56:48 -08:00
Zakhar Bessarab	ae5b85966a	lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects (#3632 ) * lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * Update docs/CHANGELOG.md Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-11 22:56:40 -08:00
Aliaksandr Valialkin	ab318660cd	lib/promscrape/discovery/gce: follow-up for `b2ccdaaa2f` - Use promutils.Labels.GetLabels() instead of comparing promutils.Labels.Labels to nil. This make the code more consistent with other places. - Mention the release where the issue has been introduced at docs/CHANGELOG.md. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624	2023-01-10 13:51:57 -08:00
Zakhar Bessarab	02f5c16433	lib/promscrape/discovery/gce: fix crash in case instance does not have any labels set (#3625 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-10 13:51:35 -08:00
Aliaksandr Valialkin	43a4dcdaf8	lib/promscrape/discovery/nomad: sync nomad_sd_configs fields with the Prometheus implementation See the list of configs supported by Prometheus at `f88a0a7d83/discovery/nomad/nomad.go (L76-L84)` - Removed "token" option. In can be set either via NOMAD_TOKEN env var or via `bearer_token` config option. - Removed "scheme" option. It is automatically detected depending on whether the `tls_config` is set. - Removed "services" and "tags" options, since they aren't supported by Prometheus. - Added "region" option. If it is missing, then the region is read from NOMAD_REGION env var. If this var is empty, then it is set to "global" in the same way as Nomad client does. See `865ee8d37c/api/api.go (L297)` and `865ee8d37c/api/api.go (L555-L556)` - If the "server" option is missing, then it is read from NOMAD_ADDR in the same way as Nomad client does - see `865ee8d37c/api/api.go (L294-L296)` This is a follow-up for `8aee209c53` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-09 21:30:19 -08:00
Roman Khavronenko	ca5136a0ee	lib/promscrape: remove `datacenter` field from nomad_sd_config (#3612 ) Looks like `datacenter` field isn't part of `/v1/services` API. See https://developer.hashicorp.com/nomad/api-docs/services#list-services and https://developer.hashicorp.com/nomad/api-docs/services#read-service Related issues: https://github.com/traefik/traefik/issues/9109 https://github.com/prometheus/prometheus/issues/11776 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-09 21:24:46 -08:00
Aliaksandr Valialkin	7792ba3272	lib/promscrape/discoveryutils: cleanup after `5df9fddaf2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468	2023-01-07 01:27:16 -08:00
Zakhar Bessarab	e8624fd781	lib/promscrape/discoveryutils: use correct timeout for blocking requests (#3609 ) Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-07 01:27:10 -08:00
Aliaksandr Valialkin	0a14b7bb82	lib/promscrape: reduce the number of concurrently executed processScrapedData calls from 2x of the number of CPUs to the number of CPUs This should reduce the maximum memory usage for processScrapedData() function by 2x. The only part, which can be IO-bound in the processScrapedData() is pushData() call, when it buffers data to persistent queue if the remote storage cannot keep up with the data ingestion speed. In this case it is OK if the scrape pace will be limited.	2023-01-07 00:17:52 -08:00
Aliaksandr Valialkin	7fb02f536a	lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples This should reduce memory usage when scraping big number of targets, since this limits the summary memory usage during concurrent parsing and relabeling by the number of available CPU cores.	2023-01-06 23:01:18 -08:00
Aliaksandr Valialkin	ec7a3b79ab	lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 21:53:08 -08:00
Aliaksandr Valialkin	54410bf51b	lib/promscrape: follow-up after `bced9fb978` - Document the bugfix at docs/CHANGELOG.md - Wait until all the worker goroutines are done in consulWatcher.mustStop() - Do not log `context canceled` errors when discovering consul serviceNames - Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client, since this handling is automatically performed by net/http.Transport. See DisableCompression option at https://pkg.go.dev/net/http#Transport . - Remove explicit handling of the proxyURL, since it is automatically handled by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport . - Expliticly set MaxIdleConnsPerHost, since its default value equals to 2. Such a small value may result in excess tcp connection churn when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client. - Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client. - Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468	2023-01-05 21:23:21 -08:00
Zakhar Bessarab	de5aad2cde	lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568 )	2023-01-05 21:23:15 -08:00
Aliaksandr Valialkin	750d309f63	lib/promscrape/discovery/nomad: follow-up after `48f371a46c` - Remove undocumented `username` and `password` config options from `nomad_sd_config`. TODO: probably, remove these options from `consul_sd_config` too? These options exist there for backwards compatibility purposes. - Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels These labels contain AllocID and JobID fields for the discovered Nomad services. - Various typo fixes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 18:09:23 -08:00
Karan Sharma	8f42c5a024	lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549 ) Add nomad_sd_config support for service discovery	2023-01-05 18:07:02 -08:00
Aliaksandr Valialkin	6eda4c6da2	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2023-01-05 16:46:03 -08:00
Aliaksandr Valialkin	1af6e0b233	lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 This allows copy-n-pasting the url to another browser window and seeing the same result. The limit in 1000 chars is selected in order to prevent from potential issues with systems which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string If the limit is exceeded, then query args are sent via POST method and aren't visible in the url. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 16:45:42 -08:00
Zakhar Bessarab	99f9b02283	lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin	1d16cc9349	lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin	634e24e685	lib/promscrape: follow-up for `a7e29c38bc` - Document the bugfix at docs/CHANGELOG.md - Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 02:51:45 -08:00
Zakhar Bessarab	52226c392f	lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595 ) * lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/targetstatus: address review comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-05 02:48:39 -08:00
Aliaksandr Valialkin	ebb8aeb0cf	app/vmselect: remove dependency on lib/promscrape from app/vmselect	2023-01-03 23:27:36 -08:00
Aliaksandr Valialkin	d3f8298739	lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..))	2023-01-03 22:15:49 -08:00
Aliaksandr Valialkin	8e9548f050	lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize This is a follow-up for `7ad9fff7e5`	2022-12-28 14:42:45 -08:00
Clément Nussbaumer	04d536c15a	fix(promscrape): check MaxScrapeSize after gzip decompression (#3550 )	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	c2fc996e01	lib/promscrape/discovery/azure: typo fix	2022-12-21 21:25:25 -08:00
Aliaksandr Valialkin	9330da3195	lib/promscrape/discovery/consul: expose service tags in individual labels `__meta_consul_tag_<tagname>` This simplifies copying service tags to target labels with the following relabeling rule: - action: labelmap regex: __meta_consul_tag_(.+) See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus	2022-12-19 13:02:56 -08:00
Aliaksandr Valialkin	2e597555ec	lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page	2022-12-16 16:44:12 -08:00
Aliaksandr Valialkin	1a88fe5b1f	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:29:57 -08:00
Aliaksandr Valialkin	a521135b7b	lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 12:47:47 -08:00
Aliaksandr Valialkin	97b41e727c	lib/promscrape: implement target-level and metric-level relabel debugging Target-level debugging is performed by clicking the 'debug' link at the corresponding target on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page. Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407 See https://docs.victoriametrics.com/vmagent.html#relabel-debug	2022-12-10 02:25:56 -08:00
Aliaksandr Valialkin	04abd5e113	docs/CHANGELOG.md: document the bugfix at `05b42601c3` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247	2022-12-08 18:35:47 -08:00
Zakhar Bessarab	c939a8e8a2	lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403 ) * lib/promscrape/discovery/azure: remove API server from URL returned by azure * lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer	2022-12-08 18:35:46 -08:00
Aliaksandr Valialkin	a13f16e48a	lib/promscrape: allow using `sample_limit` and `series_limit` options in stream parsing mode Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 17:04:12 -08:00
Aliaksandr Valialkin	6910b1de2e	all: typo fix: `the the` -> `the`	2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin	be6da5053f	lib/promscrape: optimize service discovery speed - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin	2524d94fe1	lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels	2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin	8ce5b095b7	lib/promscrape: add `exported_` prefix to metric names exported by scrape targets if they clash with automatically generated metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:34 -08:00
Roman Khavronenko	d1169c1559	vmagent: expose metrics for tracking config state (#3375 ) Expose `vm_relabel_config_` and `vm_promscrape_config_` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin	c33bcae457	lib/promscrape/discovery/gce: do not pass filter arg when discovering zones The filter arg isn't supported by zones API in GCE. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202	2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin	047fe3ee67	lib/promscrape: add a benchmark for internLabelStrings()	2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin	24213eaeba	lib/promscrape: add more cases to TestAddRowToTimeseries This is a follow-up for `16fdd2af8a`	2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL	87375b004a	test(golang): add test to function addRowToTimeseries (#3282 ) Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>	2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin	abf7e4e72f	lib/promscrape/discovery/consul: add `__meta_consul_partition` label in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/11482	2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin	ac5528cb46	lib/promscrape: properly add `exported_` prefix to labels, which clash with target labels if `honor_labels: true` option isn't set. The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function. The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_` prefix to labels inside `labels` slice become invisible in the returned `dst` labels. While at it, properly handle some corner cases: - Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix. - Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278 Thanks to @jplanckeel for the initial attempt to fix this issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281	2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin	ba739c8052	lib/promscrape/discovery/kubernetes: do not print an empty `kubeconfig_file` option in yaml at `/config` page	2022-10-28 22:15:30 +03:00
Aliaksandr Valialkin	450a32970a	lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax This is a follow-up for `02096e06d0`	2022-10-26 14:51:02 +03:00
Nikolay	ea0596d9d8	lib/promscrape/discovery/kubernetes: correctly wrap error (#3250 ) * lib/promscrape/discovery/kubernetes: correctly wrap error follow-up after `1304824201` * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin	d0288ea417	all: log error when environment variables referred from `-promscrape.config` are missing This should prevent from using incorrect config files	2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin	087393bcef	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin	3b828535f0	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin	3987b0abd1	lib/promscrape: allow specifying full target url in `__address__` label Previously the `__address__` label could contain only `host:port` part of the target url, while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__` labels. Now it is possible to set the full url in `__address__` label. This makes valid the following scrape config, which is frequently used by novice users: scrape_configs: - job_name: foo static_configs: - targets: - http://host1/metrics1 - https://host2/metrics2	2022-10-07 22:46:29 +03:00
Aliaksandr Valialkin	f0a748a3aa	lib/promscrape/discovery/azure: remove unneeded conversion to string	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	735de9ee54	lib/promscrape: add `external_labels` from `global` section of `-promscrape.config` after the relabeling is applied to the scraped metrics This aligns with Prometheus behaviour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	c628f5b6eb	lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does See https://github.com/prometheus/prometheus/pull/11326	2022-09-30 20:49:08 +03:00
Aliaksandr Valialkin	b4bb1477fe	lib/regexutil: cache MatchString results for unoptimized regexps This increases relabeling performance by 3x for unoptimized regexs	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	c0aa10bd73	lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil lib/bytesutil is more appropriate place for InternString() function	2022-09-30 07:34:59 +03:00
Aliaksandr Valialkin	4afa25fb38	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	d1b9cbcef4	lib/promscrape: typo fix after `74c00a8762`	2022-09-14 15:08:42 +03:00
Aliaksandr Valialkin	2351468bc4	lib/promscrape: read response body into memory in stream parsing mode before parsing it This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:29:39 +03:00
Aliaksandr Valialkin	592612b63f	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error This should improve debugging issues with Kubernetes API server	2022-09-13 19:37:40 +03:00
Bryce Lampe	5f7f1d5aea	Support "HTTP" and "HTTPS" schemes (#3019 ) * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:23:52 +03:00
Aliaksandr Valialkin	04761419ba	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache This is a follow-up for `c06e7a142c`	2022-08-27 00:29:59 +03:00
Aliaksandr Valialkin	86394b4179	lib/promscrape: optimize discoveryutils.SanitizeLabelName() Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for `02ca2342ab`	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	06f6de6d47	all: use os.{Read\|Write}File instead of ioutil.{Read\|Write}File The ioutil.{Read\|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for `02ca2342ab`	2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	1a00c9ef03	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_pod_container_image` label in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin	2fb63dda83	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_service_port_number` label to `role: service` in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin	2b58bd9876	lib/promscrape/discovery/dns: add support for resolving MX records See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after `2c553d5a2f` - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950 ) * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Aliaksandr Valialkin	77bd4e37cc	lib/promscrape/discovery/kubernetes: add missing `__meta_kubernetes_ingress_class_name` label for `role: ingress` See `7e65ad3e43` and `7e1111ff14`	2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom `endpoint` option in ec2_sd_configs This option was ignored since `d289ecded1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	80ecfcf759	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin	85b04732ed	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin	17290a4598	lib/promscrape/discovery/yandexcloud: further code cleanup after `83a4abda3f`	2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after `6e5ac32fba` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923 ) * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	1b5799f894	lib/promscrape: set `up=0` for partially failed scrape in stream parsing mode This behaviour aligns with Prometheus behavior	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	2d1366353c	lib/promscrape: reload all the scrape configs when the `global` section is changed inside `-promscrape.config` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:42 +03:00
Nikolay	c007b129cb	lib/promscrape: adds azure service discovery (#2743 ) * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:45:43 +03:00
Aliaksandr Valialkin	5794886662	lib/promscrape: properly set Host header when sending requests via http proxy Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin	4d03ac90fc	lib/promscrape/discovery/kubernetes: properly populate service-level labels for `role: endpointslice` targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin	c4cc45d7f8	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to `role: endpoints` and `role: endpointlice` targets in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/10759	2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin	f9303e494c	lib/promscrape: fix a test after `c66f676f3b`	2022-07-06 13:25:17 +03:00
Aliaksandr Valialkin	498c6d6e72	lib/promscrape: push `scrape_samples_limit` metric to remote storage if `sample_limit` option is set in `scrape_config` for this target See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
ttyv	00956e585d	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786 ) Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	71b0dfdefa	lib/promscrape: always send stale markers with the real scrape timestamp This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	fe2269b999	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package This package already has the same name, so there is no need in explicit name	2022-06-21 20:24:28 +03:00
Roman Khavronenko	2b5e1dee91	vmagent: update SD duration histogram metric if SD is active (#2677 ) The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:53:06 +03:00
Aliaksandr Valialkin	b6e3c12811	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin	68b6ddfb14	all: follow-up after `8edb390e21` - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin	3dbb19d624	lib/promscrape/discovery/kubernetes: follow-up after `006b8c7534` - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin	6c2fb9d8c4	lib/promscrape: add `-promscrape.cluster.name` command-line flag This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 01:11:23 +03:00
Dmytro Kozlov	ce8aade80e	lib/promscrape: adds service discovery visualization for /targets page(#2675 ) * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-04 01:11:23 +03:00
Nikolay	72e43ef2fe	lib/promscrape/discovery/kubernetes: follow-up after `0b5c874911` (#2672 )	2022-06-04 01:11:23 +03:00
hadesy	28d4624f60	promscrape/discovery: support kubeconfig (#2533 )	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	afced37c0b	all: add initial support for query tracing See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin	386f6110ec	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2022-06-01 01:43:25 +03:00
Dmytro Kozlov	cd1fa2e4cd	issue-2594: use embedded for static files (#2650 ) embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	6add79143b	removed redundant return (fixed linter) (#2647 ) * removed redundant return * updated lint package version	2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin	f149d56ac2	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 23:00:30 +03:00
Roman Khavronenko	7406665fc3	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615 ) * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:17:21 +03:00
Boris Petersen	3a8b4fab97	Add ability to sign requests for all AWS services (#2604 ) This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin	116c0b8f2e	docs/vmagent.md: typo fix in the description for `-promscrape.cluster.replicationFactor` command-line flag	2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin	0d0561ca8c	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit `67b10896d2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin	af0da45d3e	lib/promscrape: rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics	2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin	9d40bb7137	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in `ec2_sd_config` via `az_filters` section	2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin	2ce1d09135	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call Previously filters wheren't passed to this call after the commit `0e09fdb8b0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin	873f55bac5	lib/awsapi: pass `filtersQueryString` arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: `0e09fdb8b0` `d289ecded1`	2022-05-05 10:29:47 +03:00
Nikolay	7e58cba6cf	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458 ) * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:28:37 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530 ) standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Dima Lazerka	837e440865	Fix targetstatus qtpl paths (#2517 ) Ran `make quicktemplate-gen` from the root directory	2022-04-29 11:18:14 +03:00
Aliaksandr Valialkin	aa82987d70	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin	c2b13e6a04	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin	a89e31b304	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin	cc6eae6992	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin	60f74dab56	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated This is a follow-up after `26b78ad707`	2022-04-22 13:23:37 +03:00
Aliaksandr Valialkin	fea9d1e6ee	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for `2341bd48d7`	2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin	1e0517b9cd	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:41:23 +03:00
Aliaksandr Valialkin	1ae16bf671	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:27 +03:00
Aliaksandr Valialkin	e9f08b1e6a	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin	909a3ee0e4	lib/promscrape: follow-up after `91e290a8ff`	2022-04-20 16:12:26 +03:00
Nikolay	429848a67d	lib/promscrape: reduce latency for k8s GetLabels (#2454 ) replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:12:25 +03:00
Dmytro Kozlov	9dbfd99777	lib/promscrape: simply update UI (#2479 ) * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 15:38:04 +03:00
Aliaksandr Valialkin	45385a5dc6	lib/promscrape: optimize getScrapeWork() function Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:34:18 +03:00

1 2 3 4 5 ...

658 commits