1564 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	56ce7ce85b	lib/promscrape: typo fix after 74c00a8762	2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin	74c00a8762	lib/promscrape: read response body into memory in stream parsing mode before parsing it ... This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin	ccad651a61	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error ... This should improve debugging issues with Kubernetes API server	2022-09-13 19:36:55 +03:00
Aliaksandr Valialkin	ce2c07c5a7	lib/mergeset: atomically remove part dirs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	042a532f70	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	68e32b0764	lib/storage: atomically remove parts inside partitions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	340ada871d	lib/storage: atomically remove partitions, which went outside the configured retention ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin	978dcb4574	lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there ... Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5f28ca1f42	lib/storage: atomically remove snapshot directories ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5fa9525498	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block ... This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:08:09 +03:00
Bryce Lampe	74f8e12e87	Support "HTTP" and "HTTPS" schemes (#3019) ... * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin	30b8d91727	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache ... This is a follow-up for c06e7a142c	2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin	c06e7a142c	lib/promscrape: optimize discoveryutils.SanitizeLabelName() ... Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin	a2cd79576f	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix ... This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:30 +03:00
Aliaksandr Valialkin	f49c9bb700	lib/promrelabel: optimize common regex mismatch cases for action: replace and action: labelmap	2022-08-26 15:45:31 +03:00
Aliaksandr Valialkin	4c6916f32a	lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop,labelkeep,drop and keep ... This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:23:45 +03:00
Aliaksandr Valialkin	7afe8450fc	lib/promrelabel: optimize matching for commonly used regex patterns in if option ... The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo.*" and "foo.+" - substring match, e.g. ".*foo.*" and ".+foo.+" - alternate values match, e.g. "foo|bar|baz"	2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin	0ad3bbadd3	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
Aliaksandr Valialkin	b373661988	lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep} with anchored regex prefix ... The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:23:55 +03:00
Aliaksandr Valialkin	0d4ea03a73	lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values ... For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo|bar|baz"	2022-08-24 17:54:29 +03:00
Aliaksandr Valialkin	0d46e24af5	lib/storage: increase the maximum possible or values extracted from regexp from 20 to 100 ... This should improve time series search speed for regexp filters with big number of `or` values.	2022-08-24 17:15:25 +03:00
Aliaksandr Valialkin	fdbf5b5795	lib/storage: ignore start text and end text anchors in getOrValues(regexp) function ... This is OK, since the anchors are implicitly applied to the whole regexp. This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors. For example, `{label="^(foo|bar)$"}`	2022-08-24 17:12:52 +03:00
Aliaksandr Valialkin	796aa310c2	app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set ... These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:44:04 +03:00
Aliaksandr Valialkin	1f89278d88	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin	2c3a89339d	all: use os.ReadDir instead of ioutil.ReadDir ... The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for 02ca2342ab	2022-08-22 00:02:25 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:52:35 +03:00
Roman Khavronenko	d59d829cdb	lib/storage: bump max merge concurrency for small parts to 15 (#2997) ... * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:32:08 +03:00
Aliaksandr Valialkin	8550c44e31	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling ... For example: - target_label: composite-label replacement: "{{source_label1}}-{{source_label2}}"	2022-08-21 22:50:01 +03:00
Roman Khavronenko	31f922944e	lib/storage: fix the search for empty label name (#2991) ... * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-17 21:32:25 +03:00
Aliaksandr Valialkin	7d26414b2e	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit ... The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin	bb68ab99fa	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin	b0e1bb517e	lib/storage: typo fix in comments after f830edc0bc	2022-08-16 13:44:45 +03:00
Aliaksandr Valialkin	f830edc0bc	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:32:40 +03:00
Aliaksandr Valialkin	c3f8481011	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin	95d36da358	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin	c4fcd9f1c5	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin	511805d88d	lib/promscrape/discovery/dns: add support for resolving MX records ... See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:32:34 +03:00
Roman Khavronenko	a0e7432e42	lib/storage: prevent excessive loops when storage is in RO (#2962) ... * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:00 +03:00
Aliaksandr Valialkin	46d7792b72	lib/promscrape: follow-up after 2c553d5a2f ... - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:46:26 +03:00
Fury	2c553d5a2f	add support to scrape multi tenant metrics (#2950) ... * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:10:18 +03:00
Roman Khavronenko	d3f13ab85b	lib/promrelabel: fix expected test result (#2957) ... follow-up after 68c4ec9472 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:47:29 +03:00
Aliaksandr Valialkin	68c4ec9472	lib/promrelabel: do not split regex into multiple lines if it contains groups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:15:26 +03:00
Aliaksandr Valialkin	892c97e350	lib/auth: follow-up after b6a6a659f4	2022-08-07 23:14:39 +03:00
Dmytro Kozlov	b6a6a659f4	lib/auth: add tests for NewToken function (#2921) ... * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:07:57 +03:00
Aliaksandr Valialkin	9fa6b25fb2	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:29 +03:00
Aliaksandr Valialkin	0ef29ceb14	lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress ... See 7e65ad3e43 and 7e1111ff14	2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin	f2816ef031	lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs ... This option was ignored since d289ecded1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin	3e8890e71b	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels ... See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin	68de1f4e4a	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default ... This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin	02de848c88	lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f	2022-08-05 10:30:47 +03:00