1597 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	4afa25fb38	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	9c6c691471	lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings ... This is a follow-up for 7c2474dac7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	7f0b95b50a	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names ... Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for fcffdba9dc Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin	41882222d3	lib/netutil/tls.go: consistently use tlsMinVersion name across source code ... This should simplify further code maintenance and refactoring This is a follow-up after 6ab1cede62	2022-09-26 17:59:07 +03:00
Dmytro Kozlov	28dcff5791	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109) ... * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:38:43 +03:00
Roman Khavronenko	fe71c73fe1	lib/mergeset: follow-up after a0e7432e42 (#3145) ... * lib/mergeset: follow-up after a0e7432e42 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:43:17 +03:00
Zakhar Bessarab	6c65ee18d9	vmbackup: configure retries for GCS remote FS (#3156)	2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin	2b98f2bc1a	lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102 See the specification https://graphite.readthedocs.io/en/latest/tags.html	2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin	dbc20091b1	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does ... This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:58:36 +03:00
匠心零度	9777c7a367	lib/querytracer: fix comment (#3135)	2022-09-22 13:59:17 +02:00
Aliaksandr Valialkin	d1b9cbcef4	lib/promscrape: typo fix after 74c00a8762	2022-09-14 15:08:42 +03:00
Aliaksandr Valialkin	2351468bc4	lib/promscrape: read response body into memory in stream parsing mode before parsing it ... This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:29:39 +03:00
Aliaksandr Valialkin	592612b63f	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error ... This should improve debugging issues with Kubernetes API server	2022-09-13 19:37:40 +03:00
Aliaksandr Valialkin	5b488a339d	lib/mergeset: atomically remove part dirs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin	fe52378f45	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic	2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin	6c9729d694	lib/storage: atomically remove parts inside partitions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:28:41 +03:00
Aliaksandr Valialkin	daa42e4f79	lib/storage: atomically remove partitions, which went outside the configured retention ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin	0a342f04b2	lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there ... Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 13:32:05 +03:00
Aliaksandr Valialkin	ff7188b6a5	lib/storage: atomically remove snapshot directories ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:25:48 +03:00
Aliaksandr Valialkin	051e722112	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block ... This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:07:49 +03:00
Bryce Lampe	5f7f1d5aea	Support "HTTP" and "HTTPS" schemes (#3019) ... * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:23:52 +03:00
Aliaksandr Valialkin	04761419ba	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache ... This is a follow-up for c06e7a142c	2022-08-27 00:29:59 +03:00
Aliaksandr Valialkin	86394b4179	lib/promscrape: optimize discoveryutils.SanitizeLabelName() ... Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin	cead9c1e67	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix ... This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:09 +03:00
Aliaksandr Valialkin	427d69e775	lib/promrelabel: optimize common regex mismatch cases for action: replace and action: labelmap	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	da7697fda4	lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop,labelkeep,drop and keep ... This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	e1bd38fa97	lib/promrelabel: optimize matching for commonly used regex patterns in if option ... The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo.*" and "foo.+" - substring match, e.g. ".*foo.*" and ".+foo.+" - alternate values match, e.g. "foo|bar|baz"	2022-08-26 14:55:13 +03:00
Aliaksandr Valialkin	c49751adf8	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:43 +03:00
Aliaksandr Valialkin	909e681024	lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep} with anchored regex prefix ... The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:24:38 +03:00
Aliaksandr Valialkin	d60654eb0a	lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values ... For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo|bar|baz"	2022-08-24 17:55:54 +03:00
Aliaksandr Valialkin	891eb608df	lib/storage: increase the maximum possible or values extracted from regexp from 20 to 100 ... This should improve time series search speed for regexp filters with big number of `or` values.	2022-08-24 17:16:29 +03:00
Aliaksandr Valialkin	1b14cf18b6	lib/storage: ignore start text and end text anchors in getOrValues(regexp) function ... This is OK, since the anchors are implicitly applied to the whole regexp. This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors. For example, `{label="^(foo|bar)$"}`	2022-08-24 17:16:28 +03:00
Aliaksandr Valialkin	7b9ba456ff	app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set ... These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	88e0fe9469	all: use os.ReadDir instead of ioutil.ReadDir ... The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for 02ca2342ab	2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin	06f6de6d47	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:55:20 +03:00
Roman Khavronenko	fc2b8b4efd	lib/storage: bump max merge concurrency for small parts to 15 (#2997) ... * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:37:14 +03:00
Aliaksandr Valialkin	1c7f402598	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling ... For example: - target_label: composite-label replacement: {{source_label1}}-{{source_label2}}	2022-08-21 22:49:24 +03:00
Roman Khavronenko	2c59c83191	lib/storage: fix the search for empty label name (#2991) ... * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-19 11:05:09 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit ... The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	1a363192ff	lib/storage: typo fix in comments after f830edc0bc	2022-08-16 13:45:32 +03:00
Aliaksandr Valialkin	dc929e0d16	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:34:23 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	1a00c9ef03	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin	2fb63dda83	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin	2b58bd9876	lib/promscrape/discovery/dns: add support for resolving MX records ... See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin	10402459d8	lib/vmselectapi: do not log connection accept/close from vmselect ... These log messages became too spammy in production clusters after the commit 190c8b463c , which closes idle connections from vmselect to vmstorage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-08-12 09:15:29 +03:00
Aliaksandr Valialkin	1b39be3305	lib/vmselectapi: add rpc call prefix to the trace of the rpc call in order to make it more clear	2022-08-12 00:20:49 +03:00
Roman Khavronenko	f42853275f	lib/storage: prevent excessive loops when storage is in RO (#2962) ... * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:47 +03:00