1565 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	7b9ba456ff	app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set ... These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	88e0fe9469	all: use os.ReadDir instead of ioutil.ReadDir ... The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for 02ca2342ab	2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin	06f6de6d47	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:55:20 +03:00
Roman Khavronenko	fc2b8b4efd	lib/storage: bump max merge concurrency for small parts to 15 (#2997) ... * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:37:14 +03:00
Aliaksandr Valialkin	1c7f402598	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling ... For example: - target_label: composite-label replacement: {{source_label1}}-{{source_label2}}	2022-08-21 22:49:24 +03:00
Roman Khavronenko	2c59c83191	lib/storage: fix the search for empty label name (#2991) ... * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-19 11:05:09 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit ... The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code ... The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	1a363192ff	lib/storage: typo fix in comments after f830edc0bc	2022-08-16 13:45:32 +03:00
Aliaksandr Valialkin	dc929e0d16	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:34:23 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	1a00c9ef03	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin	2fb63dda83	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does ... See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin	2b58bd9876	lib/promscrape/discovery/dns: add support for resolving MX records ... See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin	10402459d8	lib/vmselectapi: do not log connection accept/close from vmselect ... These log messages became too spammy in production clusters after the commit 190c8b463c , which closes idle connections from vmselect to vmstorage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-08-12 09:15:29 +03:00
Aliaksandr Valialkin	1b39be3305	lib/vmselectapi: add rpc call prefix to the trace of the rpc call in order to make it more clear	2022-08-12 00:20:49 +03:00
Roman Khavronenko	f42853275f	lib/storage: prevent excessive loops when storage is in RO (#2962) ... * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:47 +03:00
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after 2c553d5a2f ... - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950) ... * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Roman Khavronenko	f31132b70b	lib/promrelabel: fix expected test result (#2957) ... follow-up after 68c4ec9472 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:48:14 +03:00
Aliaksandr Valialkin	9039f23bd1	lib/promrelabel: do not split regex into multiple lines if it contains groups ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:16:15 +03:00
Aliaksandr Valialkin	a17030090b	lib/auth: follow-up after b6a6a659f4	2022-08-07 23:15:25 +03:00
Dmytro Kozlov	a266e3e136	lib/auth: add tests for NewToken function (#2921) ... * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:15:23 +03:00
Aliaksandr Valialkin	fd1ac20760	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:41 +03:00
Aliaksandr Valialkin	77bd4e37cc	lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress ... See 7e65ad3e43 and 7e1111ff14	2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs ... This option was ignored since d289ecded1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	80ecfcf759	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels ... See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin	85b04732ed	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default ... This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin	17290a4598	lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f	2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923) ... * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	db049fed84	lib/mergeset: cleanup after de6dd1cd5a ... Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	ba3ca5d1cd	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:43:24 +03:00
Aliaksandr Valialkin	a1e49606ed	app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin	9f95099cf4	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:13:41 +03:00
Aliaksandr Valialkin	586d267a44	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache ... This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:11:14 +03:00
Aliaksandr Valialkin	962ed46583	lib/mergeset: optimize mergeInmemoryBlocks() function ... Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin	3bbe9054d3	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge ... Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin	547cb1edce	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin	5f2b5bd173	lib/mergeset: add tests and benchmarks for commonPrefixLen function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:25:23 +03:00
Aliaksandr Valialkin	749e825020	lib/pushmetrics: make fmt	2022-07-26 20:41:23 +03:00
Aliaksandr Valialkin	9f1e558c58	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	1b5799f894	lib/promscrape: set up=0 for partially failed scrape in stream parsing mode ... This behaviour aligns with Prometheus behavior	2022-07-22 13:38:25 +03:00
Roman Khavronenko	01755fac38	vmalert: remove dependency on datasource pkg from config (#2905) ... * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 13:38:25 +03:00
Roman Khavronenko	d0abdc2b5b	vmalert: allow configuring custom headers per group (#2901) ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:48:05 +03:00
Aliaksandr Valialkin	f00a6bf837	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin	2d1366353c	lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:42 +03:00
Boris Petersen	61e5f89cfb	fix assume role when running in ECS. (#2876) ... This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:37:33 +03:00