github-mirrors/VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

Author	SHA1	Message	Date
Aliaksandr Valialkin	518c340ae3	lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax This is a follow-up for `02096e06d0`	2022-10-26 14:49:33 +03:00
Aliaksandr Valialkin	02096e06d0	lib/envflag: allow referring environment variables in command-line flags	2022-10-26 01:52:05 +03:00
Aliaksandr Valialkin	c4265322f4	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin	d9bbf24183	app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop Previously netstorage.MustStop() call didn't free up all the resources, so the subsequent call to nestorage.Init() would panic. This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.	2022-10-25 17:47:17 +03:00
Aliaksandr Valialkin	8e998aa1a1	lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289	2022-10-24 16:40:20 +03:00
Aliaksandr Valialkin	dba218a8ce	lib/storage: skip blocks outside the configured retention during search Blocks outside the configured retention are eventually deleted during background merge. But such blocks may reside in the storage for long time until background merge. Previously VictoriaMetrics could spend additional CPU time on processing such blocks during search queries. Now these blocks are skipped.	2022-10-24 02:52:44 +03:00
Aliaksandr Valialkin	e2f0b76ebf	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg This makes code easier to read. This is a follow-up after `d2d30581a0`	2022-10-24 01:31:04 +03:00
Aliaksandr Valialkin	89a1108b1a	lib/storage: small code cleanups	2022-10-24 01:17:47 +03:00
Aliaksandr Valialkin	05512fdd74	lib/storage: re-use newTestStorage() instead of manually initializing Storage mock This is a follow-up for `d2d30581a0`	2022-10-23 16:24:00 +03:00
Aliaksandr Valialkin	d2d30581a0	lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback This improves code readability a bit.	2022-10-23 16:10:04 +03:00
Aliaksandr Valialkin	54f35c175c	lib/storage: small refactoring: move retentionDeadline to blockStreamMerger This allows defining per-block retention in the future by updating the getRetentionDeadline function	2022-10-23 16:10:02 +03:00
Aliaksandr Valialkin	187e294a53	lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop	2022-10-23 14:08:57 +03:00
Aliaksandr Valialkin	d0a9ca1bc2	lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms	2022-10-23 12:48:00 +03:00
Aliaksandr Valialkin	5e4dfe50c6	lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function The searchTSIDs function was searching for metricIDs matching the the given tag filters and then was locating the corresponding TSID entries for the found metricIDs. The TSID entries aren't needed when searching for time series names (aka MetricName), so this commit removes the uneeded TSID search from the implementation of /api/v1/series API. This improves perfromance of /api/v1/series calls. This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls, since now these calls cache small metricIDs instead of big TSID entries in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs) without the need to compress the saved entries in order to save cache space. This commit also removes concurrency limiter during searching for matching time series, which was introduced in `8f16388428`, since the concurrency for all the read queries is already limited with -search.maxConcurrentRequests command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2022-10-23 12:23:47 +03:00
Aliaksandr Valialkin	4128ad71e2	lib/storage: move common code to newRawRowsBlock() function	2022-10-21 14:46:55 +03:00
Aliaksandr Valialkin	b5674164c6	lib/storage: simplify code a bit after `3f5959c053`	2022-10-21 14:39:27 +03:00
Aliaksandr Valialkin	fd7c86ae25	lib/{mergeset,storage}: simplify the code a bit after `ae55ad8749`	2022-10-21 14:33:03 +03:00
Aliaksandr Valialkin	99d67ac8ad	lib/storage: validate timestamps in the block only if they use encoding, which needs validation This reduces CPU usage when there is no sense in validating timestamps. This is a follow-up for `5fa9525498` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011	2022-10-21 00:52:32 +03:00
Aliaksandr Valialkin	3f5959c053	lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate This should improve background merge rate under high load a bit	2022-10-20 23:28:24 +03:00
Aliaksandr Valialkin	891ff6af2a	lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes. This This change shouldn't increase memory usage for caches, since the prev cache cleaner should free up memory by deleting unused prev cache as soon as possible. See `08ca45d238` for details on prev cache cleaner.	2022-10-20 21:48:25 +03:00
Aliaksandr Valialkin	08ca45d238	lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine This makes the code more clear after `d906d8573e`	2022-10-20 21:45:29 +03:00
Aliaksandr Valialkin	4cd173bbaa	lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals if they hang for some reason in graceful shutdown procedure.	2022-10-20 21:40:20 +03:00
Aliaksandr Valialkin	150e99d403	lib/{mergeset,storage}: avoid `unaligned 64-bit atomic operation` panic on 32-bit platforms The panic has been introduced in `68f3a02589` While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs This should improve scalability on systems with many CPU cores	2022-10-20 16:25:43 +03:00
Aliaksandr Valialkin	d906d8573e	lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache This means that the majority of requests are successfully served from the current cache, so the previous cache can be reset in order to free up memory.	2022-10-20 10:47:58 +03:00
Aliaksandr Valialkin	817aeafd69	lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses This should improve cache scalability on systems with many CPU cores.	2022-10-20 09:12:17 +03:00
Aliaksandr Valialkin	9c02c39487	lib/workingsetcache: randomize interval for swapping curr and prev caches This should make CPU usage smoother over time, since different caches will be swapped at different times.	2022-10-20 08:42:43 +03:00
Nikolay	1059c4d84a	lib/promscrape/discovery/kubernetes: correctly wrap error (#3250 ) * lib/promscrape/discovery/kubernetes: correctly wrap error follow-up after `1304824201` * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-18 20:37:42 +03:00
Aliaksandr Valialkin	069401a304	all: log error when environment variables referred from `-promscrape.config` are missing This should prevent from using incorrect config files	2022-10-18 10:47:16 +03:00
Aliaksandr Valialkin	fb50730ba7	lib/storage: double the number of rawRows shards on multi-core systems This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage	2022-10-17 18:19:51 +03:00
Aliaksandr Valialkin	ae55ad8749	lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list	2022-10-17 18:01:26 +03:00
Aliaksandr Valialkin	b6e8c1403a	lib/promrelabel: add relabeling tests when the source label is missing	2022-10-17 14:47:52 +03:00
Aliaksandr Valialkin	2e3be68617	lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map This prevents from possible corruption of the internal cache map when the underlying byte slice used by the string key is modified. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227	2022-10-14 09:51:19 +03:00
Nikolay	b856581ad3	lib/backup: set s3 default region to us-west-2 (#3224 ) * lib/backup: set s3 default region to us-west-2 it should fix an error with region detection for bucket, if AWS_REGION env var is not set * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-13 10:30:07 +03:00
Aliaksandr Valialkin	185cff307b	lib/mergeset: mention in the error message the path to the part, which triggered the error This should improve debuggability	2022-10-12 09:54:21 +03:00
Aliaksandr Valialkin	50f5eae0e0	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:51:16 +03:00
Aliaksandr Valialkin	5269b1ad77	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:36:14 +03:00
Aliaksandr Valialkin	f9df0cae16	lib/promscrape: allow specifying full target url in `__address__` label Previously the `__address__` label could contain only `host:port` part of the target url, while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__` labels. Now it is possible to set the full url in `__address__` label. This makes valid the following scrape config, which is frequently used by novice users: scrape_configs: - job_name: foo static_configs: - targets: - http://host1/metrics1 - https://host2/metrics2	2022-10-07 22:43:04 +03:00
Aliaksandr Valialkin	711698b858	lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511	2022-10-07 01:02:06 +03:00
Zakhar Bessarab	176f10f5b2	app/vmbackup: fix compatibility with latest azure sdk (#461 )	2022-10-07 01:02:03 +03:00
Aliaksandr Valialkin	d9282027e6	app: follow-up after `ec04fcac93` * Optimize fast path for /api/v1/import when importing numeric values * Move the docs about the change from features to bugfixes at docs/CHANGELOG.md * Update tests at lib/protoparser/vmimport Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161	2022-10-06 14:52:02 +03:00
Dmytro Kozlov	ec04fcac93	Properly parse json when export import metric (#3180 ) * app/vmselect: properly work when export import json from `api/v1/{export, import}` API * app/vmselect: update convert function * app/vmselect: export null if `math.IsNaN(v)` * app/vmselect: get float from json * lib/protoparser: add test * docs: add change log * lib/protoparser: make export import api compatible	2022-10-06 13:54:20 +03:00
Zakhar Bessarab	97239e05ce	lib/backup/s3remote: fix error checking for alternative S3 providers (#3191 )	2022-10-06 13:36:40 +03:00
Aliaksandr Valialkin	1e93ad84e3	lib/backup/azremote: remove unused methods after the `262ce77e2d`	2022-10-06 13:08:58 +03:00
Zakhar Bessarab	262ce77e2d	lib/backup: add support of Azure Blob Storage (#460 ) * lib/backup: add support of Azure Blob Storage * lib/backup: add enterprise support of Azure Blob Storage	2022-10-06 00:32:46 +03:00
Aliaksandr Valialkin	0dc93cca7f	app/vmagent/remotewrite: allow specifying per-`-remoteWrite.url` disk limits for persistent queue with pending data This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071 Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970	2022-10-01 18:40:59 +03:00
Aliaksandr Valialkin	c1fa9828b3	lib/flagutil: rename Array to ArrayString This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:26:36 +03:00
Zakhar Bessarab	87c77727e4	vmbackup: update AWS SDK to v2 (#3174 ) * lib/backup/s3remote: update AWS SDK to v2 * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * lib/backup/s3remote: refactor error handling Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-01 17:12:07 +03:00
Aliaksandr Valialkin	725dfb0ed6	lib/httpserver: use 302 redirects instead of 301 redirects Incorrect 301 redirects can be cached by user agents such as web browsers. This can complicate recovery procedure after the incorrect redirect is fixed, e.g. web browser cache must be reset. The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752	2022-10-01 16:53:35 +03:00
Aliaksandr Valialkin	4998402004	lib/promscrape: add `external_labels` from `global` section of `-promscrape.config` after the relabeling is applied to the scraped metrics This aligns with Prometheus behaviour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137	2022-10-01 16:13:19 +03:00
Aliaksandr Valialkin	3a98ef2f5f	lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests	2022-10-01 16:05:51 +03:00
Aliaksandr Valialkin	f86070169d	lib/promscrape/discovery/azure: remove unneeded conversion to string	2022-10-01 16:04:37 +03:00
Aliaksandr Valialkin	db16759c68	lib/storage: optimize matching speed for non-trivial regexp filters Wrap re.Match into bytesutil.FastStringMatcher. This increases performance for `{foo=~"complex_regex_here"}` filters by up to 4x.	2022-10-01 12:06:06 +03:00
Aliaksandr Valialkin	e8a64f6e7a	lib/promrelabel: remove redundant memory allocations by using interned strings	2022-10-01 11:50:21 +03:00
Aliaksandr Valialkin	73dc17ef64	lib/promrelabel: add a benchmark for realistic Kubernetes relabeling The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes This benchmark has been copied from `d521933053/model/relabel/relabel_test.go (L505)` See also https://github.com/prometheus/prometheus/pull/11147	2022-10-01 10:38:22 +03:00
Aliaksandr Valialkin	c54e14cdec	lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does See https://github.com/prometheus/prometheus/pull/11326	2022-09-30 20:48:32 +03:00
Nikolay	33f40f4a5f	app/vminsert: allows parsing tenant id from labels (#3009 ) * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 18:35:53 +03:00
Aliaksandr Valialkin	171dd14aa3	lib/promrelabel: go fmt	2022-09-30 12:28:55 +03:00
Aliaksandr Valialkin	a18d6d5ccc	lib/promrelabel: optimize `action: replace` for non-trivial regex values Cache `action: replace` results for non-trivial regexs and return them next time instead of performing CPU-intensive regex replacement. Optimize also `action: labelmap_all` and `action: replace_all` in the same way.	2022-09-30 12:25:05 +03:00
Aliaksandr Valialkin	146021a076	lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at `17289ff481`	2022-09-30 10:49:18 +03:00
Aliaksandr Valialkin	899d2c40fb	lib/promrelabel: optimize `action: labelmap` for non-trivial regexs	2022-09-30 10:43:31 +03:00
Aliaksandr Valialkin	17289ff481	lib/regexutil: cache MatchString results for unoptimized regexps This increases relabeling performance by 3x for unoptimized regexs	2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin	fda60b3d4d	lib/promrelabel: properly parse regex with escaped $ at the end Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131 Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179	2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin	593da3603e	lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil lib/bytesutil is more appropriate place for InternString() function	2022-09-30 07:44:35 +03:00
Nikolay	f61b8cec69	lib/awsapi: fixes sign encoding (#3183 ) * lib/awsapi: fixes sign encoding previously white spaces at filter were incorrectly encoded encoding tip was copied from aws signing lib For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171 * Update lib/awsapi/sign.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 07:43:44 +03:00
Aliaksandr Valialkin	6a32a64073	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin	92b3622253	lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings This is a follow-up for `7c2474dac7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105	2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin	ef435f8cc4	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for `fcffdba9dc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin	6411bbcce7	lib/netutil/tls.go: consistently use tlsMinVersion name across source code This should simplify further code maintenance and refactoring This is a follow-up after `6ab1cede62`	2022-09-26 17:58:01 +03:00
Dmytro Kozlov	6ab1cede62	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109 ) * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:35:45 +03:00
Roman Khavronenko	e96ccf3f71	lib/mergeset: follow-up after `a0e7432e42` (#3145 ) * lib/mergeset: follow-up after `a0e7432e42` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:39:56 +03:00
Zakhar Bessarab	f022296d96	vmbackup: configure retries for GCS remote FS (#3156 )	2022-09-26 16:28:20 +03:00
Aliaksandr Valialkin	41f8c2987d	lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102 See the specification https://graphite.readthedocs.io/en/latest/tags.html	2022-09-26 15:17:25 +03:00
Aliaksandr Valialkin	7c2474dac7	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:57:23 +03:00
匠心零度	3d5509a720	lib/querytracer: fix comment (#3135 )	2022-09-22 19:19:48 +03:00
Aliaksandr Valialkin	56ce7ce85b	lib/promscrape: typo fix after `74c00a8762`	2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin	74c00a8762	lib/promscrape: read response body into memory in stream parsing mode before parsing it This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin	ccad651a61	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error This should improve debugging issues with Kubernetes API server	2022-09-13 19:36:55 +03:00
Aliaksandr Valialkin	ce2c07c5a7	lib/mergeset: atomically remove part dirs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	042a532f70	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	68e32b0764	lib/storage: atomically remove parts inside partitions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	340ada871d	lib/storage: atomically remove partitions, which went outside the configured retention Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin	978dcb4574	lib/storage: properly remove cache directory contents if `reset_cache_on_startup` file is located there Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5f28ca1f42	lib/storage: atomically remove snapshot directories Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	5fa9525498	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:08:09 +03:00
Bryce Lampe	74f8e12e87	Support "HTTP" and "HTTPS" schemes (#3019 ) * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin	30b8d91727	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache This is a follow-up for `c06e7a142c`	2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin	c06e7a142c	lib/promscrape: optimize discoveryutils.SanitizeLabelName() Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin	a2cd79576f	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:30 +03:00
Aliaksandr Valialkin	f49c9bb700	lib/promrelabel: optimize common regex mismatch cases for `action: replace` and `action: labelmap`	2022-08-26 15:45:31 +03:00
Aliaksandr Valialkin	4c6916f32a	lib/promrelabel: use regexutil.PromRegex for regex matching in actions `labeldrop`,`labelkeep`,`drop` and `keep` This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:23:45 +03:00
Aliaksandr Valialkin	7afe8450fc	lib/promrelabel: optimize matching for commonly used regex patterns in `if` option The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo." and "foo.+" - substring match, e.g. ".foo.*" and ".+foo.+" - alternate values match, e.g. "foo\|bar\|baz"	2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin	0ad3bbadd3	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:12 +03:00
Aliaksandr Valialkin	b373661988	lib/promrelabel: optimize `action: {drop,keep,labeldrop,labelkeep}` with anchored `regex` prefix The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:23:55 +03:00
Aliaksandr Valialkin	0d4ea03a73	lib/promrelabel: optimize `action: {labeldrop,labelkeep,keep,drop}` with `regex` containing alternate values For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo\|bar\|baz"	2022-08-24 17:54:29 +03:00
Aliaksandr Valialkin	0d46e24af5	lib/storage: increase the maximum possible `or` values extracted from regexp from 20 to 100 This should improve time series search speed for regexp filters with big number of `or` values.	2022-08-24 17:15:25 +03:00
Aliaksandr Valialkin	fdbf5b5795	lib/storage: ignore `start text` and `end text` anchors in getOrValues(regexp) function This is OK, since the anchors are implicitly applied to the whole regexp. This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors. For example, `{label="^(foo\|bar)$"}`	2022-08-24 17:12:52 +03:00
Aliaksandr Valialkin	796aa310c2	app/vmstorage: expose `vm_{hourly,daily}_series_limit_{max,current}_series` metrics if `-storage.max{Hourly,Daily}Series` limits are set These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:44:04 +03:00
Aliaksandr Valialkin	1f89278d88	all: subsitute ioutil.ReadAll with io.ReadAll ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for `02ca2342ab`	2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin	2c3a89339d	all: use os.ReadDir instead of ioutil.ReadDir The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for `02ca2342ab`	2022-08-22 00:02:25 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read\|Write}File instead of ioutil.{Read\|Write}File The ioutil.{Read\|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for `02ca2342ab`	2022-08-21 23:52:35 +03:00
Roman Khavronenko	d59d829cdb	lib/storage: bump max merge concurrency for small parts to 15 (#2997 ) * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:32:08 +03:00
Aliaksandr Valialkin	8550c44e31	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the `replacement` field during relabeling For example: - target_label: composite-label replacement: "{{source_label1}}-{{source_label2}}"	2022-08-21 22:50:01 +03:00
Roman Khavronenko	31f922944e	lib/storage: fix the search for empty label name (#2991 ) * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-17 21:32:25 +03:00
Aliaksandr Valialkin	7d26414b2e	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin	bb68ab99fa	lib/promscrape: retry http requests if the server returns 429 status code The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin	b0e1bb517e	lib/storage: typo fix in comments after `f830edc0bc`	2022-08-16 13:44:45 +03:00
Aliaksandr Valialkin	f830edc0bc	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when `match[]` filter matches small number of time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:32:40 +03:00
Aliaksandr Valialkin	c3f8481011	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin	95d36da358	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_pod_container_image` label in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin	c4fcd9f1c5	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_service_port_number` label to `role: service` in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin	511805d88d	lib/promscrape/discovery/dns: add support for resolving MX records See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:32:34 +03:00
Roman Khavronenko	a0e7432e42	lib/storage: prevent excessive loops when storage is in RO (#2962 ) * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:00 +03:00
Aliaksandr Valialkin	46d7792b72	lib/promscrape: follow-up after `2c553d5a2f` - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:46:26 +03:00
Fury	2c553d5a2f	add support to scrape multi tenant metrics (#2950 ) * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:10:18 +03:00
Roman Khavronenko	d3f13ab85b	lib/promrelabel: fix expected test result (#2957 ) follow-up after `68c4ec9472` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:47:29 +03:00
Aliaksandr Valialkin	68c4ec9472	lib/promrelabel: do not split regex into multiple lines if it contains groups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:15:26 +03:00
Aliaksandr Valialkin	892c97e350	lib/auth: follow-up after `b6a6a659f4`	2022-08-07 23:14:39 +03:00
Dmytro Kozlov	b6a6a659f4	lib/auth: add tests for NewToken function (#2921 ) * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:07:57 +03:00
Aliaksandr Valialkin	9fa6b25fb2	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:29 +03:00
Aliaksandr Valialkin	0ef29ceb14	lib/promscrape/discovery/kubernetes: add missing `__meta_kubernetes_ingress_class_name` label for `role: ingress` See `7e65ad3e43` and `7e1111ff14`	2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin	f2816ef031	lib/promscrape/discovery/ec2: properly handle custom `endpoint` option in ec2_sd_configs This option was ignored since `d289ecded1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin	3e8890e71b	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin	68de1f4e4a	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin	02de848c88	lib/promscrape/discovery/yandexcloud: further code cleanup after `83a4abda3f`	2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin	83a4abda3f	lib/promscrape/discovery/yandexcloud: follow-up after `6e5ac32fba` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:26:43 +03:00
Igor Tiunov	6e5ac32fba	YC service discovery (#2923 ) * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin	d5df08e9c2	lib/mergeset: cleanup after `de6dd1cd5a` Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:23:01 +03:00
Aliaksandr Valialkin	7c99b9eaad	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:42:43 +03:00
Aliaksandr Valialkin	6b0550c023	app/{vmselect,vmalert}: properly generate http redirects if `-http.pathPrefix` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin	5a4c58f9a2	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:12:29 +03:00
Aliaksandr Valialkin	78520f2702	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:08:51 +03:00
Aliaksandr Valialkin	de6dd1cd5a	lib/mergeset: optimize mergeInmemoryBlocks() function Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-27 23:58:05 +03:00
Aliaksandr Valialkin	a3f5822dc2	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-27 23:05:02 +03:00
Aliaksandr Valialkin	be1c82beb1	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:20:27 +03:00
Aliaksandr Valialkin	5c84f09762	lib/mergeset: add tests and benchmarks for commonPrefixLen function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:24:51 +03:00
Aliaksandr Valialkin	f5676123cc	lib/pushmetrics: `make fmt`	2022-07-26 20:40:19 +03:00
Aliaksandr Valialkin	da11056d85	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:24:24 +03:00
Aliaksandr Valialkin	ad6b3cd47d	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:36:06 +03:00
Aliaksandr Valialkin	4c2f9a1a2e	lib/promscrape: set `up=0` for partially failed scrape in stream parsing mode This behaviour aligns with Prometheus behavior	2022-07-22 13:29:44 +03:00
Roman Khavronenko	2914ce5ca5	vmalert: remove dependency on datasource pkg from config (#2905 ) * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 10:44:55 +02:00
Aliaksandr Valialkin	4ce5875fa8	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:36:27 +03:00
Roman Khavronenko	88edb3f6cf	vmalert: allow configuring custom headers per group (#2901 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 15:59:55 +02:00
Aliaksandr Valialkin	0fd86e2364	lib/promscrape: reload all the scrape configs when the `global` section is changed inside `-promscrape.config` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:07 +03:00
Boris Petersen	2f9668eba5	fix assume role when running in ECS. (#2876 ) This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:33:52 +03:00
Aliaksandr Valialkin	814bb1685f	all: fix other typos in the same way as `6f4d9b2a48` does	2022-07-18 12:08:15 +03:00
zhenyuxie	f3ea7823f3	fix inmemoryBlock's Less method (#2881 )	2022-07-18 11:56:17 +03:00
Nikolay	7301aa678c	lib/promscrape: adds azure service discovery (#2743 ) * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:43:18 +03:00
guidao	91faa152a5	add next retention metric (#2863 ) Co-authored-by: wangfeng <wangfeng@zhihu.com>	2022-07-13 12:37:04 +03:00
Dmytro Kozlov	306ec10c39	lib/mergeset: fix linter error (#2864 )	2022-07-13 12:31:35 +03:00
Aliaksandr Valialkin	17b5ac1608	lib/mergeset: optimize merge speed a bit Use heap.Fix instead of heap.Pop + heap.Push when merging blocks	2022-07-12 12:50:26 +03:00
Aliaksandr Valialkin	5c8eee26bf	all: `make fmt` via the upcoming Go1.19	2022-07-11 19:22:15 +03:00
Aliaksandr Valialkin	f97355d9fb	lib/promscrape: properly set Host header when sending requests via http proxy Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin	10cb67adb5	app/{vmagent,vminsert}: follow-up after `d19e46de55` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:30:58 +03:00
Aliaksandr Valialkin	01f55bc66b	lib/promscrape/discovery/kubernetes: properly populate service-level labels for `role: endpointslice` targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin	b186b63e07	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to `role: endpoints` and `role: endpointlice` targets in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/10759	2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin	e6ba2af7a1	lib/promscrape: fix a test after `c66f676f3b`	2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin	c66f676f3b	lib/promscrape: push `scrape_samples_limit` metric to remote storage if `sample_limit` option is set in `scrape_config` for this target See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin	77cbbacfdb	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics This reduces the number of args to vmselectapi calls	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	e1b8059086	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin	a60e03b3a7	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() This improves the API consistency	2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin	edc76286ac	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit `55e7afae3a`, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:54:03 +03:00
Aliaksandr Valialkin	855436efd2	lib/promauth: refactor NewConfig in order to improve maintainability 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin	c392d6d173	app/vmagent/remotewrite: add `-remoteWrite.header` command-line flag for setting additional http headers to send to -remoteWrite.url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:23 +03:00
Aliaksandr Valialkin	e40b40afe6	Revert "lib/promscrape, vmagent: fix path to files (#2801 )" This reverts commit `0a8e35835c`. Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005	2022-06-30 18:23:56 +03:00
Aliaksandr Valialkin	3e2dd85f7d	all: readability improvements for query traces - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:20:33 +03:00
Dmytro Kozlov	0a8e35835c	lib/promscrape, vmagent: fix path to files (#2801 ) vmagent: respect `-pathPrefix` flag for static files and links	2022-06-30 16:22:54 +02:00
ttyv	bdf9f4669a	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786 ) Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin	a350d1e81c	lib/storage: return marshaled metric names from SearchMetricNames Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin	2c836bd398	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:50:53 +03:00
Aliaksandr Valialkin	e578549b8a	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin	a963b2a0aa	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	ba514284f1	lib/storage: add querytracer to more contexts querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin	134751e43e	all: locate throttled loggers via logger.WithThrottler() only once and then use them This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 13:45:50 +03:00
Aliaksandr Valialkin	52eadb729e	lib/promscrape: always send stale markers with the real scrape timestamp This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:34:18 +03:00
Aliaksandr Valialkin	1c4f67c5d2	lib/promauth: add ability to send additional http headers in requests to scrape targets This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin	e6ed92529b	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package This package already has the same name, so there is no need in explicit name	2022-06-21 20:23:32 +03:00
Loki's Wager	ac411be904	BugFix part_header.go (#2763 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:56:41 +03:00
Aliaksandr Valialkin	49586566a3	docs: follow-up after `e4d6b750f6`	2022-06-20 17:14:43 +03:00
Nikolay	e4d6b750f6	lib/httpserver: adds flagsAuthKey command-line flag (#2758 ) * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:09:32 +03:00
Aliaksandr Valialkin	b958fc7846	lib/storage: properly take into account already registered series when `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are enabled The commit `5fb45173ae` takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:47:47 +03:00
Aliaksandr Valialkin	55e7afae3a	lib/storage: create per-day indexes together with global indexes when registering new time series Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:42:10 +03:00
Aliaksandr Valialkin	5fb45173ae	lib/storage: do not register new series if `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are exceeded Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin	62e2371a67	lib/storage: reset metric id caches for the previous and the current hour Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin	c18f8cccfa	lib/promrelabel: support `action: graphite` relabeling Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:24:22 +03:00
Aliaksandr Valialkin	ec7963208d	app/vmselect: accept `focusLabel` query arg at /api/v1/status/tsdb This allows filling the seriesCountByFocusLabelValue list in the /api/v1/status/tsdb response with label values for the specified focusLabel, which contain the highest number of time series. TODO: add this to Cardinality explorer at VMUI - https://docs.victoriametrics.com/#cardinality-explorer	2022-06-14 18:36:54 +03:00
Aliaksandr Valialkin	b6c1ca12b7	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:32:38 +03:00
Aliaksandr Valialkin	a75e59700f	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:27:59 +03:00
Aliaksandr Valialkin	52cf05c6d2	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:27:40 +03:00
Aliaksandr Valialkin	374beb350e	app/vmselect: optimize `/api/v1/labels` and `/api/v1/label/.../values` handlers when `match[]` query arg is passed to them	2022-06-12 04:32:13 +03:00
Aliaksandr Valialkin	2bcb960f17	all: improve query tracing coverage for indexdb search Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:07:07 +03:00
Howie	76f05f8670	feat: rule limit (#2676 ) vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 08:21:30 +02:00
Aliaksandr Valialkin	12ac255dae	lib/querytracer: make it easier to use by passing trace context message to New and NewChild The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:06:52 +03:00
Dmytro Kozlov	018d2303c4	Cardinality explorer (#2625 ) * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:43:05 +03:00
Roman Khavronenko	63b538ecd1	vmagent: update SD duration histogram metric if SD is active (#2677 ) The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:46:44 +03:00
Roman Khavronenko	1ee1e986da	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673 ) Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 14:55:09 +03:00
Aliaksandr Valialkin	a5814fe16a	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:22:07 +03:00
Aliaksandr Valialkin	8608dd093c	all: follow-up after `8edb390e21` - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 00:57:09 +03:00
Aliaksandr Valialkin	6f0a0e3072	lib/promscrape/discovery/kubernetes: follow-up after `006b8c7534` - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:40:52 +03:00
Aliaksandr Valialkin	cfefdde042	lib/promauth: follow-up after `006b8c7534` - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:01:16 +03:00
Aliaksandr Valialkin	0922ed2b7e	lib/promscrape: add `-promscrape.cluster.name` command-line flag This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 00:37:01 +03:00
Dmytro Kozlov	8edb390e21	lib/promscrape: adds service discovery visualization for /targets page(#2675 ) * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-03 15:38:45 +02:00
Nikolay	a18914abee	lib/promscrape/discovery/kubernetes: follow-up after `0b5c874911` (#2672 )	2022-06-01 20:44:45 +02:00
hadesy	006b8c7534	promscrape/discovery: support kubeconfig (#2533 )	2022-06-01 20:34:00 +02:00
Aliaksandr Valialkin	ca689fec54	docs/CHANGELOG.md: follow-up after `2177089f94`	2022-06-01 14:51:26 +03:00
Aliaksandr Valialkin	ea06d2fd3c	lib/storage: stop background merge when storage enters read-only mode This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:36:45 +03:00
Roman Khavronenko	642eb1c534	lib/storage: make `indexdb/tagFilters` cache size configurable (#2667 ) The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 10:07:53 +02:00
Roman Khavronenko	2177089f94	promrelabel: add support of `lowercase` and `uppercase` relabeling actions (#2665 ) * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 10:02:37 +02:00
Aliaksandr Valialkin	41958ed5dd	all: add initial support for query tracing See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin	d2567ccdd6	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2022-06-01 01:42:34 +03:00
Aliaksandr Valialkin	a1add5c2c7	lib/storage: `make fmt`	2022-05-31 12:54:37 +03:00
Aliaksandr Valialkin	bac75ea8a2	lib/storage: do not take into account series from the next day when `match[]` filter is passed to /api/v1/status/tsdb	2022-05-31 12:15:26 +03:00
Dmytro Kozlov	11f91532c5	issue-2594: use embedded for static files (#2650 ) embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 01:55:28 +02:00
Dmytro Kozlov	1eb29794e6	removed redundant return (fixed linter) (#2647 ) * removed redundant return * updated lint package version	2022-05-26 16:24:01 +02:00
Aliaksandr Valialkin	796804e4b0	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 22:59:36 +03:00
Aliaksandr Valialkin	f6d11a49aa	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag This is a follow-up for `0fbf59199a` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:05:29 +03:00
阳明	0fbf59199a	lib/storage: Remove the effect of time zone on next retention period (#2568 ) (#2574 )	2022-05-25 15:08:24 +03:00
Roman Khavronenko	d5eb6afe26	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615 ) * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:01:37 +03:00
Boris Petersen	3df8caca15	Add ability to sign requests for all AWS services (#2604 ) This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-18 14:58:31 +02:00
Aliaksandr Valialkin	a0727ab1b1	docs/vmagent.md: typo fix in the description for `-promscrape.cluster.replicationFactor` command-line flag	2022-05-12 18:50:29 +03:00
Aliaksandr Valialkin	9ea3f0c0d3	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:33:22 +03:00
Aliaksandr Valialkin	123aa4c79e	lib/promscrape: properly implement ScrapeConfig.clone() Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit `67b10896d2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:05:40 +03:00
Aliaksandr Valialkin	1dc4cc243b	lib/promscrape: rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics	2022-05-06 15:33:13 +03:00
Aliaksandr Valialkin	d5b55fe22d	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in `ec2_sd_config` via `az_filters` section	2022-05-06 12:43:29 +03:00
Aliaksandr Valialkin	97f9c2f667	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call Previously filters wheren't passed to this call after the commit `0e09fdb8b0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:00:23 +03:00
Aliaksandr Valialkin	d285c2fea7	lib/awsapi: pass `filtersQueryString` arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: `0e09fdb8b0` `d289ecded1`	2022-05-05 10:29:34 +03:00
Dmytro Kozlov	7dd9f3b98e	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535 ) * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:03 +03:00
Nikolay	d289ecded1	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458 ) * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:24:19 +03:00
Nikolay	3575aabeaf	lib/promscrape: adds correct http status codes for redirect (#2530 ) standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 13:31:31 +03:00
Aliaksandr Valialkin	0d86644d65	lib/storage: leave the last sample per each discrete interval during the deduplicaton This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:50:45 +03:00
Artem Navoiev	37cf509c3a	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487 ) * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:00:15 +03:00
Aliaksandr Valialkin	20bc2a2c44	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:56:39 +03:00
Dima Lazerka	84683b8569	Fix targetstatus qtpl paths (#2517 ) Ran `make quicktemplate-gen` from the root directory	2022-04-29 10:36:03 +03:00
Aliaksandr Valialkin	6be07e8c25	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:26:01 +03:00
Aliaksandr Valialkin	9fe1bf5d53	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:23:19 +03:00
Aliaksandr Valialkin	eb5d7ad089	lib/promauth: add support for `min_version` option at `tls_config` section in the same way as Prometheus does	2022-04-23 00:16:39 +03:00
Aliaksandr Valialkin	174431e31b	lib/promauth: add support for `proxy_url` option at `oauth2` section in the same way as Prometheus does	2022-04-23 00:00:44 +03:00
Aliaksandr Valialkin	18b14aad8e	lib/promauth: add support for `tls_config` section at `oauth2` config in the same way as Prometheus does	2022-04-22 23:51:07 +03:00
Aliaksandr Valialkin	6f79b2b68b	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:14:17 +03:00
Aliaksandr Valialkin	15190fcdae	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:41 +03:00
Aliaksandr Valialkin	57a0aa204d	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:22:01 +03:00
Aliaksandr Valialkin	67b10896d2	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated This is a follow-up after `26b78ad707`	2022-04-22 13:19:43 +03:00
Aliaksandr Valialkin	98129d4a8e	app/vmstorage: expose `vm_indexdb_items_added_total` and `vm_indexdb_items_added_size_bytes_total` counters at `/metrics` page These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:18:39 +03:00
Aliaksandr Valialkin	167d1bea8f	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for `2341bd48d7`	2022-04-21 13:06:22 +03:00
Aliaksandr Valialkin	c75d0095f5	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:40:53 +03:00
Aliaksandr Valialkin	82e34984dd	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:33 +03:00
Aliaksandr Valialkin	a2de31f8d3	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:40:25 +03:00
Aliaksandr Valialkin	2341bd48d7	lib/promscrape: follow-up after `91e290a8ff`	2022-04-20 16:11:37 +03:00
Nikolay	91e290a8ff	lib/promscrape: reduce latency for k8s GetLabels (#2454 ) replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:09:40 +03:00
Aliaksandr Valialkin	3d0549c982	lib/promscrape: optimize getScrapeWork() function Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:37:00 +03:00
Aliaksandr Valialkin	4513893ead	lib/promscrape: use a hash over target labels as a key for dropped targets' map This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:37:00 +03:00

... 3 4 5 6 7 ...

1838 commits