github-mirrors/VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00

Author	SHA1	Message	Date
Aliaksandr Valialkin	be896ddfd4	lib/protoparser/clusternative: typo fix in the comment: thic -> this	2023-01-06 18:16:25 -08:00
Aliaksandr Valialkin	ec7a3b79ab	lib/promscrape/discovery/{consul,nomad}: wait until the deleted serviceWatchers are stopped inside updateServices() call Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 21:53:08 -08:00
Aliaksandr Valialkin	54410bf51b	lib/promscrape: follow-up after `bced9fb978` - Document the bugfix at docs/CHANGELOG.md - Wait until all the worker goroutines are done in consulWatcher.mustStop() - Do not log `context canceled` errors when discovering consul serviceNames - Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client, since this handling is automatically performed by net/http.Transport. See DisableCompression option at https://pkg.go.dev/net/http#Transport . - Remove explicit handling of the proxyURL, since it is automatically handled by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport . - Expliticly set MaxIdleConnsPerHost, since its default value equals to 2. Such a small value may result in excess tcp connection churn when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client. - Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client. - Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468	2023-01-05 21:23:21 -08:00
Zakhar Bessarab	de5aad2cde	lib/promscrape/discoveryutils: switch to native http client from fasthttp (#3568 )	2023-01-05 21:23:15 -08:00
Roman Khavronenko	57277ed6bc	vmstorage: add more context to the flock acquiring msg (#3584 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3578 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-05 18:32:53 -08:00
Aliaksandr Valialkin	750d309f63	lib/promscrape/discovery/nomad: follow-up after `48f371a46c` - Remove undocumented `username` and `password` config options from `nomad_sd_config`. TODO: probably, remove these options from `consul_sd_config` too? These options exist there for backwards compatibility purposes. - Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels These labels contain AllocID and JobID fields for the discovered Nomad services. - Various typo fixes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367	2023-01-05 18:09:23 -08:00
Karan Sharma	8f42c5a024	lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549 ) Add nomad_sd_config support for service discovery	2023-01-05 18:07:02 -08:00
Aliaksandr Valialkin	6eda4c6da2	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2023-01-05 16:46:03 -08:00
Aliaksandr Valialkin	1af6e0b233	lib/promrelabel: pass query args via query string at /metric-relabel-debug and /target-relabel-debug pages if their length doesnt exceed 1000 This allows copy-n-pasting the url to another browser window and seeing the same result. The limit in 1000 chars is selected in order to prevent from potential issues with systems which limit the url length such as Internet Explorer - see https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string If the limit is exceeded, then query args are sent via POST method and aren't visible in the url. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 16:45:42 -08:00
Zakhar Bessarab	99f9b02283	lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586 ) Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin	1d16cc9349	lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin	634e24e685	lib/promscrape: follow-up for `a7e29c38bc` - Document the bugfix at docs/CHANGELOG.md - Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580	2023-01-05 02:51:45 -08:00
Zakhar Bessarab	52226c392f	lib/promscrape/targetstatus: fix crash during droppedTarget registration (#3595 ) * lib/promscrape/targetstatus: fix crash during droppedTarget registration in case original labels are not present Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/promscrape/targetstatus: address review comment Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>	2023-01-05 02:48:39 -08:00
Aliaksandr Valialkin	c97d6ed6a4	lib/streamaggr: sort `by` and `without` labels in the aggregate output metric name Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460	2023-01-05 02:08:59 -08:00
Aliaksandr Valialkin	ccec8c26ed	lib/streamaggr: remove unused fields	2023-01-04 13:33:21 -08:00
Aliaksandr Valialkin	ebb8aeb0cf	app/vmselect: remove dependency on lib/promscrape from app/vmselect	2023-01-03 23:27:36 -08:00
Aliaksandr Valialkin	3369371636	app/{vmagent,vminsert}: add support for streaming aggregation See https://docs.victoriametrics.com/stream-aggregation.html Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460	2023-01-03 22:22:07 -08:00
Aliaksandr Valialkin	d3f8298739	lib/bytesutil: add InternBytes() function as a shortcut to InternString(ToUnsafeString(..))	2023-01-03 22:15:49 -08:00
Aliaksandr Valialkin	a7942c6c0d	lib/promrelabel: allow calling Match on nil IfExpression This simplifies the caller side of IfExpression	2022-12-30 16:47:59 -08:00
Roman Khavronenko	c22122212d	csvimport: support empty values (#3565 ) Before, if the imported line contained multiple metrics and one or more of them had an empty values - the whole line was ignored. Now, only metrics with empty values are ignored, and the rest of the metrics are accepted successfully. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-29 11:56:02 -08:00
Aliaksandr Valialkin	8e9548f050	lib/promscrape: log the actual response size in the error message when the response size exceeds -promscrape.maxScrapeSize This is a follow-up for `7ad9fff7e5`	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	8dc04a86f6	lib/{storage,mergeset}: tune the threshold for assisted merge The https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425#issuecomment-1359117221 reveals that CPU usage for incoming queries may significantly increase when the number of in-memory parts becomes too big. This commit reduces the maximum number of in-memory parts before starting the assisted merge during data ingestion. This should reduce CPU usage for incoming queries, since they need to inspect lower number of in-memory parts. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-28 14:42:45 -08:00
Clément Nussbaumer	04d536c15a	fix(promscrape): check MaxScrapeSize after gzip decompression (#3550 )	2022-12-28 14:42:45 -08:00
Aliaksandr Valialkin	ae0a77d778	lib/snapshot: improve log message on unexpected status code during attempts to create or delete snapshots Use "unexpected status code returned from %q: %d; expecting %d" log message format instead of less clear format "unexpected status code returned from %q; expecting %d; got %d" This is a follow-up for `c612bb165e`	2022-12-28 11:46:23 -08:00
Zakhar Bessarab	990c874b25	lib/snapshot: fix error message format for failed HTTP request (#3559 )	2022-12-28 11:44:06 -08:00
Aliaksandr Valialkin	c2fc996e01	lib/promscrape/discovery/azure: typo fix	2022-12-21 21:25:25 -08:00
Aliaksandr Valialkin	7888712185	lib/promrelabel: `make fmt` after `d3de110070`	2022-12-21 20:25:37 -08:00
Aliaksandr Valialkin	cc482b89a3	lib/promrelabel: add support for `keepequal` and `dropequal` relabeling actions These actions are supported by Prometheus starting from v2.41.0 See https://github.com/prometheus/prometheus/pull/11564 , https://github.com/prometheus/prometheus/issues/11556 and https://github.com/prometheus/prometheus/issues/3756 Side note: It's a pity that Prometheus developers decided inventing `keepequal` and `dropequal` relabeling actions instead of adding support for `keep_if_equal` and `drop_if_equal` relabeling actions supported by VictoriaMetrics since June 2020 - see `2a39ba639d` .	2022-12-21 20:06:09 -08:00
Aliaksandr Valialkin	fcee36081b	lib/bytesutil: make sure that the cleanup code is performed only by a single goroutine out of many concurrently running goroutines Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466	2022-12-21 12:58:27 -08:00
Zakhar Bessarab	decf46d72b	app/vmbackupmanager: add metrics for better observability (#488 ) * app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response * app/vmbackupmanager: drop old metrics before creating new ones * app/vmbackupmanager: use `_total` postfix for counter metrics * app/vmbackupmanager: remove `_total` postfix for gauge-like metrics * app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention * app/vmbackupmanager: address review feedback * app/vmbackupmanager: fix metric name * app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage * app/vmbackupmanager: improve performance for backup size calculation * app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic * {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package * app/vmbackupmanager: fix creating new metrics instead of reusing existing ones * lit/formatutil: add comment to make linter happy * app/vmbackupmanager: address review feedback	2022-12-20 14:18:43 -08:00
Aliaksandr Valialkin	1ff62629f4	lib/storage: clear the err if it is set to io.EOF when searching for the TSID by metricID This is expected error after when recently added indexdb data isn't available for search yet or wasn't flushed to disk after unclean shutdown of VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515	2022-12-20 14:05:53 -08:00
Aliaksandr Valialkin	2184de3bf2	lib/storage: do not check for the result returned by db.doExtDB() where this isn't necessary This simplifies the code a bit	2022-12-19 13:23:30 -08:00
Aliaksandr Valialkin	9330da3195	lib/promscrape/discovery/consul: expose service tags in individual labels `__meta_consul_tag_<tagname>` This simplifies copying service tags to target labels with the following relabeling rule: - action: labelmap regex: __meta_consul_tag_(.+) See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus	2022-12-19 13:02:56 -08:00
Aliaksandr Valialkin	11bd290201	lib/storage: search for TSIDs for the given metricIDs in the previous indexdb if they aren't found in the current indexdb The issue triggers after the indexdb rotation for time series, which stop receiving new samples. This results in missing data for such time series in query responses. This commit should address the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502 The issue has been introduced in `2dd93449d8`	2022-12-19 11:56:49 -08:00
Aliaksandr Valialkin	8c08d625ee	lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid This should help improving performance at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-19 10:31:39 -08:00
Aliaksandr Valialkin	512c73cef9	lib/storage: properly set buf capacity inside marshalMetricID Previously it was always set to 0. In theory this could result into incorrect marshaling of metricIDs. The issue has been introduced in `5e4dfe50c6`	2022-12-19 10:31:38 -08:00
Aliaksandr Valialkin	2fad03d85e	lib/logger: follow-up for `72f8fce107` - Document the change at docs/CHANELOG.md - Log fatal errors if the -loggerJSONFields contains unexpected values - Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348	2022-12-16 17:44:05 -08:00
Michal Kralik	cffd2f79a1	lib/logger: support for renaming json fields (#3488 )	2022-12-16 17:43:39 -08:00
Aliaksandr Valialkin	70c720d640	lib/logger: follow-up for `72f8fce107` - Document the change at docs/CHANELOG.md - Log fatal errors if the -loggerJSONFields contains unexpected values - Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348	2022-12-16 17:42:31 -08:00
Aliaksandr Valialkin	2e597555ec	lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page	2022-12-16 16:44:12 -08:00
Aliaksandr Valialkin	fbeebe4869	lib/storage: skip missing tsids in the current block header by using binary search This improves performance by up to 10x when big number of the requested TSIDs are missing in the searched parts. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-14 22:07:55 -08:00
Aliaksandr Valialkin	1a88fe5b1f	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:29:57 -08:00
Aliaksandr Valialkin	3a28a52667	lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes	2022-12-14 17:53:18 -08:00
Zakhar Bessarab	1e58eabde6	lib/backup/azremote: fix copying for parts larger than 256M by using async copy (#3479 ) * lib/backup/azremote: fix copying for parts larger than 256M by using async copy * lib/backup/azremote: add description of an error for log message	2022-12-13 09:36:36 -08:00
Aliaksandr Valialkin	ea7940e5a7	lib/mergeset: reduce the parts threshold before starting assisted merges This should improve query speed in general case. This is a follow-up for `d1af6046c7`	2022-12-13 09:14:08 -08:00
Aliaksandr Valialkin	2a190f6451	lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead Blocked small merges may result into big number of small parts, which, in turn, may result in increased CPU and memory usage during queries, since queries need to inspect all the existing small parts. The issue has been introduced in `8189770c50` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-12 17:01:33 -08:00
Aliaksandr Valialkin	f3e5c9c246	lib/bytesutil: cache results for all the input strings, which were passed during the last 5 minutes from FastStringMatcher.Match(), FastStringTransformer.Transform() and InternString() Previously only up to 100K results were cached. This could result in sub-optimal performance when more than 100K unique strings were actually used. For example, when the relabeling rule was applied to a million of unique Graphite metric names like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 This commit should reduce the long-term CPU usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3466 after all the unique Graphite metrics are registered in the FastStringMatcher.Transform() cache. It is expected that the number of unique strings, which are passed to FastStringMatcher.Match(), FastStringTransformer.Transform() and to InternString() during the last 5 minutes, is limited, so the function results fit memory. Otherwise OOM crash can occur. This should be the case for typical production workloads.	2022-12-12 14:47:00 -08:00
Aliaksandr Valialkin	87390443d1	lib/protoparser/datadog: do not re-use previously parsed field values if they are missing in the currently parsed message Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432	2022-12-11 13:09:41 -08:00
Aliaksandr Valialkin	a521135b7b	lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 12:47:47 -08:00
Aliaksandr Valialkin	97b41e727c	lib/promscrape: implement target-level and metric-level relabel debugging Target-level debugging is performed by clicking the 'debug' link at the corresponding target on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page. Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407 See https://docs.victoriametrics.com/vmagent.html#relabel-debug	2022-12-10 02:25:56 -08:00
Aliaksandr Valialkin	04abd5e113	docs/CHANGELOG.md: document the bugfix at `05b42601c3` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247	2022-12-08 18:35:47 -08:00
Zakhar Bessarab	c939a8e8a2	lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403 ) * lib/promscrape/discovery/azure: remove API server from URL returned by azure * lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer	2022-12-08 18:35:46 -08:00
Aliaksandr Valialkin	6e390f3b99	lib/querytracer: fix remaining tests after `49ebc48809`	2022-12-08 18:18:50 -08:00
Aliaksandr Valialkin	e56d5e1918	lib/storage: follow-up after `7c0ae3a86a` - Update docs at https://docs.victoriametrics.com/#deduplication - Optimize the deduplication loop a bit Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333	2022-12-08 18:18:49 -08:00
Roman Khavronenko	909cd04c55	lib/storage: keep sample with the biggest value on timestamp conflict (#3421 ) The change leaves raw sample with the biggest value for identical timestamps per each `-dedup.minScrapeInterval` discrete interval when the deduplication is enabled. ``` benchstat old.txt new.txt name old time/op new time/op delta DeduplicateSamples/minScrapeInterval=1s-10 817ns ± 2% 832ns ± 3% ~ (p=0.052 n=10+10) DeduplicateSamples/minScrapeInterval=2s-10 1.56µs ± 1% 2.12µs ± 0% +35.19% (p=0.000 n=9+7) DeduplicateSamples/minScrapeInterval=5s-10 1.32µs ± 3% 1.65µs ± 2% +25.57% (p=0.000 n=10+10) DeduplicateSamples/minScrapeInterval=10s-10 1.13µs ± 2% 1.50µs ± 1% +32.85% (p=0.000 n=10+10) name old speed new speed delta DeduplicateSamples/minScrapeInterval=1s-10 10.0GB/s ± 2% 9.9GB/s ± 3% ~ (p=0.052 n=10+10) DeduplicateSamples/minScrapeInterval=2s-10 5.24GB/s ± 1% 3.87GB/s ± 0% -26.03% (p=0.000 n=9+7) DeduplicateSamples/minScrapeInterval=5s-10 6.22GB/s ± 3% 4.96GB/s ± 2% -20.37% (p=0.000 n=10+10) DeduplicateSamples/minScrapeInterval=10s-10 7.28GB/s ± 2% 5.48GB/s ± 1% -24.74% (p=0.000 n=10+10) ``` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-12-08 18:18:36 -08:00
Aliaksandr Valialkin	323fc14e0a	lib/querytracer: fix tests after `49ebc48809`	2022-12-08 17:21:47 -08:00
Aliaksandr Valialkin	a13f16e48a	lib/promscrape: allow using `sample_limit` and `series_limit` options in stream parsing mode Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 17:04:12 -08:00
Aliaksandr Valialkin	255c04bc20	lib/querytracer: put the version of VictoriaMetrics in the first message of query trace This should simplify further debugging, since the first thing to start the debugging by query trace is to know the version of VictoriaMetrics, which produced this trace.	2022-12-07 09:49:51 -08:00
Pedro Gonçalves	84cfe4fcaf	Datadog - Add device as a tag if it's present as a field in the series object (#3431 ) * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin	0a9992a9c6	lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown	2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin	7d5c64eb7a	all: add `-inmemoryDataFlushInterval` command-line flag for controlling the frequency of saving in-memory data to disk The main purpose of this command-line flag is to increase the lifetime of low-end flash storage with the limited number of write operations it can perform. Such flash storage is usually installed on Raspberry PI or similar appliances. For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data. The in-memory data is searchable in the same way as the data stored on disk. VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal. The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337	2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin	9ac1174493	lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function	2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin	a13d21513e	lib/mergeset: panic when too long item is passed to Table.AddItems()	2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin	dccd70ce10	lib/storage: remove duplicate logging for filepath on errors	2022-12-03 23:15:28 -08:00
Aliaksandr Valialkin	813e8402f6	lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args	2022-12-03 23:10:26 -08:00
Aliaksandr Valialkin	bb93494eac	lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers This simplifies the code	2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin	106332cd9f	lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation	2022-12-03 23:03:30 -08:00
Aliaksandr Valialkin	ea55c16422	lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart This allows packing in-memory blocks with different compression levels depending on its contents. This may save memory usage.	2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin	eca7f32151	lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part Previously only data was compressed with the given compressLevel	2022-12-03 22:35:16 -08:00
Aliaksandr Valialkin	7ffa66d249	lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part This results in more correct reporting of memory usage for in-memory parts	2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin	886ce94739	lib/mergeset: reduce the time needed for the slowest tests	2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin	10a17bfa16	lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}	2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin	233301a549	lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention	2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin	fd9d0a550b	lib/storage: remove logging redundant path values in a single error message	2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin	82f64072d2	lib/filestream: remove logging redundant path values in a single error message	2022-12-03 22:02:04 -08:00
Aliaksandr Valialkin	81400c80f0	lib/fs: remove logging redundant path values in a single error message	2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin	dc890ab80c	lib/backup: remove logging duplicate path values in a single error message	2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin	6910b1de2e	all: typo fix: `the the` -> `the`	2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin	0b8e7deabd	lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message appears in the log and then upgrade to newer release.	2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin	8e9822bc7f	lib/storage: speed up search for data block for the given tsids Use binary search instead of linear scan for looking up the needed data block inside index block. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425	2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin	d8303845ef	lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC	2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin	d8d4d21d7a	lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts Move the common code into releasePartsToMerge() method and consistently use it throughout the code.	2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin	be6da5053f	lib/promscrape: optimize service discovery speed - Return meta-labels for the discovered targets via promutils.Labels instead of map[string]string. This improves the speed of generating meta-labels for discovered targets by up to 5x. - Remove memory allocations in hot paths during ScrapeWork generation. The ScrapeWork contains scrape settings for a single discovered target. This improves the service discovery speed by up to 2x.	2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin	2524d94fe1	lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels	2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin	027ab74efb	lib/httpserver: link to url format docs in error message emitted on parse error for the provided url path This should help users identifying and fixing improperly set up urls. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402	2022-11-28 18:49:03 -08:00
Aliaksandr Valialkin	8ce5b095b7	lib/promscrape: add `exported_` prefix to metric names exported by scrape targets if they clash with automatically generated metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:34 -08:00
匠心零度	d4808d5b84	lib/storage: remove extra error check (#3396 )	2022-11-28 17:07:11 +01:00
Aliaksandr Valialkin	c5eebaffd8	app/{vminsert,vmagent}: follow-up after `53a63c6c4c` Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:42:38 -08:00
Zakhar Bessarab	e407e7243a	{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348 ) * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * {app/vmstorage,app/vmselect}: add API to get list of existing tenants * app/vmselect: fix error message * {app/vmstorage,app/vmselect}: fix error messages * app/vmselect: change log level for error handling * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-11-25 10:32:45 -08:00
Roman Khavronenko	d1169c1559	vmagent: expose metrics for tracking config state (#3375 ) Expose `vm_relabel_config_` and `vm_promscrape_config_` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin	c33bcae457	lib/promscrape/discovery/gce: do not pass filter arg when discovering zones The filter arg isn't supported by zones API in GCE. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202	2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin	48dda17e36	lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes. This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-17 21:55:11 +02:00
Aliaksandr Valialkin	047fe3ee67	lib/promscrape: add a benchmark for internLabelStrings()	2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin	b0b8f05fa4	lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() The issue has been introduced in `58b40f514c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin	c8d7e1312c	lib/workingsetcache: add `-prevCacheRemovalPercent` command-line flag for tuning memory usage vs CPU usage ratio Reduce the default value of this flag from 1% to 0.2% after `71335e6024` This flag should help determining the best ratio for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:41:37 +02:00
Aliaksandr Valialkin	06300fe9b8	lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf Previously indexBlock.bhs pointed to the buffer, which could be changed over time. This could result in incorrect time series search over time. This is a follow-up for `58b40f514c` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin	454060fd78	lib/mergeset: remove string allocation and copying when unmarshaling blockHeader This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343	2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin	a5a0fa75bf	lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1% It has been appeared that some production workloads could suffer for some time after every reset of the previous cache when it gets less than 5% of requests after the needed item isn't found in the current cache. This could result in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM usage needed for reading, unpacking and caching the missed data from disk. This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%. This should reduce the possible negative impact after each cache reset by at least 5x, while reducing the total memory used by caches. This is a follow-up for `d906d8573e`	2022-11-10 12:38:53 +02:00
Aliaksandr Valialkin	24213eaeba	lib/promscrape: add more cases to TestAddRowToTimeseries This is a follow-up for `16fdd2af8a`	2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL	87375b004a	test(golang): add test to function addRowToTimeseries (#3282 ) Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>	2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin	2091693f16	lib/protoparser/opentsdb: follow-up after `04b0e4e7bf` - Simplify the parser code to be less error prone - Document the change - Add a test for OpenTSDB put line with trailing whitespace without tags Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290	2022-11-09 15:36:15 +02:00
Roman Khavronenko	71dfe4d697	protoparser/opentsdb: allow lines without tags (#3303 ) According to http://opentsdb.net/docs/build/html/api_telnet/put.html "At least one tag pair must be present". However, in VictoriaMetrics datamodel tags aren't required. This could be confusing for users. Allowing accept lines without tags seems to do no harm. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 15:36:13 +02:00
Aliaksandr Valialkin	abf7e4e72f	lib/promscrape/discovery/consul: add `__meta_consul_partition` label in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/11482	2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin	d3035b1ca1	lib/storage: follow-up for `790768f20b` - Document the bugfix at docs/CHANGELOG.md - Simplify the bugfix a bit	2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin	be78950011	lib/storage: typo fix after 32d48f8dfbb03174858c00bdfe6d9d22431dc8d8	2022-11-07 13:58:13 +02:00
Aliaksandr Valialkin	9d901ee55a	lib/envtemplate: allow non-env var names inside "%{ ... }"	2022-11-07 13:16:00 +02:00
Aliaksandr Valialkin	99e6a937a5	lib/storage: remove unused isFull field from hourMetricIDs struct	2022-11-07 13:15:59 +02:00
Aliaksandr Valialkin	4a6d5ab1b1	lib/promrelabel: go fmt after `5cec9706dc`	2022-10-29 05:17:49 +03:00
Aliaksandr Valialkin	a72bf87e04	lib/promrelabel: add a test from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3251	2022-10-29 04:34:08 +03:00
Aliaksandr Valialkin	eae334f70e	lib/envflag: small refactoring after `518c340ae3` and `02096e06d0`	2022-10-29 02:29:19 +03:00
Aliaksandr Valialkin	ac5528cb46	lib/promscrape: properly add `exported_` prefix to labels, which clash with target labels if `honor_labels: true` option isn't set. The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function. The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_` prefix to labels inside `labels` slice become invisible in the returned `dst` labels. While at it, properly handle some corner cases: - Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix. - Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278 Thanks to @jplanckeel for the initial attempt to fix this issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281	2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin	ba739c8052	lib/promscrape/discovery/kubernetes: do not print an empty `kubeconfig_file` option in yaml at `/config` page	2022-10-28 22:15:30 +03:00
Aliaksandr Valialkin	450a32970a	lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax This is a follow-up for `02096e06d0`	2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin	36b92f07f7	lib/envflag: allow referring environment variables in command-line flags	2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin	ecb71a7221	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin	4f53147ed4	app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop Previously netstorage.MustStop() call didn't free up all the resources, so the subsequent call to nestorage.Init() would panic. This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.	2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin	a6d4711ac6	lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289	2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin	51f2e473f5	lib/storage: skip blocks outside the configured retention during search Blocks outside the configured retention are eventually deleted during background merge. But such blocks may reside in the storage for long time until background merge. Previously VictoriaMetrics could spend additional CPU time on processing such blocks during search queries. Now these blocks are skipped.	2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin	2fc82b846e	lib/storage: do not pass retentionMsecs and isReadOnly args explicitly - access them via Storage arg This makes code easier to read. This is a follow-up after `d2d30581a0`	2022-10-24 01:32:56 +03:00
Aliaksandr Valialkin	d51f9b9284	lib/storage: small code cleanups	2022-10-24 01:17:58 +03:00
Aliaksandr Valialkin	5ace1587e6	lib/storage: re-use newTestStorage() instead of manually initializing Storage mock This is a follow-up for `d2d30581a0`	2022-10-23 16:24:42 +03:00
Aliaksandr Valialkin	57ea7a3ee8	lib/storage: pass Storage to table and partition instead of getDeletedMetricIDs callback This improves code readability a bit.	2022-10-23 16:11:02 +03:00
Aliaksandr Valialkin	63419d8e7c	lib/storage: small refactoring: move retentionDeadline to blockStreamMerger This allows defining per-block retention in the future by updating the getRetentionDeadline function	2022-10-23 16:11:01 +03:00
Aliaksandr Valialkin	31071347ca	lib/storage: use a single reference to the currently merged block - bsm.Block during the block merge loop	2022-10-23 14:09:14 +03:00
Aliaksandr Valialkin	5d0a91afd5	lib/storage: properly pass uint64 constant to fmt.Errorf on 32-bit platforms	2022-10-23 12:48:43 +03:00
Aliaksandr Valialkin	2dd93449d8	lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function The searchTSIDs function was searching for metricIDs matching the the given tag filters and then was locating the corresponding TSID entries for the found metricIDs. The TSID entries aren't needed when searching for time series names (aka MetricName), so this commit removes the uneeded TSID search from the implementation of /api/v1/series API. This improves perfromance of /api/v1/series calls. This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls, since now these calls cache small metricIDs instead of big TSID entries in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs) without the need to compress the saved entries in order to save cache space. This commit also removes concurrency limiter during searching for matching time series, which was introduced in `8f16388428`, since the concurrency for all the read queries is already limited with -search.maxConcurrentRequests command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin	fe5611d6e1	lib/storage: free up memory occupied by Storage.pendingHourEntries after a temporary spike in its memory usage This reduces vmstorage memory usage by up to 20% in production workload	2022-10-21 14:59:14 +03:00
Aliaksandr Valialkin	32b6ce691b	lib/storage: move common code to newRawRowsBlock() function	2022-10-21 14:46:06 +03:00
Aliaksandr Valialkin	2f8861ed9c	lib/storage: simplify code a bit after `3f5959c053`	2022-10-21 14:39:44 +03:00
Aliaksandr Valialkin	1fb2be0cae	lib/{mergeset,storage}: simplify the code a bit after `ae55ad8749`	2022-10-21 14:33:15 +03:00
Aliaksandr Valialkin	af648279ce	lib/storage: validate timestamps in the block only if they use encoding, which needs validation This reduces CPU usage when there is no sense in validating timestamps. This is a follow-up for `5fa9525498` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011	2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin	edf3b7be47	lib/storage: try generating initial parts from inmemory rows with identical sizes under high ingestion rate This should improve background merge rate under high load a bit	2022-10-20 23:27:44 +03:00
Aliaksandr Valialkin	4d71023eb9	lib/workingsetcache: increase default cache expiration from 10 minutes to 20 minutes This increases the maximum time for cache population with new entries from 20 minutes to 40 minutes. This This change shouldn't increase memory usage for caches, since the prev cache cleaner should free up memory by deleting unused prev cache as soon as possible. See `08ca45d238` for details on prev cache cleaner.	2022-10-20 21:59:08 +03:00
Aliaksandr Valialkin	9a52b56b89	lib/workingsetcache: move the cleaner for the prev cache into a separate goroutine This makes the code more clear after `d906d8573e`	2022-10-20 21:59:02 +03:00
Aliaksandr Valialkin	324e119172	lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals if they hang for some reason in graceful shutdown procedure.	2022-10-20 21:58:49 +03:00
Aliaksandr Valialkin	6855de311c	lib/{mergeset,storage}: avoid `unaligned 64-bit atomic operation` panic on 32-bit platforms The panic has been introduced in `68f3a02589` While at it, add padding to shard structs in order to avoid false sharing on mordern CPUs This should improve scalability on systems with many CPU cores	2022-10-20 16:24:46 +03:00
Aliaksandr Valialkin	526bc8a8b0	lib/workingsetcache: drop the previous cache whenever it recieves less than 5% of requests comparing to the current cache This means that the majority of requests are successfully served from the current cache, so the previous cache can be reset in order to free up memory.	2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin	42cda38dbc	lib/workingsetcache: use per-bucket stats counters instead of global stats counters for cache hits/misses This should improve cache scalability on systems with many CPU cores.	2022-10-20 10:48:46 +03:00
Aliaksandr Valialkin	f22bea242f	lib/workingsetcache: randomize interval for swapping curr and prev caches This should make CPU usage smoother over time, since different caches will be swapped at different times.	2022-10-20 10:48:46 +03:00
Nikolay	ea0596d9d8	lib/promscrape/discovery/kubernetes: correctly wrap error (#3250 ) * lib/promscrape/discovery/kubernetes: correctly wrap error follow-up after `1304824201` * Update docs/CHANGELOG.md Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin	d0288ea417	all: log error when environment variables referred from `-promscrape.config` are missing This should prevent from using incorrect config files	2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin	e4e2d1fcde	lib/protoparser/clusternative: allocate unmarshalWork after reading the data from input connection This shortens the time when unmarshalWork is in use. This also reduces the number of unmarshalWork objects in the pool, and its memory usage.	2022-10-18 00:24:04 +03:00
Aliaksandr Valialkin	481ca746ba	lib/protoparser/clusternative: reuse unmarshalWork in order to reduce memory allocations	2022-10-18 00:06:56 +03:00
Aliaksandr Valialkin	6f69a88a5a	lib/storage: double the number of rawRows shards on multi-core systems This should increase data ingestion scalability on multi-core systems at the cost of slightly higher memory usage	2022-10-17 18:19:28 +03:00
Aliaksandr Valialkin	68f3a02589	lib/{storage,mergeset}: do not hold per-shard lock in fast path when adding per-shard items to the flush list	2022-10-17 18:01:55 +03:00
Aliaksandr Valialkin	c4a3d8b169	lib/promrelabel: add relabeling tests when the source label is missing	2022-10-17 14:48:29 +03:00
Aliaksandr Valialkin	ed324aad66	lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map This prevents from possible corruption of the internal cache map when the underlying byte slice used by the string key is modified. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227	2022-10-14 09:52:18 +03:00
Nikolay	07140e0877	lib/backup: set s3 default region to us-west-2 (#3224 ) * lib/backup: set s3 default region to us-west-2 it should fix an error with region detection for bucket, if AWS_REGION env var is not set * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin	7a6e5f9224	lib/mergeset: mention in the error message the path to the part, which triggered the error This should improve debuggability	2022-10-12 09:54:42 +03:00
Aliaksandr Valialkin	087393bcef	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin	3b828535f0	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin	3987b0abd1	lib/promscrape: allow specifying full target url in `__address__` label Previously the `__address__` label could contain only `host:port` part of the target url, while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__` labels. Now it is possible to set the full url in `__address__` label. This makes valid the following scrape config, which is frequently used by novice users: scrape_configs: - job_name: foo static_configs: - targets: - http://host1/metrics1 - https://host2/metrics2	2022-10-07 22:46:29 +03:00
Aliaksandr Valialkin	f926db1de4	lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511	2022-10-07 01:04:37 +03:00
Zakhar Bessarab	a5861407cc	app/vmbackup: fix compatibility with latest azure sdk (#461 )	2022-10-07 01:04:37 +03:00
Aliaksandr Valialkin	958c1f291c	app: follow-up after `ec04fcac93` * Optimize fast path for /api/v1/import when importing numeric values * Move the docs about the change from features to bugfixes at docs/CHANGELOG.md * Update tests at lib/protoparser/vmimport Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161	2022-10-06 14:54:15 +03:00
Dmytro Kozlov	4064db27a8	Properly parse json when export import metric (#3180 ) * app/vmselect: properly work when export import json from `api/v1/{export, import}` API * app/vmselect: update convert function * app/vmselect: export null if `math.IsNaN(v)` * app/vmselect: get float from json * lib/protoparser: add test * docs: add change log * lib/protoparser: make export import api compatible	2022-10-06 14:54:14 +03:00
Zakhar Bessarab	db791a254b	lib/backup/s3remote: fix error checking for alternative S3 providers (#3191 )	2022-10-06 13:37:23 +03:00
Aliaksandr Valialkin	cc0d70c3d6	lib/backup/azremote: remove unused methods after the `262ce77e2d`	2022-10-06 13:30:47 +03:00
Zakhar Bessarab	6a6dcc059b	lib/backup: add support of Azure Blob Storage (#460 ) * lib/backup: add support of Azure Blob Storage * lib/backup: add enterprise support of Azure Blob Storage	2022-10-06 00:36:19 +03:00
Aliaksandr Valialkin	b857365b84	app/vmagent/remotewrite: allow specifying per-`-remoteWrite.url` disk limits for persistent queue with pending data This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071 Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970	2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin	6f9ce3f6d6	lib/flagutil: rename Array to ArrayString This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:28:19 +03:00
Zakhar Bessarab	5b7e8d1309	vmbackup: update AWS SDK to v2 (#3174 ) * lib/backup/s3remote: update AWS SDK to v2 * Update lib/backup/s3remote/s3.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> * lib/backup/s3remote: refactor error handling Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-10-01 17:13:04 +03:00
Aliaksandr Valialkin	93e84a1c57	lib/httpserver: use 302 redirects instead of 301 redirects Incorrect 301 redirects can be cached by user agents such as web browsers. This can complicate recovery procedure after the incorrect redirect is fixed, e.g. web browser cache must be reset. The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752	2022-10-01 16:56:43 +03:00
Aliaksandr Valialkin	f0a748a3aa	lib/promscrape/discovery/azure: remove unneeded conversion to string	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	735de9ee54	lib/promscrape: add `external_labels` from `global` section of `-promscrape.config` after the relabeling is applied to the scraped metrics This aligns with Prometheus behaviour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	e5aa34b2e3	lib/promrelabel: export MustParseMetricWithLabels function, which can be used for simplifying tests	2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin	b96fe2e265	lib/storage: optimize matching speed for non-trivial regexp filters Wrap re.Match into bytesutil.FastStringMatcher. This increases performance for `{foo=~"complex_regex_here"}` filters by up to 4x.	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	969ae90941	lib/promrelabel: remove redundant memory allocations by using interned strings	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	d8d455856c	lib/promrelabel: add a benchmark for realistic Kubernetes relabeling The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes This benchmark has been copied from `d521933053/model/relabel/relabel_test.go (L505)` See also https://github.com/prometheus/prometheus/pull/11147	2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin	c628f5b6eb	lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does See https://github.com/prometheus/prometheus/pull/11326	2022-09-30 20:49:08 +03:00
Nikolay	505d359b39	app/vminsert: allows parsing tenant id from labels (#3009 ) * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin	39ba55dbb3	lib/promrelabel: go fmt	2022-09-30 12:28:27 +03:00
Aliaksandr Valialkin	9fc2817f41	lib/promrelabel: optimize `action: replace` for non-trivial regex values Cache `action: replace` results for non-trivial regexs and return them next time instead of performing CPU-intensive regex replacement. Optimize also `action: labelmap_all` and `action: replace_all` in the same way.	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	f38c9db74d	lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at `17289ff481`	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	fa46c28c5f	lib/promrelabel: optimize `action: labelmap` for non-trivial regexs	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	b4bb1477fe	lib/regexutil: cache MatchString results for unoptimized regexps This increases relabeling performance by 3x for unoptimized regexs	2022-09-30 12:28:25 +03:00
Aliaksandr Valialkin	f1eebc0a99	lib/promrelabel: properly parse regex with escaped $ at the end Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131 Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179	2022-09-30 08:20:57 +03:00
Nikolay	909709346e	lib/awsapi: fixes sign encoding (#3183 ) * lib/awsapi: fixes sign encoding previously white spaces at filter were incorrectly encoded encoding tip was copied from aws signing lib For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171 * Update lib/awsapi/sign.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 07:49:18 +03:00
Aliaksandr Valialkin	c0aa10bd73	lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil lib/bytesutil is more appropriate place for InternString() function	2022-09-30 07:34:59 +03:00
Aliaksandr Valialkin	4afa25fb38	lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	9c6c691471	lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings This is a follow-up for `7c2474dac7` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105	2022-09-28 10:39:42 +03:00
Aliaksandr Valialkin	7f0b95b50a	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for `fcffdba9dc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:02:11 +03:00
Aliaksandr Valialkin	41882222d3	lib/netutil/tls.go: consistently use tlsMinVersion name across source code This should simplify further code maintenance and refactoring This is a follow-up after `6ab1cede62`	2022-09-26 17:59:07 +03:00
Dmytro Kozlov	28dcff5791	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109 ) * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:38:43 +03:00
Roman Khavronenko	fe71c73fe1	lib/mergeset: follow-up after `a0e7432e42` (#3145 ) * lib/mergeset: follow-up after `a0e7432e42` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 16:43:17 +03:00
Zakhar Bessarab	6c65ee18d9	vmbackup: configure retries for GCS remote FS (#3156 )	2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin	2b98f2bc1a	lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102 See the specification https://graphite.readthedocs.io/en/latest/tags.html	2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin	dbc20091b1	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:58:36 +03:00
匠心零度	9777c7a367	lib/querytracer: fix comment (#3135 )	2022-09-22 13:59:17 +02:00
Aliaksandr Valialkin	d1b9cbcef4	lib/promscrape: typo fix after `74c00a8762`	2022-09-14 15:08:42 +03:00
Aliaksandr Valialkin	2351468bc4	lib/promscrape: read response body into memory in stream parsing mode before parsing it This reduces scrape duration for targets returning big responses. The response body was already read into memory in stream parsing mode before this change, so this commit shouldn't increase memory usage.	2022-09-14 13:29:39 +03:00
Aliaksandr Valialkin	592612b63f	lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error This should improve debugging issues with Kubernetes API server	2022-09-13 19:37:40 +03:00
Aliaksandr Valialkin	5b488a339d	lib/mergeset: atomically remove part dirs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin	fe52378f45	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic	2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin	6c9729d694	lib/storage: atomically remove parts inside partitions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 15:28:41 +03:00
Aliaksandr Valialkin	daa42e4f79	lib/storage: atomically remove partitions, which went outside the configured retention Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin	0a342f04b2	lib/storage: properly remove cache directory contents if `reset_cache_on_startup` file is located there Previously the cache directory was removed. This could result in error when the cache directory is mounted to a separate filesystem.	2022-09-13 13:32:05 +03:00
Aliaksandr Valialkin	ff7188b6a5	lib/storage: atomically remove snapshot directories Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 13:25:48 +03:00
Aliaksandr Valialkin	051e722112	lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block This should reduce chances of unnoticed on-disk data corruption. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011 This change modifies the format for data exported via /api/v1/export/native - now this data contains MaxTimestamp and PrecisionBits fields from blockHeader. This is OK, since the native export format is undocumented.	2022-09-06 13:07:49 +03:00
Bryce Lampe	5f7f1d5aea	Support "HTTP" and "HTTPS" schemes (#3019 ) * Support "HTTP" and "HTTPS" schemes * Update lib/promscrape/config.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2022-08-27 02:23:52 +03:00
Aliaksandr Valialkin	04761419ba	lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache This is a follow-up for `c06e7a142c`	2022-08-27 00:29:59 +03:00
Aliaksandr Valialkin	86394b4179	lib/promscrape: optimize discoveryutils.SanitizeLabelName() Cache sanitized label names and return them next time. This reduces the number of allocations and speeds up the SanitizeLabelName() function for common case when the number of unique label names is smaller than 100k	2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin	cead9c1e67	lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix This should improve slow path speed for regexps without literal prefixes	2022-08-26 21:48:09 +03:00
Aliaksandr Valialkin	427d69e775	lib/promrelabel: optimize common regex mismatch cases for `action: replace` and `action: labelmap`	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	da7697fda4	lib/promrelabel: use regexutil.PromRegex for regex matching in actions `labeldrop`,`labelkeep`,`drop` and `keep` This makes possible optimizing additional cases inside regexutil.PromRegex	2022-08-26 15:48:11 +03:00
Aliaksandr Valialkin	e1bd38fa97	lib/promrelabel: optimize matching for commonly used regex patterns in `if` option The following regex patterns are optimized: - literal string match, e.g. "foo" - prefix match, e.g. "foo." and "foo.+" - substring match, e.g. ".foo.*" and ".+foo.+" - alternate values match, e.g. "foo\|bar\|baz"	2022-08-26 14:55:13 +03:00
Aliaksandr Valialkin	c49751adf8	lib/regexutil: add Simplify() function for simplifying the regular expression	2022-08-26 11:57:43 +03:00
Aliaksandr Valialkin	909e681024	lib/promrelabel: optimize `action: {drop,keep,labeldrop,labelkeep}` with anchored `regex` prefix The following commonly used relabeling rules must work faster now: - action: labeldrop regex: "^foo.+$" - action: labeldrop regex: "^bar.*"	2022-08-25 23:24:38 +03:00
Aliaksandr Valialkin	d60654eb0a	lib/promrelabel: optimize `action: {labeldrop,labelkeep,keep,drop}` with `regex` containing alternate values For example, the following relabeling rule must work much faster now: - action: labeldrop regex: "foo\|bar\|baz"	2022-08-24 17:55:54 +03:00
Aliaksandr Valialkin	891eb608df	lib/storage: increase the maximum possible `or` values extracted from regexp from 20 to 100 This should improve time series search speed for regexp filters with big number of `or` values.	2022-08-24 17:16:29 +03:00
Aliaksandr Valialkin	1b14cf18b6	lib/storage: ignore `start text` and `end text` anchors in getOrValues(regexp) function This is OK, since the anchors are implicitly applied to the whole regexp. This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors. For example, `{label="^(foo\|bar)$"}`	2022-08-24 17:16:28 +03:00
Aliaksandr Valialkin	7b9ba456ff	app/vmstorage: expose `vm_{hourly,daily}_series_limit_{max,current}_series` metrics if `-storage.max{Hourly,Daily}Series` limits are set These metrics allow alerting when the number of unique series approach the limit. For example, the following query alerts when the number of series reaches 90% of the configured limit: vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9	2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for `02ca2342ab`	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	88e0fe9469	all: use os.ReadDir instead of ioutil.ReadDir The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for `02ca2342ab`	2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin	06f6de6d47	all: use os.{Read\|Write}File instead of ioutil.{Read\|Write}File The ioutil.{Read\|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for `02ca2342ab`	2022-08-21 23:55:20 +03:00
Roman Khavronenko	fc2b8b4efd	lib/storage: bump max merge concurrency for small parts to 15 (#2997 ) * lib/storage: bump max merge concurrency for small parts to 15 The change is based on the feedback from users on github. Thier examples show, that limit of 8 sometimes become a bottleneck. Users report that without limit concurrency can climb up to 15-20 merges at once. Signed-off-by: hagen1778 <roman@victoriametrics.com> * Update lib/storage/partition.go Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-21 23:37:14 +03:00
Aliaksandr Valialkin	1c7f402598	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the `replacement` field during relabeling For example: - target_label: composite-label replacement: {{source_label1}}-{{source_label2}}	2022-08-21 22:49:24 +03:00
Roman Khavronenko	2c59c83191	lib/storage: fix the search for empty label name (#2991 ) * lib/storage: fix the search for empty label name Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-19 11:05:09 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	aa37e6b438	lib/promscrape: retry http requests if the server returns 429 status code The 429 status code means that the server is overwhelmed with requests. The client can retry the request after some wait time. Implement this strategy for service discovery and scrape requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940	2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin	1a363192ff	lib/storage: typo fix in comments after `f830edc0bc`	2022-08-16 13:45:32 +03:00
Aliaksandr Valialkin	dc929e0d16	lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when `match[]` filter matches small number of time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978	2022-08-16 13:34:23 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	1a00c9ef03	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_pod_container_image` label in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11034	2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin	2fb63dda83	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_service_port_number` label to `role: service` in the same way as Prometheus 2.38 does See https://github.com/prometheus/prometheus/pull/11002	2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin	2b58bd9876	lib/promscrape/discovery/dns: add support for resolving MX records See https://github.com/prometheus/prometheus/pull/10099	2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin	10402459d8	lib/vmselectapi: do not log connection accept/close from vmselect These log messages became too spammy in production clusters after the commit `190c8b463c` , which closes idle connections from vmselect to vmstorage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-08-12 09:15:29 +03:00
Aliaksandr Valialkin	1b39be3305	lib/vmselectapi: add `rpc call` prefix to the trace of the rpc call in order to make it more clear	2022-08-12 00:20:49 +03:00
Roman Khavronenko	f42853275f	lib/storage: prevent excessive loops when storage is in RO (#2962 ) * lib/storage: prevent excessive loops when storage is in RO Returning nil error when storage is in RO mode results into excessive loops and function calls which could result into CPU exhaustion. Returning an err instead will trigger delays in the for loop and save some resources. Signed-off-by: hagen1778 <roman@victoriametrics.com> * document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-08-09 12:17:47 +03:00
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after `2c553d5a2f` - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950 ) * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Roman Khavronenko	f31132b70b	lib/promrelabel: fix expected test result (#2957 ) follow-up after `68c4ec9472` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-08-08 13:48:14 +03:00
Aliaksandr Valialkin	9039f23bd1	lib/promrelabel: do not split regex into multiple lines if it contains groups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928	2022-08-08 03:16:15 +03:00
Aliaksandr Valialkin	a17030090b	lib/auth: follow-up after `b6a6a659f4`	2022-08-07 23:15:25 +03:00
Dmytro Kozlov	a266e3e136	lib/auth: add tests for NewToken function (#2921 ) * lib/auth: add tests from NewToken function * lib/auth: update test, fix problem with type conversion * lib/auth: update test description * lib/auth: simplify failure tests	2022-08-07 23:15:23 +03:00
Aliaksandr Valialkin	fd1ac20760	lib/logger: prettify logging the defined command-line flags	2022-08-07 22:58:41 +03:00
Aliaksandr Valialkin	77bd4e37cc	lib/promscrape/discovery/kubernetes: add missing `__meta_kubernetes_ingress_class_name` label for `role: ingress` See `7e65ad3e43` and `7e1111ff14`	2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom `endpoint` option in ec2_sd_configs This option was ignored since `d289ecded1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	80ecfcf759	lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels See https://github.com/prometheus/prometheus/issues/9187	2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin	85b04732ed	lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default This aligns with Prometheus behaviour. See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config	2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin	17290a4598	lib/promscrape/discovery/yandexcloud: further code cleanup after `83a4abda3f`	2022-08-05 10:31:19 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after `6e5ac32fba` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923 ) * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	db049fed84	lib/mergeset: cleanup after `de6dd1cd5a` Remove unused getInmemoryPart and putInmemoryPart functions Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin	ba3ca5d1cd	lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function	2022-08-04 17:43:24 +03:00
Aliaksandr Valialkin	a1e49606ed	app/{vmselect,vmalert}: properly generate http redirects if `-http.pathPrefix` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918	2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin	9f95099cf4	lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index	2022-08-02 09:13:41 +03:00
Aliaksandr Valialkin	586d267a44	lib/storage: do not compress small number of tsids when storing them in tagFiltersCache This speeds up tsids retreival from the cache for 0-2 tsids	2022-07-30 00:11:14 +03:00
Aliaksandr Valialkin	962ed46583	lib/mergeset: optimize mergeInmemoryBlocks() function Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs. Just merge inmemoryBlock structs directly. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249	2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin	3bbe9054d3	lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge Just read the current item directly from blockStreamReader.Block.Items with the helper method - blockStreamReader.CurrItem()	2022-07-28 00:05:43 +03:00
Aliaksandr Valialkin	547cb1edce	benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 22:19:26 +03:00
Aliaksandr Valialkin	5f2b5bd173	lib/mergeset: add tests and benchmarks for commonPrefixLen function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254 This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913	2022-07-27 21:25:23 +03:00
Aliaksandr Valialkin	749e825020	lib/pushmetrics: `make fmt`	2022-07-26 20:41:23 +03:00
Aliaksandr Valialkin	9f1e558c58	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	1b5799f894	lib/promscrape: set `up=0` for partially failed scrape in stream parsing mode This behaviour aligns with Prometheus behavior	2022-07-22 13:38:25 +03:00
Roman Khavronenko	01755fac38	vmalert: remove dependency on datasource pkg from config (#2905 ) * vmalert: remove dependency on datasource pkg from config Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-22 13:38:25 +03:00
Roman Khavronenko	d0abdc2b5b	vmalert: allow configuring custom headers per group (#2901 ) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-21 20:48:05 +03:00
Aliaksandr Valialkin	f00a6bf837	all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url	2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin	2d1366353c	lib/promscrape: reload all the scrape configs when the `global` section is changed inside `-promscrape.config` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884	2022-07-18 17:15:42 +03:00
Boris Petersen	61e5f89cfb	fix assume role when running in ECS. (#2876 ) This fixes #2875 Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-18 12:37:33 +03:00
Aliaksandr Valialkin	979444b4ed	all: fix other typos in the same way as `6f4d9b2a48` does	2022-07-18 12:10:41 +03:00
zhenyuxie	14c6212a61	fix inmemoryBlock's Less method (#2881 )	2022-07-18 12:00:45 +03:00
Nikolay	c007b129cb	lib/promscrape: adds azure service discovery (#2743 ) * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:45:43 +03:00
guidao	f2d24a660b	add next retention metric (#2863 ) Co-authored-by: wangfeng <wangfeng@zhihu.com>	2022-07-13 12:41:22 +03:00
Dmytro Kozlov	5256af2291	lib/mergeset: fix linter error (#2864 )	2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin	7cbcbea49d	lib/mergeset: optimize merge speed a bit Use heap.Fix instead of heap.Pop + heap.Push when merging blocks	2022-07-12 12:52:36 +03:00
Aliaksandr Valialkin	eab8ebbe11	all: `make fmt` via the upcoming Go1.19	2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin	5794886662	lib/promscrape: properly set Host header when sending requests via http proxy Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794	2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin	95add1e8e4	app/{vmagent,vminsert}: follow-up after `d19e46de55` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:32:11 +03:00
Aliaksandr Valialkin	4d03ac90fc	lib/promscrape/discovery/kubernetes: properly populate service-level labels for `role: endpointslice` targets Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823	2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin	c4cc45d7f8	lib/promscrape/discovery/kubernetes: allow attaching node-level labels to `role: endpoints` and `role: endpointlice` targets in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/10759	2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin	f9303e494c	lib/promscrape: fix a test after `c66f676f3b`	2022-07-06 13:25:17 +03:00
Aliaksandr Valialkin	195dccf678	app/vmselect: add ability to query `vmselect` from another `vmselect`	2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin	498c6d6e72	lib/promscrape: push `scrape_samples_limit` metric to remote storage if `sample_limit` option is set in `scrape_config` for this target See https://github.com/VictoriaMetrics/operator/issues/497	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	b4489028f3	lib/storage: typo fix in MetricName.Unmarshal error	2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin	1ec4dfd678	lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics This reduces the number of args to vmselectapi calls	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	2e721f7d16	lib/vmselectapi: rename Server.MustClose to more clear Server.MustStop	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	270e555f47	lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call	2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin	78eeca6f0d	lib/vmselectapi: rename deleteMetrics to more correct deleteSeries	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	5afa54e845	lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes() This improves the API consistency	2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin	78f9a8aafd	lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index Previously the time series could be put into dateMetricIDCache without registering in the per-day inverted index if GetOrCreateTSIDByName finds TSID entry in the global index. This could lead to missing series in query results. The issue has been introduced in the commit `55e7afae3a`, which has been included in VictoriaMetrics v1.78.0	2022-07-05 14:56:55 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin	7fc03a1deb	app/vmagent/remotewrite: add `-remoteWrite.header` command-line flag for setting additional http headers to send to -remoteWrite.url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin	4fb0f15322	all: readability improvements for query traces - show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value - limit the maximum length of queries and filters shown in trace messages	2022-06-30 18:19:43 +03:00
ttyv	00956e585d	lib/promscrape: fix vmagent tickerCh reload behaviour (#2786 ) Co-authored-by: Dmitriy <dab@ttyv.ru>	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	7d5d33fd71	lib/storage: return marshaled metric names from SearchMetricNames Previously SearchMetricNames was returning unmarshaled metric names. This wasn't great for vmstorage, which should spend additional CPU time for marshaling the metric names before sending them to vmselect. While at it, remove possible duplicate metric names, which could occur when multiple samples for new time series are ingested via concurrent requests. Also sort the metric names before returning them to the client. This simplifies debugging of the returned metric names across repeated requests to /api/v1/series	2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin	15da802f5f	lib/storage: put into query trace the number of found entries in SearchMetricNames	2022-06-28 14:52:39 +03:00
Aliaksandr Valialkin	399d4c36ae	app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day	2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin	64505e924d	app/vmstorage: extract vmselect api server into a separate package - lib/vmselectapi This opens doors for implementing vmselect api server at vmselect level, so top-level vmselect could query lower-level vmselect nodes in the same way as it queries vmstorage nodes. This will create the ability to create highly available querying architecture when multiple independent VictoriaMetrics clusters with the same data are located in distinct availability zones. In this case we can use top-level vmselect instead of Promxy for simultaneous querying of all the clusters in all the AZs.	2022-06-27 14:20:41 +03:00
Aliaksandr Valialkin	6386f117c8	all: show timeRange in traces in human-readable format instead of timestamps in milliseconds	2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin	926fccbb8d	lib/storage: add querytracer to more contexts querytracer has been added to the following storage.Storage methods: - RegisterMetricNames - DeleteMetrics - SearchTagValueSuffixes - SearchGraphitePaths	2022-06-27 12:53:49 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	71b0dfdefa	lib/promscrape: always send stale markers with the real scrape timestamp This guarantees that query won't return data just after the series is disappeared.	2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin	fe2269b999	all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package This package already has the same name, so there is no need in explicit name	2022-06-21 20:24:28 +03:00
Loki's Wager	ca4730c00f	BugFix part_header.go (#2763 ) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757 Co-authored-by: haotingyi <haotingyi@corp.netease.com>	2022-06-21 15:59:11 +03:00
Aliaksandr Valialkin	288d13af8d	lib/netutil: parallelize background pings for remote addresses This should improve the time needed for determining unavailale remote addresses across big numer of ConnPool's. This is a follow-up for `a1629bd3be` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-21 13:32:27 +03:00
Aliaksandr Valialkin	a1629bd3be	lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped until the background goroutine determines that the given address can be successfully dialed. This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711 This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756 The main differences are: - The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool. This makes possible re-using this feature everywhere lib/netutil.ConnPool is used. - The check doesn't take into account handshake errors for already established connections. Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.	2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin	45e9732764	docs: follow-up after `e4d6b750f6`	2022-06-20 17:15:52 +03:00
Nikolay	15662c0f29	lib/httpserver: adds flagsAuthKey command-line flag (#2758 ) * lib/httpserver: adds flagsAuthKey command-line flag It protects /flags endpoint with authKey. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-20 17:15:51 +03:00
Aliaksandr Valialkin	b28c6febf9	app/{vminsert,vmselect}: add `-vmstorageDialTimeout` command-line flag for tuning the maximum time needed for establishing connections to vmstorage Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2022-06-20 15:17:34 +03:00
Aliaksandr Valialkin	270ad39359	lib/storage: properly take into account already registered series when `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are enabled The commit `5fb45173ae` takes into account only newly registered series when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series. This commit returns back accounting for already registered series when applying cardinality limits.	2022-06-20 13:53:41 +03:00
Aliaksandr Valialkin	7a79e7c0ef	lib/storage: create per-day indexes together with global indexes when registering new time series Previously the creation of per-day indexes and global indexes for the newly registered time series was decoupled. Now global indexes and per-day indexes for the current day are created toghether for new time series. This should speed up registering new time series a bit.	2022-06-19 22:32:41 +03:00
Aliaksandr Valialkin	88e1221b35	lib/storage: do not register new series if `-storage.maxHourlySeries` or `-storage.maxDailySeries` limits are exceeded Previously samples for new series weren't added as expected when series limits were reached, but new series were still registered in indexdb.	2022-06-19 22:03:02 +03:00
Aliaksandr Valialkin	c5ac176153	lib/storage: reset metric id caches for the previous and the current hour Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698	2022-06-19 22:02:51 +03:00
Aliaksandr Valialkin	450aa0ae5a	lib/promrelabel: support `action: graphite` relabeling Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737	2022-06-16 20:25:49 +03:00
Aliaksandr Valialkin	45fa9d798d	app/vmselect: accept `focusLabel` query arg at /api/v1/status/tsdb	2022-06-14 18:39:00 +03:00
Aliaksandr Valialkin	fb77843639	lib/storage: show top labels with the highest number of series in cardinality explorer	2022-06-14 16:34:13 +03:00
Aliaksandr Valialkin	3167fbc21d	lib/storage: improve error message when -search.max* command-line flag values are exceeded	2022-06-14 13:28:21 +03:00
Nikolay	e23af8f05c	lib/httpserver: backport changes from master branch (#2697 ) * lib/httpserver: backport changes from master branch adds basicAuth adds authKey check for /metrics and /debug/pprof requests it should improve security for cluster components * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-14 13:02:44 +03:00
Aliaksandr Valialkin	4af43a4a75	lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range	2022-06-12 14:28:37 +03:00
Aliaksandr Valialkin	61e03f172b	app/vmselect: optimize `/api/v1/labels` and `/api/v1/label/.../values` handlers when `match[]` query arg is passed to them	2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin	cb39eada77	all: improve query tracing coverage for indexdb search Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-09 20:04:02 +03:00
Howie	4afd7aa695	feat: rule limit (#2676 ) vmalert: support `limit` param in groups definition `limit` param limits number of time series samples produced by a single rule during execution. On reaching the limit rule will return an err. Signed-off-by: lihaowei <haoweili35@gmail.com>	2022-06-09 13:15:33 +03:00
Aliaksandr Valialkin	a9ea3fee38	lib/querytracer: make it easier to use by passing trace context message to New and NewChild The context message can be extended by calling Donef. If there is no need to extend the message, then just call Done.	2022-06-08 21:16:12 +03:00
Dmytro Kozlov	f2754c3e90	Cardinality explorer (#2625 ) * Cardinality explorer * vmui, vmselect: updated field name, added description to spinner * make vmui-update * updated const name, make vmui-update * lib/storage: changes calculation for totalSeries values * added static files * wip * wip * wip * wip * docs/CHANGELOG.md: document cardinality explorer feature See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233 Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-06-08 18:54:27 +03:00
Roman Khavronenko	2b5e1dee91	vmagent: update SD duration histogram metric if SD is active (#2677 ) The change updates histogram for registering SD update duration only SD is considered as `active`. SD is active if at least one scraper for this SD has started. This change supposed to reduce metrics cardinality produced by duration histogram which gets updated even if SD isn't configured. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:53:06 +03:00
Roman Khavronenko	5f33445f66	lib/storage: limit max mergeConcurrency value for systems with high number of CPUs (#2673 ) Workers count for merges affects the max part size during merges. Such behaviour protects storage from running out of disk space for scenario when all workers are merging parts with the max size. This works very well for most cases. But for systems where high number of CPUs is allocated for vmstorage components this could significantly impact the max part size and result in more unmerged parts than expected. While checking multiple production highly loaded setups it was discovered that `max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2, and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4. The change in this commit limits the max value for concurrency accordingly. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-06-07 15:02:55 +03:00
Aliaksandr Valialkin	b6e3c12811	lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string This improves code readability and maintainability a bit, since the format string is passed as string literal into fmt.Errorf.	2022-06-07 01:24:14 +03:00
Aliaksandr Valialkin	68b6ddfb14	all: follow-up after `8edb390e21` - Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb - Add /service-discovery page for API compatibility with Prometheus - Properly load bootstrap.min.css from /prometheus/targets - Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there	2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin	3dbb19d624	lib/promscrape/discovery/kubernetes: follow-up after `006b8c7534` - make more clear error logs - simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct	2022-06-06 14:41:28 +03:00
Aliaksandr Valialkin	dd0d773c13	lib/promauth: follow-up after `006b8c7534` - Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig. Print hashes instead of real values because of security considerations. - Properly update Config.tlsCertDigets when `key` and `cert` values are set. This allows properly updating scrape targets after these values are updated in configs. - Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert, because these values are immutable. - Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page. - Generate proper error messages on incorrect `key`, `cert` or `ca` values.	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	6c2fb9d8c4	lib/promscrape: add `-promscrape.cluster.name` command-line flag This flag is used for proper data de-duplication when the same target is scraped from multiple vmagent clusters. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679	2022-06-04 01:11:23 +03:00
Dmytro Kozlov	ce8aade80e	lib/promscrape: adds service discovery visualization for /targets page(#2675 ) * lib/promscrape: updated template * lib/promscrape: fixed click on unhealthy and all btns * app/vmselect: jquery scripts into static folder Co-authored-by: f41gh7 <nik@victoriametrics.com>	2022-06-04 01:11:23 +03:00
Nikolay	72e43ef2fe	lib/promscrape/discovery/kubernetes: follow-up after `0b5c874911` (#2672 )	2022-06-04 01:11:23 +03:00
hadesy	28d4624f60	promscrape/discovery: support kubeconfig (#2533 )	2022-06-04 01:11:23 +03:00
Aliaksandr Valialkin	cc226e6ebe	docs/CHANGELOG.md: follow-up after `2177089f94`	2022-06-01 14:57:39 +03:00
Roman Khavronenko	e9ee043879	lib/storage: make `indexdb/tagFilters` cache size configurable (#2667 ) The default size of `indexdb/tagFilters` now can be overridden via `storage.cacheSizeIndexDBTagFilters` flag. Please, be careful with changing default size since it may lead to inefficient work of the vmstorage or OOM exceptions. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663 Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Roman Khavronenko	bca90d7148	promrelabel: add support of `lowercase` and `uppercase` relabeling actions (#2665 ) * promrelabel: add support of `lowercase` and `uppercase` relabeling actions https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/storage: make golangci-lint happy Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Nikolay <nik@victoriametrics.com>	2022-06-01 14:57:39 +03:00
Aliaksandr Valialkin	fedfc9e686	lib/storage: stop background merge when storage enters read-only mode This should prevent from `no space left on device` errors when VictoriaMetrics under-estimates the additional disk space needed for background merge. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603	2022-06-01 14:22:12 +03:00
Aliaksandr Valialkin	afced37c0b	all: add initial support for query tracing See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403	2022-06-01 02:31:44 +03:00
Aliaksandr Valialkin	386f6110ec	lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum In this case there is no need in converting int64 to int	2022-06-01 01:43:25 +03:00
Aliaksandr Valialkin	945e9fa8c4	lib/storage: `make fmt`	2022-05-31 12:42:48 +03:00
Aliaksandr Valialkin	727cc119b6	lib/storage: do not take into account series from the next day when `match[]` filter is passed to /api/v1/status/tsdb	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	cd1fa2e4cd	issue-2594: use embedded for static files (#2650 ) embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries. Co-authored-by: f41gh7 <nik@victoriametrics.com> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594	2022-05-31 12:42:48 +03:00
Dmytro Kozlov	6add79143b	removed redundant return (fixed linter) (#2647 ) * removed redundant return * updated lint package version	2022-05-30 12:25:58 +03:00
Aliaksandr Valialkin	f149d56ac2	lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575 The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 . Thanks to @jelmd .	2022-05-25 23:00:30 +03:00
Aliaksandr Valialkin	38beb9fe04	lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag This is a follow-up for `0fbf59199a` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574	2022-05-25 16:07:14 +03:00
阳明	e4df648ea0	lib/storage: Remove the effect of time zone on next retention period (#2568 ) (#2574 )	2022-05-25 15:10:19 +03:00
Roman Khavronenko	7406665fc3	lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery (#2615 ) * lib/promscrape/discovery/kubernetes: properly updates discovered scrape works previously, added or updated scrapeworks may override previuosly discovered. it happens because swosByKey may contain small subset of kubernetes objects with it's labels. It happens for objectsUpdated and objectsAdded maps, which include only changed elements * Properly calculate vm_promscrape_discovery_kubernetes_scrape_works Co-authored-by: f41gh7 <nik@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-21 01:17:21 +03:00
Boris Petersen	3a8b4fab97	Add ability to sign requests for all AWS services (#2604 ) This adds the ability to utilize sigv4 signing for all AWS services not just "aps". When the newly introduced property "service" is not set it will default to "aps". Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-05-20 14:20:00 +03:00
Aliaksandr Valialkin	116c0b8f2e	docs/vmagent.md: typo fix in the description for `-promscrape.cluster.replicationFactor` command-line flag	2022-05-12 18:51:20 +03:00
Aliaksandr Valialkin	d8a276fbe4	lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get() This should reduce potential spikes in the number of established connections in the following cases: - when the connection establishing procedure becomes temporarily slow - after a temporary spike in the rate of ConnPool.Get() calls See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552	2022-05-11 14:11:06 +03:00
Aliaksandr Valialkin	0d0561ca8c	lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.	2022-05-09 15:34:56 +03:00
Aliaksandr Valialkin	810dd74fb9	lib/promscrape: properly implement ScrapeConfig.clone() Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields - their contents was replaced with `<secret>` value. This led to inability to use passwords and secrets in `-promscrape.config` file. The bug has been introduced in v1.77.0 in the commit `67b10896d2` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551	2022-05-07 00:06:19 +03:00
Aliaksandr Valialkin	af0da45d3e	lib/promscrape: rename `promscrape_stale_samples_created_total` metric to `vm_promscrape_stale_samples_created_total`, so its name is consistent with the rest of `vm_promscrape_` metrics	2022-05-06 15:33:43 +03:00
Aliaksandr Valialkin	9d40bb7137	lib/promscrape/discovery/ec2: add ability to filter Availability Zones in `ec2_sd_config` via `az_filters` section	2022-05-06 12:44:01 +03:00
Aliaksandr Valialkin	2ce1d09135	lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call Previously filters wheren't passed to this call after the commit `0e09fdb8b0` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2022-05-05 11:01:17 +03:00
Aliaksandr Valialkin	873f55bac5	lib/awsapi: pass `filtersQueryString` arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query The filters shouldn't be passed to DescribeAvailabilityZones API call. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 Related commits: `0e09fdb8b0` `d289ecded1`	2022-05-05 10:29:47 +03:00
Dmytro Kozlov	4f40dc9829	{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder (#2535 ) * {vmbackup, vmbackup/snapshot}: validate snapshot name * vmbackup/snapshot: added another checks * backup/actions: added check that we ignore backup_complete.ignore file * vmbackup: moved snapshot to lib directory * lib/snapshot: added functions description * lib/snapshot: fixed typo * vmbackup: code cleanup * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 22:12:48 +03:00
Nikolay	7e58cba6cf	{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite (#2458 ) * {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite moves aws related code into separate lib from lib/promscrape it allows to write data from vmagent to the AWS managed prometheus (cortex) https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287 * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-04 20:28:37 +03:00
Nikolay	51a77759c1	lib/promscrape: adds correct http status codes for redirect (#2530 ) standard http client accepts multiple http status codes as redirect it should fix issue with incorrect redirects https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482	2022-05-03 14:01:57 +03:00
Aliaksandr Valialkin	361b08c30e	lib/storage: leave the last sample per each discrete interval during the deduplicaton This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness	2022-05-02 21:59:31 +03:00
Aliaksandr Valialkin	190c8b463c	lib/netutil: close connections in ConnPool if they are idle for more than 30 seconds Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508	2022-05-02 15:01:52 +03:00
Artem Navoiev	11db05a4ff	lib/{storage,flagutil} - Add option for snapshot autoremoval (#2487 ) * lib/{storage,flagutil} - Add option for snapshot autoremoval - add prometheus-like duration as command flag - add option to delete stale snapshots - update duration.go flag to re-use own code * wip * lib/flagutil: re-use Duration.Set() call in NewDuration * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-05-02 11:24:12 +03:00
Aliaksandr Valialkin	a436836402	lib/flagutil: re-use Duration.Set() call in NewDuration	2022-05-02 10:58:08 +03:00
Dima Lazerka	837e440865	Fix targetstatus qtpl paths (#2517 ) Ran `make quicktemplate-gen` from the root directory	2022-04-29 11:18:14 +03:00
Aliaksandr Valialkin	aa82987d70	lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing This reflects the logic used in Prometheus. See https://github.com/prometheus/prometheus/pull/10080	2022-04-26 15:27:42 +03:00
Aliaksandr Valialkin	a85ef60b4b	lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string	2022-04-23 00:24:13 +03:00
Aliaksandr Valialkin	4c3cd96db5	lib/promauth: add support for `min_version` option at `tls_config` section in the same way as Prometheus does	2022-04-23 00:24:11 +03:00
Aliaksandr Valialkin	808a2f3b61	lib/promauth: add support for `proxy_url` option at `oauth2` section in the same way as Prometheus does	2022-04-23 00:01:53 +03:00
Aliaksandr Valialkin	4ade8511e2	lib/promauth: add support for `tls_config` section at `oauth2` config in the same way as Prometheus does	2022-04-23 00:01:52 +03:00
Aliaksandr Valialkin	c2b13e6a04	lib/promscrape/discovery/kubernetes: limit the minimum sleep time between updating dependent ScrapeWork objects Previously the sleep time could be dropped to nanoseconds, which could result in CPU time waste	2022-04-22 23:15:34 +03:00
Aliaksandr Valialkin	a89e31b304	lib/promscrape/discovery/kubernetes: allow attaching node-level labels and annotations to discovered pod targets in the same way as Prometheus 2.35 does See https://github.com/prometheus/prometheus/issues/9510 and https://github.com/prometheus/prometheus/pull/10080	2022-04-22 20:15:34 +03:00
Aliaksandr Valialkin	cc6eae6992	lib/promscrape/discovery/kubernetes: improve the performance of urlWatcher.reloadObjects() on multi-CPU systems Parallelize the generation of ScrapeWork objects there. Previously they were generated in a single goroutine.	2022-04-22 13:23:39 +03:00
Aliaksandr Valialkin	60f74dab56	lib/promscrape: prevent from memory leaks on -promscrape.config reload when only a small part of scrape jobs is updated This is a follow-up after `26b78ad707`	2022-04-22 13:23:37 +03:00
Aliaksandr Valialkin	ed1b394a1a	app/vmstorage: expose `vm_indexdb_items_added_total` and `vm_indexdb_items_added_size_bytes_total` counters at `/metrics` page These counters can be used for monitoring the rate of addition of new entries in indexdb (aka inverted index). See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2471	2022-04-21 13:19:42 +03:00
Aliaksandr Valialkin	fea9d1e6ee	lib/promscrape/discovery/kubernetes: properly update endpoints and endpointslice objects when the related pod or service objects are updated Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240 This is a follow-up for `2341bd48d7`	2022-04-21 13:06:49 +03:00
Aliaksandr Valialkin	1e0517b9cd	lib/promscrape: remove possible data race when cleaning up internStringsMap	2022-04-20 18:41:23 +03:00
Aliaksandr Valialkin	1ae16bf671	lib/promscrape: zero out labels after duplicate removal inside mergeLabels()	2022-04-20 18:35:27 +03:00
Aliaksandr Valialkin	e9f08b1e6a	lib/promscrape/discovery/kubernetes: do not pre-allocate memory for ScrapeWork objects There is high chance that ScrapeWork objects won't be generated because of relabeling	2022-04-20 16:42:41 +03:00
Aliaksandr Valialkin	909a3ee0e4	lib/promscrape: follow-up after `91e290a8ff`	2022-04-20 16:12:26 +03:00
Nikolay	429848a67d	lib/promscrape: reduce latency for k8s GetLabels (#2454 ) replaces internStringMap with sync.Map - it greatly reduces lock contention concurently reload scrape work for api watcher - each object labels added by dedicated CPU changes can be tested with following script https://gist.github.com/f41gh7/6f8f8d8719786aff1f18a85c23aebf70	2022-04-20 16:12:25 +03:00
Dmytro Kozlov	9dbfd99777	lib/promscrape: simply update UI (#2479 ) * lib/promscrape: simply update UI * lib/promscrape: added vm icon	2022-04-20 15:38:04 +03:00
Aliaksandr Valialkin	45385a5dc6	lib/promscrape: optimize getScrapeWork() function Reduce the number of memory allocations in this function. This improves its performance by up to 50%. This should improve service discovery speed when big number of potential targets with big number of meta-labels are generated by service discovery. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2270	2022-04-20 15:34:18 +03:00
Aliaksandr Valialkin	bfa0b8f710	lib/promscrape: use a hash over target labels as a key for dropped targets' map This reduces the number of allocations and improves the performance for updating dropped targets' map. This map is exposed at /api/v1/targets as in droppedTargets list.	2022-04-20 15:23:54 +03:00
Aliaksandr Valialkin	d0bac8e224	all: typo fix: Kuberntes -> Kubernetes	2022-04-20 10:51:41 +03:00
Dmytro Kozlov	17552dba8b	lib/promscrape: Enable filters for endpoint and labels (#2466 ) * lib/promscrape: Enable filters for endpoint and labels * lib/promscrape: cleanup * lib/promscrape: update template * lib/promscrape: move logic filter logic to backend * lib/promscrape: updated placeholder * lib/promscrape: updated placeholder * lib/promscrape: use two different fields for filters, updated form, added error on parsing queries * lib/promscrape: rename functions * lib/promscrape: removed unused values * wip * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-19 18:27:44 +03:00
Nikolay	628905f080	lib/promscrape: adds job restart method (#2455 ) * lib/promscrape: adds job restart method it must restart only ScrapeConfig with changed content this change greatly reduce time, that needed for job restart and it should decrease possible data loss when config frequently changed at kubernetes based deployments Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 20:29:33 +03:00
Aliaksandr Valialkin	7debf57ca6	lib/httpserver: clarify that `-tls` flag enables TLS for http requests to `-httpListenAddr`	2022-04-16 16:59:41 +03:00
Aliaksandr Valialkin	a7689e1b0c	app/vmstorage: add support for mTLS cipher suites via `-cluster.tlsCipherSuites` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404	2022-04-16 16:36:38 +03:00
Aliaksandr Valialkin	27e74f25d6	lib/httpserver: follow up after `def0032c7d`	2022-04-16 15:52:44 +03:00
Dmytro Kozlov	26ae50ec26	lib/httpserver: added tlsCipherSuites flag (#2468 ) * lib/httpserver: added tlsCipherSuites flag * lib/httpserver: compare lower case strings * lib/httpserver: use EqualFold * lib/httpserver: used flagutil.NewArray, supported only strings cipher suites * lib/httpserver: updated flag description, added flag to documentation * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-16 15:52:42 +03:00
Aliaksandr Valialkin	c50e48a74c	lib/promscrape: follow-up after `baa1c24b36`	2022-04-16 14:26:38 +03:00
Nikolay	a56ee034af	lib/promscrape: removes omitempty for ScrapeConfig (#2457 ) This change fixes incorrect marshalling for ScrapeConfig it affects http endpoint and ScrapeConfig checksum. With omitempty, custom Marshaller is not called if field is not a pointer. Previously this issue happened at vmalert	2022-04-16 14:26:36 +03:00
Aliaksandr Valialkin	4a3172f150	lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* This allows Go complier to generate more optimal code without bound checks	2022-04-12 12:56:52 +03:00
Aliaksandr Valialkin	70ad171070	lib/promscrape: follow-up after `7e79adfb55`	2022-04-12 12:37:03 +03:00
Nikolay	e26bcb8bbb	lib/promscrape: allows to use k8s pod name as clusterMemberNum (#2436 ) * lib/promscrape: allows to use k8s pod name as clusterMemberNum it must improve user expirience and simplify clustering scrapers. it must allow to use vmagent cluster with distroless images https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2359 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-04-12 12:37:02 +03:00
Aliaksandr Valialkin	81b7a31cb1	app/vmstorage: properly handle `maxSeries` limit passed from vmselect to vmstorage	2022-04-12 11:19:07 +03:00
Aliaksandr Valialkin	e3bf464f11	lib/protoparser/native: follow-up after `fe01f4803d`	2022-04-11 19:27:53 +03:00
Nikolay	39225fc809	lib/protoparser/native: fixes parseStream dead-lock (#2423 ) previously, if native block cannot be unmarshaled, wg.Done wasn't called by unmarshal work. It leads to connection blocking and possible dead-lock at client side	2022-04-11 19:27:51 +03:00
Aliaksandr Valialkin	edb139cfe4	lib/memory: export `process_memory_limit_bytes` metric, which shows the amounts of memory the current process has access to This metric is equivalent to `vm_available_memory_bytes`, but it has better name, since the metric is related to a process, not VictoriaMetrics itself. Leave `vm_available_memory_bytes` for backwards compatibility.	2022-04-07 15:24:08 +03:00
Aliaksandr Valialkin	cb319b15bb	lib/storage: increase the number of rawRowsShard shards on systems with more than 4 CPU cores This should improve data ingestion scalability on systems with many CPU cores	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	8ef9348801	lib/mergeset: use more rawItemsShard shards on multi-CPU systems This should improve the scalability for registering of new time series on multi-CPU system	2022-04-06 19:50:41 +03:00
Aliaksandr Valialkin	db00ddd23e	lib/mergeset: skip common prefixes when comparing inmemoryBlock items This should improve the performance for items sorting inside inmemoryBlock.MarshalUnsortedData if they have common prefix. While at it, improve the performance for inmemoryBlock.updateCommonPrefix for sorted items. This should improve performance for inmemoryBlock.MarshalSortedData during background merge.	2022-04-06 18:55:25 +03:00
Aliaksandr Valialkin	88c2631320	lib/protoparser: remove superflowous memory allocations during protocol parsing	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	123a88bb65	lib/storage: reuse sync.WaitGroup objects This reduces GC load by up to 10% according to memory profiling	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	f526c7814e	lib/cgroup: reduce the default GOGC value from 50% to 30% This reduces memory usage under production workloads by up to 10%, while CPU spent on GC remains roughly the same. The CPU spent on GC can be monitored with go_memstats_gc_cpu_fraction metric	2022-04-06 14:00:50 +03:00
Aliaksandr Valialkin	0f1ebd911d	lib/workingsetcache: reuse prev cache after its reset This should reduce memory churn rate	2022-04-05 20:39:44 +03:00
Aliaksandr Valialkin	ac93c36be7	lib/workingsetcache: check more frequently for cache size overflow This should reduce the probability of cache size limit overflow	2022-04-05 18:05:33 +03:00
Nikolay	7eb49d204f	vmctl verify-blocks command (#2390 ) * lib/protoparser: changes ParseStream for native format uses reader instead of http.Request updates app/vmagent and app/vmagent method usage * app/vmctl: add verify-block subcommand it allows to check exported from VictoriaMetrics data block in native format https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2362 Update app/vmctl/README.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2022-04-05 17:46:36 +03:00
Aliaksandr Valialkin	fca0cb8156	lib/workingsetcache: reduce the expiration duration from 20 minutes to 10 minutes This should reduce memory usage for the cache under high churn rate	2022-04-05 17:08:43 +03:00
Aliaksandr Valialkin	8752cce157	app/vminsert: reduce the max packet size, which vminsert can send to vmstorage This reduces the max memory usage for vminsert and vmstorage under heavy ingestion rate by up to 50% on production workload	2022-04-05 15:39:58 +03:00
Nikolay	4cf6219e07	lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache (#2293 ) * lib/{storage,regexpcache}: replaces regexpCacheMap with LRU cache It should decrease memory usage for regexp caching with storing cacheEntry by pointer - golang map should be able to effectivly shrink it's size original issue with this case - unexpected map grows and storage OOM Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Adds missing metrics for regexp cache and regexpPrefixes cache * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-03-26 12:57:27 +02:00
Aliaksandr Valialkin	b843f0e229	app/vmselect: add fine-grained limits for the number of returned/scanned time series for various APIs	2022-03-26 11:28:14 +02:00
Aliaksandr Valialkin	a8a4581c37	lib/blockcache: properly remove references to deleted parts Previously references to deleted parts may remain active as cache.m keys. This could prevent from proper memory de-allocation. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This is a follow-up for `88605a7ea2`	2022-03-18 17:07:54 +02:00
Aliaksandr Valialkin	e35c9124b7	lib/storage: reduce the interval for checking for free disk space from 30 seconds to 1 second This should reduce the probability of out of disk space panics when -storage.minFreeDiskSpaceBytes is set to low values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2305	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	7c92aaeaa4	lib/blockcache: properly release memory occupied by deleted entries Proviously the deleted entries could remain referenced via lastAccessHeap for long time. This could lead to increased memory usage for the following caches starting from v1.73.0: * indexdb/indexBlocks * indexdb/dataBlocks * storage/indexBlocks Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2242 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-03-18 16:53:19 +02:00
Aliaksandr Valialkin	a6d65fc824	lib/storage: typo fix after `e7831ae154`	2022-03-18 16:53:19 +02:00
jduncan0000	e7831ae154	Fix for issue #2255 - matchTagFilters for positive empty-match filters (#2304 ) * fix for issue 2255 - matchTagFilters for positive empty-match filters * add example to comments * formatting * add test for positive empty match * formatting	2022-03-18 13:08:54 +02:00
Aliaksandr Valialkin	698458b742	lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig	2022-03-17 19:46:20 +02:00
Aliaksandr Valialkin	191977b324	lib/storage: trashing -> thrashing typo in docs This is a follow-up for `918ed5cb32`	2022-03-16 13:28:29 +02:00
Vic (Shihang) Li	9767fcd837	fix: change thrashing typo (#2317 )	2022-03-16 13:05:55 +02:00
Aliaksandr Valialkin	3f999b11f7	lib/mergeset: remove aux buffers from inmemoryPart This should reduce the size of inmemoryPart items and may improve performance a bit during registering new time series Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf68da79e	lib/mergeset: eliminate copying of itemsData and lensData from storageBlock to inmemoryBlock This should improve performance when registering new time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	ecf4f7bf21	lib/mergeset: consistency renaming: ip->mp for inmemoryPart vars	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	f4e466955d	lib/mergeset: move storageBlock from inmemoryPart to a sync.Pool The lifetime of storageBlock is much shorter comparing to the lifetime of inmemoryPart, so sync.Pool usage should reduce overall memory usage and improve performance because of better locality of reference when marshaling inmemoryBlock to inmemoryPart. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247	2022-03-03 17:12:25 +02:00
Aliaksandr Valialkin	b47f18f555	lib/{mergeset,storage}: tune compression levels for small blocks This should reduce CPU usage spent on compression	2022-02-25 15:34:13 +02:00
Aliaksandr Valialkin	28b610db07	lib/storage: document why job-like and instance-like labels must be stored at mn.Tags[0] and mn.Tags[1] Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2244	2022-02-25 13:21:53 +02:00
Aliaksandr Valialkin	d1881fa582	lib/storage: add a comment to indexSearch.containsTimeRange() on why it allows false positives Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2239	2022-02-24 12:48:33 +02:00
Aliaksandr Valialkin	02a922b53f	lib/storage: properly handle series selector matching multiple metric names plus a negative filter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238 This is a follow-up for `00cbb099b6`	2022-02-24 12:11:53 +02:00
Aliaksandr Valialkin	1967b9c211	lib/mergeset: remove superflouos sorting of inmemoryBlock.data at inmemoryBlock.sort() There is no need to sort the underlying data according to sorted items there. This should reduce cpu usage when registering new time series in `indexdb`. Thanks to @ahfuzhang for the suggestion at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2245	2022-02-24 11:19:44 +02:00
Aliaksandr Valialkin	2431c9cf81	lib/promrelabel: add support for conditional relabeling via `if` filter Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998	2022-02-24 02:35:13 +02:00
Aliaksandr Valialkin	6fd85117ac	lib/workingsetcache: do not rotate cache if it is in `whole` state This should reduce the maximum memory usage for the cache in `whole` state	2022-02-23 22:55:10 +02:00
Aliaksandr Valialkin	244c23ea2c	lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes This should reduce memory usage under high time series churn rate	2022-02-23 13:42:27 +02:00
Aliaksandr Valialkin	fcaa0c5202	lib/storage: optimize `/api/v1/status/tsdb` call by skipping all the artificially created tag entries at once This is a follow-up for `b71be42d90`	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	986c12487a	lib/mergeset: typo fix after `b6ed9afd6d`	2022-02-21 19:00:04 +02:00
Aliaksandr Valialkin	d4df32cc6b	lib/blockcache: evict entries from the cache in LRU order This should improve hit rate for smaller caches	2022-02-21 19:00:04 +02:00
Roman Khavronenko	5a4b16794d	Consul SD - update services on the watcher's start (#2202 ) * lib/discovery/consul: update services on the watcher's start Previously, watcher's start was only initing goroutines for discovery but not waiting for the first iteration to end. It means first Consul discovery wasn't returning discovered targets until the next iteration. The change makes the watcher's start blocking until we get first discovery iteration done and all registries updated. Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmalert: remove workarounds for consul SD Now when consul SD lib properly updates services on the first start, we don't need workarounds in vmalert. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/discovery/consul: update after review Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 15:33:33 +02:00
Roman Khavronenko	bd7837d524	lib: allow to configure cache size by type (#2206 ) * lib: allow to configure cache size by type https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1940 Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-21 13:55:51 +02:00
Aliaksandr Valialkin	5260d7a954	lib/storage: typo fix after `c3affb0c4f`	2022-02-17 12:56:33 +02:00
Aliaksandr Valialkin	d9bdb42219	lib/storage: simplify code for searching for label values This is a follow-up after `9dd191b27c`	2022-02-17 12:39:14 +02:00
Aliaksandr Valialkin	2ebc3d21c3	lib/storage: properly skip composite tag entries when searching for tag names or tag values This is a follow-up for `b71be42d90` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 23:02:18 +02:00
Aliaksandr Valialkin	3107224306	lib/blockcache: fix TestCache by ensuring that the cache size can be divided by the number of cache shards Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2204	2022-02-16 18:48:09 +02:00
Aliaksandr Valialkin	63bc89dd81	lib/storage: document why tsid cache is reset before saving it to disk Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2205	2022-02-16 18:37:29 +02:00
Aliaksandr Valialkin	ee066aa0d5	lib/storage: use binary search instead of full scan for skipping artificial tags when searching for tag names or tag values This should improve performance for /api/v1/labels and /api/v1/label/<label_name>/values See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2200	2022-02-16 18:17:27 +02:00
Roman Khavronenko	8cbcc560b9	vmagent: fix js error on CollapseAll/ExpandAll buttons click (#2192 ) * vmagent: fix js error on CollapseAll/ExpandAll buttons click `Uncaught TypeError: Cannot read properties of null (reading 'style')` Signed-off-by: hagen1778 <roman@victoriametrics.com> * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com> Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-15 12:54:39 +02:00
Corporte Gadfly	cf8df24227	match fileSDCheckInterval with prometheus file_sd_config default (#2188 )	2022-02-15 12:05:57 +02:00
Aliaksandr Valialkin	5d8ea8c918	docs/CHANGELOG.md: document `3d890e89f1`	2022-02-14 17:42:33 +02:00
Nikolay	748034e7af	Adds server certificate reload for lib/http (#2186 ) * Adds server certificate reload for lib/http https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2171 * Update lib/httpserver/httpserver.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-14 17:42:33 +02:00
Nikolay	c11d0949c8	fixes all_tenants query option usage for openstack service discovery (#2184 ) explicit use configuration parametr instead of conditional add https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2182	2022-02-14 13:13:03 +02:00
Aliaksandr Valialkin	31b42e9c57	lib/promscrape: add `expand all` and `collapse all` buttons to `/targets` page See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2021	2022-02-12 18:42:01 +02:00
Aliaksandr Valialkin	53c2135d2a	lib/storage: tune the logic for pre-populating of the per-day inverted index for the next day - Postpone the pre-poulation to the last hour of the current day. This should reduce the number of useless entries in the next per-day index, which shouldn't be created there, when the corresponding time series are stopped to be pushed during the current day. - Make the pre-population more smooth in time by using the hash of MetricID instead of MetricID itself when calculating the need for for the given MetricID pre-population. - Sync the logic for pre-population of the next day inverted index with the logic of pre-populating tsid cache after indexdb rotation. This should improve code maintainability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401	2022-02-12 16:39:33 +02:00
artifactori	943fc056ca	Show gce sdconfig zone on vmagent:8429/config (#2178 ) * vmagent: add test for marshalling gce sdconfig with ZoneYAML * vmagent: implement MarshalYAML for ZoneYAML on gce sdconfig	2022-02-12 00:48:38 +02:00
Roman Khavronenko	d107f86fbc	lib/index: reduce read/write load after indexDB rotation (#2177 ) * lib/index: reduce read/write load after indexDB rotation IndexDB in VM is responsible for storing TSID - ID's used for identifying time series. The index is stored on disk and used by both ingestion and read path. IndexDB is stored separately to data parts and is global for all stored data. It can't be deleted partially as VM deletes data parts. Instead, indexDB is rotated once in `retention` interval. The rotation procedure means that `current` indexDB becomes `previous`, and new freshly created indexDB struct becomes `current`. So in any time, VM holds indexDB for current and previous retention periods. When time series is ingested or queried, VM checks if its TSID is present in `current` indexDB. If it is missing, it checks the `previous` indexDB. If TSID was found, it gets copied to the `current` indexDB. In this way `current` indexDB stores only series which were active during the retention period. To improve indexDB lookups, VM uses a cache layer called `tsidCache`. Both write and read path consult `tsidCache` and on miss the relad lookup happens. When rotation happens, VM resets the `tsidCache`. This is needed for ingestion path to trigger `current` indexDB re-population. Since index re-population requires additional resources, every index rotation event may cause some extra load on CPU and disk. While it may be unnoticeable for most of the cases, for systems with very high number of unique series each rotation may lead to performance degradation for some period of time. This PR makes an attempt to smooth out resource usage after the rotation. The changes are following: 1. `tsidCache` is no longer reset after the rotation; 2. Instead, each entry in `tsidCache` gains a notion of indexDB to which they belong; 3. On ingestion path after the rotation we check if requested TSID was found in `tsidCache`. Then we have 3 branches: 3.1 Fast path. It was found, and belongs to the `current` indexDB. Return TSID. 3.2 Slow path. It wasn't found, so we generate it from scratch, add to `current` indexDB, add it to `tsidCache`. 3.3 Smooth path. It was found but does not belong to the `current` indexDB. In this case, we add it to the `current` indexDB with some probability. The probability is based on time passed since the last rotation with some threshold. The more time has passed since rotation the higher is chance to re-populate `current` indexDB. The default re-population interval in this PR is set to `1h`, during which entries from `previous` index supposed to slowly re-populate `current` index. The new metric `vm_timeseries_repopulated_total` was added to identify how many TSIDs were moved from `previous` indexDB to the `current` indexDB. This metric supposed to grow only during the first `1h` after the last rotation. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-12 00:34:44 +02:00
Aliaksandr Valialkin	21a42e990f	lib/storage: fix broken BenchmarkHeadPostingForMatchers for `{i=~".*"}` after `f4dead529f` The commit `f4dead529f` makes such query to return nothing instead of all the time series. This aligns more with Prometheus behaviour.	2022-02-12 00:28:21 +02:00
Roman Khavronenko	791cad8c2e	lib/promscrape: support prometheus-like duration in scrape configs (#2169 ) * lib/promscrape: support prometheus-like duration in scrape configs The change allows to specify duration values like `1d`, `1w` for fields `scrape_interval`, `scrape_timeout`, etc. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817#issuecomment-1033384766 Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/blockcache: make linter happy Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/promscrape: support prometheus-like duration in scrape configs * add support for extra fields `scrape_align_interval` and `scrape_offset`; * support Prometheus duration parsing for `__scrape_interval__` and `__scrape_duration__` labels; Signed-off-by: hagen1778 <roman@victoriametrics.com> * wip * wip * docs/CHANGELOG.md: document the feature Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 16:17:51 +02:00
Aliaksandr Valialkin	727b29d4a3	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_endpointslice_{label,annotation}*` labels to be consistent with other `role` values for Kubernetes service discovery	2022-02-11 14:56:10 +02:00
Nikolay	265938a385	fixes service discovery for kubernetes (#2173 ) * fixes service discovery for kubernetes now it must take in account all pods that belong to the discovered endpoint and endpointslice adds simple test for endpoints https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134 * wip * docs/CHANGELOG.md: document the change Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin	895c9f4f11	lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin	102c9a4bf9	lib/blockcache: use higher number of shards for higher number of CPU cores This should reduce mutex contention and increase performance Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-10 19:11:11 +02:00
Aliaksandr Valialkin	ad19c9d302	lib/promscrape: fix errors in test config The errors were discovered after enabling strict parse mode by default. See `9bb60ab00f`	2022-02-08 20:10:28 +02:00
Aliaksandr Valialkin	fae3040868	lib/blockcache: split the cache into multiple shards This should reduce contention on cache mutex on hosts with many CPU cores, which, in turn, should increase overall throughput for the cache. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-08 19:48:32 +02:00
Aliaksandr Valialkin	a0a56d6c1c	lib/mergeset: tune sizes for `indexdb/dataBlocks` and `indexdb/indexBlocks` according to production workload This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742	2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin	eed66b6640	lib/promscrape: set `-promscrape.config.strictParse` to true by default This allows detecting long-living silent errors in -promscrape.config	2022-02-08 15:42:33 +02:00
Aliaksandr Valialkin	8863339b6b	lib/blockcache: `make fmt`	2022-02-08 15:42:31 +02:00
Aliaksandr Valialkin	1caee74235	lib/blockcache: eliminate possible race when Cache.Put is called for the same entry from multiple goroutines The race could result in incorrect cache size tracking, which, in turn, could result in too frequent cache cleaning	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	10476738a8	lib/blockcache: increase the lifetime for rarely accessed blocks from 2 minutes to 5 minutes This should improve data ingestion speed if time series samples are ingested with interval bigger than 2 minutes. The actual interval could exceed 2 minutes if the original interval between samples doesn't exceed 2 minutes in the case of slow inserts. Slow inserts may appear in the following cases: * Big number of new time series are pushed to VictoriaMetrics, so they couldn't be registered in 2 minutes. * MetricName->tsid cache reset on indexdb rotation or due to unclean shutdown. In this case VictoriaMetrics needs to load MetricName->tsid entries for all the incoming series from IndexDB. IndexDB uses the block cache for increasing lookup performance. If the cache has no the needed block, then IndexDB reads and unpacks the block from disk. This requires an extra disk read IO and CPU. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007 This also should increase performance for periodically executed queries with intervals from 2 minutes to 5 minutes. See the previous similar commit - `43103be011` It is possible that the timeout can be increased further. Let's collect production numbers for this change so the timeout could be adjusted further.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	b7cefff7b0	lib/workingsetcache: use the original cache size limits when rotating caches Previously limits for new caches were taken from cache stats. These limits could mismatch the original limits. This could result in failed cache load if the stored cache has been created with the limits obtained from cache stats.	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	87071640a7	lib/blockcache: return proper number of entries from the cache This has been broken in `0d7374ad2f`	2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin	34d14c4940	all: substitute zeroTime with time.Time{}, since this generates more optimal binary code	2022-02-07 14:36:41 +02:00
Aliaksandr Valialkin	e2d12a25e0	lib/netutil: increase dial timeout from 1 second to 5 seconds There are real-world cases when TCP connection needs more than 1 second to be established.	2022-02-07 12:33:40 +02:00
Aliaksandr Valialkin	d24e5d9efd	lib/promscrape: show the total number of scrapes and the total number of scrape errors per target at /targets page This information may be useful when debugging unreliable scrape targets	2022-02-03 20:23:27 +02:00
Aliaksandr Valialkin	678b3e71db	lib/promscrape: provide the ability to fetch target responses on behalf of vmagent or single-node VictoriaMetrics This feature may be useful when debugging metrics for the given target located in isolated environment	2022-02-03 19:02:12 +02:00
Aliaksandr Valialkin	5f266370c5	all: follow-up after `4bdd10ab90` Properly use new bytesutil.Resize* functions	2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin	d8d59ff760	lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	02b2bfcff3	lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations Follow-up for `f4989edd96` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	084664d780	lib/encoding: substitute `64-bits.LeadingZeros64()` with `bits.Len64()`	2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin	0fbfa8c245	lib/storage: avoid allocations of tsidPrev on every blockStreamReader.NextBlock() call This is a follow-up for `00b7c97d2a` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-31 22:47:16 +02:00
Aliaksandr Valialkin	a02dde6cc7	lib/cgroup: fall back to runtime.NumCPU() when determining process_cpu_cores_available metric if it is impossible to determine cpu quota via cgroups Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:31:12 +02:00
Aliaksandr Valialkin	566e12874d	lib/cgroup: expose `process_cpu_cores_available` metric This metric shows the number of CPU cores available to the process. This allows creating alerting rules on CPU saturation with the following query: rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107	2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin	776b7bc9f8	lib/storage/table.go: add missing `tb.ptwsLock.Unlock()` before the return This is a follow-up for `a1083d0531` See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2103	2022-01-28 12:12:58 +02:00
匠心零度	a1083d0531	optimized code (#2103 ) * optimized code ,because only the first error,so no need var errors []error * optimized code ,because only the first error,so no need var errors []error Co-authored-by: lirenzuo <lirenzuo@shein.com>	2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin	6232eaa938	lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice. This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed after slize resizing. Switch such places to bytesutil.ResizeNoCopy(). Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability. Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls. This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin	7ec0705b98	lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock This should reduce the memory required for indexdb/dataBlocks cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin	65176726b3	lib/logger: removed broken test after `746ee191e8`	2022-01-24 12:15:11 +02:00
Aliaksandr Valialkin	49650fe6aa	lib/logger/throttler.go: show the original location of the error and warning message Previously the location inside LogThrottler implementation was shown. This could complicate debugging.	2022-01-23 13:55:48 +02:00
Aliaksandr Valialkin	233101137d	lib/blockcache: optimize blockcache a bit - Optimize Cache.RemoveBlocksFromPart(), so it doesn't need to iterate over all the cached blocks. - Cache blocks if there were no cache misses during the last 2 minutes. This may be the case when new blocks are added simultaneously to the storage and to the cache. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-23 13:08:55 +02:00
Aliaksandr Valialkin	9edf407144	lib/mergeset: tune caches size limits for `indexdb/dataBlocks` and `indexdb/indexBlocks` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin	4e05298756	lib/storage: properly limit cardinality when ingesting multiple samples for the same time series in a single request	2022-01-21 12:38:22 +02:00
Aliaksandr Valialkin	e3277918e4	lib/storage: verify that blocks in a single part are sorted by TSID when reading sequential blocks from the part This may help narrowing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:28 +02:00
Aliaksandr Valialkin	54ee71e16d	lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used. This may help nailing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082	2022-01-20 20:37:24 +02:00
Aliaksandr Valialkin	5159a9451f	lib/blockcache: add missing dependency after `145337792d` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:51:03 +02:00
Aliaksandr Valialkin	6ae584b9b3	lib/{mergeset,storage}: properly limit cache sizes for indexdb Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`, since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded, which could result to out of memory errors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007	2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin	da95516a1f	lib/promscrape: expose promscrape_stale_samples_created_total metric for monitoring the number of created stale samples	2022-01-14 01:00:40 +02:00
Aliaksandr Valialkin	dd91759f1f	lib/promscrape/discovery/kubernetes: add `__meta_kubernetes_node_provider_id` label for discovered Kubernetes nodes in the same way as Prometheus does See https://github.com/prometheus/prometheus/pull/9603	2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin	bc18368c15	lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881	2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin	de8299f465	lib/promscrape/discovery/dockerswarm: follow up after `68a117a25a` - Document the bugfix at docs/CHANGELOG.md - Set __address__ field after copying commonLabels to the resulting map of discovered labels. This makes sure that the correct __address__ label is used.	2022-01-11 09:22:03 +02:00
Alexander Shtuchkin	45a92e6ce1	Fix for #2038 : Make correct __address__ value for dockerswarm promscrape (#2041 )	2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin	fa89f3e5a5	lib/promscrape: do not send staleness markers on graceful shutdown This follows Prometheus behavior. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079	2022-01-07 01:19:06 +02:00
Aliaksandr Valialkin	80fc3fda07	lib/storage: follow-up for `38bf5fc136`	2022-01-05 16:02:17 +02:00
weng zhao	1e0fe615ad	vmstorage: fix query like `{foo=~"bar\|"}` return extra timeseries cause by negative filter transformation malfunction (#2032 ) 1. L2749 make kb.B remain the value of comonPrefix instead of tf.prefix 2. L2762 avoid change tf.value from "bar\|" to ".+r\|"	2022-01-05 15:57:54 +02:00
Aliaksandr Valialkin	c1722003a2	lib/promscrape: scrape replicated targets at different offsets in vmagent replicated clustering mode This guarantees that the deduplication consistently leaves samples from the same vmagent replica. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets	2021-12-23 00:21:41 +02:00
Nikolay	6cdc934c3d	adds restore.lock (#1988 ) * adds restore.lock it must prevent from running storage after incomplete restore process https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 * return back flock file deletion * Apply suggestions from code review * wip * docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958 Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-22 13:10:56 +02:00
Aliaksandr Valialkin	727797a6fd	all: use logger.WithThrottler() where appropriate	2021-12-21 17:10:54 +02:00
Aliaksandr Valialkin	4dbf12254d	lib/promscrape: take into account the original job_name when creating an unique key per each scrape target This should handle the case when the original job_name has been changed in -promscrape.config , while the resulting job label remains the same because it is overriden via relabeling.	2021-12-21 16:42:42 +02:00
Roman Khavronenko	23e1de06ee	vmagent: add error log for skipped data block when rejected by receiv… (#1956 ) * vmagent: add error log for skipped data block when rejected by receiving side Previously, rejected data blocks were silently dropped - only metrics were update. From operational perspective, having an additional logging for such cases is preferable. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: throttle log messages about skipped blocks The new type of logger was added to logger pacakge. This new type supposed to control number of logged messages by time. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/logger: make LogThrottler public, so its methods can be inspected by external packages Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-21 16:42:38 +02:00
Aliaksandr Valialkin	053e85ff3d	all: typo fix: unexected -> unexpected	2021-12-20 17:40:13 +02:00
Aliaksandr Valialkin	406cb06f8c	lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981	2021-12-20 17:26:07 +02:00
Aliaksandr Valialkin	f22aab411b	lib/storage: properly update per-part `min_dedup_interval` file contents after merge Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value This is a follow-up for `4ff647137a`	2021-12-17 20:12:18 +02:00
Aliaksandr Valialkin	5bd4e47a9e	lib/promscrape: allow up to 5 redirects when scraping a target by default See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945	2021-12-16 00:14:45 +02:00
Aliaksandr Valialkin	d36fdbe537	lib/storage: deduplicate samples more thoroughly Previously some duplicate samples may be left on disk for time series with high churn rate. This may result in higher disk space usage.	2021-12-15 16:00:30 +02:00
Aliaksandr Valialkin	bc3923111b	lib/storage: return dedup interval in milliseconds from GetDedupInterval() This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.	2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin	cdfe854c9b	lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() This improves the code readability and debuggability, since the output of these functions stops depending on global state.	2021-12-14 20:52:29 +02:00
Aliaksandr Valialkin	c922c7af9a	lib/storage: convert alternate regexps into Graphite wildcards inside `__graphite__` pseudo-label For example, `{__graphite__=~"foo.(bar\|baz)"}` is automatically converted to `{__graphite__=~"foo.{bar,baz}"}` before execution. This allows using multi-value Grafana template variables such as `{__graphite__=~"foo.($app)"}`.	2021-12-14 19:55:59 +02:00
Aliaksandr Valialkin	38f5bc7451	lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' This allows copy-pasting the url to http server from logs.	2021-12-08 16:15:12 +02:00
Aliaksandr Valialkin	9aa9b081a4	app/vminsert: add `-maxLabelValueLen` command-line flag See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908	2021-12-06 11:42:24 +02:00
Aliaksandr Valialkin	51f2eb3c46	lib/workingsetcache: fix `unaligned 64-bit atomic operation` panic on 32-bit architectures The panic has been introduced in `7275ebf91a`	2021-12-03 01:22:30 +02:00
Aliaksandr Valialkin	d40441947a	app: allow specifying http and https urls in the following command-line flags * -promscrape.config * -relabelConfig * -remoteWrite.relabelConfig * -remoteWrite.urlRelabelConfig	2021-12-03 00:11:47 +02:00
Aliaksandr Valialkin	daaea1eb2c	app/vmauth: follow-up for `13368bed18` * Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md * Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file should be read from the given path. For example, in `-promscrape.config` at `vmagent`.	2021-12-02 23:34:15 +02:00
Aliaksandr Valialkin	b885a3b6e9	lib/httpserver: expose `/-/healthy` and `/-/ready` endpoints as Prometheus does This improves integration with third-party solutions, which rely on these endpoints. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833	2021-12-02 14:37:50 +02:00
Aliaksandr Valialkin	c540235470	app: use relative paths instead of absolute paths for the supported http handlers on the main page This allows hiding VictoriaMetrics components behind proxies, which serve pages at different path prefixes See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858	2021-12-02 13:54:15 +02:00
Aliaksandr Valialkin	d1289383eb	lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp	2021-12-02 13:44:01 +02:00
Aliaksandr Valialkin	37a2bea072	lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865	2021-12-02 13:33:50 +02:00
Aliaksandr Valialkin	2b7dee15dd	app/{vmbackup,vmrestore}: export internal metrics at `/metrics` http handler	2021-12-02 11:56:34 +02:00
Aliaksandr Valialkin	ab4be24397	app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query: vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9	2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin	d4655beae8	lib/fs: add `vm_filestream_read_duration_seconds_total` and `vm_filestream_write_duration_seconds_total` metrics These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`	2021-12-02 09:13:20 +02:00
Aliaksandr Valialkin	2e43cd9d62	lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges	2021-12-01 12:30:03 +02:00
Nikolay	cf1d2f289b	removes FileSize from backup part key (#1872 ) * removes FileSize from backup part key it should fix download restoration for backups * Update lib/backup/common/part.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-12-01 12:30:03 +02:00
Aliaksandr Valialkin	71c0f7cce3	lib/storage: take into account `-storage.minFreeDiskSpaceBytes` when performing big merges Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-11-30 12:56:53 +02:00
guidao	6fa7ad69fc	fix #1830 (#1861 ) Co-authored-by: wangfeng <wangfeng@zhihu.com>	2021-11-30 01:16:15 +02:00
Aliaksandr Valialkin	975498d402	lib/protoparser/prometheus: follow-up for `8e338632a3` Do not spend CPU time on error message formatting if error logger is disabled	2021-11-30 00:51:15 +02:00
Nikolay	40f0726147	Changes unmarshallRow logger to noop for getRowsDiff (#1835 )	2021-11-30 00:51:14 +02:00
Aliaksandr Valialkin	4ad397188e	lib/protoparser: do not log `connection reset by peer` error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections This error is expected, so there is no need in spamming the log with this error.	2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin	e93f46187d	lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent	2021-11-17 16:42:12 +02:00
Lan	6662714c6c	Add flag of S3ForcePathStyle (#1802 )	2021-11-17 01:10:22 +02:00
Aliaksandr Valialkin	4fb19fe34b	all: consistently return `application/json` content-type without `charset=utf-8` The `application/json` content-type has utf-8 encoding by default. See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin	f41c02e475	lib/promscrape: improve logging for `scrape_config_files` parse errors Log the actual file path, which led to the parse error. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789	2021-11-08 13:34:26 +02:00
Aliaksandr Valialkin	847004fa77	app/{vminsert,vmagent}: hide passwords and auth tokens by default at `/config` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764	2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin	bc72b83102	lib/promauth: do not show empty values in `oauth2` config section at `/config` page	2021-11-05 12:54:10 +02:00
Aliaksandr Valialkin	d445d22c0c	lib/promscrape: add `-promscrape.maxResponseHeadersSize` command-line flag for tuning the maximum http response headers size from Prometheus scrape targets	2021-11-03 22:27:55 +02:00
Aliaksandr Valialkin	6873d6d893	lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp	2021-10-28 12:48:34 +03:00
Aliaksandr Valialkin	105deb164c	lib/logger: show only explicitly set command-line flags in logs This reduces initial verbosity in logs	2021-10-28 11:03:21 +03:00
Aliaksandr Valialkin	b626d6d606	lib/promscrape: add `collapse` and `expand` buttons per each group of targets from the same scrape job	2021-10-27 20:04:03 +03:00
Aliaksandr Valialkin	2ebee4e741	app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules Previously only the lower part of 64-bit hash was used for calculating the offset. This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash for calculating the offset.	2021-10-27 20:04:02 +03:00
Aliaksandr Valialkin	92d01db85a	lib/protoparser/prometheus: optimize GetRowsDiff() function This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 , since the provided profile shows that the majority of CPU and memory is spent in this function during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.	2021-10-27 18:55:25 +03:00
Aliaksandr Valialkin	16f1aaf0b5	lib/protoparser/prometheus: add a benchmark for GetRowsDiff	2021-10-27 18:55:23 +03:00
Aliaksandr Valialkin	99784b21c1	all: fix build issues and tests for Apple M1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653	2021-10-27 15:07:19 +03:00
Aliaksandr Valialkin	ad445a06cd	lib/promscrape: properly show `proxy_url` option value at `/config` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755	2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin	b08f51f5d3	lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set The response body isn't used if -promscrape.noStaleMarkers is set after the commit `2876137c92` , so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694	2021-10-22 16:49:21 +03:00
Aliaksandr Valialkin	6bc10f0623	lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels. See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets This is a follow-up after `9882cda8b9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728	2021-10-22 13:55:39 +03:00
Aliaksandr Valialkin	a8bcc3c276	lib/promscrape: reduce memory usage if `-promscrape.noStaleMarkers` command-line flag is passed Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled. This should reduce memory usage when the scraped targets return large responses.	2021-10-22 13:22:08 +03:00
Nikolay	83e1dfccba	adds tab as second separator for graphite text protocol (#1733 ) * adds tab as second separator for graphite text protocol * changes indexFunc for indexAny * Update lib/protoparser/graphite/parser_test.go Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-10-22 12:29:27 +03:00
Aliaksandr Valialkin	d56e676d71	lib/flagutil: do not expose sensitive info (passwords, keys and urls) at /flags page	2021-10-20 00:51:15 +03:00
Aliaksandr Valialkin	5705f4b6d1	lib/httpserver: expose command-line flags at `/flags` page This should simplify debugging. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-20 00:46:54 +03:00
Aliaksandr Valialkin	a105b71116	lib/envflag: use flag.Set for setting the flags from env vars This should make visible the set flags at flag.Visit(), which is used later for logging and exporting the `is_set` label for these flags at /metrics page	2021-10-20 00:46:53 +03:00
Aliaksandr Valialkin	93511b4be7	lib/storage: log a warning when the -storageDataPath has less than -storage.minFreeDiskSpaceBytes This should improve the debuggability of the readonly feature. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727	2021-10-19 23:58:09 +03:00
Aliaksandr Valialkin	ea69eef375	lib/promscrape/discovery/kubernetes: log a warning if `role: endpoints` discovers more than 1000 targets per a single endpoint In this case `role: endpointslice` must be used instead. See the following references: * https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity * https://github.com/kubernetes/kubernetes/pull/99975 * https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398	2021-10-19 13:22:28 +03:00
Nikolay	e84a063209	changes job source for /target api (#1723 ) use jobNameOriginal instead of relabeled as prometheus does https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707	2021-10-19 09:00:05 +03:00
Aliaksandr Valialkin	fbcc8b5c7d	lib/promscrape: set `honor_timestamps: true` by default if this option isnt set explicitly in scrape configs This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-10-16 20:48:53 +03:00
Aliaksandr Valialkin	0a9be5ef9d	lib/promscrape: expose `promscrape_series_limit_max_series` and `promscrape_series_limit_current_series` metrics per each scrape target with the enabled unique series limiter	2021-10-16 19:14:13 +03:00
Aliaksandr Valialkin	99011c6b63	lib/promscrape: always initialize http client for stream parsing mode Stream parsing mode can be automatically enabled when scraping targets with big response bodies exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.	2021-10-16 13:19:48 +03:00
Aliaksandr Valialkin	0f4fda1bda	lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse This should reduce memory usage when scraping targets with big response bodies.	2021-10-16 13:00:11 +03:00
Aliaksandr Valialkin	0452a8d4e8	lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode at the cost of the increased memory usage, since now the potentially big response is kept in the lastScrape byte slice per each scrapeWork. In practice the memory usage increase shouldn't be big, since the response size is usually much smaller than the parsed metrics from this response after the relabeling, which usually adds a big pile of target-specific labels per each metric.	2021-10-15 15:26:24 +03:00
Aliaksandr Valialkin	3e9beb0f8d	lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name This is a follow-up for `31b42b30b6`	2021-10-15 12:27:31 +03:00
Aliaksandr Valialkin	25421fa2ae	lib/promscrape: add `-promscrape.minResponseSizeForStreamParse` command-line option for automatic switching to stream parsing mode when scraping targets with big responses This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes. This is common case in Kubernetes monitoring.	2021-10-14 12:30:55 +03:00
Aliaksandr Valialkin	bee130cc78	lib/promscrape: return error if `sample_limit` or `series_limit` options are set when stream parsing mode is enabled	2021-10-14 12:30:54 +03:00
Aliaksandr Valialkin	5b7d90d178	lib/promscrape: add ability to show the original labels for discovered targets at /targets page See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698	2021-10-13 16:44:34 +03:00
Roman Khavronenko	5dab25e8ad	lib/promscrape: make errcheck happy (#1703 )	2021-10-13 15:11:45 +03:00
Aliaksandr Valialkin	c3a729d458	lib/promscrape: shard targets among cluster nodes after relabeling is applied This guarantees that targets with the same set of labels go to the same vmagent node. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495	2021-10-12 17:06:37 +03:00
Aliaksandr Valialkin	aeedfe2fe2	app/vmagent: expose -promscrape.config contents at /config page as Prometheus does See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695	2021-10-12 16:27:37 +03:00
Aliaksandr Valialkin	84aa08d93a	lib/promscrape: use Prometheus format for target labels at `/targets` page This should simplify copy-pasting the labels to/from PromQL / MetricsQL	2021-10-11 12:42:18 +03:00
Aliaksandr Valialkin	a7a1305395	lib/storage: fix unaligned access on 32-bit architectures. The bug has been introduced at `a171916ef5`	2021-10-08 19:38:20 +03:00
Aliaksandr Valialkin	a47754b689	lib/protoparser/clusternative: typo fix after `4fddcf4c83`	2021-10-08 15:38:47 +03:00
Aliaksandr Valialkin	4fddcf4c83	app/{vminsert,vmstorage}: follow-up after `a171916ef5` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269	2021-10-08 14:09:51 +03:00
Nikolay	a171916ef5	Adds read-only mode for vmstorage node (#1680 ) * adds read-only mode for vmstorage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269 * changes order a bit * moves isFreeDiskLimitReached var to storage struct renames functions to be consistent change protoparser api - with optional storage limit check for given openned storage * renames freeSpaceLimit to ReadOnly	2021-10-08 12:52:56 +03:00
Ziqi Zhao	1db3aeab36	fix some typos (#1678 ) Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>	2021-10-06 14:43:56 +03:00
Aliaksandr Valialkin	522a404b79	lib/promscrape: reduce memory allocations in mergeLabels() after `48e3e6c8df`	2021-09-30 16:56:43 +03:00
Aliaksandr Valialkin	7b69d478ec	lib/protoparser: go fmt	2021-09-29 21:17:49 +03:00
Aliaksandr Valialkin	6167890d0e	lib/protoparser/prometheus: compare invalid Prometheus lines in full	2021-09-29 19:41:23 +03:00
Aliaksandr Valialkin	8dcf814c48	app/{vmbackup,vmrestore}: switch from `gcs://...` to `gs://...` urls for backups to GCS The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls, while leaving support for `gcs://` urls for backwards compatibility.	2021-09-29 12:12:37 +03:00
Nikolay	9be5689b3f	changes auth validation for openstack (#1663 ) * changes auth validation for openstack must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655 * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2021-09-29 00:33:38 +03:00
Aliaksandr Valialkin	4e65bfcc00	app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206	2021-09-29 00:12:26 +03:00
Aliaksandr Valialkin	d15d036a5a	lib/storage: properly handle `{__name__=~"prefix(suffix1\|suffix2)",other_label="..."}` queries They were broken in the commit `00cbb099b6` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644	2021-09-23 21:52:31 +03:00
Aliaksandr Valialkin	d8de26bbfd	lib/promscrape: add `vm_promscrape_max_scrape_size_exceeded_errors_total` metric for counting of the failed scrapes due to the exceeded response size Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639	2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin	86bafe796c	lib/httpserver: add `-enterprise` and/or `-cluster` suffixes to `short_version` label of `vm_app_version` metric See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635	2021-09-21 23:12:50 +03:00
Aliaksandr Valialkin	c3e1f87048	lib/promrelabel: fix parsing `regex: true` in relabeling rules	2021-09-21 23:01:40 +03:00
Nikolay	dd53abf36d	changes protoparser apis for accepting reading from io.Reader (#1624 ) adds InsertHandlerForReader apis to vmagent	2021-09-20 14:54:20 +03:00
Nikolay	1ab2f844a2	makes filters optional for ec2 api requests (#1627 ) filters can be applied only for DescribeInstances requests, like prometheus does. related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626	2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin	1493461244	lib/storage: follow up after `00cbb099b6`	2021-09-14 14:23:02 +03:00
faceair	61a51f7c15	lib/storage: optimize convert multiple values regexp filter to composite tag filter (#1610 ) * lib/storage: optimize convert multiple values regexp filter to composite tag filter * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-09-14 14:23:01 +03:00
Aliaksandr Valialkin	184e145570	docs: consistency renaming: Influx -> InfluxDB	2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin	b684624f67	lib/promscrape/discovery/docker: support host networking mode See https://github.com/prometheus/prometheus/issues/9116	2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin	6ed9f10da5	lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery See https://github.com/prometheus/prometheus/issues/8902	2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin	d90834da70	lib/promscrape: generate `scrape_timeout_seconds` metric per each scrape target in the same way as Prometheus 2.30 does See https://github.com/prometheus/prometheus/pull/9247	2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin	279f37c9e7	lib/promscrape: `make fmt`	2021-09-12 13:35:21 +03:00
Aliaksandr Valialkin	6c97388dde	lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling See https://github.com/prometheus/prometheus/pull/8911	2021-09-12 13:35:20 +03:00
Aliaksandr Valialkin	09670479cd	lib/promscrape: reduce CPU usage for common case when calculating `scrape_series_added` metric Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics. The main idea is to perform the calculations on scrape_series_added and series_limit only if the set of metrics exposed by the target has been changed. Scrape targets rarely change the set of exposed metrics, so this optimization should reduce CPU usage in general case.	2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin	c339642858	lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric	2021-09-11 11:03:38 +03:00
Aliaksandr Valialkin	6d6cf1b6e0	lib/storage: verify that the tsidsFound contain the needed tsids in tests added at `f4dead529f`	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	5aaaa686a4	lib/promscrape: send stale markers for disappeared metrics like Prometheus does	2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin	c2f37f049b	lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz\|",x=~"y\|"} Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2021-09-09 21:12:53 +03:00
Aliaksandr Valialkin	d613a018c8	lib/promscrape: add ability to set `series_limit` and `stream_parse` options via relabeling This allows managing these options on a per-target basis. Typical use case: to manage these options for pods via Kubernetes annotations.	2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin	b64866e64c	lib/promscrape: add `scrape_` prefix to `job` and `target` labels exported by `promscrape_series_limit_rows_dropped_total` metric This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.	2021-09-09 17:31:04 +03:00
Aliaksandr Valialkin	75c3514c5c	lib/promrelabel: add `keep_metrics` and `drop_metrics` actions to relabeling rules These actions simlify metrics filtering. For example, - action: keep_metrics regex: 'foo\|bar\|baz' would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted. The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above: - action: keep_metrics regex: - foo - bar - baz	2021-09-09 16:25:09 +03:00
mxlxm	42e07cfaea	reset deadline, fix #1562 . (#1597 ) * reset deadline, fix #1562. reset deadline before we put it back to pool. * make errcheck happy	2021-09-07 20:54:17 +03:00
Aliaksandr Valialkin	c4df601f43	lib/promscrape: add the ability to limit the number of unique series per each scrape target The number of series per target can be limited with the following options: * Global limit with `-promscrape.maxSeriesPerTarget` command-line option. * Per-target limit with `max_series: N` option in `scrape_config` section. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561	2021-09-01 16:08:12 +03:00
Aliaksandr Valialkin	146c14d879	lib/promscrape/discovery/kubernetes: return back support `role: endpointslices`, since it is used by VictoriaMetrics operator This is a follow up commit after `31b42b30b6`	2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin	18d7adf731	lib/protoparser/opentsdb: follow-up after `8ee75ca45a`	2021-08-29 11:50:01 +03:00
envzhu	00dddfe02f	lib/protoparser/opentsdb: accept multiple spaces between fields in a row as a deliminator. (#1575 )	2021-08-29 11:50:00 +03:00
Aliaksandr Valialkin	ca61d7c82b	lib/promscrape/discovery/kubernetes: rename `role: endpointslices` to `role: endpointslice` to be consistent with Prometheus See `2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)`	2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin	327034b54f	lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for `role: ingress` and `role: endpointslices` This should fix service discovery for these roles in Kubernetes v1.22 and newer versions. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122 The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205	2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin	7fdb4db73d	lib/promscrape: add ability to load scrape configs from multiple files See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559	2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin	4a2d7aec7f	lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config	2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin	b885bd9b7d	lib/{mergeset,storage}: improve the detection of the needed free space for background merge This should prevent from possible out of disk space crashes during big merges. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560	2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin	67bc407747	lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets Store the scraped response body instead of storing the parsed and relabeld metrics. This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics. This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics. This should also reduce CPU usage, since the marshaling of the parsed and relabeld metrics has been substituted by response body copying. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526	2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin	c3b24882a7	lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape This will make timestamps for stale markers more consistent for timestamps for other samples	2021-08-19 14:19:54 +03:00
Aliaksandr Valialkin	8ee575dee9	lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does	2021-08-18 22:00:46 +03:00
Aliaksandr Valialkin	5d92fafc40	app/vmselect: add `-search.noStaleMarkers` command-line flag for disabling stale markers handling in queries This option allows reducing CPU usage a bit when VictoriaMetrics is used for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.	2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin	f21fad53b4	lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag This option can be useful when vmagent consumes too much additional memory for staleness markers functionality and when staleness markers aren't needed.	2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin	db34c40aec	lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments). Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509	2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin	5f13c519ee	lib/promscrape: restore red highlighting for DOWN targets at /targets page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461	2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin	c1f81f08d4	all: add support for Prometheus staleness markers Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin	90efb5831b	lib/envflag: add a link to docs for -envflag.enable	2021-08-11 10:32:40 +03:00
Aliaksandr Valialkin	b877538622	app/vmagent: follow-up after `fe445f753b` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491	2021-08-05 09:51:00 +03:00
Omar Ghader	fe445f753b	feature: Add multitenant for vmagent (#1505 ) * feature: Add multitenant for vmagent * Minor fix * Fix rcs index out of range * Minor fix * Fix multi Init * Fix multi Init * Fix multi Init * Add default multi * Adjust naming * Add TenantInserted metrics * Add TenantInserted metrics * fix: remove unused metrics for vmagent * fix: remove unused metrics for vmagent Co-authored-by: mghader <marc.ghader@ubisoft.com> Co-authored-by: Sebastian YEPES <syepes@gmail.com>	2021-08-05 09:44:29 +03:00
Aliaksandr Valialkin	77bb9e1656	lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 See https://github.com/prometheus/prometheus/pull/8978	2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin	336a2aa2e0	lib/promscrape/discovery/ec2: add `__meta_ec2_availability_zone_id` label as Prometheus 2.29 does	2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin	c473d8ffe1	li/storage: re-use the per-day inverted index search code for searching in global index This allows removing a big pile of outdated code for global index search. This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486	2021-07-30 10:28:20 +03:00
Nikolay	6d47e750be	adds check for region with custom s3 endpoint (#1465 )	2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin	1950f57316	lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-27 12:03:31 +03:00
Aliaksandr Valialkin	92628f9f07	lib/promrelabel: add tests for verifying that regex works as expected in single quotes and double quotes	2021-07-27 10:53:03 +03:00
Aliaksandr Valialkin	5d255846ac	all: add `go:build` lines for Go1.17 See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin	c857e05604	lib/promscrape: add missing whitespace at /targets page before `up` word	2021-07-26 12:23:06 +03:00
Aliaksandr Valialkin	376af3c956	lib/workingsetcache: switch from split cache to full cache after the cache size exceeds 95% of split capacity Previously the switch occurred when the cache size becomes 100% of its capacity. The cache size could never reach 100% capacity. This could prevent from switching from the split cache to full cache, thus reducing the cache effectiveness.	2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin	9d3f9da5ad	lib/storage: make sure the second call to DeduplicateSamples and deduplicateSamplesDuringMerge doesnt change samples	2021-07-15 12:18:38 +03:00
Aliaksandr Valialkin	e992754e79	lib/storage: remove cache directory if it contains reset_cache_on_startup file See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447	2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin	e6edb85fa2	lib/httpserver: add `is_set` label to `flag` metrics This label allows determining the set flags with the query `flag{is_set="true"}`	2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin	51cd19d2e3	lib/storage: reset perKeyMisses stats less frequently This should reduce CPU usage for queries executed with intervals higher than 30 seconds	2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin	3f705fe8d7	lib/storage: properly limit the size of `storage/date_metricID` cache	2021-07-12 14:25:28 +03:00
Aliaksandr Valialkin	ef3c58d7a3	lib/storage: properly determine when the deduplication is needed in needsDedup Previously needsDedup() could return true if the de-duplication wasn't needed for the following case: d < interval / \ \| v \| v \| interval interval Now it properly returns false for this case	2021-07-12 10:54:51 +03:00
Aliaksandr Valialkin	41754e12f8	lib/mergeset: cache indexBlock items only on the second request This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.	2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin	ceda2b1df4	lib/httpserver: print full requestURI in httpserver.Errorf This should simplify debugging.	2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin	9826f7c1be	lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) This should reduce the cache size and memory usage for the indexdb/dataBlocks cache	2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin	74ace9340d	lib/storage: periodically reset prefetchedMetricIDs cache in order to limit its size under high churn rate	2021-07-07 10:59:39 +03:00
Aliaksandr Valialkin	a846febc89	Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" This reverts commit `7c6d3981bf`. Reason for revert: high contention at bucket16Pool on systems with big number of CPU cores. This slows down query processing significantly.	2021-07-06 18:26:56 +03:00
Aliaksandr Valialkin	b805a675f3	lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects This should reduce memory usage on systems with big number of CPU cores, since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains a separate pool inmemoryPart objects per each CPU core. Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized by big sizes of inmemoryPart objects.	2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin	d8e7c1ef27	lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method This reduces the load on memory allocator in Go runtime in production workload.	2021-07-06 16:33:24 +03:00
Aliaksandr Valialkin	db6bd69475	lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small. This results in the increased load on memory allocation code in Go runtime. Increase the pool capacity in order to reduce the load on Go runtime.	2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin	fd32855a6c	lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered (aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.	2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin	22c6e64bbc	lib/storage: consistency renaming: tagCache -> tagFiltersCache This improves code readability	2021-07-06 11:03:30 +03:00
Aliaksandr Valialkin	21abf487c3	lib/workingsetcache: properly update stats for requests and cache misses Previously the stats for cache misses could be improperly counted, because it had inflated cache misses if the entry was missing in the curr cache, but was existing in the prev cache. The same applies to cache requests - they were inflated if the entry was missing in the curr cache.	2021-07-06 10:54:38 +03:00
Aliaksandr Valialkin	e5031d9aee	lib/workingsetcache: fix cache capacity calculations after `4f0003f182`	2021-07-05 17:16:35 +03:00
Aliaksandr Valialkin	bd71f102e8	lib/workingsetcache: typo fixes after `d0c830039d`	2021-07-05 15:35:51 +03:00
Aliaksandr Valialkin	4b25e627f8	lib/workingsetcache: properly switch to `whole` mode Previously the switch from `split` to `whole` mode had been performed too early, e.g. when the current cache size became bigger than 1/4 of the allowed cache size. Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size. This change can reduce memory usage for data ingestion path when big number of active time series are ingested.	2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin	51516b96e6	lib/storage: tune cache sizes according to production workload	2021-07-05 15:14:45 +03:00
Aliaksandr Valialkin	f12f97daa1	lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes One minute cache timeout result in slower queries in some production workloads where the interval between query execution is in the range 1 minute - 2 minutes.	2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin	377bb06b47	lib/cgroup: set GOGC to 50 by default if it isn't set This should reduce memory usage for typical VictoriaMetrics workloads by up to 50%	2021-07-05 12:34:01 +03:00
Aliaksandr Valialkin	8055439fe4	lib/storage: properly detect free disk space shortage during data merge Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373	2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin	6fc3696260	lib/promscrape/discovery/consul: use case-insensitive comparison for service names Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424	2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin	61e483a01c	lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime This is a follow-up for `b84aea1e6e`	2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin	72de54f93e	lib/promauth: cache the client TLS certificate for up to a second This should reduce CPU usage when TLS connections are established at a high rate.	2021-07-02 13:20:18 +03:00
Aliaksandr Valialkin	1c12c0f79c	lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2021-07-01 15:43:43 +03:00
Nikolay	6bd2309449	fixes /targets button style (#1423 ) * fixes /targets button style https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422 * updates boostrap version	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	71c856beb8	lib/workingsetcache: reset the cache mode when the cache is reset This should reduce memory usage if the working set is reduced after the cache reset.	2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin	bced9ee666	lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute This should reduce memory usage on a system with high number of active time series and a high churn rate. One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).	2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin	b7c0b3dde3	lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs This should reduce memory usage for the pool on systems with big number of CPU cores. The sync.Pool maintains per-CPU pools, so the total number of objects in the pool is proportional to the number of available CPU cores. The channel limits the number of pooled objects by its own capacity. This means smaller number of pooled objects on average.	2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin	2edfea8c36	lib/promscrape/discovery/docker: fix golint warning: `struct field Id should be ID`	2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin	609ad6d9bf	lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache This should prevent from stats overwriting when the previous indexdb is queried.	2021-06-29 13:11:32 +03:00
Aliaksandr Valialkin	0c4c630839	lib/promscrape: typo fix in `/targets` output The typo has been introduced in `fb72a2133f` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408	2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin	97d1ccfc8e	lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common This is a follow up after `c85a5b7fcb`	2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin	4461e20e7d	lib/promscrape: consistently sort service discovery routines This should simplify further maintenance of the code	2021-06-25 13:22:16 +03:00
Lu Jiajing	12b4cbb68f	Support Docker ServiceDiscovery (#1402 ) * add docker discovery * add test * add labels test and add scrape work * remove TODO * refactor to merge apiConfig and sdConfig * apply suggestion	2021-06-25 13:22:16 +03:00
Nikolay	501429c3ff	adds missing MustStop call to do and http sd (#1404 )	2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin	b84aea1e6e	lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores	2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin	a22f37599b	lib/storage: tune tag filters search logic Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624 The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats. This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters for further queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin	f10fa0d1d7	lib/promscrape/discovery/consul: properly pass namespace to Consul watcher Follow-up for `58a2989fe7`	2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin	4adf6c9766	lib/promscrape/discovery/http: follow up after `e307bbb29a`	2021-06-22 13:42:10 +03:00
Nikolay	e03a3d3a36	adds http_sd (#1399 ) * adds http_sd * adds X-Prometheus-Refresh-Interval-Seconds header * Update lib/promscrape/discovery/http/api.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin	3ab3902f17	lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does	2021-06-22 13:18:51 +03:00
Nikolay	827a2396d2	adds consul enterprise namespace support (#1400 ) * adds consul enterprise namespace support * Update lib/promscrape/discovery/consul/consul.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin	f9069ba32a	lib/promscrape: show jobs with empty scrape targets on /targets page	2021-06-18 10:54:12 +03:00
Nikolay	9ea1dca3dd	fixes DO service discovery labels (#1389 ) adds test for digitalocean sd	2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin	a207be3ffb	lib/storage: fix infinite loop introduced in `aa9b56a046` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin	0efd37cec1	lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second. These buffers are flushed to disk in parallel starting from the commit `56b6b893ce` . This resulted in increased write disk IO usage on systems with many cpu cores as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 . This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk. This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338 See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244	2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin	b133de1e37	lib/storage: move deletedMetricIDs set from indexDB to Storage This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB). This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation. See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 . Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 . The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart. This should be OK in most cases.	2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin	ebaf68bcb0	lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error This is a follow-up for `af90c3c43b`	2021-06-14 15:20:38 +03:00
faceair	2ea187e801	lib/protoparser: stop read when callback error (#1380 )	2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin	5f91a701fa	lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377	2021-06-14 14:04:35 +03:00
Nikolay	e42da47608	adds digital ocean sd (#1376 ) * adds digital ocean sd config * adds digital ocean sd https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367 * typo fix	2021-06-14 13:19:29 +03:00
Aliaksandr Valialkin	df057177a0	lib/promscrape: increase the duration for reading the full response in stream parsing mode Increase the duration from 10x to 30x of the configured `scrape_interval'. This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin	074b11fa69	lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin	87d221f78a	lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365	2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin	0672cfffa2	app/vmauth: properly handle http.ErrAbortHandler panic This panic can be raised by the reverseProxy on aborted request to the backend. So handle it (e.g. suppress) at reverseProxy.ServeHTTP call. Do not suppress the panic at lib/httpserver generic HTTP handler, since it may result in an inconsistent state left after the panicking handler. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353	2021-06-11 12:54:37 +03:00
Aliaksandr Valialkin	ce10bdc82a	lib/storage: reset cache on disk during series deletion and during indexdb rotation This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347	2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin	eb335d2c29	lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard	2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin	d06c0e7a94	lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores	2021-06-11 10:49:02 +03:00
Nikolay	2c1611d316	disables panic for net/httpAbortHandler (#1355 )	2021-06-09 12:12:45 +03:00
Aliaksandr Valialkin	1e4a64844d	lib/storage: properly account the number of loops spent when matching for `or suffixes` This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338	2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin	e7d353ee6a	lib/promrelabel: add tests for labelsToString() function	2021-06-04 20:42:14 +03:00
Aliaksandr Valialkin	269e35d676	app/{vmagent,vminsert}: follow-up after `2fe045e2a4` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343	2021-06-04 20:33:22 +03:00
jelmd	d8b46908db	new feature: debug relabeling (#1344 ) * new feature: relabel logging Use scrape_configs[x].relabel_debug = true to log metric names inkl. labels before and after relabeling. After relabeling related metrics get dropped, i.e. not submitted to servers. * vminsert wants relabel logging, too.	2021-06-04 20:33:21 +03:00
Nikolay	3d89c01d07	fixes solaris build (#1345 )	2021-06-04 11:56:06 +03:00
Hason Chan	439c2ed510	fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350 )	2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin	fc2565b4ee	lib/storage: reduce memory allocations when syncing dateMetricIDCache	2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin	0b9f0de0a1	lib/promscrape: fix tests after `f0c21b6300`	2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin	6865f3b497	Revert "lib/mergeset: remove a pool for inmemoryBlock structs" This reverts commit `793fe39921`. Reason to revert: production testing revealed possible slowdown when registering big number of new time series	2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin	7b33bc67a1	lib/mergeset: remove a pool for inmemoryBlock structs The pool for inmemoryBlock struct doesn't give any performance gains in production workloads, while it may result in excess memory usage for inmemoryBlock structs inside the pool during background merge of indexdb.	2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin	97de72054e	docs: document `f0c21b6300`	2021-05-27 15:04:13 +03:00
faceair	b801b299f0	lib/promscrape: apply body size & sample limit to stream parse (#1331 ) * lib/promscrape: apply body size limit to stream parse Signed-off-by: faceair <git@faceair.me> * lib/promscrape: apply sample limit to stream parse Signed-off-by: faceair <git@faceair.me>	2021-05-27 15:04:11 +03:00
Aliaksandr Valialkin	49490ae5a7	lib/protoparser/clusternative: remove duplicate `cannot read packet size` phrase from the log message Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	c85084b659	lib/handshake: pass io.EOF unmodified to the caller for BufferedConn.Read, so it could properly detect the end of stream	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	10b2855949	lib/storage: fix spelling typo: `borken->broken` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336	2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin	6b90570ed3	lib/uint64set: store pointers to bucket16 instead of bucket16 objects in bucket32 This speeds up bucket32.addBucketAtPos() when bucket32.buckets contains big number of items, since the copying of bucket16 pointers is much faster than the copying of bucket16 objects. This is a cpu profile for copying bucket16 objects: 10ms 13.43s (flat, cum) 32.01% of Total 10ms 120ms 650: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 651: b.b16his[pos] = hi . 13.31s 652: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . . 653: b16 := &b.buckets[pos] . . 654: *b16 = bucket16{} . . 655: return b16 . . 656:} This is a cpu profile for copying pointers to bucket16: 10ms 1.14s (flat, cum) 2.19% of Total . 100ms 647: b.b16his = append(b.b16his[:pos+1], b.b16his[pos:]...) . . 648: b.b16his[pos] = hi 10ms 700ms 649: b.buckets = append(b.buckets[:pos+1], b.buckets[pos:]...) . 330ms 650: b16 := &bucket16{} . . 651: b.buckets[pos] = b16 . . 652: return b16 . . 653:}	2021-05-25 14:27:52 +03:00
Aliaksandr Valialkin	1c16cbacf5	lib/storage: do not stop data ingestion on the first error in Storage.AddRows Continue data ingestion for the rest of blocks.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	2601844de3	lib/storage: limit the number of rows per each block in Storage.AddRows() This should reduce memory usage when ingesting big blocks or rows.	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	95b735a883	lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows This should reduce memory usage a bit on data ingestion path	2021-05-24 15:32:24 +03:00
Aliaksandr Valialkin	0f84503880	lib/bloomfilter: fix TestLimiterConcurrent	2021-05-24 05:18:29 +03:00
Aliaksandr Valialkin	745eda9e87	lib/fs: do not pass done callback to tryRemoveAll() func This improves code readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-24 05:00:53 +03:00
Aliaksandr Valialkin	402a8ca710	lib/storage: do not populate MetricID->MetricName cache during data ingestion This cache isn't needed during data ingestion, so there is no need in spending RAM on it. This reduces RAM usage on data ingestion path by 30%	2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin	0fc857d363	lib/{mergeset,storage}: reduce the number of IFNO log messages like `merged ... items across ... blocks in ... seconds` Log these messages if the merge takes more than 30 seconds instead of 10 seconds.	2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin	71ff7ee18d	lib/promauth: follow-up after `5b8176c68e`	2021-05-22 18:02:03 +03:00
Nikolay	2780d6dbcd	basic OAuth2 support for remoteWrite and scrape targets (#1316 ) * adds OAuth2 support for remoteWrite and scrapping * adds tests changes init	2021-05-22 18:02:01 +03:00
Aliaksandr Valialkin	89e1a45cdb	lib/fs: concurrently remove up to 1024 blocked NFS directories Previously the blocked directories were removed sequentially by a single goroutine. This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second, when big number of LSM parts are created and removed at high rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:58:08 +03:00
Aliaksandr Valialkin	23355ca34c	lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full This should reduce the probability of the panic on a highly loaded VictoriaMetrics accepting millions of samples per second. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:21:35 +03:00
Aliaksandr Valialkin	d77db9d813	all: do not skip SIGHUP signal during service initialization This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin	69e365cd48	Makefile: update golangci-lint from v1.29.0 to v1.40.1	2021-05-20 18:30:24 +03:00
Aliaksandr Valialkin	da0b32c31a	app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour These numbers are exposed via the following metrics: - vmagent_hourly_series_limit_current_series - vmagent_daily_series_limit_current_series Expose also the limits via the following metrics: - vmagent_hourly_series_limit_max_series - vmagent_daily_series_limit_max_series	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	165a9f9200	app/vmstorage: add ability to limit series cardinality via `-storage.maxHourlySeries` and `-storage.maxDailySeries` command-line flags	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	7aad5c3f76	app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis	2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin	110a888e39	lib/promscrape/discovery/kubernetes: make `golangci-lint` happy by removing empty branches	2021-05-20 12:00:17 +03:00
Aliaksandr Valialkin	e228f479a5	lib/storage: remove possible data race when logging dropped labels	2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin	9d97f44772	lib/promscrape/discovery/kubernetes: reload objects on object parse error Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 23:27:24 +03:00
Aliaksandr Valialkin	74ef40034c	lib/httpserver: typo fix in `-http.shutdownDelay` command-line flag description: servier -> server	2021-05-18 16:25:27 +03:00
Aliaksandr Valialkin	c507faec0b	lib/promscrape/discovery/kubernetes: simplify the reload logic for urlWatcher.objectsByKey	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	0f54c0121b	lib/promscrape/discovery/kubernetes: properly update vm_promscrape_discovery_kubernetes_scrape_works metric Previously it wasn't descreased during config update.	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	9f62d348db	lib/promscrape/discovery/kubernetes: log errors and stop service discovery when unexpected updates are received from Kubernetes API server Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-18 15:41:51 +03:00
Aliaksandr Valialkin	6ea191d196	docs: dealay -> delay	2021-05-18 01:07:32 +03:00
Aliaksandr Valialkin	c4ed50ae54	lib/promrelabel: add tests for conditional removal of label on another label match Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1294	2021-05-18 00:23:23 +03:00
Aliaksandr Valialkin	8764b0ae21	lib/promscrape/discovery/kubernetes: key ScrapeWork objects by urlWatcher instead of namespace This makes the code less fragile if urlWatcher would depend on additional to namepsace properties. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1170 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-17 23:49:48 +03:00
Aliaksandr Valialkin	e08287f017	lib/promscrape: reload auth tokens from files every second Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart. Now there is no need in vmagent restart. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297	2021-05-14 20:03:35 +03:00
Aliaksandr Valialkin	a6cb4f10a7	app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300	2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin	e3f61d540b	lib/promscrape: limit `scrape_timeout` by `scrape_interval` like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1281	2021-05-13 16:10:42 +03:00
匠心零度	d5285ecaf0	fix vagent imbalance problem (#1292 ) /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=0 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=1 -promscrape.config=/path/to/config.yml ... /path/to/vmagent -promscrape.cluster.membersCount=3 -promscrape.cluster.replicationFactor=2 -promscrape.cluster.memberNum=2 -promscrape.config=/path/to/config.yml ... Co-authored-by: lirenzuo <lirenzuo@shein.com>	2021-05-13 11:19:30 +03:00
Aliaksandr Valialkin	f13585dc5d	vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:09 +03:00
Aliaksandr Valialkin	d13906bf1f	lib/promscrape: exponentially increase retry interval on unsuccesful requests to scrape targets or to service discovery services This should reduce CPU load at vmagent and at remote side when the remote side doesn't accept HTTP requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289	2021-05-13 10:47:07 +03:00
Aliaksandr Valialkin	66c6976723	lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for `b50024812e`	2021-05-13 09:27:35 +03:00
Nikolay	8743bf541f	adds cgroupsv2 support (#1283 ) * adds cgroupv2 limits support https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1269 * small fix * changes Atoi to ParseUint	2021-05-13 09:27:33 +03:00
Aliaksandr Valialkin	2839055513	lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss	2021-05-13 09:01:05 +03:00
Aliaksandr Valialkin	008ae25b3a	lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate These functions are non-trivial, while their code has minimal differences. It is better from maintainability PoV to merge these functions into a single function.	2021-05-12 18:01:08 +03:00
Nikolay	be87be34a4	Adds tsdb match filters (#1282 ) * init work on filters * init propose for status filters * fixes tsdb status adds test * fix bug * removes checks from test	2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin	027607db3e	lib/promscrape/discovery/kubernetes: refresh endpoints and endpointslices scrape targets every 5 seconds, since they may depend on changed service and pod objects This should make endpoints and endpointslices scrape targets eventually consistent with the maximum delay of 5 seconds after the related service or pod object changes. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-12 14:12:43 +03:00
Aliaksandr Valialkin	1d32b008c6	lib/httpserver: add new X-Server-Hostname header instead of overwriting already exsiting header This makes possible tracking origins of chained requests over multiple hops.	2021-05-11 23:47:19 +03:00
Aliaksandr Valialkin	f1317f7c6c	lib/httpserver: return X-Server-Hostname http header in all the responses for better debuggability	2021-05-11 22:04:41 +03:00
Aliaksandr Valialkin	4e59cf4380	lib/storage: properly apply time range when matching an empty filter It must match all the time series on the given time range. Previously it was matched to all the time series without the restriction on the given time range.	2021-05-11 01:09:35 +03:00
Aliaksandr Valialkin	326cf83eb4	lib/storage: remove dead code after the commit `3ccf7ea20c`	2021-05-08 20:15:59 +03:00
Aliaksandr Valialkin	9c505d27dd	lib/ingestserver: properly close incoming connections during graceful shutdown	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	4a5f45c77e	app/vminsert: add support for data ingestion via other vminsert nodes	2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin	e6c19cb09d	lib/promscrape/discovery/kubernetes: start watchers for pods and services before starting watchers for endpoints This should eliminate possible race when an update on endpoints depends on pods and/or services, which are missing in the cache yet. This could result in missing targets based on endpoints or endpointslices. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240	2021-05-05 12:23:16 +03:00

... 13 14 15 16 17 ...

2476 commits