github-mirrors/VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

Author	SHA1	Message	Date
Alexander Marshalov	d321ea91f2	fixed typos in documentation and commandline flags descriptions (#4275 )	2023-05-10 02:22:06 -07:00
Aliaksandr Valialkin	079875a127	app/vmagent/remotewrite: make more user-friendly the warning message about too small -remoteWrite.maxdiskUsagePerURL value This is a follow-up for `bc17f4828c` . While at it, document the change at docs/CHANGELOG.md . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195	2023-05-09 22:48:40 -07:00
Alexander Marshalov	26fc4afff8	added new consulagent service discovery (#3953 ) (#4217 )	2023-05-08 23:43:59 -07:00
Roman Khavronenko	c6511bc2d0	Revert "http server: limit max concurrent requests (#4185 )" (#4215 ) This reverts commit `77f76371` Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 17:22:27 -07:00
Dmytro Kozlov	f425123116	app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB (#4196 ) * app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB * app/vmagent,lib/persistentqueue: linter fix * app/vmagent,lib/persistentqueue: fix comment	2023-05-08 15:45:21 -07:00
Roman Khavronenko	20b025dc88	http server: limit max concurrent requests (#4185 ) * lib/httpserver: introduce `-http.maxConcurrentRequests` command-line flag Introduce `-http.maxConcurrentRequests` command-line flag to protect VM components from resource exhaustion during unexpected spikes of HTTP requests. By default, the new flag's value is set to 0 which means no limits are applied. Signed-off-by: hagen1778 <roman@victoriametrics.com> * lib/httpserver: mention http.maxConcurrentRequests in docs Signed-off-by: hagen1778 <roman@victoriametrics.com> --------- Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-05-08 13:13:58 -07:00
Aliaksandr Valialkin	cf4701db65	lib/fs: add MustReadDir() function Use fs.MustReadDir() instead of os.ReadDir() across the code in order to reduce the code verbosity. The fs.MustReadDir() logs the error with the directory name and the call stack on error before exit. This information should be enough for debugging the cause of the error.	2023-04-14 22:11:40 -07:00
Aliaksandr Valialkin	dad13c0a91	lib/streamaggr: follow-up for `ff72ca14b9` - Make sure that the last successfully loaded config is used on hot-reload failure - Properly cleanup resources occupied by already initialized aggregators when the current aggregator fails to be initialized - Expose distinct vmagent_streamaggr_config_reload* metrics per each -remoteWrite.streamAggr.config This should simplify monitoring and debugging failed reloads - Remove race condition at app/vminsert/common.MustStopStreamAggr when calling sa.MustStop() while sa could be in use at realoadSaConfig() - Remove lib/streamaggr.aggregator.hasState global variable, since it may negatively impact scalability on system with big number of CPU cores at hasState.Store(true) call inside aggregator.Push(). - Remove fine-grained aggregator reload - reload all the aggregators on config change instead. This simplifies the code a bit. The fine-grained aggregator reload may be returned back if there will be demand from real users for it. - Check -relabelConfig and -streamAggr.config files when single-node VictoriaMetrics runs with -dryRun flag - Return back accidentally removed changelog for v1.87.4 at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3639	2023-03-31 22:54:10 -07:00
Alexander Marshalov	8c14d17694	added hot reload support for stream aggregation configs (#3969 ) (#3970 ) added hot reload support for stream aggregation configs (#3969) Signed-off-by: Alexander Marshalov <_@marshalov.org>	2023-03-31 22:31:38 -07:00
Aliaksandr Valialkin	6f5bbf096a	app/vmagent/remotewrite: cosmetic updates after `f3a51e8b1d` - Compare directory names instead of paths to directory when determining which persistent queues must be deleted This is less error-prone solution, since paths to the same directory can differ, which could lead to accidental directory removal for the existing -remoteWrite.url - Log the `removed %d dangling queues` message when at least a single queue has been removed - Consistently use filepath.Join() for creating paths to persistent queues. This is needed for Windows support (see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 ) - Clarify the description of the change at docs/CHANGELOG.md Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014	2023-03-27 18:38:53 -07:00
Zakhar Bessarab	6ed6eb0c4c	app/vmagent: add `-remoteWrite.removeDanglingQueues` flag (#4017 ) * app/vmagent: add `-remoteWrite.removeDanglingQueues` flag which allows to automatically remove dangling persistent queue contents Related issue: #4014 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * app/vmagent: address review feedback - remove persistent queues files by default - rename `remoteWrite.removeDanglingQueues` to `remoteWrite.keepDanglingQueues` - update docs to reflect changed behaviour Related issue: #4014 * Apply suggestions from code review --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-03-27 18:38:51 -07:00
Aliaksandr Valialkin	9d19c2d89b	docs/vmagent.md: clarify that there is no need to specify multiple -remoteWrite.url options when writing data to a single VictoriaMetrics cluster when data replication is needed Also add a link to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format from `getting started` section, so users could quickly find how to write data to VictoriaMetrics cluster	2023-03-24 17:56:31 -07:00
Aliaksandr Valialkin	8ed9295109	docs/vmagent.md: mention in docs that the target relabel debug page shows target url now	2023-03-20 22:20:13 -07:00
Aliaksandr Valialkin	54fe207cc0	all: follow-up for `7a3e16e774` - Sync the description for -httpListenAddr.useProxyProtocol command-line flag at vmagent and vmauth, so it is consistent with the description at vmauth and victoria-metrics - Add a sample of panic text to docs/CHANGELOG.md, so it could be googled - Mention the -httpListenAddr.useProxyProtocol command-line flag in the description for the bugfix Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335	2023-03-08 01:42:58 -08:00
Aliaksandr Valialkin	e91fe74618	all: typo fixes of the same type as in the `d056be710b`	2023-03-08 01:37:28 -08:00
Aliaksandr Valialkin	b60594a548	app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c - Document the fix - Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient() This simplifies the fix to the following diff: diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go index 099899c19..70b904af4 100644 --- a/app/vmagent/remotewrite/client.go +++ b/app/vmagent/remotewrite/client.go @@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq persiste } c.sendBlock = c.sendBlockHTTP - return c -} - -func (c client) init(argIdx, concurrency int, sanitizedURL string) { useVMProto := forceVMProto.GetOptionalArg(argIdx) usePromProto := forcePromProto.GetOptionalArg(argIdx) if useVMProto && usePromProto { @@ -173,6 +169,10 @@ func (c client) init(argIdx, concurrency int, sanitizedURL string) { } c.useVMProto = useVMProto + return c +} + +func (c client) init(argIdx, concurrency int, sanitizedURL string) {	2023-03-08 01:35:31 -08:00
Dmytro Kozlov	2cf6797a24	app/vmagent: fix panic if auth config not defined (#530 )	2023-03-08 01:35:04 -08:00
Aliaksandr Valialkin	086a4b4fca	lib/bytesutil: add `-internStringDisableCache` and `-internStringCacheExpireDuration` command-line flags This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872	2023-02-27 14:18:02 -08:00
Aliaksandr Valialkin	d9581cf3f3	app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol	2023-02-27 11:04:11 -08:00
Aliaksandr Valialkin	bbd5914eb1	all: add makefile rules for GOARCH=s390x for all the VictoriaMetrics components This is a follow-up for `007530f882`	2023-02-26 12:38:48 -08:00
Aliaksandr Valialkin	1e156ac3c3	app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-26 12:19:53 -08:00
Aliaksandr Valialkin	3563c6882e	docs: update --help descriptions after recent changes	2023-02-23 19:02:32 -08:00
Aliaksandr Valialkin	f579cac297	app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag, which can be used for forcing Prometheus remote write protocol in cases when the remote storage supports VictoriaMetrics remote write protocol. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-23 17:38:47 -08:00
Aliaksandr Valialkin	0c60e4a30a	all: consistently use http.Method{Get,Post,Put} across the codebase This is a follow-up after `9dec3c8f80`	2023-02-22 19:01:09 -08:00
my-git9	7d86c5c94a	chore: Use http constants to replace numbers (#3846 ) Signed-off-by: xin.li <xin.li@daocloud.io>	2023-02-22 18:59:32 -08:00
Aliaksandr Valialkin	1b70238dca	lib/promscrape/discovery/kuma: follow-up for `317fef95f9` - Do not generate __meta_server label, since it is unavailable in Prometheus. - Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md, so users could click it and read the docs without the need to search the corresponding docs. - Remove kumaTarget struct, since it is easier generating labels for discovered targets directly from the response returned by Kuma. This simplifies the code. - Store the generated labels for discovered targets inside atomic.Value. This allows reading them from concurrent goroutines without the need to use mutex. - Use synchronouse requests to Kuma instead of long polling, since there is a little sense in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval. - Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used. - Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus. - Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability. - Remove unused fields from discoveryRequest and discoveryResponse. - Update tests. - Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config. - Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback, since these are public types. Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label, since usually this label is set by the Prometheus itself to __address__ after the relabeling phase. See https://www.robustperception.io/life-of-a-label/ Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389 See https://github.com/prometheus/prometheus/issues/7919 and https://github.com/prometheus/prometheus/pull/8844 as a reference implementation in Prometheus	2023-02-22 17:50:54 -08:00
Aliaksandr Valialkin	b3aed1aebc	app/vmagent/remotewrite: removed unneeded code in testPushWriteRequest after `57801660ab`	2023-02-22 17:50:03 -08:00
Alexander Marshalov	173643a771	add kuma_sd_config for Kuma Control Plane targets discovery (#3389 ) (#3840 )	2023-02-22 17:41:43 -08:00
Alexander Marshalov	29fa78a310	Fix TestPushWriteRequest for remotewriteprotocol for pure-test (with native zstd) (#3850 ) app/vmagent: fix TestPushWriteRequest for pure-test (with native zstd) Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>	2023-02-22 12:16:29 -08:00
Artem Navoiev	522179fbde	docs: fix typo OpentTSDB -> OpenTSDB (#3854 ) Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-02-22 12:11:48 -08:00
Aliaksandr Valialkin	fc5ea31a38	docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol The 10x savings are reproduced only on artificial data. The savings on production data are usually in the range 2x-4x.	2023-02-20 19:11:11 -08:00
Aliaksandr Valialkin	bf359e8cc0	docs/vmagent.md: mention that Mimir doesnt support backfilling	2023-02-20 19:06:56 -08:00
Aliaksandr Valialkin	80c6d1e24c	app/vmagent: add support for VictoriaMetrics remote write protocol, which allows saving up to 10x on network bandwidth costs under high load Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225	2023-02-20 18:40:40 -08:00
Aliaksandr Valialkin	f987fb9c8b	lib/protoparser/promremotewrite: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:48:11 -08:00
Aliaksandr Valialkin	c54d17b006	lib/protoparser/native: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:44:27 -08:00
Aliaksandr Valialkin	75cf5a8939	lib/protoparser/graphite: extract stream parsing code into a separate stream package	2023-02-13 10:33:24 -08:00
Aliaksandr Valialkin	1801fa6c5c	lib/protoparser/csvimport: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:26:29 -08:00
Aliaksandr Valialkin	41feed813d	lib/protoparser/vmimport: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:22:00 -08:00
Aliaksandr Valialkin	66f0a78810	lib/protoparser/opentsdbhttp: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:15:15 -08:00
Aliaksandr Valialkin	67c0281535	lib/protoparser/opentsdb: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 10:04:14 -08:00
Aliaksandr Valialkin	1add6c3fa0	lib/protoparser/influx: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 09:59:56 -08:00
Aliaksandr Valialkin	b691d02b92	lib/protoparser/datadog: extract stream parsing code into a separate stream package This is a follow-up for `057698f7fb`	2023-02-13 09:53:20 -08:00
Roman Khavronenko	867b7e5688	lib/protoparser/prometheus: move `streamparser` to subpackage (#3814 ) `lib/protoparser/prometheus` is used by various applications, such as `app/vmalert`. The recent change to the `lib/protoparser/prometheus` package introduced a new dependency of `lib/writeconcurrencylimiter` which exposes some metrics. Because of the dependency, now all applications which have this dependency also expose these metrics. Creating a new `lib/protoparser/prometheus/stream` package helps to remove these metrics from apps which use `lib/protoparser/prometheus` as dependency. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761 Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-02-13 09:44:47 -08:00
Aliaksandr Valialkin	34379d4cf1	all: run `apk update && apk upgrade` in base Alpine Docker image in order to get all the recent security fixes	2023-02-09 14:03:02 -08:00
Aliaksandr Valialkin	dc7b92bc34	docs/vmagent.md: clarify that automatically generated metrics contain all the target-specific labels, including `instance` and `job`	2023-02-09 11:05:57 -08:00
Aliaksandr Valialkin	cd25945ff6	app/vmagent: properly return 200 response code when importing data via Prometheus PushGateway protocol This is the same fix as has been already applied to app/vminsert at `cdb6d651e9` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2023-01-27 14:40:26 -08:00
Aliaksandr Valialkin	4cf4c307ea	docs: update command-line descriptions after `73256fe438`	2023-01-27 00:01:14 -08:00
Aliaksandr Valialkin	1a1f3e7149	app/vmagent/prometheusimport: delete the temporary directory created by vmagent after the test is complete This is a follow-up for `1cfa183c2b`	2023-01-26 23:25:34 -08:00
Nikolay	ebebaecd94	lib/netutil: init implimentation of proxy protocol (#3687 ) * lib/netutil: init implimentation of proxy protocol https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335 * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-26 23:25:22 -08:00
Roman Khavronenko	29ea196bf0	docs: specify the time window for `series_limit` (#3708 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-01-25 09:31:09 -08:00
Aliaksandr Valialkin	bd809db4d9	docs: update the list of command-line flags according to the latest changes	2023-01-25 09:22:23 -08:00
Aliaksandr Valialkin	5defa99a2e	lib/streamaggr: add ability to de-duplicate input samples before aggregation	2023-01-25 09:22:03 -08:00
Aliaksandr Valialkin	417b90b8ad	app/vmagent/{promremotewrite,vmimport}: remove unused functions InsertHandlerForReader() Thanks to `1cfa183c2b` , where the first such function has been removed	2023-01-23 22:42:46 -08:00
Aliaksandr Valialkin	4b3a207705	app/{vmagent,vminsert}: follow-up for `1cfa183c2b` - Call httpserver.GetQuotedRemoteAddr() and httpserver.GetRequestURI() only when the error occurs. This saves CPU time on fast path when there are no parsing errors. - Create a helper function - httpserver.LogError() - for logging the error with the request uri and remote addr context.	2023-01-23 22:41:08 -08:00
Artem Navoiev	0ac0cfdc69	add error handler for parsing prometheus text format to vmagent and v… (#3693 ) * add error handler for parsing prometheus text format to vmagent and vminsert Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * typo Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * fix variables naming and error message Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-23 22:36:23 -08:00
Artem Navoiev	e91814e332	vmagent: add minimal scrape file exampe for vmagent quick start (#3627 ) * vmagent: add minimal scrape file exampe for vmagent quick start Signed-off-by: Artem Navoiev <tenmozes@gmail.com> * replace example with link to your prometheus.yml in docker Signed-off-by: Artem Navoiev <tenmozes@gmail.com> Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2023-01-11 01:23:07 -08:00
Aliaksandr Valialkin	aa027529eb	lib/httpserver: directly pass flag value to CheckAuthFlag() There is no sense in passing a pointer to flag value there. This is a follow-up for `4225a0bd75`	2023-01-10 15:59:55 -08:00
Zakhar Bessarab	10f314cdbd	Use `httpAuth.` flags as a fallback for endpoints protected by `AuthKey` flags (#3582 ) * {lib/server, app/}: use `httpAuth.` flag as fallback for `AuthKey` if it is not set * lib/ingestserver/opentsdbhttp: fix opentdb HTTP handler not respecting `httpAuth.` flags Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2023-01-10 15:57:55 -08:00
Aliaksandr Valialkin	5876821a16	all: small improvements in error messages and command-line flag descriptions related to concurrency limiters	2023-01-07 00:12:24 -08:00
Aliaksandr Valialkin	b275983403	lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit Previously the -maxConcurrentInserts was limiting the number of established client connections, which write data to VictoriaMetrics. Some of these connections could be idle. Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting the number of such connections. So now the -maxConcurrentInserts command-line option limits the number of concurrently executed insert requests, not including idle connections. It is recommended removing -maxConcurrentInserts command-line option, since the default value for this option should work good for most cases.	2023-01-06 22:07:16 -08:00
Karan Sharma	8f42c5a024	lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549 ) Add nomad_sd_config support for service discovery	2023-01-05 18:07:02 -08:00
Aliaksandr Valialkin	1d16cc9349	lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin	ac890b3081	docs: update `-help` outputs for vm* tools	2023-01-03 23:27:31 -08:00
Aliaksandr Valialkin	3369371636	app/{vmagent,vminsert}: add support for streaming aggregation See https://docs.victoriametrics.com/stream-aggregation.html Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460	2023-01-03 22:22:07 -08:00
Aliaksandr Valialkin	cfba42091e	app/vmagent/remotewrite: improve descriptions for `-remoteWrite.relabelConfig` and `-remoteWrite.urlRelabelConfig` - Cross-reference these command-line flags. - Add a link to https://docs.victoriametrics.com/vmagent.html#relabeling	2023-01-03 22:05:39 -08:00
Artem Navoiev	393f4ab86f	update links to grafana dashboards (#3534 ) docs: update links to grafana dashboards Signed-off-by: Artem Navoiev <tenmozes@gmail.com>	2022-12-28 11:22:02 -08:00
Roman Khavronenko	998d89685f	vmagent: respect `-usePromCompatibleNaming` if no relabeling is set (#3511 ) * vmagent: respect `-usePromCompatibleNaming` if no relabeling is set See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493 Signed-off-by: hagen1778 <roman@victoriametrics.com> * vmagent: upd test Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-12-20 14:34:29 -08:00
Aliaksandr Valialkin	1a88fe5b1f	lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures This fixes handling of values bigger than 2GiB for the following command-line flags: - -storage.minFreeDiskSpaceBytes - -remoteWrite.maxDiskUsagePerURL	2022-12-14 19:29:57 -08:00
Aliaksandr Valialkin	3a28a52667	lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes	2022-12-14 17:53:18 -08:00
Aliaksandr Valialkin	ed7d204274	docs/vmagent.md: small formatting fix	2022-12-14 14:25:46 -08:00
Aliaksandr Valialkin	cca16cb498	docs/vmagent.md: clarify that relabeling is actually a debugging at `relabel debug` section	2022-12-13 15:46:30 -08:00
Aliaksandr Valialkin	7d996ced51	app/{vmagent,vminsert}/datadog: make the `host` label optional in DataDog data ingestion protocol Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432	2022-12-10 23:33:17 -08:00
Aliaksandr Valialkin	00d953ab66	docs: clarify that single-node VictoriaMetrics also provides functionality for relabel debugging Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 20:50:51 -08:00
Aliaksandr Valialkin	a521135b7b	lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407	2022-12-10 12:47:47 -08:00
Aliaksandr Valialkin	97b41e727c	lib/promscrape: implement target-level and metric-level relabel debugging Target-level debugging is performed by clicking the 'debug' link at the corresponding target on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page. Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407 See https://docs.victoriametrics.com/vmagent.html#relabel-debug	2022-12-10 02:25:56 -08:00
Aliaksandr Valialkin	676de127aa	all: update Go builder from v1.19.3 to v1.19.4 See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved	2022-12-08 17:04:41 -08:00
Aliaksandr Valialkin	a13f16e48a	lib/promscrape: allow using `sample_limit` and `series_limit` options in stream parsing mode Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458	2022-12-08 17:04:12 -08:00
Pedro Gonçalves	84cfe4fcaf	Datadog - Add device as a tag if it's present as a field in the series object (#3431 ) * Datadog - Add device as a tag if it's present as a field in the series object * address PR comments	2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin	6910b1de2e	all: typo fix: `the the` -> `the`	2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin	8ce5b095b7	lib/promscrape: add `exported_` prefix to metric names exported by scrape targets if they clash with automatically generated metrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406	2022-11-28 18:37:34 -08:00
Aliaksandr Valialkin	c5eebaffd8	app/{vminsert,vmagent}: follow-up after `53a63c6c4c` Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415	2022-11-25 16:42:38 -08:00
Pedro Gonçalves	47a0d845c4	Adding pushgateway basic capabilities to vmagent (#3360 ) * init pushgateway implementation * Initial implementation of pushgateway in vmagent * Initial implementation of pushgateway in vmagent	2022-11-25 16:37:28 -08:00
Aliaksandr Valialkin	c7bd0f5082	docs/vmagent.md: typo fix	2022-11-24 17:19:56 -08:00
Aliaksandr Valialkin	afc35485c1	app/vminsert: add missing vm_relabel_config_* metrics after `03d88bc066` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345	2022-11-22 00:48:13 +02:00
Roman Khavronenko	d1169c1559	vmagent: expose metrics for tracking config state (#3375 ) Expose `vm_relabel_config_` and `vm_promscrape_config_` metrics for tracking relabel and scrape configuration hot-reloads. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin	6fe8eec745	all: add a link to https://docs.victoriametrics.com/enterprise.html into description for enterprise flags	2022-11-21 15:44:54 +02:00
Roman Khavronenko	8ee464b22b	bump go version to 1.19.3 (#3327 ) Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-11-09 11:56:38 +02:00
Aliaksandr Valialkin	8ea84432ef	docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html	2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin	c7360e1fa0	app/vmagent/remotewrite: typo fix after `c914e4dace`	2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin	207b4a6c58	app/vmagent/remotewrite: typo fix after `50f5eae0e0`	2022-10-13 10:19:46 +03:00
Aliaksandr Valialkin	087393bcef	lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places Sort labels explicitly after calling the ParsedConfigs.Apply() when needed. This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.	2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin	3b828535f0	lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis Add support for no_stale_markers option at scrape_config section. See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers	2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin	98a4ab796c	all: update the minimum required Go verson from 1.19.1 to 1.19.2 This is needed because of security vulnerabilities found in Go 1.19.1 See https://go.dev/doc/devel/release#go1.19.2	2022-10-07 22:46:44 +03:00
Aliaksandr Valialkin	e9f1213cc9	docs/vmagent.md: fix incorrect url for multitenant writes to VictoriaMetrics cluster	2022-10-01 18:51:57 +03:00
Aliaksandr Valialkin	b857365b84	app/vmagent/remotewrite: allow specifying per-`-remoteWrite.url` disk limits for persistent queue with pending data This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071 Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970	2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin	6f9ce3f6d6	lib/flagutil: rename Array to ArrayString This makes the ArrayString more consistent with other Array* types. While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071	2022-10-01 18:28:19 +03:00
Aliaksandr Valialkin	d81285c814	app/vmagent: accept requests with `/prometheus` and `/influx` path prefixes in the same way as VictoriaMetrics does This allows using vmagent as a drop-in replacement for VictoriaMetrics for push protocols.	2022-10-01 14:11:09 +03:00
Aliaksandr Valialkin	b5fe42514d	docs/vmagent.md: formatting fixes	2022-10-01 13:36:48 +03:00
Aliaksandr Valialkin	9676519830	app/vmagent/README.md: apply typo fix from `1fda517af9`	2022-10-01 12:31:42 +03:00
Nikolay	505d359b39	app/vminsert: allows parsing tenant id from labels (#3009 ) * app/vminsert: allows parsing tenant id from labels it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load and it cannot handle more then 100 different tenants. This functional hidden with flag and do not change vminsert default behaviour https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970 * Update docs/Cluster-VictoriaMetrics.md Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip * app/vminsert/netstorage: clean remaining labels in order to free up GC * docs/Cluster-VictoriaMetrics.md: typo fix * wip * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin	7f0b95b50a	lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names Optimize this function by using results cache for input strings. Use this function all over the code. This is a follow-up for `fcffdba9dc` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113	2022-09-28 10:02:11 +03:00
Dmytro Kozlov	28dcff5791	lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109 ) * lib/{httpserver,netutil}: allow to define min and max TLS version of the http server * lib/httpserver: added descriptions about tls supported versions * lib/netutil: check minimal tls version, added supported tls versions to error * wip Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-09-26 17:38:43 +03:00
Aliaksandr Valialkin	dbc20091b1	lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105 Thanks to @PerGon for the idea and initial implementation.	2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin	6a6cf9c590	app/{vmagent,vminsert}: add `-usePromCompatibleNaming` command-line flag for normalizing metric names and label names in the ingested samples Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113 Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146	2022-09-26 13:13:31 +03:00
Aliaksandr Valialkin	336007182c	app/vmagent/remotewrite: `go fmt` after `2b55d167d7`	2022-09-19 15:14:03 +03:00
Aliaksandr Valialkin	34db3fdd3f	app/vmagent/remotewrite: add benchmarks for comparing the performance of standard Snappy encoder with github.com/klauspost/compress/s2 encoder The standard Snappy encoder from github.com/golang/snappy shows quite good performance number for compressing the Prometheus remote_write proto messages according to the added benchmarks, so there is no need in switching to github.com/klauspost/compress/s2 yet.	2022-09-19 14:27:56 +03:00
Aliaksandr Valialkin	1275e0580e	docs: update docs with explicitly marked enterprise command-line flags for VictoriaMetrics and vmagent	2022-09-15 13:19:01 +03:00
Roman Khavronenko	d071e39694	bump Go version to 1.19.1 (#3108 ) The reason is to cover vulnerability GO-2022-0969 Found in: net/http@go1.18.5 Fixed in: net/http@go1.19.1 More info: https://pkg.go.dev/vuln/GO-2022-0969 Signed-off-by: hagen1778 <roman@victoriametrics.com> Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-09-14 13:43:27 +03:00
Aliaksandr Valialkin	6ebe4f5392	docs/Single-server-VictoriaMetrics.md: add a note that cardinality explorer at cluster version of VictoriaMetrics may return lower than expected number of unique label values See the corresponding comment in the code: `5a6e617b5e/app/vmselect/netstorage/netstorage.go (L1039-L1045)`	2022-09-06 14:48:21 +03:00
Aliaksandr Valialkin	d4cf52f092	app/vmagent: follow-up after `2b22aa1537` - Document the change at docs/CHANGELOG.md - Move auth token parsing from app/vmagent/opentsdbhttp/ to app/vmagent/main.go, since it must be parsed only when multitenancy support is enabled at vmagent side. See https://docs.victoriametrics.com/vmagent.html#multitenancy	2022-08-24 16:21:23 +03:00
Jianyun Cheng	eccae22522	[vmagent] make opentsdb insert url support multitenant (#3015 )	2022-08-24 16:21:22 +03:00
Aliaksandr Valialkin	e2d8916935	docs: mention that it is safe sharing the collected profiles from security PoV The collected profiles do not contain sensitive information	2022-08-24 14:08:30 +03:00
Aliaksandr Valialkin	547d655d29	docs/vmagent.md: fix alerting query when scraped samples are dropped because of exceeded series limit This is a follow-up after `7d26414b2e`	2022-08-24 01:23:37 +03:00
Aliaksandr Valialkin	1905618d10	all: subsitute ioutil.ReadAll with io.ReadAll ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for `02ca2342ab`	2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin	1c7f402598	app/vmagent: add ability to construct a label from multiple existing labels by referring them in the `replacement` field during relabeling For example: - target_label: composite-label replacement: {{source_label1}}-{{source_label2}}	2022-08-21 22:49:24 +03:00
Aliaksandr Valialkin	29eb9d4602	docs/relabeling.md: add a cookbook for common relabeling tasks	2022-08-18 01:14:30 +03:00
Aliaksandr Valialkin	1812d33a2d	lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit The following metrics are generated: - scrape_series_limit - scrape_series_current - scrape_series_limit_samples_dropped These metrics simplify alerting on targets, which expose too many time series See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details	2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin	c27bd63f6c	lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html	2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin	ceb158c5bb	docs/vmagent.md: typo fix in __meta_kubernetes_annotation_prometheus_io_tenant label name It must be __meta_kubernetes_pod_annotation_prometheus_io_tenant	2022-08-08 14:53:28 +03:00
Aliaksandr Valialkin	310779d8b5	lib/promscrape: follow-up after `2c553d5a2f` - fix broken tests - cosmetic code cleanup - document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy - document the change at https://docs.victoriametrics.com/CHANGELOG.html	2022-08-08 14:49:16 +03:00
Fury	59fdb4cb72	add support to scrape multi tenant metrics (#2950 ) * add support to scrape multi tenant metrics * add support to scrape multi tenant metrics Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>	2022-08-08 14:49:15 +03:00
Aliaksandr Valialkin	69d62d5736	docs: sync -help output with the latest changes	2022-08-08 14:05:36 +03:00
Aliaksandr Valialkin	221dd3a224	all: bump the minimum supported version of Go from 1.17 to 1.18 This is needed because some dependencies uses generics, which have been appeared in Go1.18 This is a follow-up for `caf3dd4fa2`	2022-08-08 13:45:39 +03:00
Aliaksandr Valialkin	5e43145031	docs: add docs for scrape_configs section	2022-08-07 00:04:55 +03:00
Aliaksandr Valialkin	ecbe1ddf1b	lib/promscrape/discovery/ec2: properly handle custom `endpoint` option in ec2_sd_configs This option was ignored since `d289ecded1` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287	2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin	8ddad31eef	lib/promscrape/discovery/yandexcloud: follow-up after `6e5ac32fba` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386	2022-08-04 22:28:21 +03:00
Igor Tiunov	0ba86fe87e	YC service discovery (#2923 ) * YC service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386 * Fixed linter suggestions * fixed golint errors	2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin	9f1e558c58	all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency	2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin	c0c9f30870	lib/pushmetrics: properly handle errors when initializing pushmetrics	2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin	fe68bb3ba7	all: follow-up after `46f803fa7a` Add -pushmetrics.* command-line flags to all the VictoriaMetrics apps	2022-07-21 20:18:25 +03:00
Roman Khavronenko	23e85e0fc5	vmagent: expose metric `vmagent_remotewrite_queues` (#2871 ) The new metric `vmagent_remotewrite_queues` exports a static value of number of configured remote write queus. This metric is useful to calculate total saturation per each configured URL with given number of queues. See corresponding changes to vmagent alerts and dashboard. Signed-off-by: hagen1778 <roman@victoriametrics.com>	2022-07-18 14:41:04 +03:00
cui fliter	e0d30f6ec5	fix some typos (#2882 ) Signed-off-by: cui fliter <imcusg@gmail.com>	2022-07-18 12:10:40 +03:00
Aliaksandr Valialkin	70b9925bf7	app: fix `make publish-*` after `ed93330e66` Add missing `-linux` substring to built binary names for copying into Docker images	2022-07-14 11:01:34 +03:00
Aliaksandr Valialkin	e6ee940daf	docs: update descriptions for command-line flags according to the latest changes	2022-07-14 00:57:44 +03:00
Nikolay	c007b129cb	lib/promscrape: adds azure service discovery (#2743 ) * lib/promscrape: adds azure service discovery Adds azure service discovery mechanism implements authorization with oauth and msi lists virtual machines and virtual machines managed by scaleSet https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364 * makes linter happy * Apply suggestions from code review Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> * wip Co-authored-by: Roman Khavronenko <roman@victoriametrics.com> Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>	2022-07-13 23:45:43 +03:00
Boris Petersen	cbf01ee4ae	fix typo introduced in pr #2604 (#2866 ) Signed-off-by: Boris Petersen <boris.petersen@idealo.de>	2022-07-13 23:45:42 +03:00
Aliaksandr Valialkin	da6c85a2f6	all: follow-up for `d99ba3481b`	2022-07-13 17:17:08 +03:00
Dmytro Kozlov	4e4def9df8	Rename release packages (#2810 ) * makefile: add os to each release file * makefile: update vmutils arm64 * makefile: update victoria-metrics release process * makefile: update publish with os * makefile: update publish with os * makefile: change tar library * update release logic * copy all releases * sort command by GOOS * rollback commands * rollback OSARCH * fix commands * cleanup * fix windows build * sort build by GOOS, update README.md	2022-07-13 17:11:01 +03:00
Aliaksandr Valialkin	3662ce8323	docs: make more clear the relation between replication and deduplication This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2852	2022-07-11 19:23:24 +03:00
Aliaksandr Valialkin	69718a6c16	docs/vmagent.md: clarify that vmagent supports Prometheus-compatible service discovery	2022-07-07 21:22:02 +03:00
Aliaksandr Valialkin	2a53b8961e	docs/vmagent.md: typo fixes after `ef2eeeb642`	2022-07-07 21:17:13 +03:00
Aliaksandr Valialkin	372ecc5c0e	docs/vmagent.md: actualize docs	2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin	95add1e8e4	app/{vmagent,vminsert}: follow-up after `d19e46de55` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839	2022-07-07 01:32:11 +03:00
Aliaksandr Valialkin	ecc11dc32d	lib/promauth: refactor NewConfig in order to improve maintainability 1. Split NewConfig into smaller functions 2. Introduce Options struct for simplifying construction of the Config with various options This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684	2022-07-04 14:31:43 +03:00
Aliaksandr Valialkin	1e6b0a1f54	app/vmagent/remotewrite: do not shadow `headers` global variable in getAuthConfig	2022-06-30 20:18:43 +03:00
Aliaksandr Valialkin	0f6c17ed06	app/vmagent/remotewrite: clarify descriptions for `-remoteWrite.*` options, which must be set per each `-remoteWrite.url`	2022-06-30 20:18:42 +03:00
Aliaksandr Valialkin	7fc03a1deb	app/vmagent/remotewrite: add `-remoteWrite.header` command-line flag for setting additional http headers to send to -remoteWrite.url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805	2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin	92e6c55b75	docs: add Troubleshooting doc This doc contains troubleshooting guides for typical problems with VictoriaMetrics.	2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin	6c66804fd3	all: locate throttled loggers via logger.WithThrottler() only once and then use them This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler() is called frequently from concurrent goroutines.	2022-06-27 12:34:30 +03:00
Aliaksandr Valialkin	3ae6300497	lib/promauth: add ability to send additional http headers in requests to scrape targets This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header	2022-06-22 20:40:50 +03:00

1 2 3 4 5 ...

589 commits