610 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	e151c5c644	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin	3107c633e3	app/vmselect/promql: typo fix after 3e557c9861	2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin	3e557c9861	app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration ... Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin	b1f6843bd0	app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time ... This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin	039c9d2441	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin	2a45871823	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin	461481fbdf	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery	2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin	4c8b49b193	app/vminsert: export vm_relabel_metrics_dropped_total metric that shows the number of metrics dropped due to relabeling	2020-07-23 14:57:53 +03:00
Aliaksandr Valialkin	e79de9774b	app/vmselect: typo fix after 34563916f7	2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin	34563916f7	app/vmselect: reduce memory usage when querying big number of time series with long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin	9257eee982	app/vminsert: do not call ApplyRelabeling function if relabeling is disabled ... This should reduce CPU usage a bit when `-relabelConfig` isn't set	2020-07-23 13:39:44 +03:00
Aliaksandr Valialkin	6f05c4d351	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin	2f612e0c67	app/vminsert: fix relabeling for metrics ingested via Influx line protocol ... Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels if a single Influx line protocol message contains multiple field values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638	2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin	4f526cc816	app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin	dfb113f175	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries ... This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.	2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin	31ae5911a8	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin	caa2952aa6	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin	e00cfc854d	app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value ... This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin	b9c8f6bf34	app/vmselect: log the total available memory for concurrent requests on not enough memory errors ... This should simplify root cause analysis	2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin	ad6290953c	app/vmagent: add -remoteWrite.proxyURL command-line option ... This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy. This is similar to `proxy_url` option in `remote_write` section of Prometheus. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write	2020-07-20 19:28:49 +03:00
Aliaksandr Valialkin	efcbb51968	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-20 17:08:34 +03:00
Roman Khavronenko	ed0df37ee7	app/vmagent: mention grafana dashboard in README (#639)	2020-07-20 17:07:27 +03:00
Aliaksandr Valialkin	11be704109	app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries	2020-07-20 15:49:27 +03:00
Aliaksandr Valialkin	5a4675c528	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range ... This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin	ecb1b2564a	app/vmselect/promql: add mode() aggregate function	2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin	b35cb293f5	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin	1c641037e8	app/vmselect/promql: add mode_over_time(m[d]) function ... See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin	6b5ad535ae	app/vmselect/promql: optimize group(rollup(m)) calculations	2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin	8949d65ad1	app/vmselect/promql: check that any() doesn't touch metric name	2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin	aa5d88055d	app/vmselect/promql: add group() aggregate function to MetricsQL ... This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin	df01836818	app/vmselect/promql: keep all labels for time series from any() call	2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin	8c14ca93fa	app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call ... Previously this could lead to `out of range` panic	2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin	e4e1cd1de2	app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage	2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin	ed7580ad22	app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf when wrapping errors	2020-07-15 13:56:47 +03:00
Aliaksandr Valialkin	328814ee60	docs/vmagent.md: make filtering rules for init container pods less confusing	2020-07-14 20:32:47 +03:00
Aliaksandr Valialkin	b442a42d8e	app/vmagent/remotewrite: return proper value from tssRelabelPool.New ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-14 14:29:20 +03:00
Aliaksandr Valialkin	4081e2295e	app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626	2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin	25f80d320b	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time ... Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin	55d83e777d	app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9	2020-07-13 21:40:38 +03:00
faceair	1033dc7e2a	fix empty response template (#617)	2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin	619b0a25c9	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-13 21:25:11 +03:00
ofen	666c795b98	Update README.md (#621) ... Troubleshooting section updated to help out with duplicate targets detection	2020-07-13 21:18:54 +03:00
Aliaksandr Valialkin	a730b3f6a1	app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig options are set ... Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race and improper relabeling. Now each goroutine has its own copy of tss during relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-10 15:16:59 +03:00
Aliaksandr Valialkin	508ad46e0e	app/vmagent/remotewrite: typo fix in -remoteWrite.showURL help message	2020-07-10 14:07:08 +03:00
Aliaksandr Valialkin	cba820e390	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin	9c350bc20d	app/vmselect/promql: add missing tests for ifnot binary operation	2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin	256fd9a87e	app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation	2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin	2d9b3ad5b3	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin	742da690f4	app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575	2020-07-08 18:55:38 +03:00
DexterZhang	99f54e44ff	feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598)	2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin	6daa5f7500	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:42:05 +03:00
Roman Khavronenko	703def4b2e	app/vmalert: add retries to remotewrite (#605) ... * app/vmalert: add retries to remotewrite Remotewrite pkg now does limited number of retries if write request failed. This suppose to make vmalert state persisting more reliable. New metrics were added to remotewrite in order to track rows/bytes sent/dropped. defaultFlushInterval was increased from 1s to 5s for sanity reasons. * fix * wip * wip * wip * fix bits alignment bug for 32-bit systems * fix mistakenly dropped field	2020-07-05 18:46:52 +03:00
Aliaksandr Valialkin	de137aef98	app/victoria-metrics: fix tests after the commit acf828a759	2020-07-05 18:24:41 +03:00
Aliaksandr Valialkin	acf828a759	app/vmselect/prometheus: small fixes on top of 8bb762124a	2020-07-05 18:17:06 +03:00
faceair	8bb762124a	fix adjust last points avoid influence earlier value (#606)	2020-07-05 17:56:54 +03:00
Ween	d59cdbe90c	[VMAlert] Fix error log when remoteWrite queue size is full (#602) ... * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors * fix error logs when queue is full Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-07-03 16:49:37 +03:00
Aliaksandr Valialkin	0b2086b7a5	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin	8f628cd805	app/victoria-metrics: removed debug log message when -selfScrapeInterval is set	2020-07-02 20:39:41 +03:00
Aliaksandr Valialkin	91b3482894	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin	e5500bfcf2	all: typo fix: exptected -> expected	2020-07-02 18:05:52 +03:00
Aliaksandr Valialkin	5d3db3ff7c	app/vmselect: add interpolate function for filling gaps with linearly interpolated values ... See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin	4dd3de9286	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:14 +03:00
BigFish	9d5f5b6878	fix: spelling mistakes (#594) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-07-01 01:35:26 +03:00
Aliaksandr Valialkin	84a37098ed	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention ... VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention if `-denyQueriesOutsideRetention` command-line flag is set.	2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin	7c2c8b2981	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Roman Khavronenko	88538df267	app/vmalert: support multiple notifier urls (#584) (#590) ... * app/vmalert: support multiple notifier urls (#584) User now can set multiple notifier URLs in the same fashion as for other vmutils (e.g. vmagent). The same is correct for TLS setting for every configured URL. Alerts sending is done in sequential way for respecting the specified URLs order. * app/vmalert: add basicAuth support for notifier client (#585) The change adds possibility to set basicAuth creds for notifier client in the same fasion as for remote write/read and datasource.	2020-06-29 22:21:03 +03:00
Roman Khavronenko	82ecfa3b32	app/vmalert: move flags description and initialization into subpackages ... The change adds no new functionality and aims to move flags definitions to subpackages that are using them. This should improve readability of the main function.	2020-06-28 12:26:22 +01:00
kreedom	dc4e3f0e0b	app/vmalert: properly set transport for HTTP clients ... Fixes issue #586	2020-06-27 08:31:54 +01:00
Aliaksandr Valialkin	8f2e88234f	docs: update the info that docker images are built on top of alpine image now ... A follow-up after the commit ff624c9125 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-26 13:54:10 +03:00
Aliaksandr Valialkin	05d754d7bb	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series ... This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:38:39 +03:00
nicbaz	774f7ca1c1	vmselect: fix label_replace when mismatch (#579) ... As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the **MetricsQL** `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin	7209d58fbd	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series ... Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:46:15 +03:00
nicbaz	72c90bfd8b	vmalert: add support for TLS configuration (#578) ... app/vmalert: add support for TLS configuration Add support for TLS optional configuration in a similar fashion to what is currently supported in other vmutils such as vmagent. TLS configuration options are distinct for datasource, remoteRead, remoteWrite as well as notifier.	2020-06-23 20:45:45 +01:00
Aliaksandr Valialkin	2a39ba639d	lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs ... These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin	7532dbcdf5	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:33:01 +03:00
kreedom	7ec6711f06	Support of custom URL path for alert (#560) ... app/vmalert: Support custom URL for alerts source Add flag `external.alert.source` for configuring custom URL for alert's source. This may be handy to re-point default source URL to other systems like Grafana. Updates #517	2020-06-21 11:32:46 +01:00
Aliaksandr Valialkin	9f3e3a4d7a	docs/vmauth.md: mention that we can provide custom integration with SAML	2020-06-19 13:14:07 +03:00
Aliaksandr Valialkin	b542e50680	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin	88e1b7d144	app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin	a12364ad37	docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible	2020-06-18 18:52:43 +03:00
Roman Khavronenko	e91d758831	vmalert-537: allow name duplication for rules within one group. (#559) ... Uniqueness of rule is now defined by combination of its name, expression and labels. The hash of the combination is now used as rule ID and identifies rule within the group. Set of rules from coreos/kube-prometheus was added for testing purposes to verify compatibility. The check also showed that `vmalert` doesn't support `query` template function that was mentioned as limitation in README.	2020-06-15 20:15:47 +01:00
Clémence Saussez	e426434770	app/vmalert: fix link to testdata (#547) ... Fix broken link to vmalert test data Signed-off-by: Clemence Saussez <clemence@zen.ly>	2020-06-10 17:26:27 +01:00
Roman Khavronenko	3e277020a5	vmalert-491: allow to configure concurrent rules execution per group. (#542) ... The feature allows to speed up group rules execution by executing them concurrently. Change also contains README changes to reflect configuration details.	2020-06-09 15:21:20 +03:00
Roman Khavronenko	ffa75c423d	vmalert-521: allow to disable rules expression validation. (#536) ... This feature may be useful for using `vmalert` with PromQL compatible datasources like Loki.	2020-06-06 21:27:09 +01:00
Aliaksandr Valialkin	f5dd2a71a6	app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 20:13:56 +03:00
Aliaksandr Valialkin	4b98e436ef	app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set	2020-06-05 20:13:55 +03:00
Aliaksandr Valialkin	0d92abfbf6	app/vmalert: print brief usage info for vmalert -help	2020-06-05 10:43:18 +03:00
Aliaksandr Valialkin	ff1a725a56	app/vmauth: print brief usage info for vmauth -help	2020-06-05 10:40:00 +03:00
Aliaksandr Valialkin	05ae1472e3	app/vmagent: print brief usage info for vmagent -help	2020-06-05 10:39:59 +03:00
Aliaksandr Valialkin	f8692a1d43	app/vmauth: log when -auth.config is reloaded in SIGHUP	2020-06-03 23:22:14 +03:00
Aliaksandr Valialkin	d2f30e8d79	app/vmalert: fix comment for UpdateWith exported methods	2020-06-01 14:35:32 +03:00
Roman Khavronenko	270552fde4	vmalert: Add recording rules support. (#519) ... * vmalert: Add recording rules support. Recording rules support required additional service refactoring since it wasn't planned to support them from the very beginning. The list of changes is following: * new entity RecordingRule was added for writing results of MetricsQL expressions into remote storage; * interface Rule now unites both recording and alerting rules; * configuration parser was moved to separate package and now performs more strict validation; * new endpoint for listing all groups and rules in json format was added; * evaluation interval may be set to every particular group; * vmalert: uncomment tests * vmalert: rm outdated TODO * vmalert: fix typos in README	2020-06-01 13:46:37 +03:00
Aliaksandr Valialkin	32652485e3	app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518	2020-05-30 14:37:12 +03:00
Aliaksandr Valialkin	d988f89415	app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent	2020-05-30 14:23:14 +03:00
Aliaksandr Valialkin	5b6a9675d8	app/vmauth: fix make run-vmauth command	2020-05-22 16:45:02 +03:00
Aliaksandr Valialkin	8905bc2a40	app/vmagent: check for error returned from flag.Set	2020-05-21 16:31:14 +03:00
Aliaksandr Valialkin	f9847352b4	app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin	dbd8beccfa	app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions ... These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:07:48 +03:00
kreedom	6b23df2bec	vmalert add quotes escape function (#510) ... * vmalert add quotes escape function Co-authored-by: kreedom	2020-05-20 22:20:31 +03:00