1059 commits

Author	SHA1	Message	Date
faceair	64e99744f1	add charset on targets response (#897)	2020-11-13 10:18:13 +02:00
Aliaksandr Valialkin	f7a6ae3d11	docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into Relabeling section	2020-11-12 12:27:13 +02:00
Aliaksandr Valialkin	069979c367	docs/vmagent.md: typo fix	2020-11-11 16:05:04 +02:00
Aliaksandr Valialkin	7a0094adae	docs/vmagent.md: add Configuration update section	2020-11-11 16:01:21 +02:00
immerrr again	1ec1a9f27f	app/vmstorage: add "/internal/force_flush" endpoint (#893)	2020-11-11 14:46:37 +02:00
Aliaksandr Valialkin	4f2c5877db	app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters	2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin	a78bf34ff3	app/vmselect: do not return isPartialResponse=true when all the storageNodes return errors	2020-11-10 18:48:57 +02:00
Aliaksandr Valialkin	8f3339fa81	app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845	2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin	6385432611	app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum	2020-11-10 01:18:59 +02:00
Roman Khavronenko	4fd2b6cd16	vmalert: explicitly set extra labels to alert entities (#886) ... The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, time series always contained only original labels and alert ID was generated from sorted labels key-values. Extra labels didn't affect the generated ID and were applied on the following actions: - templating for Summary and Annotations; - persisting state via remote write; - restoring state via remote read. Such behaviour caused difficulties on restore procedure because extra labels had to be dropped before checking the alert ID, but that not always worked. Consider the case when expression returns the following time series `up{job="foo"}` and rule has extra label `job=bar`. This would mean that restored alert ID will be always different to the real time series because of collision. To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't store original labels anymore. However, this could result into a new error situation. Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`, while rule has extra label `job=bar`. In such case, applying extra labels will result into two identical time series and `vmalert` will return error: `result contains metrics with the same labelset after applying rule labels` https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-10 00:27:56 +02:00
Aliaksandr Valialkin	a8562d643b	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin	b8083b7659	lib/promscrape: clean references to label name and label value strings after applying per-target relabeling ... This should reduce memory usage when per-target relabeling creates big number of temporary labels with long names and/or values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin	efebc3b6fb	app/vmselect/promql: code cleanup after 43823addea	2020-11-06 01:31:33 +02:00
n4mine	3127aa92b5	app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883)	2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin	767231f41f	app/vmstorage/transport: properly handle request to labelValuesOnTimeRange	2020-11-05 02:08:04 +02:00
Aliaksandr Valialkin	72011bcc45	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin	c5e6c5f5a6	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin	1336e47c86	docs/vmagent.md: update after 4c808d58bf	2020-11-04 20:33:49 +02:00
Nikolay	5b235b902b	Adds ready probe (#874) ... * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:33:48 +02:00
Nikolay	d0a9b24c5a	reduces memory usage for vmagent, (#880) ... * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin	407a46c11e	Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels" ... This reverts commit 1a80acc712.	2020-11-04 11:45:35 +02:00
Aliaksandr Valialkin	1a80acc712	docs/vmagent.md: mention about -promscrape.dropOriginalLabels	2020-11-04 11:16:16 +02:00
Aliaksandr Valialkin	887a3c317f	app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop	2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin	66de02fbb4	app/vmselect/promql: allow dropping trailing sample only for default_rollup function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850	2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin	ca2e0f1e04	app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code ... This makes consistent the behaviour with Prometheus. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873	2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin	6b623eba02	app/vmselect/promql: go fmt	2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin	7c0b658865	app/vmselect/promql: do not drop trailing datapoints for instant queries ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-11-02 00:12:53 +02:00
Roman Khavronenko	333675875f	vmalert: skip automatically added labels on alerts restore (#871) ... Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correctly purged on restore event and affected alert's ID uniqueness. This commit removes `alertgroup` label in restore function. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870	2020-11-01 23:26:00 +02:00
kreedom	40172c0721	vmbackup fix panic when no origin fs given (#859) ... * use fsnil when no origin fs	2020-11-01 23:17:01 +02:00
Aliaksandr Valialkin	ed724d25ba	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin	abdf22e0bb	app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets ... This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin	c4464594b7	app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K	2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin	9c5cd5a6c5	lib/storage: code cleanup after 5bfd4e6218	2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin	0db7c2b500	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:42:46 +03:00
kreedom	4526cf92d3	vmalert - add dryRun (#842) ... vmalert: add `dryRun` flag for rules validation without running the service	2020-10-20 10:49:22 +03:00
Seva Poliakov	e6bf9eaac7	Fix typo in vnrestore readme	2020-10-20 10:49:22 +03:00
Aliaksandr Valialkin	ee2902ddaf	app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series ... Now trailing data points are additionally dropped for time series with a single raw sample Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-17 10:44:26 +03:00
Roman Khavronenko	d6155a3f33	vmalert: update docs to highlight the state restore requirements; (#833) ... Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830	2020-10-13 18:34:00 +03:00
Aliaksandr Valialkin	b9a4601c97	app/vmselect/promql: return a single time series at max from absent() function like Prometheus does	2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin	217c192c88	app/vmselect/promql: improve time series staleness detection ... This should prevent from double counting for time series at the time when it changes label. The most common case is in K8S, which changes pod uid label with each new deployment. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748	2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin	f877e703c8	app/vmselect/promql: fix mode_over_time calculations ... Previously `mode_over_time` could return garbage due to improper shuffling of input data points.	2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin	d884ab13dc	app/vmselect/prometheus: fix golangci-lint warning	2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin	0867dea5fc	app/vmselect: add ability to export data in CSV format via /api/v1/export/csv	2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin	938b3b7ed1	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin	7f96712b38	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin	2d03d0e2dd	app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning ... Functions are: * keep_last_value * keep_next_value * interpolate * running_min * running_max * running_avg * range_min * range_max * range_avg * range_first * range_last * range_quantile * smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	3881c84afe	Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" ... This reverts commit ac45082216. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric names: * ceil * floor * round Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	79d70480b7	Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" ... This reverts commit bb61a4769b. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * clamp_min * clamp_max Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	8c37b63ea9	Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" ... This reverts commit e5202a4eae. Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users - functions, which don't change the meaning of the time series shouldn't drop metric name. Now the following functions do not drop metric name: * max_over_time * min_over_time * avg_over_time * quantile_over_time * geomean_over_time * mode_over_time * holt_winters * predict_linear Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674	2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin	de1c07b937	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin	bf6d523bef	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin	9b7ce5d004	app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics	2020-10-09 13:38:49 +03:00
Aliaksandr Valialkin	d2e917d1cb	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin	4b1c401790	app/vmalert: accept days, weeks and years in for: part of config like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817	2020-10-08 20:13:20 +03:00
Aliaksandr Valialkin	35b8ffaa17	docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors command-line flag usage	2020-10-08 19:24:05 +03:00
Aliaksandr Valialkin	0d44e371f3	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin	f9f8e4a39c	app/vmalert: do not pring description for all the flags on config errors ... The description is too big to consume by human and it just distracts humans.	2020-10-08 13:35:46 +03:00
Aliaksandr Valialkin	f6ee6efc34	app/vmselect/promql: add missing label filters to binary operands before query execution ... This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization See also https://github.com/cortexproject/cortex/issues/3253	2020-10-07 21:17:11 +03:00
Dmitry Shihovtsev	aec863e70b	Fix typos in the vmalert datasource (#814) ... * Fix typos in the vmalert datasource * Fix typo in the vmalert datasource test	2020-10-07 18:00:29 +03:00
Aliaksandr Valialkin	b51fa16177	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin	c4d309aa41	app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page	2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin	abfd3a8fab	app/{vminsert,vmselect,vmstorage}: add a link to https://victoriametrics.github.io/Cluster-VictoriaMetrics.html from main page of every cluster component	2020-10-06 15:30:07 +03:00
Nikolay Khramchikhin	b4c77fc6d2	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin	3e0c473cc9	app/vmselect/promql: fill gaps on graphs for range_* and running_* functions ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806	2020-10-02 14:02:47 +03:00
Aliaksandr Valialkin	fd7dd5064a	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:12:04 +03:00
Roman Khavronenko	368b890e11	vmalert: make maxIdleConnections configurable for datasource HTTP client (#797) ... Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795	2020-09-30 09:51:14 +03:00
Aliaksandr Valialkin	536aa8779a	app/vmstorage: rename vm_{big|small}_merge_need_free_disk_space to vm_merge_need_free_disk_space ... This simplifies alerting.	2020-09-29 22:53:33 +03:00
Aliaksandr Valialkin	097a4c10dd	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin	5dca7bbe85	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin	61f4801b93	app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output	2020-09-29 18:03:45 +03:00
Aliaksandr Valialkin	f4d74ccfec	docs/{vmbackup,vmrestore}: formatting fixes	2020-09-29 13:19:13 +03:00
Aliaksandr Valialkin	5179df7082	docs/vmbackup.md: make docs about minio config more prominent	2020-09-29 13:16:34 +03:00
Aliaksandr Valialkin	79b00f7b6b	docs: improve readability a bit	2020-09-29 13:05:05 +03:00
Nikolay Khramchikhin	d77f56fd97	update vmbackup/vmrestore README usage (#794) ... * update vmbackup/vmrestore README usage https://github.com/VictoriaMetrics/VictoriaMetrics/issues/381 with minio and configuration file examples. * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * added backup/restore docs changes * added example for relabelConfig flag * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-29 13:05:04 +03:00
Aliaksandr Valialkin	0d082cdf53	app/vmselect/netstorage: remove unused code	2020-09-29 11:55:08 +03:00
Aliaksandr Valialkin	c47972d843	app/vmselect/prometheus: check for errors returned from bufferedwriter.Write ... This makes `make errcheck` happy	2020-09-29 11:36:54 +03:00
Aliaksandr Valialkin	e66f7edfc9	app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.* ... In this case only node must be returned with stripped dot in the end of id as carbonapi does	2020-09-29 11:29:38 +03:00
Aliaksandr Valialkin	ffa6581c46	app/vminsert: refresh the list of healthy storage nodes only if the the row cannot be sent to destination storage node ... Previously the list had been generated for each rerouted row. This could consume additional CPU time during rerouting, which could lead to rerouting slowdown. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791	2020-09-29 01:29:24 +03:00
Aliaksandr Valialkin	bc37f1cbec	app/vminsert: do not pollute logs with repated cannot dial storageNode errors ... Log only the first error per -storageNode	2020-09-29 00:20:32 +03:00
Aliaksandr Valialkin	2ee0dc27a6	app/vmstorage: parallelize data processing obtained from a single connection from vminsert ... Previously vmstorage could use only a single CPU core for data processing from a single connection from vminsert. Now all the CPU cores can be used for data processing from a single connection from vminsert. This should improve the maximum data ingestion performance for a single vminsert->vmstorage connection.	2020-09-28 21:41:16 +03:00
Aliaksandr Valialkin	9d123eb22a	app/vminsert: remove useless delays when sending data to vmstorage ... This improves the maximum data ingestion performance for cluster VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791	2020-09-28 21:41:15 +03:00
Aliaksandr Valialkin	8df33bd5c1	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin	6d8c23fdbd	app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native ... This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics	2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin	db14f22fc0	app/vmselect: stop /api/v1/export/* execution if client disconnects	2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin	aadbd014ff	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin	e83947a882	app/vminsert: code prettifying	2020-09-26 04:16:14 +03:00
Aliaksandr Valialkin	8fc9b77496	app/vmagent: reduce memory usage when importing data via /api/v1/import ... Previously vmagent could use big amounts of RAM when each ingested JSON line contained many samples.	2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin	973df09686	app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls	2020-09-24 20:44:15 +03:00
Aliaksandr Valialkin	b8bce348c5	app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784	2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin	543f3aea97	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin	8627365b48	app/vmselect/prometheus: code cleanup after 3ba507000c	2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin	1fce79518a	app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin	0468cdf33e	app/vmselect/netstorage: properly pre-allocate space for sbs	2020-09-22 23:51:01 +03:00
Aliaksandr Valialkin	69eb9783e6	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:21:42 +03:00
Aliaksandr Valialkin	ed473c94ff	docs/vmagent.md: typo fix	2020-09-21 21:49:08 +03:00
Aliaksandr Valialkin	e564725641	app/vmselect/searchutils: fixed tests after 2eb72e09ab	2020-09-21 21:31:28 +03:00
Aliaksandr Valialkin	07c6226334	app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:46 +03:00
Nikolay Khramchikhin	0069353d5e	Add improvements to ec2_sd_discovery (#775) ... * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin	8cd89cb847	app/vmalert: remove unneeded UTC() call ... UTC() doesn't change the underlying timestamp, so the call isn't needed here	2020-09-21 15:56:48 +03:00
Roman Khavronenko	d111969d39	vmalert: add support for datasource.lookback flag (#779) ... New datasource flag `datasource.lookback` defines how far to look into past when evaluating queries. Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668	2020-09-21 15:56:47 +03:00
Roman Khavronenko	0042b0f307	vmalert: fix the typo in error message (#782) ... The error will be always nil so no sense in printing it.	2020-09-21 11:36:09 +03:00
Aliaksandr Valialkin	1c07d7bee7	app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit	2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin	a1bebb660c	app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API	2020-09-18 11:53:28 +03:00
Aliaksandr Valialkin	9b15b11f74	app/vmstorage: added -forceMergeAuthKey command-line flag for protecting /internal/force_merge endpoint	2020-09-17 14:24:20 +03:00
Aliaksandr Valialkin	d96858b921	lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions ... This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin	406f4fe445	app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens ... Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin	9793008734	app/vmselect: add -search.storageTimeout command-line flag for limiting the maximum duration of query execution per each -storageNode ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:33:47 +03:00
Aliaksandr Valialkin	a9205fe308	app/vmselect: prevent from closing connection to vmstorage on query timeout by setting +2 secs deadline on connection comparing to query deadline ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-16 21:14:00 +03:00
Aliaksandr Valialkin	1587f83fa0	app/vmselect/netstorage: typo fix after 03dfccfbed	2020-09-16 00:10:33 +03:00
Aliaksandr Valialkin	03dfccfbed	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:09 +03:00
Aliaksandr Valialkin	27cd5555e6	app/vmselect/netstorage: mention that RunParallel or Cancel must be called on the returned results from ProcessSearchQuery	2020-09-15 20:39:43 +03:00
Aliaksandr Valialkin	1f0c0b0f6b	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:25:34 +03:00
Roman Khavronenko	e2b31590e6	vmalert: add Group name as label to generated alerts and timeseries (#761) ... Solves #611	2020-09-11 23:41:12 +03:00
Roman Khavronenko	16e0bb496e	vmalert: update groups on config reload only if changes detected (#759) ... On config reload event `vmalert` reloads configuration for every group. While it works for simple configurations, the more complex and heavy installations may suffer from frequent config reloads. The change introduces the `checksum` field for every group and is set to md5 hash of yaml configuration. The checksum will change if on any change to group definition like rules order or annotation change. Comparing the `checksum` field on config reload event helps to detect if group should be updated. The groups update is now done concurrently, so reload duration will be limited by the slowest group now. Partially solves #691 by improving config reload speed.	2020-09-11 23:41:12 +03:00
Aliaksandr Valialkin	b776a93608	app/vmselect/promql: support composite durations like Prometheus 2.21 does ... The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin	6382e8081a	app/vmagent: allow setting multiple identical -remoteWrite.url values ... This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755	2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin	d3ad0d365e	app/vmselect: move Deadline from netstorage to searchutils ... This removes dependency on netstorage from searchutils.	2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin	58d3b82ae5	app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:28:31 +03:00
Aliaksandr Valialkin	579c20756a	app/vmselect: substitute inf values at smooth_exponential with the previous values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin	d67e6d3d2e	app/vmselect: skip infinite values when calculating smooth_exponential ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:57:53 +03:00
Aliaksandr Valialkin	06427a184f	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:59:52 +03:00
Aliaksandr Valialkin	f307e6f432	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin	f5cb213ef9	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps ... This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin	475698d2ad	docs: sync docs for vmalert, vmauth, vmbackup and vmrestore	2020-09-09 21:10:48 +03:00
Aliaksandr Valialkin	5ab57f916b	docs/vmagent.md: clarified the case when -remoteWrite.queues must be tuned ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745	2020-09-08 20:15:49 +03:00
Aliaksandr Valialkin	f3a79abfb4	app/vmselect/promql: go fmt	2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin	4f06eed1c1	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin	0d0b606455	app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin	db91045348	app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3	2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin	804304c365	app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true	2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin	478d8f8393	app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions ... These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin	3490160fd0	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin	a3cdef6b06	app/vmagent: properly flush big blocks of data ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741 Thanks to @IceRain00 for the investigation and initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742	2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin	de216bab41	app/vmagent: fix data race when accessing writeRequest.lastFlushTime	2020-09-03 12:12:09 +03:00
Nikolay Khramchikhin	80a9dc79fe	changed vmalert behaviour (#738) ... * VMAlert start with empty rules dir There are some applications (operator for instance), that generates alerts configuration at runtime and vmalert must start correctly without rules to support this behaviour. Later application will add rules files and send SIGHUP to vmalert, which will trigger reading rules files and start rules exectuion. Removing rules files with SIGHUP signal must stop rules execution and vmalert will wait for new rules. * imports sorted * added test cases for empty rules, removed blank line * fixed imports conflict * updated tests	2020-09-03 11:07:40 +03:00
Aliaksandr Valialkin	7ac10ee978	app/vmalert: imrovements over 3f932c2db1	2020-09-03 01:14:30 +03:00
DexterZhang	85f49ad439	feat: spread load of rule evaluation by group when starting new groups (#724) ... * feat: spread load of rule evaluation by group when starting new groups * review: reduce the resulting diff. * Update app/vmalert/group.go Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-03 01:14:26 +03:00
Aliaksandr Valialkin	4fa97430d7	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats ... Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin	fe08b1eb26	app/vminsert: improve error message when the data cannot be sent to vmstorage - log reroutedBR buffer size ... This should improve debuggability for improperly configured cluster	2020-08-31 17:51:44 +03:00
Aliaksandr Valialkin	6f9c1bc078	app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url	2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin	3b1ecac04b	app/vmagent: apply sane limits to -remoteWrite.queues ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707	2020-08-30 21:25:51 +03:00
Roman Khavronenko	08b76cb26f	vmalert: update -rule flag description to enforce quotes using (#709) ... Description for `-rule` flag uses as example specific chars like asterisks which could be interpreted wrong by different shells. To avoid this, description now contains quoted flag values. See also #708	2020-08-28 09:46:35 +03:00
Roman Khavronenko	4b89da9463	lib/decimal: rename significant decimal digits to significant figures (#698) ... The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin	6aab2f4989	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin	285665e93b	app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)	2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin	a2021d0dde	docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion	2020-08-14 20:48:17 +03:00
Aliaksandr Valialkin	e7c0b2ca56	docs: update docs	2020-08-14 19:14:46 +03:00
Aliaksandr Valialkin	b996280c65	app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag	2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin	60c7397be5	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components ... Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin	6721e47ae9	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin	62b6e54622	app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin	c9f5c5623f	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores ... This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin	b3d4ff7ee2	app/vmstorage: improve error logging when the request times out	2020-08-10 13:17:24 +03:00
Roman Khavronenko	78afc61896	app/vmalert: extend metrics set exported by vmalert #573 (#654) ... * app/vmalert: extend metrics set exported by `vmalert` #573 New metrics were added to improve observability: + vmalert_alerts_pending{alertname, group} - number of pending alerts per group per alert; + vmalert_alerts_acitve{alertname, group} - number of active alerts per group per alert; + vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error during prev execution, is 0 if no errors happened; + vmalert_recording_rules_error{recording, group} - is 1 if recording rule ended up with error during prev execution, is 0 if no errors happened; * vmalert_iteration_total{group, file} - now contains group and file name labels. This should improve control over specific groups; * vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups; Some collisions for alerts and recording rules are possible, because neither group name nor alert/recording rule name are unique for compatibility reasons. Commit contains list of TODOs for Unregistering metrics since groups and rules are ephemeral and could be removed without application restart. In order to unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13 * app/vmalert: extend metrics set exported by `vmalert` #573 The changes are following: * add an ID label to rules metrics, since `name` collisions within one group is a common case - see the k8s example alerts; * supports metrics unregistering on rule updates. Consider the case when one rule was added or removed from the group, or the whole group was added or removed. The change depends on https://github.com/VictoriaMetrics/metrics/pull/16 where race condition for Unregister method was fixed.	2020-08-09 09:42:05 +03:00
ofen	3fea7c39be	401 Unauthorize HTTP error added (#681) ... 401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]	2020-08-09 09:39:37 +03:00
Aliaksandr Valialkin	67cacb22ac	lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary ... This makes such binaries compatible with binaries from `master` branch (aka single-node version) See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677	2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin	95a8c492ef	app/vmselect/promql: properly handle -n^m like Prometheus does ... `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin	7f93d61a56	app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin	01000505a0	app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus ... This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin	75bff1a567	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus ... Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin	8835004a4c	app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does ... This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin	14ddb8a34e	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block ... This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin	46c98cd97a	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin	a455930ab4	app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"} to vm_cache_entries{type="storage/prefetchedMetricIDs"} to be consistent with other vm_cache_entries metrics	2020-08-06 16:34:18 +03:00
Aliaksandr Valialkin	a3e91c593b	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 ... This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin	a930460236	app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663	2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin	a04f4a3d9a	app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration	2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin	bdb881c43b	app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...)	2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin	94471a1273	app: remove duplicate *-pure makefile rules	2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin	a2aa3a60eb	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin	106e302d7a	all: add mssing APP_NAME to vm*-GOARCH builds	2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin	945645f38f	docs/{vmagent,vmalert}: add instruction on how to build for ARM	2020-07-31 09:25:41 +03:00
Aliaksandr Valialkin	0c00fe70cf	app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin	29bbab0ec9	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected ... The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin	338ee47d60	app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window ... Just divide the data point value by the window duration in this case.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	717c554fb0	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now ... See e00cfc854d for the approach used now.	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	d9037b3970	app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty	2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin	f6d4275087	app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query ... Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:25:49 +03:00
Aliaksandr Valialkin	baebe86844	app/vmagent/remotewrite: add missing resp.Body.Close() after pushing data to remote storage ... Missing body close could disable HTTP keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin	0f6f0d30d3	app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query	2020-07-28 15:14:40 +03:00
Roman Khavronenko	ec6ed467c6	app/vmalert: support external.label to specify global labelset for all rules #622 (#652) ... `external.label` flag supposed to help to distinguish alert or recording rules source in situations when more than one `vmalert` runs for the same datasource or AlertManager.	2020-07-28 14:23:04 +03:00
Aliaksandr Valialkin	9dccedc599	app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp ... This aligns `group()` behaviour to Prometheus	2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin	d5057f6d04	app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage ... Previously the request body was already consumed before the retry, so this led to the following error: http: ContentLength=... with Body length 0	2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin	b191e425b3	app/vmselect/promql: improve further the accuracy of buckets_limit() function ... The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin	43871e79c6	app/vmselect/promql: avoid dropping inf bucket in buckets_limit ... The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin	978c1e930e	app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets	2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin	51cbf27077	app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function ... Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin	cf69b1ea6f	app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k ... This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin	45334f61de	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin	3526e8768a	app/vmselect/promql: typo fix after 3e557c9861	2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin	8d1721d128	app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration ... Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin	88e8bed0c9	app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time ... This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin	fb3d1380ac	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin	dbf3038637	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin	16a4b1b20c	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks	2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin	0750d2cec1	app/vminsert: export vm_relabel_metrics_dropped_total metric that shows the number of metrics dropped due to relabeling	2020-07-23 14:58:02 +03:00
Aliaksandr Valialkin	55ed07add1	app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g	2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin	7aa5b48508	app/vmselect: reduce memory usage when querying big number of time series with long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin	49a0011837	app/vminsert: do not call ApplyRelabeling function if relabeling is disabled ... This should reduce CPU usage a bit when `-relabelConfig` isn't set	2020-07-23 13:35:36 +03:00
Aliaksandr Valialkin	c91ccce50c	app/vminsert: fix relabeling for metrics ingested via Influx line protocol ... Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels if a single Influx line protocol message contains multiple field values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638	2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin	b8303afcd8	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin	20d0c41ac5	app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin	bd4299fafe	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries ... This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile	2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin	a3f48e395e	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin	5bb4fe1ba4	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin	0755cb3b50	app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value ... This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin	71eba8dcf5	app/vmselect: log the total available memory for concurrent requests on not enough memory errors ... This should simplify root cause analysis	2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin	3b246aa569	app/vmagent: add -remoteWrite.proxyURL command-line option ... This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy. This is similar to `proxy_url` option in `remote_write` section of Prometheus. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write	2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin	8bee3ef91b	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-20 17:09:30 +03:00
Roman Khavronenko	8949ec961d	app/vmagent: mention grafana dashboard in README (#639)	2020-07-20 17:09:27 +03:00
Aliaksandr Valialkin	86b54f3768	app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries	2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin	141e84b5a4	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range ... This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin	4d2011a87d	app/vmselect/promql: add mode() aggregate function	2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin	31ef39e8da	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin	427fa43ce2	app/vmselect/promql: add mode_over_time(m[d]) function ... See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin	eb402a17bd	app/vmselect/promql: optimize group(rollup(m)) calculations	2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin	ea8dc85ba8	app/vmselect/promql: check that any() doesn't touch metric name	2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin	fc8fe38a82	app/vmselect/promql: add group() aggregate function to MetricsQL ... This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin	c64914a7e4	app/vmselect/promql: keep all labels for time series from any() call	2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin	f9b38f7f2d	app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call ... Previously this could lead to `out of range` panic	2020-07-17 00:05:24 +03:00
Aliaksandr Valialkin	14dc426b45	app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage	2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin	ce381b3868	app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf when wrapping errors	2020-07-15 13:55:13 +03:00
Aliaksandr Valialkin	e6d96bb0bd	docs/vmagent.md: make filtering rules for init container pods less confusing	2020-07-14 20:33:19 +03:00
Aliaksandr Valialkin	c2b4b9138d	app/vmagent/remotewrite: return proper value from tssRelabelPool.New ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin	86044f6561	app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626	2020-07-14 14:18:40 +03:00
Aliaksandr Valialkin	0e7b2008b2	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time ... Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin	3898cc0285	app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9	2020-07-13 21:41:17 +03:00
faceair	bf39e67ade	fix empty response template (#617)	2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin	b6a5c29549	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-13 21:26:00 +03:00
ofen	9ffa688846	Update README.md (#621) ... Troubleshooting section updated to help out with duplicate targets detection	2020-07-13 21:25:59 +03:00
Aliaksandr Valialkin	4353ff7ef1	app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig options are set ... Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race and improper relabeling. Now each goroutine has its own copy of tss during relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-10 15:17:23 +03:00
Aliaksandr Valialkin	805a90f642	app/vmagent/remotewrite: typo fix in -remoteWrite.showURL help message	2020-07-10 14:07:14 +03:00
Aliaksandr Valialkin	6373d377ef	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin	d449d0a0e1	app/vmselect/promql: add missing tests for ifnot binary operation	2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin	7e706eea13	app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation	2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin	6c1a47b5e0	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin	fb86071552	app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528	2020-07-08 19:09:31 +03:00
DexterZhang	9930ce1fa9	Feat/query list vmselect (#575) ... * feat(vmselect): add support for listing current running queries and canceling specific query * fix(vmselect): change current queries' pid from int64 counter to uuid * feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth * fix(vmselect): add more info to current queries * review: delete some unnecessary code and use function instead of init * review: returen *queriesMap in newQueriesMap * review: delete unused var in struct queriesMap, add comments to exported functions * review: add return if error occurs * feat(vmselect): truncate query string in current running query list API since the size of query string might be large; use query string's pointer in struct `query` for the same reason; add query info API to get full access of query's info;	2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin	0bff96fe4b	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:44:04 +03:00
Roman Khavronenko	9afd19d375	app/vmalert: add retries to remotewrite (#605) ... * app/vmalert: add retries to remotewrite Remotewrite pkg now does limited number of retries if write request failed. This suppose to make vmalert state persisting more reliable. New metrics were added to remotewrite in order to track rows/bytes sent/dropped. defaultFlushInterval was increased from 1s to 5s for sanity reasons. * fix * wip * wip * wip * fix bits alignment bug for 32-bit systems * fix mistakenly dropped field	2020-07-05 18:47:38 +03:00
Aliaksandr Valialkin	82871fb7a5	app/vmselect/prometheus: small fixes on top of 8bb762124a	2020-07-05 18:17:53 +03:00
faceair	17f175ff5a	fix adjust last points avoid influence earlier value (#606)	2020-07-05 18:17:52 +03:00
Ween	d28fb0baf9	[VMAlert] Fix error log when remoteWrite queue size is full (#602) ... * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors * fix error logs when queue is full Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-07-03 16:50:43 +03:00
Aliaksandr Valialkin	8bb3622e9d	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin	6ebac3ab63	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin	a45856570b	all: typo fix: exptected -> expected	2020-07-02 18:06:21 +03:00
Aliaksandr Valialkin	f10e8809c0	app/vmselect: add interpolate function for filling gaps with linearly interpolated values ... See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin	2361ad8ab4	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:34 +03:00
BigFish	aa26b94f33	fix: spelling mistakes (#594) ... Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-07-01 01:36:40 +03:00
Aliaksandr Valialkin	4cb3e7595c	app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention	2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin	0c4e8aeb2b	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin	d962568e93	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:33:46 +03:00
Roman Khavronenko	156c83d112	app/vmalert: support multiple notifier urls (#584) (#590) ... * app/vmalert: support multiple notifier urls (#584) User now can set multiple notifier URLs in the same fashion as for other vmutils (e.g. vmagent). The same is correct for TLS setting for every configured URL. Alerts sending is done in sequential way for respecting the specified URLs order. * app/vmalert: add basicAuth support for notifier client (#585) The change adds possibility to set basicAuth creds for notifier client in the same fasion as for remote write/read and datasource.	2020-06-29 22:21:56 +03:00
Roman Khavronenko	bbeab70de6	app/vmalert: move flags description and initialization into subpackages ... The change adds no new functionality and aims to move flags definitions to subpackages that are using them. This should improve readability of the main function.	2020-06-29 22:18:29 +03:00
kreedom	63c36e2e69	app/vmalert: properly set transport for HTTP clients ... Fixes issue #586	2020-06-29 22:18:25 +03:00
Aliaksandr Valialkin	2b504f17de	docs: update the info that docker images are built on top of alpine image now ... A follow-up after the commit ff624c9125 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-26 13:52:25 +03:00
Aliaksandr Valialkin	a586b8b6d4	app/vminsert/netstorage: do not re-route every time series to more than two vmstorage nodes when certain vmstorage nodes are temporarily slower than the rest of them ... Previously vminsert may spread data for a single time series across all the available vmstorage nodes when vmstorage nodes couldn't handle the given ingestion rate. This could lead to increased usage of CPU and memory on every vmstorage node, since every vmstorage node had to register all the time series seen in the cluster. Now a time series may spread to maximum two vmstorage nodes under heavy load. Every time series is routed to a single vmstorage node under normal load.	2020-06-25 16:42:37 +03:00
Aliaksandr Valialkin	12b87b2088	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series ... This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:37:35 +03:00
nicbaz	46c5c0772c	vmselect: fix label_replace when mismatch (#579) ... As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the **MetricsQL** `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:54:29 +03:00
nicbaz	ea2ed4b7e8	vmalert: add support for TLS configuration (#578) ... app/vmalert: add support for TLS configuration Add support for TLS optional configuration in a similar fashion to what is currently supported in other vmutils such as vmagent. TLS configuration options are distinct for datasource, remoteRead, remoteWrite as well as notifier.	2020-06-23 22:47:23 +03:00
Aliaksandr Valialkin	0fdbe5de25	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series ... Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin	3a444bb7bb	lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs ... These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:19 +03:00
kreedom	f227799c87	Support of custom URL path for alert (#560) ... app/vmalert: Support custom URL for alerts source Add flag `external.alert.source` for configuring custom URL for alert's source. This may be handy to re-point default source URL to other systems like Grafana. Updates #517	2020-06-21 16:33:58 +03:00
Aliaksandr Valialkin	70bf8218bb	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin	2fc2679a3f	app/vminsert/netstorage: remove possible race condition when broken connection may be recovered before acquiring storageNode.bcLock	2020-06-20 16:38:08 +03:00
Aliaksandr Valialkin	9409a31c07	docs/vmauth.md: mention that we can provide custom integration with SAML	2020-06-19 13:13:53 +03:00
Aliaksandr Valialkin	4400700832	app/vminsert: properly replicate data for the last RF-1 storage nodes for -replicationFactor=RF ... Previously the data for the last `RF-1` storage noes has been incorrectly replicated to the first storage node.	2020-06-19 12:40:22 +03:00
Aliaksandr Valialkin	4f673a5201	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin	6939e36fdd	app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin	85c1ccb8b8	app/vminsert/netstorage: add missing return in storageNode.checkHealth on connection failure	2020-06-18 20:51:51 +03:00
Aliaksandr Valialkin	464682f380	app/vminsert/netstorage: periodically check for each -storageNode health, so it could be marked as healthy when it is ready to accept data ... This fixes uneven data routing in cluster version when `-replicationFactor` is set to 1 (default value), i.e. when the replication is disabled. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-18 20:42:43 +03:00
Roman Khavronenko	1a01fe2cf2	vmalert-537: allow name duplication for rules within one group. (#559) ... Uniqueness of rule is now defined by combination of its name, expression and labels. The hash of the combination is now used as rule ID and identifies rule within the group. Set of rules from coreos/kube-prometheus was added for testing purposes to verify compatibility. The check also showed that `vmalert` doesn't support `query` template function that was mentioned as limitation in README.	2020-06-18 18:54:35 +03:00
Aliaksandr Valialkin	87151e825e	docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible	2020-06-18 18:54:34 +03:00
Aliaksandr Valialkin	cc2225cc49	app/vmselect: fix the error after 936f35920a	2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin	936f35920a	app/vmselect/prometheus: allow returning partial response from /api/v1/export if -search.denyPartialResponse=false ... This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.	2020-06-12 21:11:48 +03:00
Clémence Saussez	0b53e380cf	app/vmalert: fix link to testdata (#547) ... Fix broken link to vmalert test data Signed-off-by: Clemence Saussez <clemence@zen.ly>	2020-06-10 19:37:21 +03:00
Roman Khavronenko	d71b6e6584	vmalert-491: allow to configure concurrent rules execution per group. (#542) ... The feature allows to speed up group rules execution by executing them concurrently. Change also contains README changes to reflect configuration details.	2020-06-09 15:22:11 +03:00
Roman Khavronenko	5c049bf4dd	vmalert-521: allow to disable rules expression validation. (#536) ... This feature may be useful for using `vmalert` with PromQL compatible datasources like Loki.	2020-06-09 15:19:25 +03:00
Aliaksandr Valialkin	c1be462d42	app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 20:14:07 +03:00
Aliaksandr Valialkin	7680b7155d	app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set	2020-06-05 20:14:05 +03:00
Aliaksandr Valialkin	01719f4949	app/vmstorage/transport: simplify setupTfss in order to prevent the possibility of nil tfs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534	2020-06-05 13:17:26 +03:00
Aliaksandr Valialkin	e4cef1b678	app/vmstorage: prevent from serving conns from vminsert and vmselect after the server is closed ... Previously it was possible that the connection is served after the server is closed if the following steps are performed: 1) Server accepts new connection. 2) Server.MustClose() is called and successfully finished. 3) Server starts processing the connection accepted at step 1. There could be various crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534 since the storage may be already closed. Now the server closes the connection at step 3 without processing it.	2020-06-05 11:55:48 +03:00
Aliaksandr Valialkin	58069f5a6a	app/vmalert: print brief usage info for vmalert -help	2020-06-05 10:43:24 +03:00
Aliaksandr Valialkin	3848ea3a4a	app/vmauth: print brief usage info for vmauth -help	2020-06-05 10:40:11 +03:00
Aliaksandr Valialkin	8ad8ca350a	app/vmagent: print brief usage info for vmagent -help	2020-06-05 10:40:10 +03:00
Aliaksandr Valialkin	3d0a0b3785	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:14:04 +03:00
DexterZhang	fa103875a0	feat(vmselect): add tmp block dir size metrics vm_tmp_blocks_files_size_total (#527) ... * feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total` * refactor(vmselect): use free space instead of used space in tmp block file metrics * fix: add `bytes` suffix to tmp dir free space metric	2020-06-04 13:05:50 +03:00
Aliaksandr Valialkin	faea804b88	app/vmauth: log when -auth.config is reloaded in SIGHUP	2020-06-03 23:22:20 +03:00
Aliaksandr Valialkin	045b87c662	app/vmalert: fix comment for UpdateWith exported methods	2020-06-01 14:35:03 +03:00
Aliaksandr Valialkin	43b14b9569	app/vminsert/netstorage: free up unused memory in buffer after memory usage spikes	2020-06-01 14:33:35 +03:00
Roman Khavronenko	44c51c627f	vmalert: Add recording rules support. (#519) ... * vmalert: Add recording rules support. Recording rules support required additional service refactoring since it wasn't planned to support them from the very beginning. The list of changes is following: * new entity RecordingRule was added for writing results of MetricsQL expressions into remote storage; * interface Rule now unites both recording and alerting rules; * configuration parser was moved to separate package and now performs more strict validation; * new endpoint for listing all groups and rules in json format was added; * evaluation interval may be set to every particular group; * vmalert: uncomment tests * vmalert: rm outdated TODO * vmalert: fix typos in README	2020-06-01 13:53:46 +03:00
Aliaksandr Valialkin	37aa4fe282	app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518	2020-05-30 14:37:02 +03:00
Aliaksandr Valialkin	a646131a33	app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent	2020-05-30 14:22:38 +03:00
Aliaksandr Valialkin	f41a01332a	app/vminsert/netstorage: evenly distribute rerouted rows among all the availalbe storage nodes ... Previously such rows were distributed to the original storage node or to the next storage node. This may result to uneven load among the remaining storage nodes.	2020-05-30 13:51:09 +03:00
Aliaksandr Valialkin	02b2064d8e	app/vminsert/netstorage: do not increment vm_rpc_rows_lost_total when all the vmstorage nodes are unavailable, since vminsert retries sending the data instead of dropping it	2020-05-28 22:36:56 +03:00
Aliaksandr Valialkin	7a61357b5d	app/vminsert/netstorage: make sure that the the data is always replicated among -replicationFactor vmstorage nodes ... Previously vminsert could write multiple copies of the data to a single vmstorage node when the ingestion rate exceeds the maximum throughput for connections to vmstorage nodes.	2020-05-28 19:59:07 +03:00
Aliaksandr Valialkin	77e5165e7b	app/vminsert: add -replicationFactor command-line flag for enabling data replication among available -storageNode instances	2020-05-27 17:29:44 +03:00
Aliaksandr Valialkin	b4e3bffe4b	app/vminsert/netstorage: emit warnings instead of errors when re-routing data to healthy storage nodes	2020-05-27 16:31:41 +03:00
Aliaksandr Valialkin	75f2f3b09d	app/vminsert/netstorage: improve ingestion performance when a single vmstorage node is slower than other vmstorage nodes ... Previously the ingestion performance has been limited by the slowest vmstorage node. Now vminsert should re-route data from the slowest vmstorage node to the remaining nodes.	2020-05-27 15:08:22 +03:00
Aliaksandr Valialkin	9844845d79	app/vminsert: tune the maximum summary buffer size for pending data to 1/4 of available RAM, since 1/2 of RAM is too big considering GOGC overhead	2020-05-25 02:00:37 +03:00
Aliaksandr Valialkin	4a82631e44	app/vminsert: limit the summary buffer sizes for all the storage nodes to a half of the allowed memory	2020-05-25 01:39:33 +03:00
Aliaksandr Valialkin	4bd3d4b148	app/vminsert/netstorage: do not return error from storageNode.flushBufLocked when the buffer has been successfully re-routed to healthy nodes ... This should reduce the number of false errors in the log and the number of falsely lost rows	2020-05-22 18:29:43 +03:00
Aliaksandr Valialkin	6edc33d9bb	app/vminsert/netstorage: capture the first error instead of the last error when sending data to vmstorage ... The first error has more chances to point to the real root cause of the issue.	2020-05-22 17:49:33 +03:00
Aliaksandr Valialkin	bb4a2bf1aa	app/vmauth: fix make run-vmauth command	2020-05-22 16:45:19 +03:00
Aliaksandr Valialkin	dcbdc009f5	app/vmagent: check for error returned from flag.Set	2020-05-21 16:30:48 +03:00
Aliaksandr Valialkin	b59e089ac7	app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:18 +03:00
Aliaksandr Valialkin	901093279e	app/vmstorage/transport: update stale comment - vmstorage now sends small ack packets to vminsert	2020-05-21 14:04:52 +03:00
kreedom	2752d6cb26	vmalert add quotes escape function (#510) ... * vmalert add quotes escape function Co-authored-by: kreedom	2020-05-21 12:10:35 +03:00
Aaron France	b26245c48b	Update README.md	2020-05-21 12:10:33 +03:00
Aliaksandr Valialkin	d83c68ca03	app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions ... These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin	8ff28f5b91	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin	ddc9e69bd6	docs/vmagent.md: mention an alternative to refresh_interval option in scrape configs	2020-05-19 23:10:16 +03:00
Aliaksandr Valialkin	7d46dd452a	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin	37068064dd	app/vmselect/promql: fix outilersk calculations	2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin	fc81ea38d4	app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin	9ca781b8f0	app/vmalert/notifier: go fmt	2020-05-19 13:00:18 +03:00
kreedom	27911ae179	vmalert - add expr to variables, add escape functions (#495) ... * vmalert - add expr to variables, add escape functions Co-authored-by: kreedom	2020-05-19 11:55:03 +03:00
Roman Khavronenko	c7f3e58032	vmalert: avoid sending resolves for pending alerts (#498) ... Before the change we were sending notifications to notifier if following conditions are met: * alert is in Fire state * alert is in Inactive state We were sending Inactive notifications to resolve alert ASAP. Unfortunately, we were sending resolves for Pending alerts that become Inactive, which is wrong. In this change we delete alert from the active list if it was Pending and become Inactive. In this way we now have Inactive alerts only if they were in state Fire before. See test change for example.	2020-05-19 11:55:00 +03:00
Roman Khavronenko	e5f5342e18	vmalert: fix potential race during configuration reloads (#497) ... Configuration reload and rules evaluation can't be executed in same time now. This may make reload time longer but prevents from potential races.	2020-05-19 11:54:55 +03:00
Aliaksandr Valialkin	b99d03a956	app/vmalert: run make quicktemplate-gen from the root dir of the repository	2020-05-16 22:45:45 +03:00
Aliaksandr Valialkin	2784015a4d	all: print --help output to stdout instead of stderr ... This is easier to grep and pipe	2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin	dbf8048134	app/vmrestore: document better that vmrestore works like rsync --delete, i.e. it deletes files in -storageDataPath, which are missing in the backup	2020-05-16 09:02:46 +03:00
Aliaksandr Valialkin	e544155a82	app/vmagent/Makefile: fix make run-vmagent rule	2020-05-15 19:35:16 +03:00
Aliaksandr Valialkin	6c43ba1cb1	app/vmagent/remotewrite: remove unused import after the commit 93267f143f	2020-05-15 17:42:31 +03:00
Aliaksandr Valialkin	1d71253653	app/vmagent/remotewrite: allow ingesting time series with multiple samples at once ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481	2020-05-15 17:37:27 +03:00
Aliaksandr Valialkin	a853869e75	app/vmstorage/transport: prevent from uncontrolled memory usage growth when vminsert sends big packets with too long labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/490	2020-05-15 15:42:54 +03:00
Aliaksandr Valialkin	1e5c1d7eaa	app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:12:24 +03:00
Aliaksandr Valialkin	d6b9a49481	app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:46:57 +03:00
Roman Khavronenko	e850bf0eff	vmalert: fix the access to rules slice element by wrong index (#486) ... During group's update rules deletion was causing slice mutations while slice index was assumed to be unchanged. This caused "slice bounds out of range" errors when multiple rules were deleted sequentially.	2020-05-15 13:26:06 +03:00
hagen1778	d369450f27	vmalert: update README	2020-05-15 13:26:04 +03:00
Aliaksandr Valialkin	3845420a8f	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:06:50 +03:00
Roman Khavronenko	e208e76222	vmalert: check if remoteRead object was initied before calling Restore (#473) ... The check for non-nil remoteRead was mistakenly dropped during refactoring which caused panics when `vmalert` wasn't configured with `remoteRead` flag.	2020-05-13 22:57:26 +03:00
Roman Khavronenko	1523890742	vmalert: fix flag names and description in README (#475) ... Change also adds the recommendation for `remotewrite` queue error.	2020-05-13 22:57:20 +03:00
肖贝贝	8c3e9adf7f	Feat/vmalert add max queue size (#472) ... * feat: add remoteWrite.maxQueueSize to reduce queue full * rename remote(write|read) flags to remote(Write|Read) for the sake of consistency Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-13 22:57:16 +03:00
Aliaksandr Valialkin	bac9a684e8	docs/vmbackup.md: add a link to vmbackuper tool	2020-05-13 22:57:11 +03:00
Aliaksandr Valialkin	f3d9a5b0ec	app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint	2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin	3b0f66a227	app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set ... This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:03:45 +03:00
Aliaksandr Valialkin	18a0caee43	app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one	2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin	3d3f41b961	app/vmstorage/transport: fix panic during server stop on 32-bit arches ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2020-05-12 20:21:40 +03:00
Aliaksandr Valialkin	81b8811cf4	app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag ... Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin	408ade27a9	app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y	2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin	21c2982ac8	app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation	2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin	f341c6fcc4	Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors" ... This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c. Reason for revert: this commit has been removed from single-node version	2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin	d54a93fc81	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:43 +03:00
Aliaksandr Valialkin	405cf44aed	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:53 +03:00
Aliaksandr Valialkin	da6a84e147	app/vmagent/remotewrite: properly dial TCP6 addresses set via -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469	2020-05-12 15:24:50 +03:00
Aliaksandr Valialkin	4e237b4670	app/vminsert/influx: support passing AccountID and ProjectID via plain TCP and UDP ... Now `vminsert` accepts AccountID and ProjectID via `VictoriaMetrics_AccountID` and `VictoriaMetrics_ProjectID` tags when reading Influx line protocol data via plain TCP or UDP (i.e. when `-influxListenAddr` is set).	2020-05-12 13:13:04 +03:00
Aliaksandr Valialkin	f7753b1469	lib/storage: gradually pre-populate per-day inverted index for the next day ... This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:32 +03:00
Roman Khavronenko	0157566fdb	vmalert: cleanup and restructure of code to improve maintainability (#471) ... The change introduces new entity `manager` which replaces `watchdog`, decouples requestHandler and groups. Manager supposed to control life cycle of groups, rules and config reloads. Groups export an ID method which returns a hash from filename and group name. ID supposed to be unique identifier across all loaded groups. Some tests were added to improve coverage. Bug with wrong annotation value if $value is used in templates after metrics being restored fixed. Notifier interface was extended to accept context. New set of metrics was introduced for config reload.	2020-05-11 14:35:55 +03:00
Nikolay Khramchikhin	0e8c345ffb	vmalert config reload ... added config hot reload for vmalert with sighup and api call	2020-05-11 14:35:50 +03:00