github-mirrors/VictoriaMetrics

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

Author	SHA1	Message	Date
Aliaksandr Valialkin	3b1e3a03e0	app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711	2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin	94f7d00537	docs/vmagent.md: typo fix	2020-09-21 21:49:22 +03:00
Aliaksandr Valialkin	00b5145c69	app/vmselect/searchutils: fixed tests after `2eb72e09ab`	2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin	2eb72e09ab	app/vmselect: use `time` value rounded to seconds if it isnt passed to `/api/v1/query` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720	2020-09-21 21:24:40 +03:00
Nikolay Khramchikhin	312fead9a2	Add improvements to ec2_sd_discovery (#775 ) * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin	1e1a27d803	app/vmalert: remove unneeded UTC() call UTC() doesn't change the underlying timestamp, so the call isn't needed here	2020-09-21 15:55:59 +03:00
Roman Khavronenko	5dffc7a553	vmalert: add support for `datasource.lookback` flag (#779 ) New datasource flag `datasource.lookback` defines how far to look into past when evaluating queries. Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668	2020-09-21 15:53:49 +03:00
Roman Khavronenko	82c3bbce34	vmalert: fix the typo in error message (#782 ) The error will be always nil so no sense in printing it.	2020-09-21 11:34:23 +03:00
Aliaksandr Valialkin	d50165ad59	app/vmagent: increase default value for `-remoteWrite.queues` from 1 to 4, since it has been appeared that many users hit this limit	2020-09-18 14:21:54 +03:00
Aliaksandr Valialkin	98d1cd0971	app/vmselect/graphite: return proper results `/metrics/find?query=foo.*.bar` according to Graphite Metrics API	2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin	7a134b0fd7	app/vmstorage: added `-forceMergeAuthKey` command-line flag for protecting `/internal/force_merge` endpoint	2020-09-17 14:21:53 +03:00
Aliaksandr Valialkin	1f33dd717f	lib/storage: add `/internal/force_merge` handler for running forced compactions on historical per-month partitions This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin	ab53cb6f7b	app/vmagent: substitute `-remoteWrite.url` with `secret-url` value in logs, since it may contain sensitive info such as passwords or auth tokens Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin	9f79bcf64a	app/vmselect: improve description for -search.maxQueryDuration	2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin	1fec47a289	app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series	2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin	8c3d7c1a59	app/vmselect: typo fix in -search.maxStalenessInterval description	2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin	0e533d1a9c	app/vmselect/promql: support composite durations like Prometheus 2.21 does The following durations are supported now: `1h5m35s` or `1s543ms` See https://github.com/prometheus/prometheus/releases/tag/v2.21.0 and https://github.com/prometheus/prometheus/pull/7713	2020-09-11 23:39:13 +03:00
Roman Khavronenko	6ad6480400	vmalert: add Group name as label to generated alerts and timeseries (#761 ) Solves #611	2020-09-11 20:52:56 +01:00
Roman Khavronenko	4cdffb04a4	vmalert: update groups on config reload only if changes detected (#759 ) On config reload event `vmalert` reloads configuration for every group. While it works for simple configurations, the more complex and heavy installations may suffer from frequent config reloads. The change introduces the `checksum` field for every group and is set to md5 hash of yaml configuration. The checksum will change if on any change to group definition like rules order or annotation change. Comparing the `checksum` field on config reload event helps to detect if group should be updated. The groups update is now done concurrently, so reload duration will be limited by the slowest group now. Partially solves #691 by improving config reload speed.	2020-09-11 20:14:30 +01:00
Aliaksandr Valialkin	ca856284e4	app/vmagent: allow setting multiple identical `-remoteWrite.url` values This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755	2020-09-11 15:17:22 +03:00
Aliaksandr Valialkin	a2f647d142	app/vmselect/prometheus: typo fix in the description for `-search.latencyOffset` command-line flag	2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin	2380e9b017	app/{vminsert,vmagent}: allow passing timestamp via `timestamp` query arg when ingesting data to `/api/v1/import/prometheus` See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:27:14 +03:00
Aliaksandr Valialkin	f0005c3007	app/vmselect: move Deadline from netstorage to searchutils This removes dependency on netstorage from searchutils.	2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin	2114179e19	app/vmselect: substitute inf values at smooth_exponential with the previous values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin	204ec415b4	app/vmselect: skip infinite values when calculating smooth_exponential Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757	2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin	8a8b5a73d3	app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric	2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin	f6bc608e86	app/vmselect: initial implementation of Graphite Metrics API See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin	9d8fdff6c5	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin	d7c04db1fc	docs: sync docs for vmalert, vmauth, vmbackup and vmrestore	2020-09-09 21:10:34 +03:00
Aliaksandr Valialkin	11eaa37111	docs/vmagent.md: clarified the case when `-remoteWrite.queues` must be tuned Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745	2020-09-08 20:15:27 +03:00
Aliaksandr Valialkin	62919eaf7e	app/vmselect/promql: go fmt	2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin	e6da63dffe	app/vmselect/promql: adjust `integrate()` calculations to be more similar to calculations from InfluxDB: attempt #2 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin	8e85b56737	app/vmselect/promql: adjust `integrate()` calculations to be more similar to calculations from InfluxDB Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701	2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin	c0343a661b	app/vmselect/promql: increase floating point calculations accuracy by dividing by `1e3` instead of multiplying by `1e-3`	2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin	f4e7e5fb90	app/vmselect/promql: add `count_le_over_time(m[d], le)` and `count_gt_over_time(m[d], gt)` functions These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`. These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.	2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin	e706e59d49	app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does	2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin	ddabc13796	app/vmagent: properly flush big blocks of data Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741 Thanks to @IceRain00 for the investigation and initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742	2020-09-03 12:12:39 +03:00
Aliaksandr Valialkin	7a839b461f	app/vmagent: fix data race when accessing writeRequest.lastFlushTime	2020-09-03 12:12:37 +03:00
Nikolay Khramchikhin	764b3d4fda	changed vmalert behaviour (#738 ) * VMAlert start with empty rules dir There are some applications (operator for instance), that generates alerts configuration at runtime and vmalert must start correctly without rules to support this behaviour. Later application will add rules files and send SIGHUP to vmalert, which will trigger reading rules files and start rules exectuion. Removing rules files with SIGHUP signal must stop rules execution and vmalert will wait for new rules. * imports sorted * added test cases for empty rules, removed blank line * fixed imports conflict * updated tests	2020-09-03 11:04:42 +03:00
Aliaksandr Valialkin	5f16ceb294	app/vmalert: imrovements over `3f932c2db1`	2020-09-03 01:00:55 +03:00
DexterZhang	3f932c2db1	feat: spread load of rule evaluation by group when starting new groups (#724 ) * feat: spread load of rule evaluation by group when starting new groups * review: reduce the resulting diff. * Update app/vmalert/group.go Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-03 00:58:54 +03:00
Aliaksandr Valialkin	f41b36bb9a	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:43:21 +03:00
Aliaksandr Valialkin	5af777469a	app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url	2020-08-30 21:40:22 +03:00
Aliaksandr Valialkin	2149733bd2	app/vmagent: apply sane limits to `-remoteWrite.queues` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707	2020-08-30 21:25:37 +03:00
Roman Khavronenko	10601bc652	vmalert: update `-rule` flag description to enforce quotes using (#709 ) Description for `-rule` flag uses as example specific chars like asterisks which could be interpreted wrong by different shells. To avoid this, description now contains quoted flag values. See also #708	2020-08-20 22:36:38 +01:00
Roman Khavronenko	f5f59896ec	lib/decimal: rename `significant decimal digits` to `significant figures` (#698 ) The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:21:35 +03:00
Aliaksandr Valialkin	147c35ebd4	all: allow using `KB`, `MB`, `GB`, `KiB`, `MiB` and `GiB` suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin	7d89fafe1a	app/vmselect/promql: allow passing multiple args to aggregate functions such as `avg(q1, q2, q3)`	2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin	cd96248480	docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion	2020-08-14 20:47:57 +03:00
Aliaksandr Valialkin	184670fb9b	docs: update docs	2020-08-14 19:13:42 +03:00
Aliaksandr Valialkin	576da0fe46	app/{vminsert,vmagent}: improve documentation for `-influxListenAddr` command-line flag	2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin	c402265e88	all: support `%{ENV_VAR}` placeholders in yaml configs in all the vm* components Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:15:25 +03:00
Aliaksandr Valialkin	ef7e2af8f5	app: respect CPU limits set via cgroups Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 22:59:19 +03:00
Aliaksandr Valialkin	e969ef2639	app/vmselect: reduce memory usage when exporting time series with big number of samples via `/api/v1/export` if `max_rows_per_line` is set to non-zero value Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin	8adba82c02	app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores This should reduce contention on the channel with unpack work for systems with high number of CPU cores	2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin	f3d33e23c9	app/vmstorage: improve error logging when the request times out	2020-08-10 13:23:26 +03:00
Roman Khavronenko	0be5b09fb4	app/vmalert: extend metrics set exported by `vmalert` #573 (#654 ) * app/vmalert: extend metrics set exported by `vmalert` #573 New metrics were added to improve observability: + vmalert_alerts_pending{alertname, group} - number of pending alerts per group per alert; + vmalert_alerts_acitve{alertname, group} - number of active alerts per group per alert; + vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error during prev execution, is 0 if no errors happened; + vmalert_recording_rules_error{recording, group} - is 1 if recording rule ended up with error during prev execution, is 0 if no errors happened; * vmalert_iteration_total{group, file} - now contains group and file name labels. This should improve control over specific groups; * vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups; Some collisions for alerts and recording rules are possible, because neither group name nor alert/recording rule name are unique for compatibility reasons. Commit contains list of TODOs for Unregistering metrics since groups and rules are ephemeral and could be removed without application restart. In order to unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13 * app/vmalert: extend metrics set exported by `vmalert` #573 The changes are following: * add an ID label to rules metrics, since `name` collisions within one group is a common case - see the k8s example alerts; * supports metrics unregistering on rule updates. Consider the case when one rule was added or removed from the group, or the whole group was added or removed. The change depends on https://github.com/VictoriaMetrics/metrics/pull/16 where race condition for Unregister method was fixed.	2020-08-09 09:41:29 +03:00
ofen	81746d14b9	401 Unauthorize HTTP error added (#681 ) 401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]	2020-08-09 09:38:41 +03:00
Aliaksandr Valialkin	716754fae6	app/vmselect/promql: properly handle `-n^m` like Prometheus does `-n^m` must be handled as `-(n^m)` instead of `(-n)^m`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675	2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin	bb61a4769b	app/vmselect/promql: remove metric name after applying `clamp_min` and `clamp_max` functions in order to be consistent with Prometheus This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin	ac45082216	app/vmselect/promql: remove metric name after applying `ceil`, `floor` and `round` functions in order to be more consistent with Prometheus This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin	e5202a4eae	app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus Rollup functions: - avg_over_time - min_over_time - max_over_time - quantile_over_time This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin	68e4f40a72	app/vmselect: properly handle PromQL queries like `scalar1 < metric < scalar2` like Prometheus does This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/	2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin	bc8381613d	app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names	2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin	3f85c06b65	app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block This should improve query performance on systems with big number of CPU cores (16 and more)	2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin	d20c2156e4	app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans	2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin	dbbdfbe7ee	app/vmstorage: rename `vm_cache_size_entries{type="storage/prefetchedMetricIDs"}` to `vm_cache_entries{type="storage/prefetchedMetricIDs"}` to be consistent with other `vm_cache_entries` metrics	2020-08-06 16:34:24 +03:00
Aliaksandr Valialkin	8f16388428	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin	c25b0c2cd5	app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663	2020-08-04 21:00:29 +03:00
Aliaksandr Valialkin	5d0c37bec0	app/vmselect: use warning level instead of info level for logging slow queries that take longer than `-search.logSlowQueryDuration`	2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin	e6eee2bebf	app/vmselect/promql: add zscore-related functions: `zscore_over_time(m[d])` and `zscore(q) by (...)`	2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin	509d12643b	app/vmselect: show `X-Forwarded-For` contents on `/api/v1/status/active_queries` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin	d01f3c1943	all: add mssing APP_NAME to vm*-GOARCH builds	2020-07-31 13:42:18 +03:00
Aliaksandr Valialkin	3f498cf2dc	docs/{vmagent,vmalert}: add instruction on how to build for ARM	2020-07-31 09:25:22 +03:00
Aliaksandr Valialkin	f0c678c41b	app/vmselect: do not adjust `start` and `end` query args passed to `/api/v1/query_range` when `-search.disableCache` command-line flag is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563	2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin	e7959094f6	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin	328b52e5ff	app/vmselect/promql: return non-empty value from `rate_over_sum(m[d])` even if a single data point is located in the given `[d]` window Just divide the data point value by the window duration in this case.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	700737c181	app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in `increase()` is handled in another way now See `e00cfc854d` for the approach used now.	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	2f735f112d	app/vmselect/promql: fill gaps with 0 in `rate_over_sum` response when the last value before the selected time window isnt empty	2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin	0f63da3698	app/{vmagent,vminsert}: properly preserve `db` tag from query string passed to Influx line protocol query Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:25:19 +03:00
Aliaksandr Valialkin	62ed38c6f0	app/vmagent/remotewrite: add missing `resp.Body.Close()` after pushing data to remote storage Missing body close could disable HTTP keep-alive connections. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653	2020-07-28 21:00:15 +03:00
Aliaksandr Valialkin	79c30cf4cb	app/vmselect: show query origin (aka remote_addr or client address) on the `/api/v1/status/active_queries` page for every query	2020-07-28 15:13:08 +03:00
Roman Khavronenko	2f1e7298ce	app/vmalert: support `external.label` to specify global labelset for all rules #622 (#652 ) `external.label` flag supposed to help to distinguish alert or recording rules source in situations when more than one `vmalert` runs for the same datasource or AlertManager.	2020-07-28 14:20:31 +03:00
Aliaksandr Valialkin	0da202023b	app/vmselect/promql: return empty values from `group()` if all the time series have no values at the given timestamp This aligns `group()` behaviour to Prometheus	2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin	cb8c6908dc	app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage Previously the request body was already consumed before the retry, so this led to the following error: http: ContentLength=... with Body length 0	2020-07-27 17:32:46 +03:00
Aliaksandr Valialkin	894dcb7c1c	app/vmselect/promql: improve further the accuracy of buckets_limit() function The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.	2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin	215eba0b82	app/vminsert: flush bufs if needed after the current row is added Previously the data for the added row could be overwritten by the flush before the row addition is complete.	2020-07-26 12:10:11 +03:00
Aliaksandr Valialkin	edb1eca6f1	app/vmselect/promql: avoid dropping `inf` bucket in `buckets_limit` The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.	2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin	97b6f5d223	app/vmselect/promql: optimize `buckets_limit(k, buckets)` for big number of buckets	2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin	a090627059	app/vminsert: limit memory usage when ingesting data in big packets	2020-07-24 23:32:40 +03:00
Aliaksandr Valialkin	bb161497cf	app/vmselect/promql: improve the accuracy of `buckets_limit(k, buckets)` function Now it properly merges the bucket with the previous bucket after deletion.	2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin	994fa2f3bf	app/vmselect/promql: add `buckets_limit(k, buckets)` function, which limits the number of buckets per time series to `k` This function works with both Prometheus-style and VictoriaMetrics-style buckets. The function removes buckets with the lowest values in order to reserve the highest precision. The function is useful for building heatmaps in Grafana from too big number of buckets.	2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin	e151c5c644	app/vmselect: fix tests for rate_over_sum	2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin	3107c633e3	app/vmselect/promql: typo fix after `3e557c9861`	2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin	3e557c9861	app/vmselect/promql: add `rate_over_sum(m[d])` function to MetricsQL, which returns rate over sum of `m` values over `d` duration Something like `sum_over_time(m[d]) / d`, but more accurate.	2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin	b1f6843bd0	app/vmselect/promql: allow setting `[d]` window smaller than the interval between raw points for `avg_over_time` This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636	2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin	039c9d2441	lib/storage: respect `-search.maxQueryDuration` when searching for time series in inverted index Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin	2a45871823	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin	461481fbdf	app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery	2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin	4c8b49b193	app/vminsert: export `vm_relabel_metrics_dropped_total` metric that shows the number of metrics dropped due to relabeling	2020-07-23 14:57:53 +03:00
Aliaksandr Valialkin	e79de9774b	app/vmselect: typo fix after `34563916f7`	2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin	34563916f7	app/vmselect: reduce memory usage when querying big number of time series with long labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646	2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin	9257eee982	app/vminsert: do not call ApplyRelabeling function if relabeling is disabled This should reduce CPU usage a bit when `-relabelConfig` isn't set	2020-07-23 13:39:44 +03:00
Aliaksandr Valialkin	6f05c4d351	lib/storage: improve prioritizing of data ingestion over querying Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin	2f612e0c67	app/vminsert: fix relabeling for metrics ingested via Influx line protocol Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels if a single Influx line protocol message contains multiple field values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638	2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin	4f526cc816	app/vmselect/prometheus: support `d`, `w` and `y` suffixes for durations passed to `step` in `/api/v1/query_range` like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641	2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin	dfb113f175	app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries This should slightly reduce load on GC when processing queries that touch big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.	2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin	31ae5911a8	app/vmagent: add `-remoteWrite.decimalPlaces` command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin	caa2952aa6	app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query	2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin	e00cfc854d	app/vmselect/promql: skip the first value in time series passed to `increase()` if it exceeds by more than 10x the delta between the next value and the first value This should prvent from inflated `increase()` results for time series that start from big initial values. Such cases may occur when a label value changes in a metric without counter reset.	2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin	b9c8f6bf34	app/vmselect: log the total available memory for concurrent requests on `not enough memory` errors This should simplify root cause analysis	2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin	ad6290953c	app/vmagent: add `-remoteWrite.proxyURL` command-line option This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy. This is similar to `proxy_url` option in `remote_write` section of Prometheus. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write	2020-07-20 19:28:49 +03:00
Aliaksandr Valialkin	efcbb51968	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-20 17:08:34 +03:00
Roman Khavronenko	ed0df37ee7	app/vmagent: mention grafana dashboard in README (#639 )	2020-07-20 17:07:27 +03:00
Aliaksandr Valialkin	11be704109	app/vmagent/remotewrite: allow passing empty `-remoteWrite.urlRelabelConfig` entries	2020-07-20 15:49:27 +03:00
Aliaksandr Valialkin	5a4675c528	app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range This matches Prometheus behaviour. This should fix https://github.com/jacksontj/promxy/issues/329	2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin	ecb1b2564a	app/vmselect/promql: add `mode()` aggregate function	2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin	b35cb293f5	lib/httpserver: log remote address in error message from `httpserver.Errorf` This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin	1c641037e8	app/vmselect/promql: add `mode_over_time(m[d])` function See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of	2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin	6b5ad535ae	app/vmselect/promql: optimize `group(rollup(m))` calculations	2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin	8949d65ad1	app/vmselect/promql: check that `any()` doesn't touch metric name	2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin	aa5d88055d	app/vmselect/promql: add `group()` aggregate function to MetricsQL This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480	2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin	df01836818	app/vmselect/promql: keep all labels for time series from `any()` call	2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin	8c14ca93fa	app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call Previously this could lead to `out of range` panic	2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin	e4e1cd1de2	app/vmselect: fix `nil pointer dereference` panic when unsuccessfully querying `vmstorage`	2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin	ed7580ad22	app/vmalert: consistently use "%w" instead of "%s" in `fmt.Errorf` when wrapping errors	2020-07-15 13:56:47 +03:00
Aliaksandr Valialkin	328814ee60	docs/vmagent.md: make filtering rules for init container pods less confusing	2020-07-14 20:32:47 +03:00
Aliaksandr Valialkin	b442a42d8e	app/vmagent/remotewrite: return proper value from `tssRelabelPool.New` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-14 14:29:20 +03:00
Aliaksandr Valialkin	4081e2295e	app/{vminsert,vmagent}: add `-influxSkipMeasurement` command-line flag for using field name as metric name See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626	2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin	25f80d320b	app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time Such timestamps usually mean that the query contains `offset`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625	2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin	55d83e777d	app/vmselect/prometheus: minimize the diff for the change `1033dc7e2a` over `619b0a25c9`	2020-07-13 21:40:38 +03:00
faceair	1033dc7e2a	fix empty response template (#617 )	2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin	619b0a25c9	docs/vmagent.md: sync with app/vmagent/README.md	2020-07-13 21:25:11 +03:00
ofen	666c795b98	Update README.md (#621 ) Troubleshooting section updated to help out with duplicate targets detection	2020-07-13 21:18:54 +03:00
Aliaksandr Valialkin	a730b3f6a1	app/vmagent: fix data race when multiple `-remoteWrite.urlRelabelConfig` options are set Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race and improper relabeling. Now each goroutine has its own copy of tss during relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599	2020-07-10 15:16:59 +03:00
Aliaksandr Valialkin	508ad46e0e	app/vmagent/remotewrite: typo fix in `-remoteWrite.showURL` help message	2020-07-10 14:07:08 +03:00
Aliaksandr Valialkin	cba820e390	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via `/api/v1/import/prometheus`	2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin	9c350bc20d	app/vmselect/promql: add missing tests for `ifnot` binary operation	2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin	256fd9a87e	app/vmselect/promql: refactor implementations for `and` and `unless` binary operations, so they are closer to `or` implementation	2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin	2d9b3ad5b3	app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function	2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin	742da690f4	app/vmselect: add `/api/v1/status/active_queries` page with the list of currently running queries This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575	2020-07-08 18:55:38 +03:00
DexterZhang	99f54e44ff	feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598 )	2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin	6daa5f7500	lib/storage: prioritize data ingestion over heavy queries Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:42:05 +03:00
Roman Khavronenko	703def4b2e	app/vmalert: add retries to remotewrite (#605 ) * app/vmalert: add retries to remotewrite Remotewrite pkg now does limited number of retries if write request failed. This suppose to make vmalert state persisting more reliable. New metrics were added to remotewrite in order to track rows/bytes sent/dropped. defaultFlushInterval was increased from 1s to 5s for sanity reasons. * fix * wip * wip * wip * fix bits alignment bug for 32-bit systems * fix mistakenly dropped field	2020-07-05 18:46:52 +03:00
Aliaksandr Valialkin	de137aef98	app/victoria-metrics: fix tests after the commit `acf828a759`	2020-07-05 18:24:41 +03:00
Aliaksandr Valialkin	acf828a759	app/vmselect/prometheus: small fixes on top of `8bb762124a`	2020-07-05 18:17:06 +03:00
faceair	8bb762124a	fix adjust last points avoid influence earlier value (#606 )	2020-07-05 17:56:54 +03:00
Ween	d59cdbe90c	[VMAlert] Fix error log when remoteWrite queue size is full (#602 ) * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors * fix error logs when queue is full Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-07-03 16:49:37 +03:00
Aliaksandr Valialkin	0b2086b7a5	app/vminsert: prevent from adding and/or selecting labels with empty values Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin	8f628cd805	app/victoria-metrics: removed debug log message when `-selfScrapeInterval` is set	2020-07-02 20:39:41 +03:00
Aliaksandr Valialkin	91b3482894	app/vminsert: add ability to apply relabeling to all the incoming metrics if `-relabelConfig` command-line arg points to a file with a list of `relabel_config` entries See https://victoriametrics.github.io/#relabeling	2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin	e5500bfcf2	all: typo fix: exptected -> expected	2020-07-02 18:05:52 +03:00
Aliaksandr Valialkin	5d3db3ff7c	app/vmselect: add `interpolate` function for filling gaps with linearly interpolated values See https://stackoverflow.com/q/62565021/274937 for details	2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin	4dd3de9286	lib/promscrape: add ability to set `disable_compression` and `disable_keepalive` options in `scrape_config` section of the config passed to `-promscrape.config` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:14 +03:00
BigFish	9d5f5b6878	fix: spelling mistakes (#594 ) Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-07-01 01:35:26 +03:00
Aliaksandr Valialkin	84a37098ed	app/vmstorage: add `-denyQueriesOutsideRetention` command-line flag for denying queries outside the configured retention VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention if `-denyQueriesOutsideRetention` command-line flag is set.	2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin	7c2c8b2981	all: use `errors.As` for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in `fmt.Errorf` This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Roman Khavronenko	88538df267	app/vmalert: support multiple notifier urls (#584 ) (#590 ) * app/vmalert: support multiple notifier urls (#584) User now can set multiple notifier URLs in the same fashion as for other vmutils (e.g. vmagent). The same is correct for TLS setting for every configured URL. Alerts sending is done in sequential way for respecting the specified URLs order. * app/vmalert: add basicAuth support for notifier client (#585) The change adds possibility to set basicAuth creds for notifier client in the same fasion as for remote write/read and datasource.	2020-06-29 22:21:03 +03:00
Roman Khavronenko	82ecfa3b32	app/vmalert: move flags description and initialization into subpackages The change adds no new functionality and aims to move flags definitions to subpackages that are using them. This should improve readability of the main function.	2020-06-28 12:26:22 +01:00
kreedom	dc4e3f0e0b	app/vmalert: properly set transport for HTTP clients Fixes issue #586	2020-06-27 08:31:54 +01:00
Aliaksandr Valialkin	8f2e88234f	docs: update the info that docker images are built on top of `alpine` image now A follow-up after the commit `ff624c9125` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522	2020-06-26 13:54:10 +03:00
Aliaksandr Valialkin	05d754d7bb	app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series This should reduce GC pressure when processing time series with big number of rows	2020-06-24 19:38:39 +03:00
nicbaz	774f7ca1c1	vmselect: fix label_replace when mismatch (#579 ) As per documentation on `label_replace` function: "If the regular expression doesn't match then the timeseries is returned unchanged". Currently this behavior is not enforced, if a regexp on an existing tag doesn't match then the tag value is copied as-is in the destination tag. This fix first checks that the regular expression matches the source tag before applying anything. Given the current implementation, this fix also changes the behavior of the MetricsQL `label_transform` function which does not document this behavior at the moment.	2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin	7209d58fbd	app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series Previously VictoriaMetrics was processing up to 32 time series in a single goroutine. This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work, while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing. This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572	2020-06-23 22:46:15 +03:00
nicbaz	72c90bfd8b	vmalert: add support for TLS configuration (#578 ) app/vmalert: add support for TLS configuration Add support for TLS optional configuration in a similar fashion to what is currently supported in other vmutils such as vmagent. TLS configuration options are distinct for datasource, remoteRead, remoteWrite as well as notifier.	2020-06-23 20:45:45 +01:00
Aliaksandr Valialkin	2a39ba639d	lib/promrelabel: add support for `keep_if_equal` and `drop_if_equal` actions to relabel configs These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin	7532dbcdf5	app/vmselect/promql: properly override label values from `group_left` and `group_right` lists like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:33:01 +03:00
kreedom	7ec6711f06	Support of custom URL path for alert (#560 ) app/vmalert: Support custom URL for alerts source Add flag `external.alert.source` for configuring custom URL for alert's source. This may be handy to re-point default source URL to other systems like Grafana. Updates #517	2020-06-21 11:32:46 +01:00
Aliaksandr Valialkin	9f3e3a4d7a	docs/vmauth.md: mention that we can provide custom integration with SAML	2020-06-19 13:14:07 +03:00
Aliaksandr Valialkin	b542e50680	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin	88e1b7d144	app/vmselect/promql: fill gaps on right side with values from left side of `or` operator in the same way as Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552	2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin	a12364ad37	docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible	2020-06-18 18:52:43 +03:00
Roman Khavronenko	e91d758831	vmalert-537: allow name duplication for rules within one group. (#559 ) Uniqueness of rule is now defined by combination of its name, expression and labels. The hash of the combination is now used as rule ID and identifies rule within the group. Set of rules from coreos/kube-prometheus was added for testing purposes to verify compatibility. The check also showed that `vmalert` doesn't support `query` template function that was mentioned as limitation in README.	2020-06-15 20:15:47 +01:00
Clémence Saussez	e426434770	app/vmalert: fix link to testdata (#547 ) Fix broken link to vmalert test data Signed-off-by: Clemence Saussez <clemence@zen.ly>	2020-06-10 17:26:27 +01:00
Roman Khavronenko	3e277020a5	vmalert-491: allow to configure concurrent rules execution per group. (#542 ) The feature allows to speed up group rules execution by executing them concurrently. Change also contains README changes to reflect configuration details.	2020-06-09 15:21:20 +03:00
Roman Khavronenko	ffa75c423d	vmalert-521: allow to disable rules expression validation. (#536 ) This feature may be useful for using `vmalert` with PromQL compatible datasources like Loki.	2020-06-06 21:27:09 +01:00
Aliaksandr Valialkin	f5dd2a71a6	app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 20:13:56 +03:00
Aliaksandr Valialkin	4b98e436ef	app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set	2020-06-05 20:13:55 +03:00
Aliaksandr Valialkin	0d92abfbf6	app/vmalert: print brief usage info for `vmalert -help`	2020-06-05 10:43:18 +03:00
Aliaksandr Valialkin	ff1a725a56	app/vmauth: print brief usage info for `vmauth -help`	2020-06-05 10:40:00 +03:00
Aliaksandr Valialkin	05ae1472e3	app/vmagent: print brief usage info for `vmagent -help`	2020-06-05 10:39:59 +03:00
Aliaksandr Valialkin	f8692a1d43	app/vmauth: log when -auth.config is reloaded in SIGHUP	2020-06-03 23:22:14 +03:00
Aliaksandr Valialkin	d2f30e8d79	app/vmalert: fix comment for UpdateWith exported methods	2020-06-01 14:35:32 +03:00
Roman Khavronenko	270552fde4	vmalert: Add recording rules support. (#519 ) * vmalert: Add recording rules support. Recording rules support required additional service refactoring since it wasn't planned to support them from the very beginning. The list of changes is following: * new entity RecordingRule was added for writing results of MetricsQL expressions into remote storage; * interface Rule now unites both recording and alerting rules; * configuration parser was moved to separate package and now performs more strict validation; * new endpoint for listing all groups and rules in json format was added; * evaluation interval may be set to every particular group; * vmalert: uncomment tests * vmalert: rm outdated TODO * vmalert: fix typos in README	2020-06-01 13:46:37 +03:00
Aliaksandr Valialkin	32652485e3	app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on `/-/reload` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518	2020-05-30 14:37:12 +03:00
Aliaksandr Valialkin	d988f89415	app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent	2020-05-30 14:23:14 +03:00
Aliaksandr Valialkin	5b6a9675d8	app/vmauth: fix `make run-vmauth` command	2020-05-22 16:45:02 +03:00
Aliaksandr Valialkin	8905bc2a40	app/vmagent: check for error returned from flag.Set	2020-05-21 16:31:14 +03:00
Aliaksandr Valialkin	f9847352b4	app/vmagent: add `-dryRun` option for checking all the configs mentioned in command-line flags without running `vmagent` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin	dbd8beccfa	app/vmselect/promql: add `ascent_over_time(m[d])` and `descent_over_time(m[d])` functions These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss over the given duration `d`.	2020-05-21 12:07:48 +03:00
kreedom	6b23df2bec	vmalert add quotes escape function (#510 ) * vmalert add quotes escape function Co-authored-by: kreedom	2020-05-20 22:20:31 +03:00
Aaron France	619d4959c7	Update README.md	2020-05-20 09:04:31 +03:00
Aliaksandr Valialkin	70ea4e28a7	app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3	2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin	80c18b7275	docs/vmagent.md: mention an alternative to `refresh_interval` option in scrape configs	2020-05-19 23:10:06 +03:00
Aliaksandr Valialkin	538fdfe133	app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries	2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin	f52769f6ee	app/vmselect/promql: fix `outilersk` calculations	2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin	a441cdd1d9	app/vmselect/promql: add `outliersk(N, m)` aggregate function for anomaly detection across groups of similar time series	2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin	d0f08b4a58	app/vmalert/notifier: go fmt	2020-05-19 12:59:46 +03:00
kreedom	7e173655ba	vmalert - add expr to variables, add escape functions (#495 ) * vmalert - add expr to variables, add escape functions Co-authored-by: kreedom	2020-05-18 11:55:16 +03:00
Roman Khavronenko	92212f04da	vmalert: avoid sending resolves for pending alerts (#498 ) Before the change we were sending notifications to notifier if following conditions are met: * alert is in Fire state * alert is in Inactive state We were sending Inactive notifications to resolve alert ASAP. Unfortunately, we were sending resolves for Pending alerts that become Inactive, which is wrong. In this change we delete alert from the active list if it was Pending and become Inactive. In this way we now have Inactive alerts only if they were in state Fire before. See test change for example.	2020-05-17 15:13:22 +01:00
Roman Khavronenko	de60ad0cd6	vmalert: fix potential race during configuration reloads (#497 ) Configuration reload and rules evaluation can't be executed in same time now. This may make reload time longer but prevents from potential races.	2020-05-17 15:12:09 +01:00
Aliaksandr Valialkin	eac3da478e	app/vmalert: run `make quicktemplate-gen` from the root dir of the repository	2020-05-16 22:46:02 +03:00
Aliaksandr Valialkin	93c87d28f6	all: print `--help` output to stdout instead of stderr This is easier to grep and pipe	2020-05-16 11:59:33 +03:00
Aliaksandr Valialkin	8cb35974af	app/vmrestore: document better that `vmrestore` works like `rsync --delete`, i.e. it deletes files in `-storageDataPath`, which are missing in the backup	2020-05-16 09:22:17 +03:00
Aliaksandr Valialkin	3a68c47de0	app/vmagent/Makefile: fix `make run-vmagent` rule	2020-05-15 19:35:10 +03:00
Aliaksandr Valialkin	697b6af10f	app/vmagent/remotewrite: remove unused import after the commit `93267f143f`	2020-05-15 17:42:19 +03:00
Aliaksandr Valialkin	93267f143f	app/vmagent/remotewrite: allow ingesting time series with multiple samples at once Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481	2020-05-15 17:36:36 +03:00
Aliaksandr Valialkin	82ffbcb9a6	app/vmstorage: add `vm_slow_metric_name_loads_total` metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin	82ccdfaa91	app/vmstorage: add `vm_slow_row_inserts_total` and `vm_slow_per_day_index_inserts_total` metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:44:32 +03:00
Roman Khavronenko	a249cd9d22	vmalert: fix the access to rules slice element by wrong index (#486 ) During group's update rules deletion was causing slice mutations while slice index was assumed to be unchanged. This caused "slice bounds out of range" errors when multiple rules were deleted sequentially.	2020-05-15 07:55:22 +01:00
hagen1778	ef0e37cb9e	vmalert: update README	2020-05-15 09:17:28 +03:00
Aliaksandr Valialkin	0afd48d2ee	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin	894e5d2b9b	docs/vmbackup.md: add a link to vmbackuper tool	2020-05-13 22:54:22 +03:00
Roman Khavronenko	415b1ddfb5	vmalert: check if remoteRead object was initied before calling Restore (#473 ) The check for non-nil remoteRead was mistakenly dropped during refactoring which caused panics when `vmalert` wasn't configured with `remoteRead` flag.	2020-05-13 19:32:58 +01:00
Roman Khavronenko	db7dd96346	vmalert: fix flag names and description in README (#475 ) Change also adds the recommendation for `remotewrite` queue error.	2020-05-13 19:32:21 +01:00
肖贝贝	ba48438b06	Feat/vmalert add max queue size (#472 ) * feat: add remoteWrite.maxQueueSize to reduce queue full * rename remote(write\|read) flags to remote(Write\|Read) for the sake of consistency Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-13 18:58:56 +01:00
Aliaksandr Valialkin	7882a0dbbf	app/vmselect/promql: suppress "SA4006: this value of `dstValues` is never used" error in golangci-lint	2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin	96e001d254	app/vmagent: fix a bug with improper relabeling when multiple `-remoteWrite.urlRelableConfig` args are set This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin	faf92a0965	app/vmselect/promql: fix `any(..)` calculations - return all the data points instead of the first one	2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin	cc311e20fe	app/vmselect/promql: add `any(x) by (y)` aggregate function, which returns any time series from `q` for each group `y`	2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin	574289c3fb	app/vmselect/promql: support for `sum(x) by (y) limit N` syntax in order to limit the number of output time series after aggregation	2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin	0a134ace63	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin	8300cc17af	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin	6273385618	app/vmagent/remotewrite: properly dial TCP6 addresses set via `-remoteWrite.url` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469	2020-05-12 15:22:29 +03:00
Aliaksandr Valialkin	dbd0c552d5	lib/storage: gradually pre-populate per-day inverted index for the next day This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:05 +03:00
Roman Khavronenko	8c8ff5d0cb	vmalert: cleanup and restructure of code to improve maintainability (#471 ) The change introduces new entity `manager` which replaces `watchdog`, decouples requestHandler and groups. Manager supposed to control life cycle of groups, rules and config reloads. Groups export an ID method which returns a hash from filename and group name. ID supposed to be unique identifier across all loaded groups. Some tests were added to improve coverage. Bug with wrong annotation value if $value is used in templates after metrics being restored fixed. Notifier interface was extended to accept context. New set of metrics was introduced for config reload.	2020-05-10 17:58:17 +01:00
Nikolay Khramchikhin	9e8733ff65	vmalert config reload added config hot reload for vmalert with sighup and api call	2020-05-09 10:32:12 +01:00
Aliaksandr Valialkin	baedb25936	docs/vmauth.md: fix a link to docker images	2020-05-08 14:10:04 +03:00
Aliaksandr Valialkin	51291015a5	app/vmagent: return 200 from `/-/reload` endpoint as Prometheus does	2020-05-07 19:30:30 +03:00
Aliaksandr Valialkin	6afb25fd08	docs/{vmagent,vmauth}: small clarifications in the docs	2020-05-07 12:55:20 +03:00
Aliaksandr Valialkin	653d51694a	app/vmauth: prevent from attacks with `..` in path for accessing resources outside the configured `url_prefix`	2020-05-07 12:55:18 +03:00
Aliaksandr Valialkin	8a00807f60	app/vmagent: allow setting independent auth configs per each configured `-remoteWrite.url`	2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin	b69eb7bf38	app/vmagent: properly set client-side TLS certificates for `-remoteWrite.url`. Previously they were mistakenly set as server-side	2020-05-06 16:50:30 +03:00
Aliaksandr Valialkin	e8936c9cb3	docs/vmagent.md: small fixes	2020-05-06 14:49:18 +03:00
Aliaksandr Valialkin	3f52a97f9b	lib/promscrape: add Prometheus-compatible DNS-based service discovery aka `dns_sd_configs`	2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin	08320cfcf4	docs/{vmauth,vmagent}: fix ports for profiling	2020-05-05 20:15:47 +03:00
Aliaksandr Valialkin	f65930b34d	docs/vmauth.md: mention that we can help creating customized proxy	2020-05-05 12:34:42 +03:00
Aliaksandr Valialkin	266327642b	docs/{vmagent,vmauth}: add `Profiling` section	2020-05-05 11:45:13 +03:00
Aliaksandr Valialkin	0c7cddfca6	docs: add vmauth.md	2020-05-05 11:17:23 +03:00
Aliaksandr Valialkin	e767aedd17	app/vmauth: add initial version of vmauth. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details	2020-05-05 10:54:17 +03:00
Aliaksandr Valialkin	b5a780930d	docs/vmagent.md: `/targets` page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead	2020-05-05 10:54:14 +03:00
Roman Khavronenko	0ba1b5c71b	app/vmalert: restore alerts state from datasource metrics (#461 ) * app/vmalert: restore alerts state from datasource metrics Vmalert will restore alerts state for rules that have `rule.For` > 0 from previously written timeseries via `remotewrite.url` flag. * app/vmalert: mention remotewerite and remoteread configuration in README	2020-05-05 00:51:22 +03:00
Aliaksandr Valialkin	40c3ffb359	lib/promscrape: add Prometheus-compatible service discovery for Consul aka `consul_sd_configs` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin	432187ac3b	app/vminsert: add `/-/reload` handler in the same way as for `vmagent`	2020-04-30 02:15:39 +03:00
DexterZhang	67511d4165	feat(vmagent): add promscrap config reload suppport via http (#450 ) * feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload` * fix: typo fix	2020-04-30 02:00:32 +03:00
Aliaksandr Valialkin	43c39dc36c	vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp The upstream fasthttp may contain issues like `996610f021` , plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.	2020-04-29 17:33:34 +03:00
Artem Navoiev	4487b454a8	Update README.md	2020-04-29 12:39:15 +03:00
Aliaksandr Valialkin	57407cca83	app/vmselect/promql: remove `-search.maxPointsPerTimeseries` command-line flag Limit the estimated time series count after aggregation with grouping by the number of source time series.	2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin	4e4f57b121	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin	17d96e4503	app/vmselect: add `-search.estimatedSeriesCountAfterAggregation` command-line flag for tuning the probability of OOMs vs false-positive `not enough memory` errors	2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin	1397612117	app/vmalert: added missing comments for public entities	2020-04-28 11:21:07 +03:00
Roman Khavronenko	3bfa41a95c	app/vmalert: initial remote-write support for alerts state persistence. (#442 ) * app/vmalert: initial remote-write support for alerts state persistence. If `remotewrite.url` flag is set, vmalert will send alerts state via remote-write protocol to remote storage. The sending is asynchronous to avoid blocking calls in rules evaluation loop. * app/vmalert: merge with master * app/vmalert: write both `instant` and `for` alerts timeseries states in remote storage.	2020-04-28 00:18:02 +03:00
Aliaksandr Valialkin	90670cb55e	app/vmalert: include it into the next release	2020-04-28 00:10:12 +03:00
Aliaksandr Valialkin	b768bc9a6a	lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka `ec2_sd_configs`	2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin	b4afe562c1	lib/storage: postpone reading data from blocks during search This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin	0224071ebe	lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config	2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin	fcf57f9883	app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks This should minimize the number of disk seeks when reading data from temporary file.	2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin	6954d0edb7	lib/promscrape/discovery/gce: allow empty `zone` arg in `gce_sd_config` - in this case zones for the given project are automatically discovered	2020-04-26 14:34:11 +03:00
kreedom	2c18548e08	alert - rename validate function and flags (#440 ) * alert - rename validate function and flags	2020-04-26 14:15:04 +03:00
kreedom	5f61d43db9	vmalert - validate template in labels (#439 )	2020-04-26 13:53:57 +03:00
肖贝贝	eeadfccdc5	fix: fix vmalert template label not complete bug (#435 ) Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-26 13:30:10 +03:00
Aliaksandr Valialkin	1f3fd93b58	docs/{vmbackup,vmrestore}.md: update `-help` output	2020-04-24 22:44:21 +03:00
Jason Gardner	66af7e40f3	app/vmbackup: added ability to create and delete snapshots during backup (#428 ) * app/vmbackup: added ability to create and delete snapshots during backup Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422 * Add snapshot create and delete url flags * Fixed errcheck warnings in build	2020-04-24 22:35:03 +03:00
Aliaksandr Valialkin	a596aec82c	app/vmselect: fix description for `-search.resetCacheAuthKey`	2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin	9ef5935552	lib/promscrape: initial implementation for `gce_sd_configs` aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin	364db13c9c	app/vmselect: add `/api/v1/status/tsdb` page with useful stats for locating root cause for high cardinality issues See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin	9ebc937685	app/vmselect: add `-search.minStalenessInterval` command-line flag for removing gaps on graphs built from time series with irregular duration between samples Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin	fe57d46687	app/vmselect: merge `-search.maxLookback` and `-search.maxStalenessInterval` flags, since it has been appeared they have identical purpose :( Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin	851946af1e	deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var For example, the following command will build VictoriaMetrics docker image on top of alpine image: ROOT_IMAGE=alpine make package-victoria-metrics	2020-04-20 01:16:57 +03:00
Aliaksandr Valialkin	936fb0eac3	app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection This should fix the following error when sending data to remote storage: couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-17 15:52:42 +03:00
Aliaksandr Valialkin	79fb595732	docs/vmagent.md: typo fix: unvailable -> unavailable	2020-04-17 13:11:31 +03:00
Aliaksandr Valialkin	546d26523c	app/vmagent/README.md: mention about `prodmscrape.suppressScrapeErrors`	2020-04-17 13:08:21 +03:00
Aliaksandr Valialkin	f41e6a7bd9	app/vmselect: properly apply `-search.maxLookback` to queries sent to `/api/v1/query`	2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin	071fdf5518	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin	6f7f64f757	app/vmselect: handle `timestamp(metric offset X)` the same way as Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin	067c7afebc	lib/promscrape: show information on improperly configured scrape targets at the bottom of `/targets` page This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin	0b76c27fa1	docs/vmagent.md: mention that vmagent supports `kubernetes_sd_configs` now	2020-04-13 21:06:36 +03:00
hagen1778	2eed6c393f	vmalert: prepare package for external usage * update README according to changes * add Makefile with basic commands	2020-04-12 15:32:42 +03:00
kreedom	948f8b6b5f	[vmalert] fix linter issues	2020-04-12 15:08:11 +03:00
kreedom	8fca5f2819	[vmalert] add tests to webserver (#413 )	2020-04-12 14:51:03 +03:00
Roman Khavronenko	7c9405f53d	Vmalert metrics (#412 ) vmalert: add basic list of metrics	2020-04-11 20:42:01 +01:00
Roman Khavronenko	9f8cc8ae1b	Extend web responses for alerts: (#411 ) vmalert: Extend web responses for alerts * populate apiAlert object with additional fields * return all active alerts, not only firing * sort list of API alerts for deterministic output * add helper for available path list	2020-04-11 16:49:23 +01:00
kreedom	90de3086b3	[vmalert] add webserver (#410 ) * [vmalert] add webserver	2020-04-11 12:40:24 +03:00
Aliaksandr Valialkin	f94a090020	docs: update minimum supported Go version from 1.12 to 1.13	2020-04-07 13:38:37 +03:00
Roman Khavronenko	b099d84271	Vmalert/rules eval (#400 ) * Initial rules evaluation support. Rules are now store alerts state in private field `alerts`. Every evaluation updates the alerts and state. Every unique metric received from datastore represents a unique alert, uniqueness is guaranteed by hashing ordered labelset. * merge with master * cleanup * support endAt parameter as 3evaluationInterval for active alerts make golint happy	2020-04-06 14:44:03 +03:00
Aliaksandr Valialkin	895cadfae7	app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request This header is required by Cortex (and, probably, other remote storage systems). See `9c1f44d090/docs/apis.md (remote-api)` . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399	2020-04-04 16:24:56 +03:00
Aliaksandr Valialkin	57704aa584	app/victoria-metrics: add `-selfScrapeInstance` and `-selfScrapeJob` flags for tuning labels for self-scraped metrics	2020-04-04 14:57:22 +03:00
Aliaksandr Valialkin	f9b24d4899	app/vmselect/promql: keep metric name after applying `first_over_time` and `last_over_time` functions	2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin	b38d048dd9	app/vmstorage: add `vm_free_disk_space_bytes` metric for monitoring the remaining disk space at `-storageDataPath`	2020-04-01 23:08:58 +03:00
kreedom	298eb0a0f8	[vmalert] improve external url handling	2020-04-01 22:29:11 +03:00
kreedom	12fe915b48	[vmalert] add prometheus template function (#396 ) * [vmalert] add prometheus template function * make linter be happy Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-04-01 18:17:53 +03:00
Aliaksandr Valialkin	e71519b8b2	app/victoria-metrics/testdata: add a test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 12:51:25 +03:00
Aliaksandr Valialkin	318326c309	lib/storage: properly handle `{label=~"foo\|"}` filters as Prometheus does Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin	0c485f14d1	app/vmselect/prometheus: allow passing relative time to `start`, `end` and `time` args of `/api/v1/*` queries	2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin	2ebf7d86ff	app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()	2020-03-29 21:50:28 +03:00
kreedom	bf6c24d0f4	[vmalert] config parser (#393 ) * [vmalert] config parser * make linter be happy * fix test * fix sprintf add test for rule validation	2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin	bd156cd088	docs/vmagent.md: add `prometheus remote_write proxy` use case	2020-03-28 23:16:38 +02:00
Dmitry Naumov	27373807c1	Rootless docker images by default (#358 ) * Rootless docker images by default * Migrate to rootless base image Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-03-27 21:23:50 +02:00
kreedom	b22da547a2	[vmalert] - parse template annotaions (#387 ) * [vmalert] - parse template annotations	2020-03-27 18:31:16 +02:00
Aliaksandr Valialkin	3c7c71a49c	app/vmselect: adjust `label_map()` handling for corner cases The following corner cases now supported: * label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label` * label_map(q, "label", "foo", "") - removes `label="foo"` from series All the unmatched labels are kept unchanged.	2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin	4fc4912f0c	app/vmalert/datasource: typo fix in docs: Labels -> Label	2020-03-13 12:22:33 +02:00
kreedom	a746cb62b6	vmalert add vm datasource, change alertmanager (#364 ) * vmalert add vm datasource, change alertmanager * make linter be happy * make linter be happy.2 * PR comments * PR comments.1	2020-03-13 12:19:31 +02:00
Aliaksandr Valialkin	fdc2a9d1d7	app/vmselect: add `label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN)` function to MetricsQL Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin	8023ad7dbd	app/vmselect: add `-search.maxStalenessInterval` for tuning Prometheus data model closer to Influx-style data model	2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin	fcd91795d5	app/vmagent: mention that vmagent can filter data	2020-03-11 16:22:39 +02:00
Aliaksandr Valialkin	301c2acd61	app/vmstorage: return 500 status code instead of 200 status code on internal errors inside `/snapshot/*` handlers	2020-03-10 23:51:55 +02:00
Aliaksandr Valialkin	e17702fada	app/vmselect: add optional `max_rows_per_line` query arg to `/api/v1/export` This arg allows limiting the number of data points that may be exported on a single line.	2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin	1fe66fb3cc	app/{vmagent,vminsert}: add support for importing csv data via `/api/v1/import/csv`	2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin	5fe38a84eb	app/vmagent: properly apply `-remoteWrite.sendTimeout` to fasthttp.HostClient	2020-03-09 13:31:55 +02:00
Aliaksandr Valialkin	c386c5de57	app/vmagent: properly add labels set via `-remoteWrite.label` to metrics before sending them to `-remoteWrite.url`	2020-03-06 19:26:58 +02:00
Aliaksandr Valialkin	c5f894b361	Makefile: add build and test rules with enabled race detector. These rules have `-race` suffix Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin	754db0d22e	app/vmagent/README.md: small fixes	2020-03-04 18:14:47 +02:00
Aliaksandr Valialkin	772312bf7b	app/vmagent/README.md: typo fix	2020-03-04 18:05:09 +02:00
Aliaksandr Valialkin	871abfab7a	app/vmagent/README.md: clarification	2020-03-04 18:03:48 +02:00
Aliaksandr Valialkin	007c591de8	app/vmagent/README.md: add `iot and edge monitoring` use case	2020-03-04 18:01:34 +02:00
Aliaksandr Valialkin	474a09c0f1	app/vmagent/README.md: add `use cases` section	2020-03-04 17:42:27 +02:00
Aliaksandr Valialkin	032c88561b	app/vminsert/prompush: limit memory usage by pushing promscrape data in smaller blocks	2020-03-03 19:58:54 +02:00
Aliaksandr Valialkin	76036c1897	app/vmagent: add `-remoteWrite.maxDiskUsagePerURL` for limiting the maximum disk usage for each `-remoteWrite.url` buffer Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352	2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin	c31d640eb9	app/vmagent/remotewrite: do not reset empty relabelCtx	2020-03-03 15:01:03 +02:00
Aliaksandr Valialkin	02b55c72dc	app/vmagent: add `-remoteWrite.urlRelabelConfig` for applying individual relabeling for each `-remoteWrite.url` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308	2020-03-03 13:12:16 +02:00
Aliaksandr Valialkin	7d178a40bd	app/vmselect/prometheus: do not add `__name__!=` filter when searching for all the matching metric names via `/api/v1/label/__name__/values` with non-empty label filter This should reduce query time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin	f9a584b5c1	app/vmagent/remotewrite: yet another typo fix	2020-02-28 20:05:55 +02:00
Aliaksandr Valialkin	b9b46cb8dc	app/vmagent/remotewrite: typo fix	2020-02-28 19:03:16 +02:00
Aliaksandr Valialkin	db6f4e4af1	app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage	2020-02-28 18:58:01 +02:00
Aliaksandr Valialkin	5701b2f7bb	app/vmselect/prometheus: properly pass filter for `labelName=__name__` in labelValuesWithMatches Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin	18af31a4c2	all: properly split `vm_deduplicated_samples_total` among cluster components Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin	c458b521a2	app/vmagent: allow setting `-httpListenAddr` to empty string in order to disable listening for http requests Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340	2020-02-26 20:58:11 +02:00
Aliaksandr Valialkin	117c76311c	app/vmagent/README.md: list service discovery mechanisms, which will be added soon Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-02-26 19:27:08 +02:00
Aliaksandr Valialkin	be37d762cd	app/vmagent: add `-remoteWrite.maxBlockSize` command-line flag for limiting the maximum size of unpacked block to send to remote storage	2020-02-25 19:57:47 +02:00
Aliaksandr Valialkin	4e24839a2c	app/vmagent: do not allow sending unpacked requests with sizes exceeding `-maxInsertRequestSize`	2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin	6386aeb1e0	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin	e453880084	app/vmagent/README.md: mention that `vmagent` exposes target statuses at `/targets` page	2020-02-25 18:15:58 +02:00
Aliaksandr Valialkin	4c4448b66e	app/vminsert: add `/targets` handler, which exposes Prometheus targets defined in `-promscrape.config` file	2020-02-25 18:13:11 +02:00
Aliaksandr Valialkin	e1ef72af01	app/vmagent: logo fix	2020-02-25 00:09:19 +02:00
Aliaksandr Valialkin	56c70fe856	app/vmagent: update docs	2020-02-25 00:09:18 +02:00
Aliaksandr Valialkin	e7e4aa5243	app/vmagent/README.md: small fixes	2020-02-24 21:25:38 +02:00
Aliaksandr Valialkin	ae51300973	app/vmselect/promql: properly take into account the first datapoint when calculating `rollup_candlestick` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin	e65ec88779	app/vmselect/promql: do not take into account values outside the current window in `rollup_candlestick` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-23 18:03:57 +02:00
Yaroslav	a6d0645539	fix rollupOpen(), rollupHigh(), rollupLow() functions (#328 )	2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin	04762344c6	app/vmagent: initial implementation for vmagent	2020-02-23 13:36:03 +02:00
kreedom	49390b8dbc	[vmalert] integration with AlertManager (#325 )	2020-02-21 23:15:05 +02:00
Aliaksandr Valialkin	2f55cabaa4	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin	ec9bf39b5b	app/vmselect: add `-search.cacheTimestampOffset` command-line flag This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin	539139391c	app/vmselect: add `/internl/resetRollupResultCache` handler for resetting response cache Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:05 +02:00
kreedom	3c06179184	basic vmalert backbone (#317 ) * basic vmalert backbone * Resolve code review comments for vmalert backbone * Second review fixes for vmalert backbone	2020-02-16 20:59:02 +02:00
Aliaksandr Valialkin	7b1c7051a3	app/vmselect: add `sort_by_label(q, label)` and `sort_by_label_desc(q, label)` functions This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin	8466ab0034	all: allow setting flags via environment vars Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin	e210cd9da1	lib/storage: move `-dedup.minScrapeInterval` flag outside lib/storage, so it doesnt show up in `vminsert` in cluster version	2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin	a6c6a2debc	app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit `9e1119dab8`	2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin	78b62dee87	app/vmselect/promql: properly adjust time range for data to select Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin	366693b9f1	app/vmselect: unconditionally `offset -step` to `rollup_candlestick`. This makes results more consistent	2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin	525101339e	app/vmselect/promql: automatically apply `offset -step` to `rollup_candlestick` function in order to obtain the expected OHLC results See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin	ada6a3da8d	app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin	36a1ac8360	app/vmselect: take into account the time the requests wait in the queue if `-search.maxConcurrentRequests` is exceeded This will prevent from excess CPU usage for timed out queries.	2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin	834051e5b2	app/vmselect: add a placeholder for `/api/v1/metadata`, which could be requested by Grafana See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.	2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin	42864bb52f	all: do not clash flag description with back-quoted flag types See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:46:52 +02:00
Artem Navoiev	a47f292295	[vmalert] add vmalert.png.2	2020-02-02 12:17:19 +02:00
Artem Navoiev	354232b62b	[vmalert] add vmalert.png	2020-02-02 12:16:05 +02:00
Artem Navoiev	28778be0cc	[vmalert] initial	2020-02-02 12:14:09 +02:00
Aliaksandr Valialkin	90cf356ea1	app/vmselect/promql: adjust `and` and `unless` binary operator handling to be consistent with Prometheus	2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin	c3d86eef96	all: add `-dedup.minScrapeInterval` command-line flag for data de-duplication Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin	d70ba7eb37	lib/fs: optimize small reads for `ReaderAt.MustReadAt` by reading from memory-mapped space instead of reading from file descriptor This should improve performance when reading many small blocks.	2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin	d68546aa4a	lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init This should speed up Search.NextMetricBlock loop for big number of found time series.	2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin	a462355b2f	app/vmselect/promql: add `keep_next_value(q)` for filling gaps with the next non-empty value	2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin	371e86194d	app/vminsert: moved `-maxInsertRequestSize` command-line flag out of `lib/prompb` in order to prevent its inclusion in `vmselect` and `vmstorage` apps	2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin	adbbc4fa1a	app/vmselect/promql: return expected results from `increase()` over the beginning of time series, which start from big value Examples for such counters: OS-level counters for network or cpu stats.	2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin	75ad47a43c	app/victoria-metrics: check for error arg passed to filepath.Walk callback	2020-01-27 20:56:45 +02:00
Aliaksandr Valialkin	6320a19a8c	app/victoria-metrics: remove `integration` build tag from tests This simplifies testing with `go test ./app/victoria-metrics` without the need to remember to pass `-tags=integration` to Go commands.	2020-01-27 20:25:28 +02:00
Aliaksandr Valialkin	2392a359e1	app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296	2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin	9df5b2d1c3	app/victoria-metrics: add `-selfScrapeInterval` flag for self-scraping `/metrics` page Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180	2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin	6456c93dbb	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin	680080887d	all: consistently log durations in seconds with millisecond precision This should improve logs readability	2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin	9773022e50	app/vmselect: mention the original query and time range in error messages This should simplify debugging invalid or heavy queries.	2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin	efc7ad88ec	app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors	2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin	ec9651e266	app/vmselect/prometheus: increase default value `-maxExportDuration` to 30 days, since 10 minutes beat users exporting bit amounts of data	2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin	582dd01f42	app/vmselect/promql: add `range_over_time(m[d])` function for calculating value range for `m` over `d`	2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin	36973ee975	app/vmselect/promql: add `label_match(q, label, regexp)` and `label_mismatch(q, label, regexp)` functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin	227cf53ef9	app/vminsert: increase default value for `-insert.maxQueueDuration` from 30s to 60s This should help catching up with high ingestion rate after VictoriaMetrics restart.	2020-01-18 14:39:36 +02:00
Aliaksandr Valialkin	c9472e4f3a	all: use `github.com/klauspost/compress/gzip` instead of `compress/gzip` `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin	cdcacaea6d	app/vmselect/netstorage: `make fmt`	2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin	7327adbc86	app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file This should reduce memory usage on systems with more than 8GB RAM.	2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin	d0d258b314	app/vmselect: limit the default value for `-search.maxConcurrentRequests`, so it plays well on systems with more than 16 vCPUs A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value. This will give more chances for executing insert path.	2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin	d88725f133	app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects	2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin	0d7505b00e	all: mention command-line flags used for limiting the incoming request size in error messages This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin	2839f4688a	app/vmselect/promql: fix panic on `sum(aggr_over_time(...))` with incorrect number of args	2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin	8b14572f70	app/vmselect/promql: add `hoeffding_bound_upper(phi, m[d])` and `hoeffding_bound_lower(phi, m[d])` functions These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin	8eaced8cae	app/vmselect/promql: return continuous values for `min_over_time` and `max_over_time` when `step` is smaller than `scrape_interval`	2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin	ea231f8167	app/victoria-metrics: adjust integration tests after the commit 99facd71cd6ac151d512cea1df73be91c10c7f83	2020-01-11 00:58:16 +02:00
Aliaksandr Valialkin	46bfdbe6cf	app/vmselect/promql: do not take into account the previous point before time window in square brackets for `min_over_time`, `max_over_time`, `rollup_first` and `rollup_last` functions This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.	2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin	4f0a645f77	vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5 This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.	2020-01-10 23:15:15 +02:00
Aliaksandr Valialkin	b829fe5e39	app/vmselect/promql: properly handle `aggr(aggr_over_time(...))`	2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin	164278151f	app/vmselect/promql: add `aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])` function This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin	c4632faa9d	app/vmselect/promql: add `tmin_over_time(m[d])` and `tmax_over_time(m[d])` functions These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin	f358fb72d1	app/{vmbackup,vmrestore}: add `backup complete` file to backup when it is complete and check for this file before restoring from backup This should prevent from restoring from incomplete backups. Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.	2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin	d5aeda0e1a	app/vmselect/promql: skip `rate` calculation for the first point on time series	2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin	d410c78c7e	app/vmselect/promql: fix calculations for `histogram_share`	2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin	d64699bb9f	app/vmselect/promql: add missing MetricName into netstorage.Result in tests	2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin	f409f2d050	app/vmselect/promql: add `histogram_share(le, buckets)` function	2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin	b1ded7cf9a	app/vmselect/promql: add `absent_over_time(m[d])` func similar to the function in Prometheus 2.16 See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin	a8360d04c0	app/vmselect/promql: add `histogram_over_time(m[d])` rollup function	2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin	3e09d38f29	app/vmselect/promql: fix results caching for multi-arg rollup functions such as `quantile_over_time` Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.	2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin	a774120460	app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating `rate` for the first point on the time series This should provide better estimation for `rate` in the beginning of time series.	2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin	b5645ccbdf	app/vmselect/promql: increase the estimated number of time series returned by `aggr() by (something)` from 100 to 1K, since 100 may result in OOM for high number of time series	2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin	cb3a342882	app/vmselect/promql: add `share_le_over_time` and `share_gt_over_time` functions for SLI and SLO calculations	2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin	6f67e0b56b	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin	1925ee038d	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:03:59 +02:00
Mike Poindexter	bec62e4e43	Split Extended PromQL parsing to a separate library	2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin	d880325cf6	app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range This should prevent from the following panic at app/vmselect/promql/binary_op.go:255: BUG: len(leftVaues) must match len(rightValues) and len(dstValues)	2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin	9e19949c6b	app/vmselect/promql: adjust calculations for `rate` and `increase` for the first value These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.	2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin	0455c03cb9	app/vmselect/promql: properly calculate `rate` on the first data point It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape, i.e. we can assume its value was 0 at this time.	2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin	5cb8d97743	all: use gozstd instead of pure Go zstd for GOARCH=amd64	2019-12-24 12:42:42 +02:00
Aliaksandr Valialkin	5b75984aa9	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin	bea5a8700a	app/vmselect: add `-search.maxExportDuration` command-line flag for limiting `/api/v1/export` duration Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275	2019-12-20 11:35:22 +02:00
Andrii Dembitskyi	2fba7b6f35	Fix typo in log message	2019-12-19 14:33:20 +02:00
Aliaksandr Valialkin	d03827c57d	app/vminsert: return StatusNoContent http response for `/api/v1/import` to be consistent with other insert handlers	2019-12-19 01:21:54 +02:00
Aliaksandr Valialkin	96ff8d9adb	app/vmselect: add ability to pass `match[]`, `start` and `end` to `/api/v1/labels` This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`. See https://github.com/prometheus/prometheus/issues/6178 for more details.	2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin	02f6566ce1	app/vmbackup: mention that backups are possible to Ceph and Swift	2019-12-14 01:08:49 +02:00
Aliaksandr Valialkin	bc645152cb	app/vminsert: simultaneously accept `telnet put` and `HTTP /api/put` OpenTSDB metrics at `-opentsdbListenAddr` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin	cea5a14853	all: rename `Extended PromQL` to `PromQL extensions`	2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin	6ddcd162ed	all: publish Docker images for the following GOARCH: amd64, arm, arm64, ppc64le and 386 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/258	2019-12-11 23:32:59 +02:00
Aliaksandr Valialkin	73b2a3d4b7	app/vmselect/promql: return `lower` and `upper` bounds for the estimated percentile from `histogram_quantile` if third arg is passed Updates https://github.com/prometheus/prometheus/issues/5706	2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin	caa4eb72d9	app/vmselect/promql: return matrix instead of vector on subqueries to `/api/v1/query` like Prometheus does	2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin	3c076544bf	app/vmselect/promql: allow negative offsets Updates https://github.com/prometheus/prometheus/issues/6282	2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin	68e1cf8942	app/vminsert: add `/api/v1/import` handler Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6	2019-12-09 20:59:04 +02:00
Aliaksandr Valialkin	8501b4a48d	app/vminsert: consistency renaming for counters	2019-12-09 16:43:10 +02:00
Aliaksandr Valialkin	293b541784	`make vendor-update`	2019-12-07 23:10:16 +02:00
Aliaksandr Valialkin	84b57e8974	app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689	2019-12-07 23:00:40 +02:00
Aliaksandr Valialkin	c09472dfd9	app/vmselect/promql: add `{topk\|bottomk}_{min\|max\|avg\|median}` aggregate functions for returning the exact k time series on the given time range The full list of functions added: - `topk_min(k, q)` - returns top K time series with the max minimums on the given time range - `topk_max(k, q)` - returns top K time series with the max maximums on the given time range - `topk_avg(k, q)` - returns top K time series with the max averages on the given time range - `topk_median(k, q)` - returns top K time series with the max medians on the given time range - `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range - `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range - `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range - `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range	2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin	face3d57bf	app/vmselect: add placeholders for `/api/v1/rules` and `/api/v1/alerts`	2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin	20812008a7	lib/storage: remove metricID with missing metricID->metricName entry The metricID->metricName entry can be missing in the indexdb after unclean shutdown when only a part of entries for new time series is written into indexdb. Recover from such a situation by removing the broken metricID. New metricID will be automatically created for time series with the given metricName when new data point will arive to it.	2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin	9476b73527	app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals This should improve scrape interval estimation for tiem series with gaps.	2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin	811b7a8303	app/vminsert/influx: allow empty measurement in Influx line protocol In this case metric names are mapped directly from field names without any prefixes.	2019-11-30 23:18:41 +02:00
Aliaksandr Valialkin	75ff524a4e	app/vmselect/promql: fix corner case for `increase` over time series with gaps In this case `increase` could return invalid high value for the first point after the gap.	2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin	96492348cb	deployment/docker/certs: update TLS certs source from alpine:3.9 to alpine:3.10	2019-11-29 19:57:29 +02:00
Aliaksandr Valialkin	f733cb2186	lib/backup: cosmetic fixes after #243	2019-11-29 18:07:04 +02:00
glebsam	15b7406f7b	Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243 ) * Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile * make fmt	2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin	4c859d980c	app/vmselect/prometheus: consistently apply `nocache` arg to `/api/v1/query` the same way ast to `/api/v1/query_range` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241	2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin	110235f789	app/vmselect/prometheus: fix content-type for `/api/v1/export` responses The correct Content-Type should be `application/stream+json` instead of `application/json` Thanks to Joshua Ryder for pointing to this.	2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin	205233d9a7	app/vmselect/promql: remove zero timeseries from `prometheus_buckets` output	2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin	3f99f39e9b	app/vmselect/prometheus: reduce default value for `-search.latencyOffset` from 60s to 30s 30 seconds should be enough for almost all the cases	2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin	e91cb34c0e	app/vmselect/promql: allow nested parens	2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin	826dfd63a5	vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1	2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin	0401969d78	app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph This should reduce the amounts memory allocations	2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin	da98703748	app/vmselect/promql: optimize binary search over big number of samples during rollup calculations	2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin	c28876172f	app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0	2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin	50ae1879c6	app/vmselect/promql: add `histogram` aggregate function, which is useful for building heatmaps from multiple time series	2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin	8582b50360	app/vmselect/promql: do not take into account buckets with negative counters in `prometheus_buckets`	2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin	19dfe52254	app/vmselect/promql: properly handle `histogram_quantile(0, ...)` with zero buckets	2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin	4bb88843cf	app/vmselect: add `vm_per_query_{rows,series}_processed_count` histograms	2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin	7753c8c0a1	app/vmselect/promql: transparently apply `prometheus_buckets` in `histogram_quantile`	2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin	c4287b3c86	app/vmselect/promql: add `prometheus_buckets` function for converting the upcoming histogram buckets from `github.com/VictoriaMetrics/metrics` to Prometheus-compatible buckets	2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin	1f3fd2c910	app/vmselect: adjust `end` arg instead of adjusting `start` arg if start > end `start` arg has higher chances to be set properly comparing to `end` arg, so it is expected that the `end` arg could be adjusted if it was set incorrectly.	2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin	7a4635f853	all: remove the remaining mentions of cluster version	2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin	443189fb0a	app/{vmbackup,vmrestore}: add `-maxBytesPerSecond` command-line flag for limiting the used network bandwidth during backup / restore	2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin	0094bc4fc9	app/vmselect/prometheus: properly adjust too big time `time` on `/api/v1/query` Too big `time` must be adjusted to `now()-queryOffset`.	2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin	3f1637fae8	app/vmselect/promql: properly calculate `integrate(q[d])`	2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin	c56b9ed03b	app/victoria-metrics: add build rules for GOARCH=ppc64le Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235	2019-11-13 20:24:33 +02:00
Aliaksandr Valialkin	3fd32e331a	app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235	2019-11-13 20:24:26 +02:00
Aliaksandr Valialkin	119dfd01bb	lib/storage: add `vm_cache_size_bytes{type="storage/hour_metric_ids"}` metric	2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin	86a1cd700b	lib/storage: remove inmemory index for recent hour, since it uses too much memory Production workload shows that the index requires ~4Kb of RAM per active time series. This is too much for high number of active time series, so let's delete this index. Now the queries should fall back to the index for the current day instead of the index for the recent hour. The query performance for the current day index should be good enough given the 100M rows/sec scan speed per CPU core.	2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin	c57eb0ff83	lib/storage: add `-disableRecentHourIndex` flag for disabling inmemory index for recent hour This may be useful for saving RAM on high number of time series aka high cardinality	2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin	ca259864e2	lib/storage: return back inmemory inverted index for recent hour Issues fixed: - Slow startup times. Now the index is loaded from cache during start. - High memory usage related to superflouos index copies every 10 seconds.	2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin	01bb3c06c7	lib/storage: remove inmemory inverted index for recent hours Production load with >10M active time series showed it could slow down VictoriaMetrics startup times and could eat all the memory leading to OOM. Remove inmemory inverted index for recent hours until thorough testing on production data shows it works OK.	2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin	6c2303764e	Revert "lib/fs: do not postpone directory removal on NFS error" This reverts commit `4c02e496f7`. Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234	2019-11-12 16:18:09 +02:00
Aliaksandr Valialkin	661dd190bb	Refer to https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883 from multiple places in README.md	2019-11-12 13:02:39 +02:00
Oleg Kovalov	b4f44befa3	fix misspelled words (#229 )	2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin	8e8f98f712	lib/storage: add tests for dateMetricIDCache	2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin	56d7cc8a0d	app/victoria-metrics: remove deprecated fs.MustStopDirRemover from main_test.go	2019-11-10 13:37:13 +02:00
Aliaksandr Valialkin	4c02e496f7	lib/fs: do not postpone directory removal on NFS error Continue trying to remove NFS directory on temporary errors for up to a minute. The previous async removal process breaks in the following case during VictoriaMetrics start - VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them. - The transaction instructs removing old directories for parts, which were already merged into bigger part. - VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors. - VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished and finds directories, which should be removed, but weren't still removed due to NFS errors. - VictoriaMetrics panics when it finds unexpected empty directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin	5c3fa59181	app/vmrestore: the upcoming release would be 1.29.0	2019-11-10 00:20:41 +02:00
Aliaksandr Valialkin	ee7765b10d	lib/storage: implement per-day inverted index	2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin	5810ba57c2	lib/storage: use specialized cache for (date, metricID) entries This improves ingestion performance.	2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin	6ad7fe8eeb	lib/storage: export `vm_new_timeseries_created_total` metric for determining time series churn rate	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	63b05c0b9f	app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions Incremental aggregate functions don't keep all the selected time series in memory - they keep only up to GOMAXPROCS time series for incremental aggregations. Take into account that the number of time series in RAM can be higher if they are split into many groups with `by (...)` or `without (...)` modifiers. This should reduce the number of `not enough memory for processing ... data points` false positive errors.	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	d888b21657	lib/storage: add inmemory inverted index for the last hour It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	1e46961d68	app/{vmbackup,vmrestore}: add `vmbackup` and `vmrestore` tools for creating backups on s3 or gcs from instant snapshots Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	e472f0b23b	lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter The origin of the error has been detected and documented in the code, so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`, so it could be monitored and alerted on high error rates. Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`, so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.	2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin	2f58f37f07	app/vmselect/promql: add `lag(q[d])` function, which returns the lag between the current timestamp and the timstamp for the last data point in `q`	2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin	d18ea0c95b	app/vmstorage: add `-bigMergeConcurrency` and `-smallMergeConcurrency` flags for tuning the maximum number of CPU cores used during merges	2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin	b0295dbf2e	app/vmselect: add `-search.latencyOffset` flag for tuning the time after data collection when data points become visible in query results Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218	2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin	e83fe938c8	all: `make fmt`	2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin	97ce4e03a5	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin	f752479cb8	app/victoria-metrics/test: add missing docs to public funcs PopulateTimeTplString and PopulateTimeTpl	2019-10-17 00:50:46 +03:00
Aliaksandr Valialkin	61e956e175	app/victoria-metrics: add a test for `max_lookback=<duration>` query arg Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	c66a691593	app/vmselect/prometheus: add `-search.maxLookback` command-line flag for overriding dynamic calculations for max lookback interval This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically from interval between datapoints for each input time series if the flag isn't set. The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	cc21b31502	app/victoria-metrics/test: add a test for PopulateTimeTplString	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	956fdd89d3	app/vmselect/promql: take into account the previous point when calculating `max_over_time` and `min_over_time` This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns the minimum value. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204	2019-10-08 12:30:05 +03:00
Artem Navoiev	1e2c511747	Add regression test for query apo Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2019-10-07 22:18:04 +03:00
Artem Navoiev	a116f5e7c1	Add regression test for query apo (#194 ) Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184	2019-09-30 11:25:54 +03:00
Aliaksandr Valialkin	4e9e1ca0f7	app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially This became the case after `b7ee2e7af2` .	2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin	c1d3705be0	app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock This allows re-using the destination buffer for marshaling in the outer loop.	2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin	b7ee2e7af2	app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series	2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin	67d44b0845	app/vmselect/promql: do not generate timestamps for NaN values in `timestamp` function according to Prometheus logic	2019-09-27 18:54:43 +03:00
Artem Navoiev	1e6ae9eff4	Add regression test for duplicated labels and series Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/155 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172	2019-09-27 16:52:16 +03:00
Artem Navoiev	cf4786f34a	add test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-09-26 12:45:19 +03:00
Aliaksandr Valialkin	26dc21cf64	app/vmselect/promql: add `increases_over_time` and `decreases_over_time` functions `increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`. `decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.	2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin	d32f88c378	app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls	2019-09-19 22:15:59 +03:00
Aliaksandr Valialkin	d4218d27e6	app/vmselect/promql: properly handle subqueries like `aggr_func(rollup_func(metric[window:step]))` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184	2019-09-13 21:41:04 +03:00
Aliaksandr Valialkin	4d636c244d	app/vmselect/promql: binary operation fixes according to Prometheus behaviour The follosing issues were fixed: - VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers - VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list	2019-09-13 17:42:52 +03:00
Artem Navoiev	4fd741f40d	[tests] check timestamp in tests (#177 )	2019-09-08 19:48:38 +03:00
Artem Navoiev	22e3fabefd	Add OpenTSDB and Prometheus integration tests (#168 ) * [WIP] open tsdb and prometheus integration tests * app/victoria-metrics: fix race condition on parallel tests	2019-09-05 17:55:38 +03:00
Aliaksandr Valialkin	88f8670ede	lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin	6954e126fc	app/vmselect/promql: ignore grouping by destination label in `count_values`, since such a grouping is performed automatically	2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin	305e7bc981	app/vmselect/promql: do not return artificial points beyond the last point in time series	2019-09-04 16:35:34 +03:00
Aliaksandr Valialkin	9721d06c6a	app/vmselect/prometheus: do not adjust `start` and `end` args in `/api/v1/query_range` if `nocache=1` arg is set Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171	2019-09-04 13:10:09 +03:00
Aliaksandr Valialkin	db4560ca31	app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does	2019-09-03 20:42:54 +03:00
Aliaksandr Valialkin	1575a560f0	app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks The maximum inmemory file size now depends on `-memory.allowedPercent`. This should improve performance and reduce the number of filesystem calls on machines with big amounts of RAM when performing heavy queries over big number of samples and time series.	2019-09-03 13:32:09 +03:00
Aliaksandr Valialkin	3685fc18d5	Makefile: extract `app-local` and `app-local-pure` build rules	2019-08-28 01:34:58 +03:00
Aliaksandr Valialkin	ede7ad3703	app/victoria-metrics: add missing `victoria-metrics` prefix to `--version` output when building with `make victoria-metrics`	2019-08-28 01:28:08 +03:00
Aliaksandr Valialkin	9196c085a7	all: port to FreeBSD on GOARCH=amd64	2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin	603a79b357	app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries	2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin	fb909cf710	app/vminsert/influx: set `db` label only if Influx line doesnt have `db` tag	2019-08-24 13:52:48 +03:00
Aliaksandr Valialkin	9c9221d1b2	app/vminsert: skip empty tags	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	70ca018a57	app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	4266091e4f	app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	8001d29b6e	app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	9d3f1fcbb9	app/vminsert/influx: skip invalid rows and continue parsing the remaining rows Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	ba7b3806be	app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life The prefious report with escaped newline chars in influx line protocol was false alarm.	2019-08-23 18:42:05 +03:00
Aliaksandr Valialkin	7fa88c6efc	app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life	2019-08-23 18:35:41 +03:00
Aliaksandr Valialkin	4da34b11f8	app/vminsert/influx: handle `\r\n` aka `crlf` influx line endings from windows world Such lines exist in real life.	2019-08-23 18:28:49 +03:00
Aliaksandr Valialkin	a18317adbc	app/vminsert/influx: allow escaping newline char Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ , there are reports that such chars occur in real life	2019-08-23 15:14:46 +03:00
Aliaksandr Valialkin	44d7fc599d	app/vminsert/influx: skip comments starting with `#` in influx line protocol	2019-08-23 14:43:09 +03:00
Aliaksandr Valialkin	ac004665b5	all: return 503 http error if service is temporarily unavailable Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin	8c03a8c4b4	app/vminsert: allow setting the maximum number of labels per time series via `-maxLabelsPerTimeseries`	2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin	1272e407b2	app/vmselect/promql: attempt to repair invalid bucket counts passed to `histogram_quantile` Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154	2019-08-22 14:39:46 +03:00
Aliaksandr Valialkin	5f33fc8e46	app/vminsert: add ability to ingest data via HTTP OpenTSDB `/api/put` requests This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152 Thanks to nustinov@gmail.com for the initial pull request.	2019-08-22 12:28:32 +03:00
Aliaksandr Valialkin	ec8125606d	app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing `put` prefixes to each line	2019-08-21 19:14:47 +03:00
Aliaksandr Valialkin	f4a38f7fb1	app/vmselect/promql: fix panic on `-search.disableCache` Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.	2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin	ab740afd0d	app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue	2019-08-21 11:29:24 +03:00
Aliaksandr Valialkin	7b5168adfb	app/vmselect/promql: remove NaNs from `/api/v1/query_range` output like Prometheus does Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153	2019-08-20 23:01:41 +03:00
Aliaksandr Valialkin	a0d480fbf3	app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries This should reduce memory allocations for big number of timeseries	2019-08-20 23:01:39 +03:00
Aliaksandr Valialkin	d3fd113a80	app/vmselect/promql: add `label_value(q, label_name)` func, which returns numeric value labels with name `label_name` in `q`	2019-08-20 00:28:34 +03:00
Aliaksandr Valialkin	dd86e6130c	app/vmselect/promql: independently track offset hints for tStart and tEnd This should improve performance if timeseries starts or ends on the selected time range	2019-08-19 13:40:14 +03:00
Aliaksandr Valialkin	6a27657d73	app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do This should improve the performance of queries over big number of time series with big number of output points.	2019-08-19 13:03:29 +03:00
Aliaksandr Valialkin	e74fb23189	app/vmselect/promql: add `scrape_interval(q[d])` function, which would return scrape interval for `q` over `d`	2019-08-18 21:08:26 +03:00
Aliaksandr Valialkin	582fdc059a	app/vmselect/promql: hande comparisons with `NaN` similar to Prometheus Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2019-08-18 00:25:50 +03:00
Aliaksandr Valialkin	1c108fc494	app/vmselect/promql: add `lifetime(q[d])` function, which returns the lifetime of `q` over `d` in seconds. This function is useful for determining time series lifetime. `d` must exceed the expected lifetime of the time series, otherwise the function would return values close to `d`.	2019-08-16 11:59:32 +03:00
Aliaksandr Valialkin	d6b5ed6d39	app/vmselect/promql: fix corner-case calculation for `ideriv`	2019-08-16 11:59:28 +03:00
Aliaksandr Valialkin	639b14e8ab	app/vmselect/promql: properly handle corner cases for rollup functions	2019-08-15 23:29:59 +03:00
Aliaksandr Valialkin	9ecb994671	app/vmselect/promql: store compressed results in the cache This should increase rollup results cache capacity.	2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin	09fc6e22e5	all: use workingsetcache instead of fastcache This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin	b8bb74ffc6	app/vmstorage: add `vm_concurrent_addrows_*` metrics for tracking concurrency for Storage.AddRows calls Track also the number of dropped rows due to the exceeded timeout on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`	2019-08-06 15:08:33 +03:00
Aliaksandr Valialkin	5c83f8e203	app: add `vm_concurrent_` metrics for visibility in concurrency limiters for vminsert and vmselect	2019-08-05 18:30:57 +03:00
Aliaksandr Valialkin	99e048c9df	app/vmselect: allow passing `match[]`, `start` and `time` to `/api/v1/label/<label_name>/values` `/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)` call in Grafana templating.	2019-08-04 23:09:21 +03:00
Aliaksandr Valialkin	47e4b50112	app/vmselect: optimize `/api/v1/series` by skipping storage data Fetch and process only time series metainfo.	2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin	241170dc05	app/vmselect/prometheus: prevent from fetching and scanning all the data on `/api/v1/searies` call by default	2019-08-04 19:42:36 +03:00
Aliaksandr Valialkin	1c69f4eadc	app/vmselect/promql: tune automatic window adjustement Increase the windows adjustement for small scrape intervals, since they usually have higher jitter. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134	2019-08-04 19:34:05 +03:00
Aliaksandr Valialkin	8d93b15b86	app/vmselect/promql: further increase the allowed jitter for scrape interval Real-world production data shows higher jitter than 1/8 of scrape interval. This may results in gaps on the graph. So increase the allowed jitter to 1/4 of scrape interval in order to reduce the probability of gaps on the graphs over time series with high jitter for scrape_interval.	2019-08-02 20:10:23 +03:00
Aliaksandr Valialkin	a9f39168d2	app/vminsert/influx: round automatically generated timestamp according to the given `precision` arg	2019-08-02 00:24:06 +03:00

... 9 10 11 12 13 ...

1152 commits