894 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	6954d0edb7	lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered	2020-04-26 14:34:11 +03:00
kreedom	2c18548e08	alert - rename validate function and flags (#440) ... * alert - rename validate function and flags	2020-04-26 14:15:04 +03:00
kreedom	5f61d43db9	vmalert - validate template in labels (#439)	2020-04-26 13:53:57 +03:00
肖贝贝	eeadfccdc5	fix: fix vmalert template label not complete bug (#435) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-26 13:30:10 +03:00
Aliaksandr Valialkin	1f3fd93b58	docs/{vmbackup,vmrestore}.md: update -help output	2020-04-24 22:44:21 +03:00
Jason Gardner	66af7e40f3	app/vmbackup: added ability to create and delete snapshots during backup (#428) ... * app/vmbackup: added ability to create and delete snapshots during backup Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422 * Add snapshot create and delete url flags * Fixed errcheck warnings in build	2020-04-24 22:35:03 +03:00
Aliaksandr Valialkin	a596aec82c	app/vmselect: fix description for -search.resetCacheAuthKey	2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin	9ef5935552	lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin	364db13c9c	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin	9ebc937685	app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin	fe57d46687	app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( ... Leave both flags for backwards compatibility reasons. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426	2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin	851946af1e	deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var ... For example, the following command will build VictoriaMetrics docker image on top of alpine image: ROOT_IMAGE=alpine make package-victoria-metrics	2020-04-20 01:16:57 +03:00
Aliaksandr Valialkin	936fb0eac3	app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection ... This should fix the following error when sending data to remote storage: couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-17 15:52:42 +03:00
Aliaksandr Valialkin	79fb595732	docs/vmagent.md: typo fix: unvailable -> unavailable	2020-04-17 13:11:31 +03:00
Aliaksandr Valialkin	546d26523c	app/vmagent/README.md: mention about prodmscrape.suppressScrapeErrors	2020-04-17 13:08:21 +03:00
Aliaksandr Valialkin	f41e6a7bd9	app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query	2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin	071fdf5518	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin	6f7f64f757	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin	067c7afebc	lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page ... This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin	0b76c27fa1	docs/vmagent.md: mention that vmagent supports kubernetes_sd_configs now	2020-04-13 21:06:36 +03:00
hagen1778	2eed6c393f	vmalert: prepare package for external usage ... * update README according to changes * add Makefile with basic commands	2020-04-12 15:32:42 +03:00
kreedom	948f8b6b5f	[vmalert] fix linter issues	2020-04-12 15:08:11 +03:00
kreedom	8fca5f2819	[vmalert] add tests to webserver (#413)	2020-04-12 14:51:03 +03:00
Roman Khavronenko	7c9405f53d	Vmalert metrics (#412) ... vmalert: add basic list of metrics	2020-04-11 20:42:01 +01:00
Roman Khavronenko	9f8cc8ae1b	Extend web responses for alerts: (#411) ... vmalert: Extend web responses for alerts * populate apiAlert object with additional fields * return all active alerts, not only firing * sort list of API alerts for deterministic output * add helper for available path list	2020-04-11 16:49:23 +01:00
kreedom	90de3086b3	[vmalert] add webserver (#410) ... * [vmalert] add webserver	2020-04-11 12:40:24 +03:00
Aliaksandr Valialkin	f94a090020	docs: update minimum supported Go version from 1.12 to 1.13	2020-04-07 13:38:37 +03:00
Roman Khavronenko	b099d84271	Vmalert/rules eval (#400) ... * Initial rules evaluation support. Rules are now store alerts state in private field `alerts`. Every evaluation updates the alerts and state. Every unique metric received from datastore represents a unique alert, uniqueness is guaranteed by hashing ordered labelset. * merge with master * cleanup * support endAt parameter as 3*evaluationInterval for active alerts * make golint happy	2020-04-06 14:44:03 +03:00
Aliaksandr Valialkin	895cadfae7	app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request ... This header is required by Cortex (and, probably, other remote storage systems). See 9c1f44d090/docs/apis.md (remote-api) . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399	2020-04-04 16:24:56 +03:00
Aliaksandr Valialkin	57704aa584	app/victoria-metrics: add -selfScrapeInstance and -selfScrapeJob flags for tuning labels for self-scraped metrics	2020-04-04 14:57:22 +03:00
Aliaksandr Valialkin	f9b24d4899	app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions	2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin	b38d048dd9	app/vmstorage: add vm_free_disk_space_bytes metric for monitoring the remaining disk space at -storageDataPath	2020-04-01 23:08:58 +03:00
kreedom	298eb0a0f8	[vmalert] improve external url handling	2020-04-01 22:29:11 +03:00
kreedom	12fe915b48	[vmalert] add prometheus template function (#396) ... * [vmalert] add prometheus template function * make linter be happy Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-04-01 18:17:53 +03:00
Aliaksandr Valialkin	e71519b8b2	app/victoria-metrics/testdata: add a test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 12:51:25 +03:00
Aliaksandr Valialkin	318326c309	lib/storage: properly handle {label=~"foo|"} filters as Prometheus does ... Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin	0c485f14d1	app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries	2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin	2ebf7d86ff	app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()	2020-03-29 21:50:28 +03:00
kreedom	bf6c24d0f4	[vmalert] config parser (#393) ... * [vmalert] config parser * make linter be happy * fix test * fix sprintf add test for rule validation	2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin	bd156cd088	docs/vmagent.md: add prometheus remote_write proxy use case	2020-03-28 23:16:38 +02:00
Dmitry Naumov	27373807c1	Rootless docker images by default (#358) ... * Rootless docker images by default * Migrate to rootless base image Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-03-27 21:23:50 +02:00
kreedom	b22da547a2	[vmalert] - parse template annotaions (#387) ... * [vmalert] - parse template annotations	2020-03-27 18:31:16 +02:00
Aliaksandr Valialkin	3c7c71a49c	app/vmselect: adjust label_map() handling for corner cases ... The following corner cases now supported: * label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label` * label_map(q, "label", "foo", "") - removes `label="foo"` from series All the unmatched labels are kept unchanged.	2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin	4fc4912f0c	app/vmalert/datasource: typo fix in docs: Labels -> Label	2020-03-13 12:22:33 +02:00
kreedom	a746cb62b6	vmalert add vm datasource, change alertmanager (#364) ... * vmalert add vm datasource, change alertmanager * make linter be happy * make linter be happy.2 * PR comments * PR comments.1	2020-03-13 12:19:31 +02:00
Aliaksandr Valialkin	fdc2a9d1d7	app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin	8023ad7dbd	app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model	2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin	fcd91795d5	app/vmagent: mention that vmagent can filter data	2020-03-11 16:22:39 +02:00
Aliaksandr Valialkin	301c2acd61	app/vmstorage: return 500 status code instead of 200 status code on internal errors inside /snapshot/* handlers	2020-03-10 23:51:55 +02:00
Aliaksandr Valialkin	e17702fada	app/vmselect: add optional max_rows_per_line query arg to /api/v1/export ... This arg allows limiting the number of data points that may be exported on a single line.	2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin	1fe66fb3cc	app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv	2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin	5fe38a84eb	app/vmagent: properly apply -remoteWrite.sendTimeout to fasthttp.HostClient	2020-03-09 13:31:55 +02:00
Aliaksandr Valialkin	c386c5de57	app/vmagent: properly add labels set via -remoteWrite.label to metrics before sending them to -remoteWrite.url	2020-03-06 19:26:58 +02:00
Aliaksandr Valialkin	c5f894b361	Makefile: add build and test rules with enabled race detector. These rules have -race suffix ... Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin	754db0d22e	app/vmagent/README.md: small fixes	2020-03-04 18:14:47 +02:00
Aliaksandr Valialkin	772312bf7b	app/vmagent/README.md: typo fix	2020-03-04 18:05:09 +02:00
Aliaksandr Valialkin	871abfab7a	app/vmagent/README.md: clarification	2020-03-04 18:03:48 +02:00
Aliaksandr Valialkin	007c591de8	app/vmagent/README.md: add iot and edge monitoring use case	2020-03-04 18:01:34 +02:00
Aliaksandr Valialkin	474a09c0f1	app/vmagent/README.md: add use cases section	2020-03-04 17:42:27 +02:00
Aliaksandr Valialkin	032c88561b	app/vminsert/prompush: limit memory usage by pushing promscrape data in smaller blocks	2020-03-03 19:58:54 +02:00
Aliaksandr Valialkin	76036c1897	app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352	2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin	c31d640eb9	app/vmagent/remotewrite: do not reset empty relabelCtx	2020-03-03 15:01:03 +02:00
Aliaksandr Valialkin	02b55c72dc	app/vmagent: add -remoteWrite.urlRelabelConfig for applying individual relabeling for each -remoteWrite.url ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308	2020-03-03 13:12:16 +02:00
Aliaksandr Valialkin	7d178a40bd	app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter ... This should reduce query time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin	f9a584b5c1	app/vmagent/remotewrite: yet another typo fix	2020-02-28 20:05:55 +02:00
Aliaksandr Valialkin	b9b46cb8dc	app/vmagent/remotewrite: typo fix	2020-02-28 19:03:16 +02:00
Aliaksandr Valialkin	db6f4e4af1	app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage	2020-02-28 18:58:01 +02:00
Aliaksandr Valialkin	5701b2f7bb	app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343	2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin	18af31a4c2	all: properly split vm_deduplicated_samples_total among cluster components ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin	c458b521a2	app/vmagent: allow setting -httpListenAddr to empty string in order to disable listening for http requests ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340	2020-02-26 20:58:11 +02:00
Aliaksandr Valialkin	117c76311c	app/vmagent/README.md: list service discovery mechanisms, which will be added soon ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-02-26 19:27:08 +02:00
Aliaksandr Valialkin	be37d762cd	app/vmagent: add -remoteWrite.maxBlockSize command-line flag for limiting the maximum size of unpacked block to send to remote storage	2020-02-25 19:57:47 +02:00
Aliaksandr Valialkin	4e24839a2c	app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize	2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin	6386aeb1e0	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP ... Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin	e453880084	app/vmagent/README.md: mention that vmagent exposes target statuses at /targets page	2020-02-25 18:15:58 +02:00
Aliaksandr Valialkin	4c4448b66e	app/vminsert: add /targets handler, which exposes Prometheus targets defined in -promscrape.config file	2020-02-25 18:13:11 +02:00
Aliaksandr Valialkin	e1ef72af01	app/vmagent: logo fix	2020-02-25 00:09:19 +02:00
Aliaksandr Valialkin	56c70fe856	app/vmagent: update docs	2020-02-25 00:09:18 +02:00
Aliaksandr Valialkin	e7e4aa5243	app/vmagent/README.md: small fixes	2020-02-24 21:25:38 +02:00
Aliaksandr Valialkin	ae51300973	app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin	e65ec88779	app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-23 18:03:57 +02:00
Yaroslav	a6d0645539	fix rollupOpen(), rollupHigh(), rollupLow() functions (#328)	2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin	04762344c6	app/vmagent: initial implementation for vmagent	2020-02-23 13:36:03 +02:00
kreedom	49390b8dbc	[vmalert] integration with AlertManager (#325)	2020-02-21 23:15:05 +02:00
Aliaksandr Valialkin	2f55cabaa4	app/vmselect/promql: log when rollupResult cache is cleared	2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin	ec9bf39b5b	app/vmselect: add -search.cacheTimestampOffset command-line flag ... This flag can be used for removing gaps on graphs if the difference between the current time and the timestamps from the ingested data exceeds 5 minutes. This is the case when the time between data sources and VictoriaMetrics is improperly synchronized. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin	539139391c	app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312	2020-02-21 13:58:05 +02:00
kreedom	3c06179184	basic vmalert backbone (#317) ... * basic vmalert backbone * Resolve code review comments for vmalert backbone * Second review fixes for vmalert backbone	2020-02-16 20:59:02 +02:00
Aliaksandr Valialkin	7b1c7051a3	app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions ... This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin	8466ab0034	all: allow setting flags via environment vars ... Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin	e210cd9da1	lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version	2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin	a6c6a2debc	app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8	2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin	78b62dee87	app/vmselect/promql: properly adjust time range for data to select ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin	366693b9f1	app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent	2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin	525101339e	app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin	ada6a3da8d	app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309	2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin	36a1ac8360	app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded ... This will prevent from excess CPU usage for timed out queries.	2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin	834051e5b2	app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana ... See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.	2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin	42864bb52f	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:46:52 +02:00
Artem Navoiev	a47f292295	[vmalert] add vmalert.png.2	2020-02-02 12:17:19 +02:00
Artem Navoiev	354232b62b	[vmalert] add vmalert.png	2020-02-02 12:16:05 +02:00
Artem Navoiev	28778be0cc	[vmalert] initial	2020-02-02 12:14:09 +02:00
Aliaksandr Valialkin	90cf356ea1	app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus	2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin	c3d86eef96	all: add -dedup.minScrapeInterval command-line flag for data de-duplication ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin	d70ba7eb37	lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor ... This should improve performance when reading many small blocks.	2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin	d68546aa4a	lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init ... This should speed up Search.NextMetricBlock loop for big number of found time series.	2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin	a462355b2f	app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value	2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin	371e86194d	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin	adbbc4fa1a	app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value ... Examples for such counters: OS-level counters for network or cpu stats.	2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin	75ad47a43c	app/victoria-metrics: check for error arg passed to filepath.Walk callback	2020-01-27 20:56:45 +02:00
Aliaksandr Valialkin	6320a19a8c	app/victoria-metrics: remove integration build tag from tests ... This simplifies testing with `go test ./app/victoria-metrics` without the need to remember to pass `-tags=integration` to Go commands.	2020-01-27 20:25:28 +02:00
Aliaksandr Valialkin	2392a359e1	app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296	2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin	9df5b2d1c3	app/victoria-metrics: add -selfScrapeInterval flag for self-scraping /metrics page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180	2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin	6456c93dbb	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin	680080887d	all: consistently log durations in seconds with millisecond precision ... This should improve logs readability	2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin	9773022e50	app/vmselect: mention the original query and time range in error messages ... This should simplify debugging invalid or heavy queries.	2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin	efc7ad88ec	app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors	2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin	ec9651e266	app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data	2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin	582dd01f42	app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d	2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin	36973ee975	app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin	227cf53ef9	app/vminsert: increase default value for -insert.maxQueueDuration from 30s to 60s ... This should help catching up with high ingestion rate after VictoriaMetrics restart.	2020-01-18 14:39:36 +02:00
Aliaksandr Valialkin	c9472e4f3a	all: use github.com/klauspost/compress/gzip instead of compress/gzip ... `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin	cdcacaea6d	app/vmselect/netstorage: make fmt	2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin	7327adbc86	app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file ... This should reduce memory usage on systems with more than 8GB RAM.	2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin	d0d258b314	app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs ... A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value. This will give more chances for executing insert path.	2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin	d88725f133	app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects	2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin	0d7505b00e	all: mention command-line flags used for limiting the incoming request size in error messages ... This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin	2839f4688a	app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args	2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin	8b14572f70	app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions ... These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin	8eaced8cae	app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval	2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin	ea231f8167	app/victoria-metrics: adjust integration tests after the commit 99facd71cd6ac151d512cea1df73be91c10c7f83	2020-01-11 00:58:16 +02:00
Aliaksandr Valialkin	46bfdbe6cf	app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions ... This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.	2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin	4f0a645f77	vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5 ... This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.	2020-01-10 23:15:15 +02:00
Aliaksandr Valialkin	b829fe5e39	app/vmselect/promql: properly handle aggr(aggr_over_time(...))	2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin	164278151f	app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function ... This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin	c4632faa9d	app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions ... These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin	f358fb72d1	app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup ... This should prevent from restoring from incomplete backups. Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.	2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin	d5aeda0e1a	app/vmselect/promql: skip rate calculation for the first point on time series	2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin	d410c78c7e	app/vmselect/promql: fix calculations for histogram_share	2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin	d64699bb9f	app/vmselect/promql: add missing MetricName into netstorage.Result in tests	2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin	f409f2d050	app/vmselect/promql: add histogram_share(le, buckets) function	2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin	b1ded7cf9a	app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 ... See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin	a8360d04c0	app/vmselect/promql: add histogram_over_time(m[d]) rollup function	2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin	3e09d38f29	app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time ... Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.	2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin	a774120460	app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series ... This should provide better estimation for `rate` in the beginning of time series.	2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin	b5645ccbdf	app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series	2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin	cb3a342882	app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations	2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin	6f67e0b56b	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin	1925ee038d	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:03:59 +02:00
Mike Poindexter	bec62e4e43	Split Extended PromQL parsing to a separate library	2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin	d880325cf6	app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range ... This should prevent from the following panic at app/vmselect/promql/binary_op.go:255: BUG: len(leftVaues) must match len(rightValues) and len(dstValues)	2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin	9e19949c6b	app/vmselect/promql: adjust calculations for rate and increase for the first value ... These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.	2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin	0455c03cb9	app/vmselect/promql: properly calculate rate on the first data point ... It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape, i.e. we can assume its value was 0 at this time.	2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin	5cb8d97743	all: use gozstd instead of pure Go zstd for GOARCH=amd64	2019-12-24 12:42:42 +02:00
Aliaksandr Valialkin	5b75984aa9	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin	bea5a8700a	app/vmselect: add -search.maxExportDuration command-line flag for limiting /api/v1/export duration ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275	2019-12-20 11:35:22 +02:00
Andrii Dembitskyi	2fba7b6f35	Fix typo in log message	2019-12-19 14:33:20 +02:00
Aliaksandr Valialkin	d03827c57d	app/vminsert: return StatusNoContent http response for /api/v1/import to be consistent with other insert handlers	2019-12-19 01:21:54 +02:00
Aliaksandr Valialkin	96ff8d9adb	app/vmselect: add ability to pass match[], start and end to /api/v1/labels ... This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`. See https://github.com/prometheus/prometheus/issues/6178 for more details.	2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin	02f6566ce1	app/vmbackup: mention that backups are possible to Ceph and Swift	2019-12-14 01:08:49 +02:00
Aliaksandr Valialkin	bc645152cb	app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin	cea5a14853	all: rename Extended PromQL to PromQL extensions	2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin	6ddcd162ed	all: publish Docker images for the following GOARCH: amd64, arm, arm64, ppc64le and 386 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/258	2019-12-11 23:32:59 +02:00
Aliaksandr Valialkin	73b2a3d4b7	app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed ... Updates https://github.com/prometheus/prometheus/issues/5706	2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin	caa4eb72d9	app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does	2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin	3c076544bf	app/vmselect/promql: allow negative offsets ... Updates https://github.com/prometheus/prometheus/issues/6282	2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin	68e1cf8942	app/vminsert: add /api/v1/import handler ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6	2019-12-09 20:59:04 +02:00
Aliaksandr Valialkin	8501b4a48d	app/vminsert: consistency renaming for counters	2019-12-09 16:43:10 +02:00
Aliaksandr Valialkin	293b541784	make vendor-update	2019-12-07 23:10:16 +02:00
Aliaksandr Valialkin	84b57e8974	app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689	2019-12-07 23:00:40 +02:00
Aliaksandr Valialkin	c09472dfd9	app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range ... The full list of functions added: - `topk_min(k, q)` - returns top K time series with the max minimums on the given time range - `topk_max(k, q)` - returns top K time series with the max maximums on the given time range - `topk_avg(k, q)` - returns top K time series with the max averages on the given time range - `topk_median(k, q)` - returns top K time series with the max medians on the given time range - `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range - `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range - `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range - `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range	2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin	face3d57bf	app/vmselect: add placeholders for /api/v1/rules and /api/v1/alerts	2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin	20812008a7	lib/storage: remove metricID with missing metricID->metricName entry ... The metricID->metricName entry can be missing in the indexdb after unclean shutdown when only a part of entries for new time series is written into indexdb. Recover from such a situation by removing the broken metricID. New metricID will be automatically created for time series with the given metricName when new data point will arive to it.	2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin	9476b73527	app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals ... This should improve scrape interval estimation for tiem series with gaps.	2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin	811b7a8303	app/vminsert/influx: allow empty measurement in Influx line protocol ... In this case metric names are mapped directly from field names without any prefixes.	2019-11-30 23:18:41 +02:00
Aliaksandr Valialkin	75ff524a4e	app/vmselect/promql: fix corner case for increase over time series with gaps ... In this case `increase` could return invalid high value for the first point after the gap.	2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin	96492348cb	deployment/docker/certs: update TLS certs source from alpine:3.9 to alpine:3.10	2019-11-29 19:57:29 +02:00
Aliaksandr Valialkin	f733cb2186	lib/backup: cosmetic fixes after #243	2019-11-29 18:07:04 +02:00
glebsam	15b7406f7b	Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243) ... * Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile * make fmt	2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin	4c859d980c	app/vmselect/prometheus: consistently apply nocache arg to /api/v1/query the same way ast to /api/v1/query_range ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241	2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin	110235f789	app/vmselect/prometheus: fix content-type for /api/v1/export responses ... The correct Content-Type should be `application/stream+json` instead of `application/json` Thanks to Joshua Ryder for pointing to this.	2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin	205233d9a7	app/vmselect/promql: remove zero timeseries from prometheus_buckets output	2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin	3f99f39e9b	app/vmselect/prometheus: reduce default value for -search.latencyOffset from 60s to 30s ... 30 seconds should be enough for almost all the cases	2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin	e91cb34c0e	app/vmselect/promql: allow nested parens	2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin	826dfd63a5	vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1	2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin	0401969d78	app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph ... This should reduce the amounts memory allocations	2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin	da98703748	app/vmselect/promql: optimize binary search over big number of samples during rollup calculations	2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin	c28876172f	app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0	2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin	50ae1879c6	app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series	2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin	8582b50360	app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets	2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin	19dfe52254	app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets	2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin	4bb88843cf	app/vmselect: add vm_per_query_{rows,series}_processed_count histograms	2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin	7753c8c0a1	app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile	2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin	c4287b3c86	app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets	2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin	1f3fd2c910	app/vmselect: adjust end arg instead of adjusting start arg if start > end ... `start` arg has higher chances to be set properly comparing to `end` arg, so it is expected that the `end` arg could be adjusted if it was set incorrectly.	2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin	7a4635f853	all: remove the remaining mentions of cluster version	2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin	443189fb0a	app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore	2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin	0094bc4fc9	app/vmselect/prometheus: properly adjust too big time time on /api/v1/query ... Too big `time` must be adjusted to `now()-queryOffset`.	2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin	3f1637fae8	app/vmselect/promql: properly calculate integrate(q[d])	2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin	c56b9ed03b	app/victoria-metrics: add build rules for GOARCH=ppc64le ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235	2019-11-13 20:24:33 +02:00
Aliaksandr Valialkin	3fd32e331a	app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235	2019-11-13 20:24:26 +02:00
Aliaksandr Valialkin	119dfd01bb	lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"} metric	2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin	86a1cd700b	lib/storage: remove inmemory index for recent hour, since it uses too much memory ... Production workload shows that the index requires ~4Kb of RAM per active time series. This is too much for high number of active time series, so let's delete this index. Now the queries should fall back to the index for the current day instead of the index for the recent hour. The query performance for the current day index should be good enough given the 100M rows/sec scan speed per CPU core.	2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin	c57eb0ff83	lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour ... This may be useful for saving RAM on high number of time series aka high cardinality	2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin	ca259864e2	lib/storage: return back inmemory inverted index for recent hour ... Issues fixed: - Slow startup times. Now the index is loaded from cache during start. - High memory usage related to superflouos index copies every 10 seconds.	2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin	01bb3c06c7	lib/storage: remove inmemory inverted index for recent hours ... Production load with >10M active time series showed it could slow down VictoriaMetrics startup times and could eat all the memory leading to OOM. Remove inmemory inverted index for recent hours until thorough testing on production data shows it works OK.	2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin	6c2303764e	Revert "lib/fs: do not postpone directory removal on NFS error" ... This reverts commit 4c02e496f7. Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234	2019-11-12 16:18:09 +02:00
Aliaksandr Valialkin	661dd190bb	Refer to https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883 from multiple places in README.md	2019-11-12 13:02:39 +02:00
Oleg Kovalov	b4f44befa3	fix misspelled words (#229)	2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin	8e8f98f712	lib/storage: add tests for dateMetricIDCache	2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin	56d7cc8a0d	app/victoria-metrics: remove deprecated fs.MustStopDirRemover from main_test.go	2019-11-10 13:37:13 +02:00
Aliaksandr Valialkin	4c02e496f7	lib/fs: do not postpone directory removal on NFS error ... Continue trying to remove NFS directory on temporary errors for up to a minute. The previous async removal process breaks in the following case during VictoriaMetrics start - VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them. - The transaction instructs removing old directories for parts, which were already merged into bigger part. - VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors. - VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished and finds directories, which should be removed, but weren't still removed due to NFS errors. - VictoriaMetrics panics when it finds unexpected empty directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin	5c3fa59181	app/vmrestore: the upcoming release would be 1.29.0	2019-11-10 00:20:41 +02:00
Aliaksandr Valialkin	ee7765b10d	lib/storage: implement per-day inverted index	2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin	5810ba57c2	lib/storage: use specialized cache for (date, metricID) entries ... This improves ingestion performance.	2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin	6ad7fe8eeb	lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	63b05c0b9f	app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions ... Incremental aggregate functions don't keep all the selected time series in memory - they keep only up to GOMAXPROCS time series for incremental aggregations. Take into account that the number of time series in RAM can be higher if they are split into many groups with `by (...)` or `without (...)` modifiers. This should reduce the number of `not enough memory for processing ... data points` false positive errors.	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	d888b21657	lib/storage: add inmemory inverted index for the last hour ... It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	1e46961d68	app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	e472f0b23b	lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter ... The origin of the error has been detected and documented in the code, so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`, so it could be monitored and alerted on high error rates. Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`, so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.	2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin	2f58f37f07	app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q	2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin	d18ea0c95b	app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges	2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin	b0295dbf2e	app/vmselect: add -search.latencyOffset flag for tuning the time after data collection when data points become visible in query results ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218	2019-10-28 12:31:07 +02:00
Aliaksandr Valialkin	e83fe938c8	all: make fmt	2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin	97ce4e03a5	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin	f752479cb8	app/victoria-metrics/test: add missing docs to public funcs PopulateTimeTplString and PopulateTimeTpl	2019-10-17 00:50:46 +03:00
Aliaksandr Valialkin	61e956e175	app/victoria-metrics: add a test for max_lookback=<duration> query arg ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	c66a691593	app/vmselect/prometheus: add -search.maxLookback command-line flag for overriding dynamic calculations for max lookback interval ... This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically from interval between datapoints for each input time series if the flag isn't set. The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	cc21b31502	app/victoria-metrics/test: add a test for PopulateTimeTplString	2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin	956fdd89d3	app/vmselect/promql: take into account the previous point when calculating max_over_time and min_over_time ... This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns the minimum value. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204	2019-10-08 12:30:05 +03:00
Artem Navoiev	1e2c511747	Add regression test for query apo ... Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2019-10-07 22:18:04 +03:00
Artem Navoiev	a116f5e7c1	Add regression test for query apo (#194) ... Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184	2019-09-30 11:25:54 +03:00
Aliaksandr Valialkin	4e9e1ca0f7	app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially ... This became the case after b7ee2e7af2 .	2019-09-30 00:11:14 +03:00
Aliaksandr Valialkin	c1d3705be0	app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock ... This allows re-using the destination buffer for marshaling in the outer loop.	2019-09-28 21:07:13 +03:00
Aliaksandr Valialkin	b7ee2e7af2	app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series	2019-09-28 21:07:09 +03:00
Aliaksandr Valialkin	67d44b0845	app/vmselect/promql: do not generate timestamps for NaN values in timestamp function according to Prometheus logic	2019-09-27 18:54:43 +03:00
Artem Navoiev	1e6ae9eff4	Add regression test for duplicated labels and series ... Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187 cover: - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/155 - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172	2019-09-27 16:52:16 +03:00
Artem Navoiev	cf4786f34a	add test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-09-26 12:45:19 +03:00
Aliaksandr Valialkin	26dc21cf64	app/vmselect/promql: add increases_over_time and decreases_over_time functions ... `increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`. `decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.	2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin	d32f88c378	app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls	2019-09-19 22:15:59 +03:00
Aliaksandr Valialkin	d4218d27e6	app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step])) ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184	2019-09-13 21:41:04 +03:00
Aliaksandr Valialkin	4d636c244d	app/vmselect/promql: binary operation fixes according to Prometheus behaviour ... The follosing issues were fixed: - VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers - VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list	2019-09-13 17:42:52 +03:00
Artem Navoiev	4fd741f40d	[tests] check timestamp in tests (#177)	2019-09-08 19:48:38 +03:00
Artem Navoiev	22e3fabefd	Add OpenTSDB and Prometheus integration tests (#168) ... * [WIP] open tsdb and prometheus integration tests * app/victoria-metrics: fix race condition on parallel tests	2019-09-05 17:55:38 +03:00
Aliaksandr Valialkin	88f8670ede	lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown ... This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin	6954e126fc	app/vmselect/promql: ignore grouping by destination label in count_values, since such a grouping is performed automatically	2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin	305e7bc981	app/vmselect/promql: do not return artificial points beyond the last point in time series	2019-09-04 16:35:34 +03:00
Aliaksandr Valialkin	9721d06c6a	app/vmselect/prometheus: do not adjust start and end args in /api/v1/query_range if nocache=1 arg is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171	2019-09-04 13:10:09 +03:00
Aliaksandr Valialkin	db4560ca31	app/vmselect/promql: reset timeseries name on group_left and group_right as Prometheus does	2019-09-03 20:42:54 +03:00
Aliaksandr Valialkin	1575a560f0	app/vmselect/netstorage: adaptively adjust the maximum inmemory file size for storing temporary blocks ... The maximum inmemory file size now depends on `-memory.allowedPercent`. This should improve performance and reduce the number of filesystem calls on machines with big amounts of RAM when performing heavy queries over big number of samples and time series.	2019-09-03 13:32:09 +03:00
Aliaksandr Valialkin	3685fc18d5	Makefile: extract app-local and app-local-pure build rules	2019-08-28 01:34:58 +03:00
Aliaksandr Valialkin	ede7ad3703	app/victoria-metrics: add missing victoria-metrics prefix to --version output when building with make victoria-metrics	2019-08-28 01:28:08 +03:00
Aliaksandr Valialkin	9196c085a7	all: port to FreeBSD on GOARCH=amd64	2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin	603a79b357	app/vmstorage: increase default values for search.maxTagKeys, search.maxTagValues and search.maxUniqueTimeseries	2019-08-27 14:29:53 +03:00
Aliaksandr Valialkin	fb909cf710	app/vminsert/influx: set db label only if Influx line doesnt have db tag	2019-08-24 13:52:48 +03:00
Aliaksandr Valialkin	9c9221d1b2	app/vminsert: skip empty tags	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	70ca018a57	app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	4266091e4f	app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	8001d29b6e	app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows ... Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	9d3f1fcbb9	app/vminsert/influx: skip invalid rows and continue parsing the remaining rows ... Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.	2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin	ba7b3806be	app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life ... The prefious report with escaped newline chars in influx line protocol was false alarm.	2019-08-23 18:42:05 +03:00
Aliaksandr Valialkin	7fa88c6efc	app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life	2019-08-23 18:35:41 +03:00
Aliaksandr Valialkin	4da34b11f8	app/vminsert/influx: handle \r\n aka crlf influx line endings from windows world ... Such lines exist in real life.	2019-08-23 18:28:49 +03:00
Aliaksandr Valialkin	a18317adbc	app/vminsert/influx: allow escaping newline char ... Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ , there are reports that such chars occur in real life	2019-08-23 15:14:46 +03:00
Aliaksandr Valialkin	44d7fc599d	app/vminsert/influx: skip comments starting with # in influx line protocol	2019-08-23 14:43:09 +03:00
Aliaksandr Valialkin	ac004665b5	all: return 503 http error if service is temporarily unavailable ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin	8c03a8c4b4	app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries	2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin	1272e407b2	app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154	2019-08-22 14:39:46 +03:00
Aliaksandr Valialkin	5f33fc8e46	app/vminsert: add ability to ingest data via HTTP OpenTSDB /api/put requests ... This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152 Thanks to nustinov@gmail.com for the initial pull request.	2019-08-22 12:28:32 +03:00
Aliaksandr Valialkin	ec8125606d	app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing put prefixes to each line	2019-08-21 19:14:47 +03:00
Aliaksandr Valialkin	f4a38f7fb1	app/vmselect/promql: fix panic on -search.disableCache ... Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.	2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin	ab740afd0d	app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue	2019-08-21 11:29:24 +03:00
Aliaksandr Valialkin	7b5168adfb	app/vmselect/promql: remove NaNs from /api/v1/query_range output like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153	2019-08-20 23:01:41 +03:00
Aliaksandr Valialkin	a0d480fbf3	app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries ... This should reduce memory allocations for big number of timeseries	2019-08-20 23:01:39 +03:00
Aliaksandr Valialkin	d3fd113a80	app/vmselect/promql: add label_value(q, label_name) func, which returns numeric value labels with name label_name in q	2019-08-20 00:28:34 +03:00
Aliaksandr Valialkin	dd86e6130c	app/vmselect/promql: independently track offset hints for tStart and tEnd ... This should improve performance if timeseries starts or ends on the selected time range	2019-08-19 13:40:14 +03:00
Aliaksandr Valialkin	6a27657d73	app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do ... This should improve the performance of queries over big number of time series with big number of output points.	2019-08-19 13:03:29 +03:00
Aliaksandr Valialkin	e74fb23189	app/vmselect/promql: add scrape_interval(q[d]) function, which would return scrape interval for q over d	2019-08-18 21:08:26 +03:00
Aliaksandr Valialkin	582fdc059a	app/vmselect/promql: hande comparisons with NaN similar to Prometheus ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150	2019-08-18 00:25:50 +03:00
Aliaksandr Valialkin	1c108fc494	app/vmselect/promql: add lifetime(q[d]) function, which returns the lifetime of q over d in seconds. ... This function is useful for determining time series lifetime. `d` must exceed the expected lifetime of the time series, otherwise the function would return values close to `d`.	2019-08-16 11:59:32 +03:00
Aliaksandr Valialkin	d6b5ed6d39	app/vmselect/promql: fix corner-case calculation for ideriv	2019-08-16 11:59:28 +03:00
Aliaksandr Valialkin	639b14e8ab	app/vmselect/promql: properly handle corner cases for rollup functions	2019-08-15 23:29:59 +03:00
Aliaksandr Valialkin	9ecb994671	app/vmselect/promql: store compressed results in the cache ... This should increase rollup results cache capacity.	2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin	09fc6e22e5	all: use workingsetcache instead of fastcache ... This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin	b8bb74ffc6	app/vmstorage: add vm_concurrent_addrows_* metrics for tracking concurrency for Storage.AddRows calls ... Track also the number of dropped rows due to the exceeded timeout on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`	2019-08-06 15:08:33 +03:00
Aliaksandr Valialkin	5c83f8e203	app: add vm_concurrent_ metrics for visibility in concurrency limiters for vminsert and vmselect	2019-08-05 18:30:57 +03:00
Aliaksandr Valialkin	99e048c9df	app/vmselect: allow passing match[], start and time to /api/v1/label/<label_name>/values ... `/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)` call in Grafana templating.	2019-08-04 23:09:21 +03:00
Aliaksandr Valialkin	47e4b50112	app/vmselect: optimize /api/v1/series by skipping storage data ... Fetch and process only time series metainfo.	2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin	241170dc05	app/vmselect/prometheus: prevent from fetching and scanning all the data on /api/v1/searies call by default	2019-08-04 19:42:36 +03:00
Aliaksandr Valialkin	1c69f4eadc	app/vmselect/promql: tune automatic window adjustement ... Increase the windows adjustement for small scrape intervals, since they usually have higher jitter. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/139 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134	2019-08-04 19:34:05 +03:00
Aliaksandr Valialkin	8d93b15b86	app/vmselect/promql: further increase the allowed jitter for scrape interval ... Real-world production data shows higher jitter than 1/8 of scrape interval. This may results in gaps on the graph. So increase the allowed jitter to 1/4 of scrape interval in order to reduce the probability of gaps on the graphs over time series with high jitter for scrape_interval.	2019-08-02 20:10:23 +03:00
Aliaksandr Valialkin	a9f39168d2	app/vminsert/influx: round automatically generated timestamp according to the given precision arg	2019-08-02 00:24:06 +03:00
Aliaksandr Valialkin	f090b2e917	app/vmselect/promql: tolerate higher jitter in scrape interval ... Allow jitter for up to 1/8 instead of 1/16 for the scrape interval. This should imrpove graphs when `step` is smaller than the `scrape_interval`.	2019-08-01 23:26:00 +03:00
Aliaksandr Valialkin	3a38b23fa3	app/vmselect/promql: add vm_slow_queries_total metric for counting slow queries ... The query is slow if its execution time exceeds `-search.logSlowQueryDuration`	2019-07-31 03:36:37 +03:00
Aliaksandr Valialkin	8bd6f1f6df	app/vmselect/promql: return NaN from histogram_quantile if at least a single bucket is broken	2019-07-31 01:18:07 +03:00
Aliaksandr Valialkin	4aaa5c2efc	app/vmselect/promql: allow adjusting window for default rollup function ... Default rollup function is `last_over_time`. It must support adjusting the provided window in order to prevent from gaps on the graph for window values smaller than scrape interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/134	2019-07-31 00:45:54 +03:00
Aliaksandr Valialkin	10f5a26bec	app/vmselect/promql: return NaN values if invalid bucket counts are passed to histogram_quantile ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136	2019-07-30 22:05:10 +03:00
Aliaksandr Valialkin	afcc7fb167	app/vmselect/netstorage: improve error message when reading data blocks from storage ... Mention the block number in the error. This should simplify troubleshooting in this code.	2019-07-28 12:12:35 +03:00
Aliaksandr Valialkin	1eade9b358	app/vminsert: add vm_rows_per_insert summary metric ... This metric should help tuning batch sizes on clients writing data to VictoriaMetrics	2019-07-27 13:21:46 +03:00
Aliaksandr Valialkin	0168a1b658	package: various fixes ... - Use `-prod` binaries instead of development binaries for both deb and rpm packages. - Fix binary directory from /usr/sbin to /usr/local/bin as outlined in package/victoria-metrics.service - Fix binary name from `victoriametrics` to `victoria-metrics-prod` in package/victoria-metrics.service	2019-07-26 22:31:04 +03:00
Aliaksandr Valialkin	bf6cbb762c	app/vminsert: improve error messages for Influx, OpenTSDB and Graphite parsing ... Include in the error message the line which failed to parse. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/127	2019-07-26 22:08:52 +03:00
Aliaksandr Valialkin	c98725db55	app/vmstorage: consistency renaming for ignored rows metrics ... vm_too_big_timestamp_rows_total -> vm_rows_ignored_total{reason="big_timestamp"} vm_too_small_timestamp_rows_total -> vm_rows_ignored_total{reason="small_timestamp"}	2019-07-26 20:02:06 +03:00
Aliaksandr Valialkin	f586e1f83c	lib/storage: add metrics for calculating skipped rows outside the retention ... The metrics are: - vm_too_big_timestamp_rows_total - vm_too_small_timestamp_rows_total	2019-07-26 14:11:01 +03:00
Kostya Vasilyev	89eb6d78a4	RPM packaging (#122)	2019-07-25 23:47:41 +03:00
Aliaksandr Valialkin	17096b5750	app/vmselect/promql: return NaN from count() over zero time series ... This aligns `count` behavior with Prometheus.	2019-07-25 22:02:30 +03:00
Aliaksandr Valialkin	66efa5745f	app/vmselect/promql: properly calculate incremental aggregations grouped by __name__ ... Previously the following query may fail on multiple distinct metric names match: sum(count_over_time{__name__!=''}) by (__name__)	2019-07-25 21:53:20 +03:00
Aliaksandr Valialkin	162f1fb1b7	all: small updates after PR #114	2019-07-23 19:54:50 +03:00
Roman Khavronenko	5bf4e5ffb5	all: add Pure Go build (pull request #114) ... Updates #94	2019-07-23 19:26:39 +03:00
Kostya Vasilyev	8c3629a892	Debian packaging (#116) ... * initial commit of deb packaging * Incorporated feedback from @valyala: - Put data directory under /var/lib - More beef in systemd file - Packaging for arm64 - Package all target which builds and packages both amd64 and arm64 * Remove PIDFile from systemd unit, useless * per PR feedback, move debian specific files into deb subdirectory Updates #107 .	2019-07-22 17:12:48 +03:00
Aliaksandr Valialkin	575d2f0a91	app/vminsert: use netutil.TCPListener for collecting network-related metrics for Graphite and OpenTSDB TCP traffic	2019-07-15 22:58:00 +03:00
Aliaksandr Valialkin	6a943a6a58	app/vmselect/promql: remove empty time series after applying filters like q > 0 ... This should reduce CPU and RAM usage for queries over high number of time series.	2019-07-12 19:59:27 +03:00
Aliaksandr Valialkin	ab88890523	app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores ... This may reduce response times for aggregation over big number of time series with small step between output data points.	2019-07-12 15:52:22 +03:00
Aliaksandr Valialkin	62a7353479	app/vmselect/prometheus: set start arg in /api/v1/series to the minimum allowed time by default as Prometheus does ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91	2019-07-11 17:10:14 +03:00
Aliaksandr Valialkin	54bd21eb4a	app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage	2019-07-11 17:07:20 +03:00
Artem Navoiev	cd4833d3d0	integration tests	2019-07-11 15:48:08 +03:00
Aliaksandr Valialkin	101fa258e5	app/vmstorage: prepare for integration tests with multiple Init / Stop cycles	2019-07-11 15:34:50 +03:00
Aliaksandr Valialkin	a336bb4e22	app/vmselect/promql: reduce RAM usage for aggregates over big number of time series ... Calculate incremental aggregates for `aggr(metric_selector)` function instead of keeping all the time series matching the given `metric_selector` in memory.	2019-07-10 13:04:39 +03:00
Aliaksandr Valialkin	1fe6d784d8	all: consistency renaming: bytesSize -> sizeBytes	2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin	55fe36149c	app/vmselect/promql: mention -search.logSlowQueryDuration flag value in the slow query log message	2019-07-10 00:41:24 +03:00
Aliaksandr Valialkin	9203170eb2	app/vmselect/promql: extract rmoeveGroupTags function for removing unneeded tags from MetricName according to the given modifierExpr	2019-07-09 23:20:48 +03:00
Aliaksandr Valialkin	2db685c19c	app/vmselect/promql: properly preserve metric name after applying functions in any case from transformFuncsKeepMetricGroup	2019-07-09 23:10:35 +03:00
Aliaksandr Valialkin	40a6c0d672	app/vmselect/prometheus: typo fix	2019-07-07 23:34:23 +03:00
Aliaksandr Valialkin	1371024747	app/vmselect/prometheus: handle minTime and maxTime values that may be set by Promxy or Prometheus client ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/88	2019-07-07 21:53:48 +03:00
Aliaksandr Valialkin	4dbd642c86	app/vmselect/promql: remove empty timeseries left after topk call	2019-07-04 19:42:39 +03:00
Aliaksandr Valialkin	56c154f45b	all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size	2019-07-04 19:42:30 +03:00
Aliaksandr Valialkin	9a4b2b8315	app/vmselect/prometheus: update adjustLastPoints function ... - Do not overwrite last points by the previous NaNs, since this may result in empty time series. - Overwrite the last 2 points instead of 3. This should be enough in most cases.	2019-07-04 09:14:18 +03:00
Aliaksandr Valialkin	e06866005d	app/vmselect/promql: gracefully handle duplicate timestamps in irate and rollup_rate funcs ... Previously such timestamps result in `+Inf` results. Now the previous timestamp is used for the calculations.	2019-07-03 12:39:55 +03:00
Aliaksandr Valialkin	b9166a60ff	app/vmselect: do not return empty time series in /api/v1/query result	2019-07-01 17:16:34 +03:00
Aliaksandr Valialkin	74c0fb04f3	app/vmselect/promql: consistency renaming: candlestick -> rollup_candlestick	2019-06-29 03:13:02 +03:00
Aliaksandr Valialkin	55bcf60ea6	app/vmselect: fix 32bit arm build ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/83	2019-06-27 19:36:17 +03:00
Aliaksandr Valialkin	796b010139	app/vmselect: add candlestick(m[d]) func for returning open, close, low and high rollups on the given time range d ... This function is frequently used in financial apps. See https://en.wikipedia.org/wiki/Candlestick_chart	2019-06-27 18:46:13 +03:00
Jiri Tyr	e0fccc6c60	Change the default influxMeasurementFieldSeparator	2019-06-26 13:22:03 +03:00
Aliaksandr Valialkin	1f7d9a213a	app/vminsert: fix inifinite loop when reading two lines without newline in the end ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/82	2019-06-26 02:51:56 +03:00
Aliaksandr Valialkin	844ce4731e	app/vmselect/promql: suppress error when template func is used inside modifier list. Just leave it as is ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/78	2019-06-25 20:43:22 +03:00
Aliaksandr Valialkin	feacfffe89	app/vmselect/promql: increase default value for -search.maxPointsPerTimeSeries from 10k to 30k ... This may be required for subqueries with small steps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/77	2019-06-24 22:53:18 +03:00
Aliaksandr Valialkin	4bb738ddd9	app/vmselect/promql: adjust value returned by linearRegression to the end of time range like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71	2019-06-24 22:45:58 +03:00
Aliaksandr Valialkin	90e72c2a42	app/vmselect/promql: add sum2 and sum2_over_time, geomean and geomean_over_time funcs. ... These functions may be useful for statistic calculations.	2019-06-24 16:44:44 +03:00
Aliaksandr Valialkin	61926bae01	app/vmselect/promql: adjust the provided window only for range functions with dt in denominator ... This should fix range function calculations such as `changes(m[d])` where `d` is smaller than the scrape interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72	2019-06-23 19:27:31 +03:00
Aliaksandr Valialkin	ee13256f74	app/vmselect/promql: use deriv_fast instead of deriv in ttf, since deriv calculations have been changed recently	2019-06-23 15:54:18 +03:00
Aliaksandr Valialkin	3b3b2f1e6e	app/vmselect/promql: adjust ttf calculation, so deriv(freev) for freev=m[d] could be properly calculated	2019-06-23 14:31:19 +03:00
Aliaksandr Valialkin	146c6e1f72	app/vmselect/promql: typo fixes in comments	2019-06-21 23:22:59 +03:00
Aliaksandr Valialkin	d261fa2885	app/vmselect/promql: add deriv_fast function for calculating fast derivative ... `deriv_fast` calculates derivative based on the first and the last point on the interval instead of calculating linear regression based on all the data points on the interval. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73	2019-06-21 23:05:39 +03:00
Aliaksandr Valialkin	5b47c00910	app/vmselect/promql: use linear regression in deriv func like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/73	2019-06-21 22:59:46 +03:00
Aliaksandr Valialkin	9e1119dab8	app/vmselect/promql: ajdust data model to the model used in Prometheus ... Do not take into account data points on the range `[timestamp .. timestamp+step)` when calculating value on the given `timestamp`. Use only data points from the past when performing these calculations like Prometheus does. This should reduce discrepancies between results returned by VictoriaMetrics and results returned by Prometheus. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/72 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/71	2019-06-21 21:54:48 +03:00
Aliaksandr Valialkin	47a3228108	app/vmselect/promql: do not strip __name__ form time series after binary comparison operation ... Example: foo > 10 Would leave `foo` name for all the matching time series on the left.	2019-06-21 13:09:38 +03:00
Aliaksandr Valialkin	973e4b5b76	app/vmselect/promql: remove unused func keepLastValue; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:35:11 +03:00
Aliaksandr Valialkin	7aadec8e3c	app/vmselect/promql: typo fix; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:33:47 +03:00
Aliaksandr Valialkin	4b2523fb40	app/vminsert/opentsdb: remove unused const maxReadPacketSize; update https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:30:06 +03:00
Aliaksandr Valialkin	70ba36fa37	app/vmselect/prometheus: return better error messages on missing args to /api/v1/*	2019-06-20 14:07:55 +03:00
Aliaksandr Valialkin	a78b3dba7f	app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count	2019-06-19 18:36:47 +03:00
Aliaksandr Valialkin	a8d4224828	app/vminsert/graphite: allow skipping timestamps in Graphite plaintext protocol ... In this case VictoriaMetrics uses the ingestion time as a timestamp.	2019-06-18 19:04:04 +03:00
Aliaksandr Valialkin	86d3d907a5	app/vminsert/influx: add -influxSkipSingleField flag for using {measurement} instead of {measurement}{separator}{field_name} for Influx lines with a single field ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/66	2019-06-17 19:05:57 +03:00
Aliaksandr Valialkin	269285848f	app/vminsert/influx: add -influxMeasurementFieldSeparator flag for the ability to change separator for {measurement}{separator}{field_name} metric name	2019-06-14 10:00:12 +03:00
Aliaksandr Valialkin	beb479b8f1	app/vmselect/promql: use dynamic limit on memory for concurrent queries	2019-06-12 23:18:44 +03:00
Aliaksandr Valialkin	a8db528930	app/vmselect/promql: merge non-overlapping duplicate time series in group_left and group_right joins	2019-06-12 20:32:32 +03:00
Aliaksandr Valialkin	15613e5338	app/vmselect/promql: swap binary operation with modifier in the error message for improved readability	2019-06-12 17:14:39 +03:00
Aliaksandr Valialkin	3237d0309c	app/vmselect/promql: list a sample of duplicate time series in the error message for group_left or group_right ... This should improve troubleshooting for complex queries involving `group_left` and `group_right` modifiers.	2019-06-12 16:57:37 +03:00
Aliaksandr Valialkin	419197ba08	lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func ... The func syncs parent dir in order to persist directory removal in the event of power loss	2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin	ac7b186f13	all: try hard removing directory with contents ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin	1c33b5937e	app/vmselect/promql: prevent from count_values explosion of timeseries, which could result in OOM	2019-06-11 01:03:13 +03:00
Aliaksandr Valialkin	8bb6bc986d	app/vmselect/promql: skip superflouos timestamps copying in count_values	2019-06-11 00:44:01 +03:00
Aliaksandr Valialkin	d2be567482	app/vmselect/promql: remove superflouos timeseries copy in histogram_quantile func	2019-06-11 00:39:41 +03:00
Aliaksandr Valialkin	7e7d4d5275	app/vmselect/promql: remove superflouos timeseries copy in union func	2019-06-11 00:35:20 +03:00
Aliaksandr Valialkin	bf9782eaf6	app/vmselect/promql: skip NaN values in count_values func	2019-06-10 22:42:32 +03:00
Aliaksandr Valialkin	cbe692f0e2	app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values	2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin	7b6623558f	lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter ... This should improve speed for searching metrics among high number of time series with high churn rate like in big Kubernetes clusters with frequent deployments.	2019-06-10 14:07:39 +03:00
Aliaksandr Valialkin	7354090aad	app/vmstorage: add missing _total suffixes to newly added metrics	2019-06-09 22:11:36 +03:00
Aliaksandr Valialkin	d37924900b	lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)	2019-06-09 19:13:56 +03:00
Aliaksandr Valialkin	c0baa977cf	app/vminsert/concurrencylimiter: typo fix in the error message	2019-06-08 22:43:33 +03:00
Aliaksandr Valialkin	f4252f87e6	app/vminsert: really fix #60 ... ReadLinesBlock may accept dstBuf with non-zero length. In this case the last line without trailing newline isn't read. Fix this by comparing len(dstBuf) to 0 instead of its original length.	2019-06-07 23:37:03 +03:00
Aliaksandr Valialkin	0b78d228d2	app/vminsert: properly read trailing line without newline in the end ... This fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/60	2019-06-07 23:17:59 +03:00
Aliaksandr Valialkin	c1f18ee48d	app/vmselect/promql: properly handle {__name__ op "string"} queries ... This has been broken in 7294ef333ad26f4f6578b783e97649e58b1f8945 .	2019-06-07 02:02:04 +03:00
Aliaksandr Valialkin	667115a5c7	app/vmselect/prometheus: report about incorrect time or duration instead of silently using the default value ... This should prevent from incorrect usage of the querying API. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/52	2019-06-06 22:18:18 +03:00
Aliaksandr Valialkin	1458450dba	app/vmselect/promql: return the correct time series from quantile ... Previously arbitrary time series could be returned from `quantile` depending on sort order for the last data point in the selected range. Fix this by returning the calculated time series. Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/55	2019-06-06 17:07:31 +03:00
Aliaksandr Valialkin	53ea90865d	app/vmselect/promql: add -search.disableCache flag for disabling response caching ... This may be useful for data back-filling, when the response caching could interfere badly with newly added data points with timestamps in the past.	2019-06-04 17:30:45 +03:00
Aliaksandr Valialkin	17f0a53068	app/vminsert: explain that /query request emulation is required for TSBS benchmark	2019-06-03 18:40:27 +03:00
Aliaksandr Valialkin	f2cf5d8e36	app/vmselect/promql: allow escaping identifiers with \ and \xXX ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/42	2019-05-31 17:35:17 +03:00
Aliaksandr Valialkin	27f0d098bd	app/victoria-metrics: add make victoria-metrics-arm64 rule for building GOARCH=arm64 binary	2019-05-29 23:07:14 +03:00
Aliaksandr Valialkin	56b952c456	app/vminsert: add -maxConcurrentInserts command-line flag for limiting the number of concurrent inserts	2019-05-29 12:41:23 +03:00
Aliaksandr Valialkin	968d094524	app/vminsert: reduce memory usage for Influx, Graphite and OpenTSDB protocols ... Do not buffer per-connection data and just store it as it arrives	2019-05-28 18:47:23 +03:00
Aliaksandr Valialkin	e307a4d92c	lib/timerpool: use timer pool in concurrency limiters ... This should reduce the number of memory allocations in highly loaded system	2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin	0eae39daa7	app/vminsert: properly reset InsertCtx.mrs - they must be empty after Reset call	2019-05-28 16:08:01 +03:00
Aliaksandr Valialkin	4859ca0cda	app/vmselect: update comment according to the updated code	2019-05-26 22:38:58 +03:00
Aliaksandr Valialkin	feb6b203a4	app/vminsert/influx: try converting string values to numeric values, since Influx agents may send numeric values as strings ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/34	2019-05-26 22:11:19 +03:00
Aliaksandr Valialkin	5262aae5da	app/vmselect/promql: misspeling fix	2019-05-25 21:53:11 +03:00
Aliaksandr Valialkin	54fb8b21f9	all: fix misspellings	2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin	d6523ffe90	Makefile: add -s flag to go fmt in make fmt command	2019-05-25 21:43:35 +03:00
Aliaksandr Valialkin	96ac664b27	Add make victoria-metrics Makefile rule for building dev binary	2019-05-25 18:24:51 +03:00
Aliaksandr Valialkin	5cbd4cfca9	app/vmselect: log slow queries if their execution time exceeds -search.logSlowQueryDuration	2019-05-24 16:12:31 +03:00
Aliaksandr Valialkin	718ce33714	app/vmselect: consume resultsCh data in exportHandler if writeResponseFunc failed to consume it	2019-05-24 14:54:31 +03:00
Aliaksandr Valialkin	2eb3602d61	app/victoria-metrics: remove -p XXXX:XXXX from docker run options, since it is unnesessary if --net=host is set	2019-05-24 12:54:53 +03:00
Aliaksandr Valialkin	d15d0127fe	app/vmselect/promql: add alias(q, name) function that sets the given name to all the time series in q	2019-05-24 02:41:45 +03:00
Aliaksandr Valialkin	fb140eda33	app/vmselect/promql: add label_transform(q, label, regexp, replacement) function for replacing all the occurences of regexp with replacement in the given label for q	2019-05-23 16:26:19 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00