669 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	d12c4914f0	lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	64e2d66014	lib/storage: code cleanup after 5bfd4e6218	2020-11-01 23:35:06 +02:00
Nikolay	67059caa12	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-10-29 17:17:52 +03:00
Nikolay	de3fe22815	adds leading forward slash check for scrapeURL path (#855) ... * fixes in-consistency with prometheus behaviour for scrape targets url path. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835	2020-10-29 08:39:42 +03:00
Aliaksandr Valialkin	0729cc36b2	lib/memory: do not print trailing zeroes in logs for -memory.allowedPercent command-line flag	2020-10-20 14:32:07 +03:00
Aliaksandr Valialkin	5bfd4e6218	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:31:44 +03:00
faceair	84e3881c0b	disable response compression on websocket (#841)	2020-10-17 13:32:34 +03:00
Aliaksandr Valialkin	01987f8c77	lib/storage: small code adjustements after d2960a20e0 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-10-17 01:16:54 +03:00
faceair	d2960a20e0	evaluate the execution cost of all tag filters (#824) ... * evaluate the execution cost of all tag filters * fix suffixes typo	2020-10-17 00:46:55 +03:00
Nikolay Khramchikhin	e6ab69dd88	fixes openstack api endpoint with suffix trim adds openstack (#840) ... api v2.0 check	2020-10-16 21:20:57 +03:00
Aliaksandr Valialkin	63c4999e06	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:12:36 +03:00
Nikolay Khramchikhin	9bd9f67718	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 13:38:21 +03:00
Aliaksandr Valialkin	b449607181	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin	cf5f2874cd	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:12:03 +03:00
Aliaksandr Valialkin	68f0e00761	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin	561a7619a5	lib/promscrape: fix tests after 71ea4935de	2020-10-08 19:32:36 +03:00
Aliaksandr Valialkin	71ea4935de	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:58:30 +03:00
Aliaksandr Valialkin	9b0a5c1028	lib/backup/actions: improve logging to be more clear to humans	2020-10-08 14:23:07 +03:00
Aliaksandr Valialkin	af90b3121c	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:35:44 +03:00
Aliaksandr Valialkin	e19d400230	lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does ... Such timestamps are rounded to seconds like Carbon does. See b0ba62a62d/lib/carbon/protocols.py (L197)	2020-10-06 11:38:29 +03:00
Aliaksandr Valialkin	90aa2a8ffd	lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human	2020-10-06 11:34:09 +03:00
Aliaksandr Valialkin	aba899c298	lib/promscrape/discovery/openstack: code prettifying after cbe3cf683b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-05 18:11:55 +03:00
Nikolay Khramchikhin	cbe3cf683b	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:45:33 +03:00
Aliaksandr Valialkin	f42194d817	lib/promrelabel: make a copy of label with new name for action: labelmap in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812	2020-10-05 16:19:19 +03:00
Aliaksandr Valialkin	bbeac0ba46	lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807	2020-10-05 15:19:05 +03:00
Aliaksandr Valialkin	47db9bb24a	lib/decimal: add tests for negative values passed to maxUpExponent	2020-10-05 14:56:45 +03:00
Aliaksandr Valialkin	bc7d67cee2	lib/decimal: properly calibrate scale for blocks with Inf values ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805	2020-10-05 14:52:44 +03:00
Aliaksandr Valialkin	764dc2499f	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:06:23 +03:00
Aliaksandr Valialkin	10f2eedee0	lib/storage: imrpove cache effectiveness for time series ids matching the given filters ... Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day. This should reduce CPU usage in the following cases: * when querying recently added data with small churn rate for time series * when querying historical data	2020-10-01 14:38:25 +03:00
Aliaksandr Valialkin	bec9b31b81	lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space if there are multiple partitions with deferred merges due to disk space shortage	2020-09-29 22:51:43 +03:00
Aliaksandr Valialkin	a9db81c4ab	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:48:33 +03:00
Aliaksandr Valialkin	127537d631	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:12 +03:00
Aliaksandr Valialkin	76b244cfcf	lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online ... The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309	2020-09-29 13:55:26 +03:00
Aliaksandr Valialkin	1659135752	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:22 +03:00
Aliaksandr Valialkin	38789e4aa0	lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm	2020-09-29 12:40:35 +03:00
Aliaksandr Valialkin	19c0b6f3ef	lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic	2020-09-29 12:30:04 +03:00
Aliaksandr Valialkin	7cde336b33	lib/protoparser/prometheus: fix TestParseStream after 124f78857b	2020-09-29 12:11:17 +03:00
Aliaksandr Valialkin	7673839228	lib/{fs,filestream}: small consistency-related updates after cc90a548b1	2020-09-29 00:42:43 +03:00
Nikolay Khramchikhin	cc90a548b1	added openbsd implementations (#790) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:29:04 +03:00
Aliaksandr Valialkin	7500146321	lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer	2020-09-28 17:19:16 +03:00
Aliaksandr Valialkin	124f78857b	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin	5cdad60a6f	lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server ... This should reduce syscall overhead when reading big amounts of data	2020-09-28 02:07:10 +03:00
Aliaksandr Valialkin	95688cbfc5	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin	b4bf722d8f	lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream ... Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.	2020-09-26 04:21:32 +03:00
Aliaksandr Valialkin	82973f8ae7	Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" ... This reverts commit bab6a15ae0. Reason for revert: the `fetchData` arg is used in cluster branch. Leaving this arg in master branch makes smaller the diff with cluster branch.	2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin	bab6a15ae0	lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock ... This arg became unused after 23bdc1f107	2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin	24ca30bf66	lib/storage: correctly use maxBlockSize in various checks ... Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.	2020-09-24 18:12:56 +03:00
Aliaksandr Valialkin	2985077c35	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:46:34 +03:00
Aliaksandr Valialkin	c5ef0e6327	lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue	2020-09-23 02:17:47 +03:00
Aliaksandr Valialkin	5c42965853	lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728	2020-09-22 23:27:19 +03:00