2423 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	9761ffd161	lib/promscrape/discovery/kubernetes: properly handle too old resource version error message from Kubernetes watch API	2021-03-26 12:28:10 +02:00
Aliaksandr Valialkin	aa81039b42	app/vmselect: log the metric which trigger rollup result cache reset ... This should help finding the source of stale metrics	2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin	6e855d4b82	lib/storage: tune loopsCountPerMetricNameMatch according to production workload	2021-03-25 13:27:47 +02:00
Aliaksandr Valialkin	d4aadba9fa	app/vmagent: add -promscrape.consul.waitTime command-line flag for configuring Consul service discovery wait time ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144	2021-03-23 19:33:25 +02:00
Aliaksandr Valialkin	3a3d2165f9	lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock ... This should speed up Search.NextMetricBlock a bit	2021-03-23 17:56:49 +02:00
Nikolay	29f9ef9b7f	changes consul_service label value (#1143) ... according to prometheus discovery. It should mitigate issue with case sensetive services https://github.com/hashicorp/consul/issues/5707	2021-03-23 15:35:01 +02:00
Aliaksandr Valialkin	3cfb3a3683	lib/storage: respect the deadline passed to Storage.SearchMetricNames	2021-03-22 23:03:17 +02:00
Aliaksandr Valialkin	8e2afdf568	lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache	2021-03-22 22:49:18 +02:00
Aliaksandr Valialkin	910092ca4d	lib/storage: tune loopsCountPerMetricNameMatch	2021-03-22 12:53:17 +02:00
Aliaksandr Valialkin	726f6ad804	lib/storage: small code simplification after 6cee5338b2	2021-03-18 15:21:13 +02:00
Aliaksandr Valialkin	6cee5338b2	lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars ... The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137	2021-03-18 14:53:47 +02:00
Aliaksandr Valialkin	4fb049bcba	lib/fs: reduce the frequency of failed to remove directory ... due to NFS lock log warnings ... Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.	2021-03-18 13:24:46 +02:00
Aliaksandr Valialkin	45dabfac1b	lib/storage: faster move heavy filters to the end of list	2021-03-17 15:12:13 +02:00
Aliaksandr Valialkin	828669e4e1	all: make golint happy	2021-03-17 00:49:28 +02:00
Aliaksandr Valialkin	ccfb0ae2d3	lib/storage: limit loops count in order to reduce max CPU usage during filter search	2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin	576a80b3d9	lib/storage: do not modify filterLoopsCount stats with loopsCount stats ... Such a modification can result in incorrect filter sorting later	2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin	f104f3eb2a	all: make golangci-lint happy after the commit 6378205415	2021-03-17 00:24:40 +02:00
Aliaksandr Valialkin	6378205415	lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics ... This is a follow-up for 18cfc4be7b See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:16:17 +02:00
Nikolay	18cfc4be7b	Adds udp6 support for ingest servers (#1134) ... with flag -enableUDP6 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131	2021-03-17 00:03:06 +02:00
Aliaksandr Valialkin	35bb44d317	lib/uint64set: optimize bucket16.addMulti a bit	2021-03-16 21:09:07 +02:00
Aliaksandr Valialkin	fd86a7dc1d	lib/storage: time series search optimization according to production workload profiling ... Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down the function by multiple times when it finds big number of metricIDs (tens of millions).	2021-03-16 20:01:43 +02:00
Aliaksandr Valialkin	e36fbfae5b	lib/storage: further tuning for time series search	2021-03-16 18:46:22 +02:00
Aliaksandr Valialkin	dd7e82c34f	app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate	2021-03-15 22:38:50 +02:00
Aliaksandr Valialkin	37323c57c9	lib/influxutils: return response compatible with InfluxDB 1.8.4 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 22:19:59 +02:00
f41gh7	acf2a82d3c	typo fix	2021-03-15 23:03:52 +03:00
Aliaksandr Valialkin	85a95bf60c	all: various fixes in command-line flag descriptions	2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin	7c37e9aea9	app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124	2021-03-15 21:37:55 +02:00
Aliaksandr Valialkin	9c77e34ef9	lib/storage: further tuning for time series selector code	2021-03-15 20:31:34 +02:00
Aliaksandr Valialkin	fb82c4b9fa	lib/uint64set: reduce the size of bucket16 by storing smallPool by pointer. ... This reduces CPU time spent on bucket16 copying inside bucket13.addBucketAtPos.	2021-03-15 17:23:31 +02:00
Aliaksandr Valialkin	eb103e1527	lib/uint64set: optimize Set.AddMulti for large sorted sets	2021-03-15 17:10:40 +02:00
Aliaksandr Valialkin	43504ebd14	lib/uint64set: optimize bucket16.add and bucket16.addMulti a bit	2021-03-15 16:58:12 +02:00
Aliaksandr Valialkin	3ccf7ea20c	lib/storage: tune per-day index search	2021-03-15 13:31:55 +02:00
Aliaksandr Valialkin	c14dafce43	lib/promscrape: an attempt to reduce memory usage when vmagent scrapes targets with varying number of metrics ... Do not cache too big byte buffers and too big writeRequestCtx objects, since it is cheaper to re-create them instead of wasting RAM for their caching. This reverts 7f6f350ee1	2021-03-15 11:45:39 +02:00
Aliaksandr Valialkin	7f6f350ee1	lib/promscrape: return back the logic for flushing big buffers to storage from the commit 3fd8653b40 ... This should reduce memory usage when vmagent scrapes targets with big number of metrics and `-promscrape.streamParse` isn't enabled	2021-03-14 22:26:00 +02:00
Aliaksandr Valialkin	b88806ecbf	lib/promscrape/discovery/kubernetes: do not start object watcher until initial objects are loaded	2021-03-14 21:55:00 +02:00
Aliaksandr Valialkin	83edbb7cab	lib/promscrape: retry service discovery in a few seconds if it starts returning 0 targets ... This should reduce recovery time from temporary issues during service discovery	2021-03-14 21:53:23 +02:00
Aliaksandr Valialkin	bf15d6a6a2	lib/promscrape: remove duplicate target word in error message	2021-03-14 21:52:02 +02:00
Aliaksandr Valialkin	d409898515	lib/promscrape/discovery/kubernetes: further optimize kubernetes service discovery for the case with many scrape jobs ... Do not re-calculate labels per each scrape job - reuse them instead for scrape jobs with identical Kubernetes role Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-14 21:14:53 +02:00
Aliaksandr Valialkin	7a16e8e3a2	lib/promscrape/discovery: fixes after 133b288681 ... - Removed a deadlock in addAPIWatcher - Do not create unused ScrapeWork objects - Do not spend CPU resources on creating objectByKey map in addAPIWatcher This work is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1125	2021-03-13 15:18:51 +02:00
Aliaksandr Valialkin	def014eb75	lib/promscrape/discovery/kubernetes: remove debug lines left after the commit 133b288681	2021-03-12 11:22:33 +02:00
Aliaksandr Valialkin	ca4d5ce037	lib/proxy: there is no need in cloning tlsCfg, which has been created two lines above	2021-03-12 10:47:02 +02:00
Aliaksandr Valialkin	895d5d1355	lib/proxy: set proxy address in tls.Config.ServerName instead of the target address ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 10:41:33 +02:00
Aliaksandr Valialkin	a6a71ef861	lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-12 03:36:19 +02:00
Aliaksandr Valialkin	f669531506	lib/storage: further tune filters sorting logic	2021-03-12 00:53:04 +02:00
Aliaksandr Valialkin	133b288681	lib/promscrape/discovery/kubernetes: use a single watcher per apiURL ... Previously multiple scrape jobs could create multiple watchers for the same apiURL. Now only a single watcher is used. This should reduce load on Kubernetes API server when many scrape job configs use Kubernetes service discovery.	2021-03-11 16:43:04 +02:00
Aliaksandr Valialkin	c0ac740f93	lib/proxy: do not show inline basic auth passwords when logging errors related to proxy_url	2021-03-11 13:43:36 +02:00
Aliaksandr Valialkin	bebcb8130c	lib/promscrape/discovery/kubernetes: localize Bookmark parsing code ... This is a follow-up for e772d1c920	2021-03-11 13:08:08 +02:00
Aliaksandr Valialkin	e772d1c920	lib/promscrape/discovery/kubernetes: reduce load on Kubernetes API server by using watch bookmarks ... This allows continuing object watch from the last bookbark instead of reloading all the objects on watch errors or timeouts. See https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks	2021-03-10 15:06:35 +02:00
Aliaksandr Valialkin	bc5c4add89	lib/httpserver: export vm_available_memory_bytes and vm_available_cpu_cores metrics ... These metrics are useful for tracking the available memory and CPU cores for VictoriaMetrics apps.	2021-03-10 12:02:42 +02:00
Aliaksandr Valialkin	787242d7b0	lib/proxy: pass proxy hostname in Host header of the CONNECT request ... This should resolve the following issue when connecting to tls proxy: cannot validate certificate for ... because it doesn't contain any IP SANs	2021-03-09 20:39:40 +02:00
Aliaksandr Valialkin	36fd007247	lib/proxy: set missing ServerName in TLS config for proxy_url. ... While at it, allow setting Proxy-Authorization for `proxy_url` via `basic_auth` and `bearer_token` configs.	2021-03-09 18:58:18 +02:00
Nikolay	ad34f42467	Changes tlsConfig init for proxy connections (#1121) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116	2021-03-09 18:51:00 +02:00
Aliaksandr Valialkin	3fd8653b40	lib/promscrape: apply sample_limit after metric relabeling is applied as Prometheus does ... See the description for `sample_limit` option from Prometheus docs: Per-scrape limit on number of scraped samples that will be accepted. If more than this number of samples are present after metric relabeling the entire scrape will be treated as failed. 0 means no limit. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config	2021-03-09 15:47:18 +02:00
Aliaksandr Valialkin	4b18a4f026	lib/promscrape/discovery/kubernetes: remove too verbose logs about starting and stopping the watchers ... Log the number of objects loaded per each watch url Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-09 15:05:14 +02:00
John Belmonte	364fdf4a56	spelling fix: adjacent (#1115)	2021-03-09 09:18:19 +02:00
Aliaksandr Valialkin	14a399dd06	lib/promscrape: add scrape_offset option to scrape_config ... This option can be used for specifying the particular offset per each scrape interval for target scraping	2021-03-08 12:03:33 +02:00
Aliaksandr Valialkin	345980f78f	lib/storage: go fmt	2021-03-08 12:03:31 +02:00
Aliaksandr Valialkin	18fe0ff14b	lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow ... The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload	2021-03-07 21:12:35 +02:00
Aliaksandr Valialkin	ab4f090c63	lib/promscrape/discovery/kubernetes: reduce memory usage further when big number of scrape jobs are configured for the same kubernetes_sd_config role ... Serialize reloading per-role objects, so they don't occupy too much memory when objects for many scrape jobs are simultaneously refreshed. Do not reload per-role objects if they were already refreshed by concurrent goroutines. This should reduce load on Kubernetes API server when big number of scrape jobs are configured for the same Kubernetes role. This is a follow-up for 17b87725ed Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113	2021-03-07 19:51:03 +02:00
Aliaksandr Valialkin	1187ee5e16	lib/decimal: prevent exponent overflow when processing values close to zero ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1114	2021-03-05 18:52:47 +02:00
Aliaksandr Valialkin	17b87725ed	lib/promscrape/discovery/kubernetes: reduce memory usage when Kubernetes service discovery is configured on a big number of scrape jobs ... Previously vmagent was creating a separate Kubernetes object cache per each scrape job. This could result in increased memory usage when monitoring a Kubernetes cluster with big number of objects (pods / nodes / services, etc.) as seen at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1113 Now it uses a shared map of scrape objects across multiple scrape jobs.	2021-03-05 17:29:55 +02:00
Aliaksandr Valialkin	c9a25e931b	lib/promscrape/discovery/kubernetes: move apiWatcher code to a separate file	2021-03-05 12:36:05 +02:00
Aliaksandr Valialkin	444fca89f8	lib/promscrape: make cluster membership calculations consistent across 32-bit and 64-bit architectures	2021-03-05 09:06:17 +02:00
Aliaksandr Valialkin	423cd981fb	lib/promscrape: add -promscrape.cluster.replicationFactor command-line flag for replicating scrape targets among vmagent instances in the cluster	2021-03-04 10:20:15 +02:00
Aliaksandr Valialkin	9a48c1b53d	lib/promscrape/discovery/kubernetes: fix tests after e154f4a644	2021-03-03 22:41:30 +02:00
Nikolay	e154f4a644	Fix ingress discovery api (#1110)	2021-03-03 10:43:39 +02:00
Aliaksandr Valialkin	7906316741	lib/promscrape/discovery/kubernetes: properly check for nil pointer inside interface ... See https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1 This fixes a panic when the ScrapeWork is filtered out in swcFunc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1108	2021-03-03 10:33:17 +02:00
Aliaksandr Valialkin	03dceb700d	lib/promscrape: go fmt	2021-03-02 21:20:43 +02:00
Aliaksandr Valialkin	4de4da1e2a	lib/storage: typo fix: umarshal -> unmarshal	2021-03-02 20:47:59 +02:00
Aliaksandr Valialkin	062211c61c	lib/promscrape: pre-allocate space for a map in mergeLabels ... This should reduce the number of memory allocations when discovering big number of targets	2021-03-02 18:41:58 +02:00
Aliaksandr Valialkin	d1d34664b5	lib/promscrape/discovery: properly track vm_promscrape_discovery_kubernetes_objects_removed_total metric	2021-03-02 18:32:54 +02:00
Aliaksandr Valialkin	a939667ce0	lib/promrelabel: remove unneded optimizations for labeldrop and labelkeep actions ... These optimizations may slow down code execution by matching the same label against regexp two times instead of a single time	2021-03-02 17:55:43 +02:00
Aliaksandr Valialkin	6a7ef768ff	lib/promscrape/discovery/kubernetes: cache ScrapeWork objects as soon as the corresponding k8s objects are changed ... This should reduce CPU usage and memory usage when Kubernetes contains tens of thousands of objects	2021-03-02 16:42:55 +02:00
Aliaksandr Valialkin	22b1941cfc	lib/promscrape/discovery/ec2: follow-up after f6114345de	2021-03-02 13:46:26 +02:00
Nikolay	f6114345de	Adds webIndentity token for aws (#1099) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1080	2021-03-02 13:27:09 +02:00
Aliaksandr Valialkin	937f382938	lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format ... Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.	2021-03-02 13:21:43 +02:00
Aliaksandr Valialkin	019d8e88d8	lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as foo; 1 2 ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100	2021-03-01 17:16:35 +02:00
Aliaksandr Valialkin	1a3689af9a	lib/promscrape/discovery/kubernetes: deflake tests; a follow-up for 05fb08713c	2021-03-01 14:32:12 +02:00
Aliaksandr Valialkin	62ebf5c88e	lib/promscrape: explicitly stop and cleanup service discovery routines when new config is read from -promscrape.config ... This should reduce memory usage when `-promscrape.config` file frequently changes	2021-03-01 14:14:00 +02:00
Aliaksandr Valialkin	e32ad9e923	lib/promscrape: use target arg in ScrapeWork cache	2021-03-01 12:29:09 +02:00
Aliaksandr Valialkin	f5d77a7081	lib/promscrape: typo fix, which prevented from caching ScrapeWork entries	2021-03-01 12:12:56 +02:00
Aliaksandr Valialkin	e84153d5ca	lib/promscrape: add vm_promscrape_scrapework_cache_* metrics for tracking ScrapeWork cache effectiveness	2021-03-01 12:05:45 +02:00
Aliaksandr Valialkin	7f15cd7161	lib/httpserver: make make errcheck happy after the commit 9fc7726d84	2021-03-01 00:34:43 +02:00
Aliaksandr Valialkin	530e9904af	lib/promscrape: reduce CPU usage an memory allocations when constructing scrapeWorkKey	2021-02-28 22:29:58 +02:00
Aliaksandr Valialkin	9fc7726d84	lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls ... This should fix the `http: superfluous response.WriteHeader call` issue See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078	2021-02-28 19:22:50 +02:00
Aliaksandr Valialkin	e5ca8ac0db	lib/promscrape: add ability to spread scrape targets among multiple vmagent instances ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084	2021-02-28 18:41:08 +02:00
Aliaksandr Valialkin	e02d1ef93c	lib/promscrape/discovery/kubernetes: properly account the number of objects when watcher is stopped ... A follow-up for b21b110b7a	2021-02-28 17:06:02 +02:00
Aliaksandr Valialkin	b21b110b7a	lib/promscrape/discovery/kubernetes: add vm_promscrape_discovery_kubernetes_* metrics for monitoring internal state of k8s service discovery	2021-02-28 16:57:40 +02:00
Aliaksandr Valialkin	c459600346	lib/promscrape/discovery/kubernetes: remove resourceVersionMatch=NotOlderThan query arg when watching for k8s object changes, since it cannot be used when watch=1 query arg is passed	2021-02-28 16:07:14 +02:00
Aliaksandr Valialkin	59a31171e3	lib/promscrape: fix possible deadlock in parallel execution of target relabeling	2021-02-28 16:05:13 +02:00
Aliaksandr Valialkin	68a0f5ce12	lib/promscrape/discovery/kubernetes: fix deadlock in startWatcherForURL ... reloadObjects must be called without holding aw.mu lock	2021-02-28 15:26:30 +02:00
Aliaksandr Valialkin	b523e0369c	lib/promscrape/discovery/kubernetes: typo fix after 241ffd1f3b	2021-02-28 15:12:17 +02:00
Aliaksandr Valialkin	241ffd1f3b	lib/promscrape/discovery/kubernetes: pre-populate labelsByKey in reloadObject()	2021-02-28 15:09:49 +02:00
Aliaksandr Valialkin	05fb08713c	lib/promscrape/discovery/kubernetes: compare sorted sets of labels in tests ... This should deflake tests where the order of labels isn't stable	2021-02-28 14:10:19 +02:00
Aliaksandr Valialkin	af8b7e8391	lib/promscrape: add missing startWatchersForRole() call at the beginning of apiWatcher.getLabelsForRole	2021-02-28 14:00:17 +02:00
Aliaksandr Valialkin	422b31de40	lib/promscrape/discovery/kubernetes: reload k8s resources on every error ... This is needed for obtaining fresh resourceVersion	2021-02-27 01:47:27 +02:00
Aliaksandr Valialkin	7cc3d96a41	lib/fs: follow-up after f3a03c4164	2021-02-27 01:01:47 +02:00
Nikolay	f3a03c4164	Adds windows build (#1040) ... * fixes windows compilation, adds signal impl for windows, adds free space usage for windows, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036 NOTE victoria metrics database still CANNOT work under windows system, only vmagent is supported. To completly port victoria metrics, you have to fix issues with separators, parsing and posix file removall * rollback separator * Adds windows setInformation api, it must behave like unix, need to test it. changes procutil * check for invlaid param * Fixes posix delete semantic * refactored a bit * fixes openbsd build * removed windows api call * Fixes code after windows add * Update lib/procutil/signal_windows.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-27 00:37:07 +02:00
Aliaksandr Valialkin	a78948ae8b	lib/promscrape: yet another typo fix after ed8441ec52	2021-02-26 23:35:47 +02:00
Aliaksandr Valialkin	8683ea85e6	lib/fs: properly handle stale NFS file handle error during file deletion ... This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.	2021-02-26 23:25:14 +02:00
Aliaksandr Valialkin	9fa2632ac3	lib/promscrape: typo fix after ed8441ec52	2021-02-26 23:04:05 +02:00
Aliaksandr Valialkin	ed8441ec52	lib/promscrape: cache ScrapeWork ... This should reduce the time needed for updating big number of scrape targets.	2021-02-26 21:43:22 +02:00
Aliaksandr Valialkin	815666e6a6	lib/promscrape/discovery/kubernetes: cache target labels ... This should reduce CPU usage on repeated SDConfig.GetLabels() calls.	2021-02-26 20:23:28 +02:00
Aliaksandr Valialkin	19712fc2bd	lib/promscrape/discovery/kubernetes: errcheck fix	2021-02-26 17:00:08 +02:00
Aliaksandr Valialkin	c8f2f9b2e8	lib/promscrape: cleanup after 9b2246c29b ... Main points: * Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit * Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001 * Corner case fixes.	2021-02-26 16:54:05 +02:00
Nikolay	9b2246c29b	vmagent kubernetes watch stream discovery. (#1082) ... * started work on sd for k8s * continue work on watch sd * fixes * continue work * continue work on sd k8s * disable gzip * fixes typos * log errror * minor fix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-26 16:46:13 +02:00
Aliaksandr Valialkin	a93e644001	lib/promscrape: remove duplicate code a bit	2021-02-26 16:39:56 +02:00
Aliaksandr Valialkin	f7b242540b	lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel	2021-02-26 16:39:56 +02:00
Aliaksandr Valialkin	f7049e2af7	lib/promrelabel: optimize labeldrop and labelkeep relabeling for prefix.* and prefix.+ regexps	2021-02-24 17:58:28 +02:00
Aliaksandr Valialkin	2c44178645	lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache	2021-02-23 15:47:19 +02:00
faceair	15d61c4879	lib/storage: correct tagfilter match cost (#1079)	2021-02-22 21:46:56 +02:00
Aliaksandr Valialkin	d136081040	lib/promrelabel: add more optimizations for relabeling for common cases	2021-02-22 16:33:55 +02:00
Aliaksandr Valialkin	dd1e53b119	lib/promrelabel: optimize relabeling performance for common cases	2021-02-22 00:51:13 +02:00
Aliaksandr Valialkin	ff5bbc4b88	lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric	2021-02-21 23:21:42 +02:00
Aliaksandr Valialkin	636c55b526	lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation ... This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now, so GC doesn't need visiting them.	2021-02-21 22:06:47 +02:00
Aliaksandr Valialkin	388cdb1980	lib/storage: do not re-calculate stats for heavy tag filters ... This should reduce the number of slow queries when stats for heavy tag filters was recalculated.	2021-02-21 21:39:01 +02:00
Aliaksandr Valialkin	48656dcc38	lib/{mergeset,storage}: allow merging smaller number of small parts ... While this may increase CPU and disk IO usage needed for background merge, this also recudes CPU usage during queries in production. This is because such queries tend to read recently added data and it is better to have lower number of parts for such data in order to reduce CPU usage. This partially reverts ebf8da3730	2021-02-21 21:28:36 +02:00
Aliaksandr Valialkin	cb311bb156	lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains	2021-02-21 21:18:59 +02:00
Aliaksandr Valialkin	2cfb376945	lib/promscrape: typo fix after the commit f26162ec99	2021-02-19 00:33:37 +02:00
Aliaksandr Valialkin	c2678754e4	app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065	2021-02-19 00:31:34 +02:00
Aliaksandr Valialkin	f26162ec99	lib/promscrape: add scrape_align_interval config option into scrape config ... This option allows aligning scrapes to a particular intervals.	2021-02-18 23:53:44 +02:00
Aliaksandr Valialkin	f9084611bd	lib/storage: use composite index for a query with a name filter and negative filters	2021-02-18 18:57:23 +02:00
Aliaksandr Valialkin	a537c4f602	lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters ... Example: `node_cpu_seconds_total{mode!="idle"}`	2021-02-18 18:46:36 +02:00
Aliaksandr Valialkin	e540c02014	lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter. ... This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters	2021-02-18 13:58:18 +02:00
Aliaksandr Valialkin	711f8a5b8d	lib/storage: sort tag filters by the number of loops they need for the execution ... This metric should work better than the filter execution duration, since it cannot be distorted by concurrently running queries.	2021-02-18 12:47:38 +02:00
Aliaksandr Valialkin	ce99b48a9a	Revert "lib/mergeset: tune lifetime for entries inside block caches" ... This reverts commit 458c89324d. Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.	2021-02-17 20:42:21 +02:00
Aliaksandr Valialkin	939d5ffc2b	lib/storage: move composite filters to the top during sorting	2021-02-17 20:26:51 +02:00
Aliaksandr Valialkin	faad6f84a4	lib/storage: return back filter arg to getMetricIDsForTagFilter function ... The filter arg has been removed in the commit c7ee2fabb8 because it was preventing from caching the number of matching time series per each tf. Now the cache contains duration for tf execution, so the filter shouldn't break such caching.	2021-02-17 19:33:22 +02:00
Aliaksandr Valialkin	d4849561ef	app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics	2021-02-17 19:13:38 +02:00
Aliaksandr Valialkin	33806264ec	lib/storage: revert ecf132933e, since negative filters require the same amount of work as non-negative filters	2021-02-17 18:55:04 +02:00
Aliaksandr Valialkin	63fc140624	lib/storage: tag filters sorting...	2021-02-17 17:55:29 +02:00
Aliaksandr Valialkin	74424b55ee	lib/storage: further tune tag filters sorting	2021-02-17 17:28:15 +02:00
Aliaksandr Valialkin	442fcfec5a	lib/storage: tune the logic for sorting tag filters according the their exeuction times	2021-02-17 15:00:08 +02:00
Aliaksandr Valialkin	4a07820048	lib/storage: make sure that nobody uses partitions when closing the table	2021-02-17 14:59:04 +02:00
Aliaksandr Valialkin	1256931aee	lib/httpserver: make errcheck happy	2021-02-16 22:05:32 +02:00
Aliaksandr Valialkin	d61f7b7279	lib/storage: more tuning for tag filters sorting according the time they take	2021-02-16 21:22:23 +02:00
Aliaksandr Valialkin	458c89324d	lib/mergeset: tune lifetime for entries inside block caches ... This should reduce memory usage in general case without significant CPU usage increase	2021-02-16 18:11:51 +02:00
Aliaksandr Valialkin	2824856691	lib/mergeset: clarify comments in the code a bit	2021-02-16 18:02:57 +02:00
Aliaksandr Valialkin	1bf6cd814d	lib/uint64set: remove memory allocation in bucket16.appendTo when sorting smallPool	2021-02-16 15:31:49 +02:00
Aliaksandr Valialkin	8ec45ff335	lib/httpserver: cache /metrics output for a second ... This should reduce CPU load when `/metrics` output is scraped with a frequency exceeding a request per second	2021-02-16 14:56:36 +02:00
Aliaksandr Valialkin	b861a64510	lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names	2021-02-16 13:59:18 +02:00
Aliaksandr Valialkin	7faa762021	lib/mergeset: remove unused code after a4140de9e6	2021-02-16 13:40:09 +02:00
Aliaksandr Valialkin	ca191696fe	lib/storage: tune sorting for tag filters	2021-02-16 13:04:49 +02:00
Aliaksandr Valialkin	ecf132933e	lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs	2021-02-15 16:34:23 +02:00
Aliaksandr Valialkin	4e39bf148c	vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 ... The new version switches from log-linear histograms to log-based histograms, which provide up to 3.6 times better accuracy.	2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin	9f5ac603a7	lib/storage: reduce the minimum supported retention for inverted index from one month to one day	2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin	2e30202dc7	lib/flagutil: prevent from integer overflow when parsing duration	2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin	38d7e96602	lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_* and __meta_kuberntes_endpoints_annotation_* labels to role: endpoints ... This syncs kubernetes SD with Prometheus 2.25 See 617c56f55a	2021-02-15 02:51:16 +02:00
Aliaksandr Valialkin	74963f71c6	lib/logger: explicitly import "time/tzdata" package for embedding tzdata into the app ... The approach with `timetzdata` build tag didn't work for GOARCH=arm and GOARCH=ppc64le due to the issue https://github.com/golang/go/issues/44073#issuecomment-778854298	2021-02-15 01:00:01 +02:00
Aliaksandr Valialkin	71c417427c	lib/storage: sort tag filters by actual execution time instead of by the number of matching time series ... This should improve query speed for queries with regexp filters matching small number of time series on a label with big number of unique values.	2021-02-15 00:18:13 +02:00
Aliaksandr Valialkin	c727d2219b	lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters. ... For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` . This could result in missing time series for such tag filters.	2021-02-14 23:38:14 +02:00
Aliaksandr Valialkin	80dc74dbc1	lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics ... These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes. * It is better using `up == 0` query for determining failing targets. * It is better using the following query for determining targets with exceeded limit on the number of metrics: scrape_samples_scraped > 0 if up == 0	2021-02-12 05:26:04 +02:00
Aliaksandr Valialkin	0e26b7168a	lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case	2021-02-10 22:41:04 +02:00
Aliaksandr Valialkin	b51c23dc5b	lib/storage: parallelize tag filters execution a bit ... This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.	2021-02-10 18:12:25 +02:00
Aliaksandr Valialkin	c7ee2fabb8	lib/storage: remove filter arg from getMetricIDsForDateTagFilter function ... The `filter` arg breaks the logic for sorting tag filters by the matching metrics, which may result in non-optimal performance during time series search.	2021-02-10 18:12:20 +02:00
Aliaksandr Valialkin	57cac289e0	lib/storage: fix inconsistencies in error logs	2021-02-10 18:12:16 +02:00
Aliaksandr Valialkin	5d5f0b0627	lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata	2021-02-10 17:55:40 +02:00
Aliaksandr Valialkin	cdecf83ce5	app/vmstorage: export vm_composite_index_min_timestamp metric	2021-02-10 17:14:08 +02:00
Aliaksandr Valialkin	553016ea99	lib/storage: disable composite index usage when querying old data	2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin	fcb7655d1e	lib/storage: fix metric name match for composite filter	2021-02-10 01:27:45 +02:00
Aliaksandr Valialkin	c7dccebaef	lib/storage: optimize search by label filters matching big number of time series	2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin	6b4e6c229c	lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day ... This should help systems with multiple CPU cores	2021-02-10 00:01:13 +02:00
Aliaksandr Valialkin	31f6b9c977	lib/fs: remove the code for tracking whether the given memory region is in page cache ... This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.	2021-02-09 16:49:03 +02:00
Aliaksandr Valialkin	0a69122d81	lib/mergeset: remove dead code left after a4140de9e6	2021-02-09 16:33:52 +02:00
Aliaksandr Valialkin	d56390b925	optimize Storage.updatePerDateData()	2021-02-09 02:55:36 +02:00
Aliaksandr Valialkin	fda61e8e96	lib/storage: skip deduplication when creating inmemory data blocks ... The deduplication will be performed later during merging such blocks.	2021-02-09 02:25:32 +02:00
Aliaksandr Valialkin	a4140de9e6	lib/mergeset: unconditionally cache indexdb blocks ... Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage. This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.	2021-02-09 00:47:50 +02:00
Aliaksandr Valialkin	cb96a1865b	app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches	2021-02-09 00:47:00 +02:00
Aliaksandr Valialkin	c5770600a2	lib/cgroup: follow-up after b9bf3cbe3e	2021-02-08 15:54:38 +02:00
Nikolay	b9bf3cbe3e	refactored cgroups limits, (#1061) ... adds tests, remove os.Exec	2021-02-08 15:46:22 +02:00
Aliaksandr Valialkin	2242647a04	lib/storage: optimize data ingestion in the beginning of every hour ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-08 12:01:12 +02:00
Aliaksandr Valialkin	8f28a578d3	lib/logger: exit the app if unsupported timezone value has been passed to -loggerTimezone ... While at it, clarify descrption for `-loggerTimezone` command-line flag.	2021-02-07 23:35:37 +02:00
Aliaksandr Valialkin	83d3e582ab	lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion ... This should reduce possible CPU usage spikes at the beginning of every hour. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046	2021-02-04 18:48:13 +02:00
Aliaksandr Valialkin	9fb38569eb	lib/httpserver: expose process_open_fds and process_max_fds metrics ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037	2021-02-04 16:40:50 +02:00
Nikolay	48c8c5093b	fixes dockerswarm (#1053) ... fixes improper usage of host network services https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028	2021-02-04 15:56:42 +02:00
Aliaksandr Valialkin	d16f22f3a1	app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards ... Example: foo{bar{x,yz},a[b-c],*de}	2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin	a5a1b9bd66	lib/storage: fix a bug, which breaks searching by Graphite wildcard filters	2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin	6123aa3e75	sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks	2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin	157c02622b	app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax	2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin	4068f8d590	lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric	2021-02-02 16:15:25 +02:00
Aliaksandr Valialkin	bd11fd8f1d	lib/promscrape: add vm_promscrape_scrape_retries_total, vm_promscrape_discovery_retries_total and vm_promscrape_discovery_requests_total metrics	2021-02-01 20:06:27 +02:00
Aliaksandr Valialkin	f0087f0dbb	lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func	2021-02-01 14:35:39 +02:00
Aliaksandr Valialkin	b2aa80e74b	app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values ... This commit also adds --vm-round-digits command-line option to vmctl tool.	2021-02-01 14:27:09 +02:00
Aliaksandr Valialkin	d6347a3e56	lib/logger: initialize timezone by UTC in order to fix failing tests	2021-01-27 00:59:12 +02:00
Aliaksandr Valialkin	fc5b26d856	lib/promscrape: export vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics ... These metrics could be useful for determining imporperly working scrape targets. Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.	2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin	de3c662e8a	all: consistently use timers from timerpool	2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin	3149ac7a7e	lib/fs: properly initialize cleaner for pageCache bitmaps ... Previously it wasnt working because the timer was fired only once	2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin	3fe848cdd7	lib/logger: add -loggerTimezone command-line flag for adjusting timezone for timestamps in log messages	2021-01-26 22:51:54 +02:00
Aliaksandr Valialkin	8cea3c3cc4	lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection	2021-01-22 13:22:33 +02:00
faceair	b638c1eed5	lib/mergeset: add missing shouldCacheBlock (#1019)	2021-01-15 11:46:01 +02:00
Aliaksandr Valialkin	bdd0a1cdb2	lib/backup: increase backup chunk size from 128MB to 1GB ... This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing	2021-01-13 12:16:35 +02:00
Nikolay	7bf5d48315	bumps minimal tls version (#1012)	2021-01-13 00:35:47 +02:00
Aliaksandr Valialkin	31ec79eaf6	lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here ... This is a follow-up commit after c8ea697db8	2021-01-12 15:13:30 +02:00
Aliaksandr Valialkin	c8ea697db8	lib/storage: de-duplicate tags in MetricName.sortTags ... Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.	2021-01-12 15:03:42 +02:00
Nikolay	7976c22797	Fixes error handling for promscrape.streamParse (#1009) ... properly return error if client cannot read data, properly suppress scraper errors	2021-01-12 13:31:47 +02:00
Aliaksandr Valialkin	2c44f9989a	lib/promscrape: properly show scrape duration on /targets page ... Previously it has been shown as 0.000s for any scrape duration.	2021-01-11 21:14:46 +02:00
Aliaksandr Valialkin	24ffad74c1	all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000	2021-01-11 12:53:30 +02:00
Aliaksandr Valialkin	9dcb18e03d	app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage	2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin	490c69c64e	lib/storage: wait for pending transactions before closing and dropping the partition ... This deflakes `make test-full-386` test	2020-12-25 11:45:53 +02:00
Aliaksandr Valialkin	cab7e936a3	lib/storage: physically remove stale parts ... Previously they were removed from partition struct, but the corresponding directories weren't removed. This is a follow-up for 46dba00756	2020-12-24 16:51:36 +02:00
Nikolay	b21e16ad0c	fixes kubernetes_sd (#983) ... * fixes kubernetes_sd, adds missing service metadata for pod ports without endpoint https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982 * fix test	2020-12-24 11:26:14 +02:00
Aliaksandr Valialkin	820669da69	lib/promscrape: code prettifying for 8dd03ecf19	2020-12-24 10:56:10 +02:00
Nikolay	8dd03ecf19	adds proxy_url support, (#980) ... * adds proxy_url support, adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery, adds proxy_url to the scrape_config for targets scrapping, http based proxy is supported atm, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503 * fixes imports	2020-12-24 10:52:37 +02:00
Aliaksandr Valialkin	9e4ed5e591	lib/storage: do not remove parts outside the configured retention if they are currently merged ... These parts are automatically removed after the merge is complete.	2020-12-24 08:51:28 +02:00
Aliaksandr Valialkin	46dba00756	lib/storage: remove stale parts as soon as they go outside the configured retention ... Previously such parts could remain undeleted for long durations until they are merged with other parts. This should help for `-retentionPeriod` values smaller than one month.	2020-12-22 19:54:31 +02:00
Aliaksandr Valialkin	d65c03c004	lib/storage: properly determine max rows for output part when merging small parts	2020-12-18 23:14:38 +02:00
Aliaksandr Valialkin	ebf8da3730	lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage	2020-12-18 20:01:08 +02:00
Aliaksandr Valialkin	2dfa746c91	lib/promscrape: remove ID field from ScrapeWork struct. Use a pointer to ScrapeWork as a key in targetStatusMap ... This simplifies the code a bit.	2020-12-17 14:32:56 +02:00
Aliaksandr Valialkin	9abb2d6c74	lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2 ... Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.	2020-12-16 23:40:17 +02:00
BigFish	27f0261257	lib/protoparser/prometheus/parser.go (#970) ... fix parse timestamp error if there are some whitespaces after timestamp	2020-12-16 23:36:20 +02:00
Aliaksandr Valialkin	a4c7fcb5e1	lib/promscrape: properly remove deleted target from /targets page ... Previously `sw` variable wasn't captured correctly by the started goroutine.	2020-12-15 20:57:09 +02:00
Aliaksandr Valialkin	b730fc2667	lib/promscrape: properly handle scrape errors when stream parsing is enabled ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967	2020-12-15 14:08:28 +02:00
Nikolay	95c9b630cc	adds new Array Flags (#965) ... * adds ArrayDuration and ArrayBool flags, makes sendTimeout and tlsInsecure configurable per remoteWrite url * added backward compatibility testcases for ArrayDuration and ArrayBool * fixes bool flag * fixes test cases	2020-12-15 12:51:12 +02:00
Aliaksandr Valialkin	60fcac4878	lib/promscrape: add bootstrap styles to /targets html page	2020-12-15 12:37:56 +02:00
Aliaksandr Valialkin	5af2a9ca0e	lib/promscrape: formatting fixes for /tarets page	2020-12-15 11:59:04 +02:00
Aliaksandr Valialkin	020917949b	lib/promscrape: formatting fixes for /targets page	2020-12-15 11:24:18 +02:00
Aliaksandr Valialkin	d006b41eff	lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964	2020-12-14 19:25:25 +02:00
Aliaksandr Valialkin	ae972429c7	lib/promscrape: add missing whitespace between duration and ago word at /targets page	2020-12-14 14:19:58 +02:00
Aliaksandr Valialkin	069c9ade52	app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser	2020-12-14 14:06:00 +02:00
Nikolay	ce8c2dd1f1	Changes targets api (#961) ... * changes /targets api adds html response if requester accepts text/html, adds quick template for /targets api, fixes pathPrefix for / requests * changes namings * renamed targets file * Update app/victoria-metrics/main.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com> * adds trimspace to qtpl, moves content-type for targets response closer to writer * fixes bug with prefix Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-12-14 13:36:48 +02:00
Aliaksandr Valialkin	5f9d88a3cb	lib/promscrape/discovery/consul: reduce load on Consul API server by increasing timeout for blocking requests from 50 seconds to 9 minutes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-11 17:24:13 +02:00
Aliaksandr Valialkin	fd9fd191b9	lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server ... Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg. See https://www.consul.io/api-docs/health#list-nodes-for-service for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170	2020-12-08 21:52:42 +02:00
Aliaksandr Valialkin	4146fc4668	all: properly handle CPU limits set on the host system/container ... This can reduce memory usage on systems with enabled CPU limits. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946	2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin	364f30a6e7	lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork() ... This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they are no longer used. See the related commit e205975716 and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-12-08 17:50:05 +02:00
Aliaksandr Valialkin	08b71d2067	lib/promscrape: re-use strings for labels stored in ScrapeWork ... This should reduce memory usage when working with big number of scrape targets.	2020-12-08 12:22:59 +02:00
Aliaksandr Valialkin	0f1b969aa6	lib/promscrape: export vm_promscrape_scrapers_{started|stopped}_total metrics for monitoring target churn rate	2020-12-08 11:57:52 +02:00
Aliaksandr Valialkin	c7ac7c1807	lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value ... This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.	2020-12-08 11:50:48 +02:00
Aliaksandr Valialkin	05813259dc	lib/promscrape: export vm_promscrape_active_scrapers{type="<sd_type>"} metric for tracking the number of active scrapers per each service discovery type	2020-12-08 01:54:23 +02:00
Aliaksandr Valialkin	9c1c9d8e76	lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun command-line flag is passed ... Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944	2020-12-07 13:18:50 +02:00
Aliaksandr Valialkin	82972a8f2a	lib/promscrape: mention in scrape error message that scrape errors can be disabled by -promscrape.suppressScrapeErrors command-line flag	2020-12-06 23:27:58 +02:00
Aliaksandr Valialkin	299a35948c	lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set	2020-12-06 13:18:39 +02:00
Aliaksandr Valialkin	b0e4b234cb	lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol	2020-12-06 12:01:27 +02:00
Aliaksandr Valialkin	52915c8f7e	lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries ... Too low limit could result in unexpected errors when performing big number of blocking queries.	2020-12-05 12:15:52 +02:00
Aliaksandr Valialkin	eb27dbde13	lib/flagutil: make golangci-lint happy by using strings.TrimPrefix instead of manual prefix removal via strings.HasPrefix	2020-12-03 22:07:57 +02:00
Aliaksandr Valialkin	9d787f9edd	all: do not print usage info for all the flags when incorrect command-line flag is passed ... This should improve usability for VictoriaMetrics apps that have big number of command-line flags, i.e. all the apps.	2020-12-03 21:47:37 +02:00
Aliaksandr Valialkin	de0643fab5	lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher	2020-12-03 20:14:55 +02:00
Aliaksandr Valialkin	9cd8eb92f1	lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header	2020-12-03 20:05:23 +02:00
Aliaksandr Valialkin	5009b25a03	lib/promscrape: code cleanup after c6dee6c52d ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-12-03 19:50:53 +02:00
Nikolay	c6dee6c52d	Changes consul discovery api (#921) ... * adds consul watch api, it must reduce load on consul service with blocking wait requests, changed discoveryClient api with fetchResponseMeta callback. * small fix * fix after master merge * adds watch client at discovery utils * fixes consul watcher, changes namings, fixes data race * small typo fix * sanity fix * fix naming and service node update	2020-12-03 19:47:40 +02:00
Aliaksandr Valialkin	357f886f97	Revert "lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test" ... This reverts commit f3e80eb70e493620e42a0cc22a62c9af75076c77, since it isn't needed for single-node version of VictoriaMetrics	2020-11-29 01:27:25 +02:00
Aliaksandr Valialkin	ace969d595	lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test	2020-11-29 01:26:04 +02:00
Aliaksandr Valialkin	32869e4c0f	lib/promscrape: fix failing tests after a906b3862f	2020-11-29 01:26:03 +02:00
Aliaksandr Valialkin	a906b3862f	lib/protoparser/prometheus: properly parse OpenMetrics timestamps ... OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds. This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.	2020-11-27 14:54:29 +02:00
Aliaksandr Valialkin	3a32789352	lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets	2020-11-26 18:32:06 +02:00
Aliaksandr Valialkin	2cea4d403f	all: typo fix: thouthand->thousand	2020-11-26 13:33:46 +02:00
Aliaksandr Valialkin	b0a5c382ee	lib/promscrape: release http response non-200 status code	2020-11-26 13:25:17 +02:00
Aliaksandr Valialkin	b7fcdb528d	app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun	2020-11-25 22:59:13 +02:00
Aliaksandr Valialkin	1c669a69a8	lib/mergeset: tune the number of rawItemsBlocks to merge at once ... 512 blocks give higher ingestion performance and slightly lower memory usage	2020-11-25 21:52:52 +02:00
Aliaksandr Valialkin	7119f294f3	lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset	2020-11-25 21:19:43 +02:00
Aliaksandr Valialkin	8a057e705a	lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention ... This should simplify debugging when the source of the metric with unexpected timestamp must be found.	2020-11-25 14:41:37 +02:00
Aliaksandr Valialkin	b65236530c	lib/storage: typo fix in error message: allowd->allowed	2020-11-25 14:15:42 +02:00
Aliaksandr Valialkin	ae04378424	lib/protoparser/prometheus: properly parse "infinity" values in OpenMetrics format ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924	2020-11-24 19:03:38 +02:00
Aliaksandr Valialkin	bf95fbfc1d	lib/logger: disable rate limiting for error and warn logs by default	2020-11-24 12:42:25 +02:00
Aliaksandr Valialkin	78d2715d04	all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9	2020-11-24 12:42:22 +02:00
Aliaksandr Valialkin	b7f4fc6e0d	lib/protoparser/prometheus: properly parse metrics with exemplars ... Examplars have been introduced in OpenMetrics - see https://github.com/OpenObservability/OpenMetrics/blob/master/OpenMetrics.md#exemplars-1 Previously VictoriaMetrics couldn't parse the following metric foo{bar="baz"} 123 # exemplar here This commit fixes this. Note that VictoriaMetrics ignores the exemplar as for now.	2020-11-24 12:34:56 +02:00
Aliaksandr Valialkin	aa90b93778	lib/promscrape: expose __meta_ec2_ipv6_addresses label for ec2_sd_config like Prometheus will do in the next release	2020-11-23 16:56:42 +02:00
Aliaksandr Valialkin	de523c81b9	lib/promscrape: add filters option to dockerswarm_sd_config like Prometheus did in v2.23.0	2020-11-23 16:27:40 +02:00
Aliaksandr Valialkin	f765985947	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin	9d160f9048	lib/promscrape: hint that -enableTCP6 command-line flag can be used for connecting to IPv6 addresses	2020-11-21 14:39:00 +02:00
Aliaksandr Valialkin	d7932775cc	lib/promscrape/discovery/eureka: follow-up after eec76718e9	2020-11-20 14:00:12 +02:00
Nikolay	eec76718e9	Adds eureka service discovery (#913) ... * Adds eureka service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851 Netflix service discovery for AWS * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-20 13:38:12 +02:00
Aliaksandr Valialkin	97a0c80904	lib/logger: follow-up for 09105ff49c	2020-11-19 12:37:00 +02:00
Nikolay	09105ff49c	Adds log suppression per caller (#908) ... * Adds log suppression per caller https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905 * fixes style and report message	2020-11-19 12:17:23 +02:00
Aliaksandr Valialkin	a6b2b2c005	lib/logger: add -loggerWarnsPerSecondLimit command-line flag for rate limiting of WARN log messages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905	2020-11-18 03:43:37 +02:00
Aliaksandr Valialkin	266788be14	app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place ... This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.	2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin	0208d8c103	lib/storage: add a test for Storage.SearchMetricNames	2020-11-16 13:15:16 +02:00
Aliaksandr Valialkin	465923b181	app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags	2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin	48d033a198	app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API ... See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb	2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin	4aaee33860	lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page	2020-11-16 00:44:35 +02:00
Aliaksandr Valialkin	0deabbbb4a	lib/protoparser/promremotewrite: log the time spent on unsuccessful data read from the network ... This should help with debugging `connection timed out` errors.	2020-11-13 17:49:12 +02:00
Vasily	6fcbd17bdd	Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig (#796) ... * Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig * Add omitempty annotation to all the default/optional values * Fix annotations after review	2020-11-13 16:19:05 +02:00
Aliaksandr Valialkin	cdf1e6684b	lib/protoparser/opentsdbhttp: increment errors counter on unmarshal errors ... This is a follow-up for 149c0c4a6d	2020-11-13 13:23:17 +02:00
Aliaksandr Valialkin	149c0c4a6d	lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols ... The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 13:05:24 +02:00
Aliaksandr Valialkin	4f8a3af061	lib/protoparser/promremotewrite: synchronously process Prometheus remote_write requests ... There is no reason in processing these requests asynchronously in the face of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896 Synchronous processing code is easier to read and understand than the previous async code	2020-11-13 12:17:25 +02:00
Aliaksandr Valialkin	57a4af98fa	lib/protoparser/promremotewrite: forward errors, which can occur during data ingestion, to the caller of ParseStream, so it could properly return HTTP 503 status code on non-nil error ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896	2020-11-13 11:01:07 +02:00
Aliaksandr Valialkin	47a038401b	all: consistently return text-based HTTP responses with charset=utf-8 ... This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897	2020-11-13 10:35:41 +02:00
immerrr again	51c529a2b6	app/vmstorage: add "/internal/force_flush" endpoint (#893)	2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin	e205975716	lib/promscrape: make a copy of ScrapeWork from discovered []ScrapeWork slice instead of referring to an item in this slice ... This should prevent from holding previously discovered []ScrapeWork slices when a part of discovered targets changes over time. This should reduce memory usage for the case when big number of discovered scrape targets changes over time. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 16:13:05 +02:00
Aliaksandr Valialkin	6e668fd480	lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets ... This should reduce load on GC a bit when discovering big number of scrape targets	2020-11-10 15:56:51 +02:00
Aliaksandr Valialkin	bcd139362b	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-10 00:19:57 +02:00
Aliaksandr Valialkin	6c24c5caa3	lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-09 10:54:42 +02:00
Aliaksandr Valialkin	83df20b5b5	lib/promscrape: clean references to label name and label value strings after applying per-target relabeling ... This should reduce memory usage when per-target relabeling creates big number of temporary labels with long names and/or values. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 16:19:59 +02:00
Aliaksandr Valialkin	9e83335ca9	lib/promscrape/discovery/kubernetes: go fmt	2020-11-07 13:03:49 +02:00
Aliaksandr Valialkin	5407eed2f6	lib/promscrape/discovery/kubernetes: reduce memory usage for labels when discovering big number of scrape targets by using string concatenation instead of fmt.Sprintf ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825	2020-11-07 13:03:08 +02:00
Aliaksandr Valialkin	188325f0fc	lib/promscrape: eliminate data race in stream parse mode ... Previously `-promscrape.streamParse` mode could result in garbage labels for the scraped metrics because of data race. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247	2020-11-07 12:45:17 +02:00
Aliaksandr Valialkin	368b69b4c4	app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange	2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin	1cb78ba1a0	lib/storage: remove data race when updating rowsDeleted	2020-11-05 01:12:21 +02:00
Aliaksandr Valialkin	b378cd6ed8	app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set	2020-11-05 01:01:33 +02:00
Nikolay	4c808d58bf	Adds ready probe (#874) ... * adds leading forward slash check for scrapeURL path https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835 * adds ready probe for scrape config initialization, it should prevent metrics loss during vmagent rolling update, /ready api will return 425 http code, if some scrape config still waits for initialization. * updates docs * Update app/vmagent/README.md * renames var * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 20:29:18 +02:00
Aliaksandr Valialkin	b2042a1c30	lib/promscrape: docs update after e4182dd896 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 17:12:30 +02:00
Aliaksandr Valialkin	caeb74f068	app/vmselect: reduce memory usage when query touches big number of time series	2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin	ae91a6883c	lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively ... Previously such blocks were cleaned after they weren't accessed during 10 minutes. Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.	2020-11-04 17:04:04 +02:00
Nikolay	e4182dd896	reduces memory usage for vmagent, (#880) ... * reduces memory usage for vmagent, limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878 * Update app/vmagent/README.md * Update app/vmagent/README.md Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-11-04 17:03:43 +02:00
Aliaksandr Valialkin	b9e5172aa2	Revert "lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets" ... This reverts commit 5a9743211f.	2020-11-04 11:45:02 +02:00
Aliaksandr Valialkin	5a9743211f	lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 11:08:57 +02:00
Aliaksandr Valialkin	ca8b5745b5	lib/promscrape: reduce memory allocations in promLabelsString() function ... This should help with reducing memory usage in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878	2020-11-04 10:38:44 +02:00
Aliaksandr Valialkin	f3f62ab04e	lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification	2020-11-03 15:32:02 +02:00
Aliaksandr Valialkin	c87fb9191e	lib/storage: remove unneeded fmt.Sprintf	2020-11-03 14:20:31 +02:00
John Belmonte	51e661ecfe	add short_version label to vm_app_version metric (#877) ... * add short_version label to vm_app_version metric use case: Version panel of Grafana dashboard should use a live query, but currently it uses a template query which becomes stale. Grafana is not able to preform regex substitution on labels. * Update metrics.go * fix compile	2020-11-03 14:10:42 +02:00
Aliaksandr Valialkin	fe289331dd	lib/storage: remove obsolete code	2020-11-02 19:11:59 +02:00
Aliaksandr Valialkin	c0f3be824d	lib/promscrape: properly handle response body after 301 redirect ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/869	2020-11-02 01:09:52 +02:00
Aliaksandr Valialkin	e277c3d07b	lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	29e4e7f422	lib/storage: drop more samples outside the given retention during background merge ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	b7638f04a7	app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets ... This page is exposed by vmagent and by a single-node VictoriaMetrics Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	d12c4914f0	lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*	2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin	64e2d66014	lib/storage: code cleanup after 5bfd4e6218	2020-11-01 23:35:06 +02:00
Nikolay	67059caa12	fixes panic at scrape error body formating, (#868) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/864 regression after body reuse improvements	2020-10-29 17:17:52 +03:00
Nikolay	de3fe22815	adds leading forward slash check for scrapeURL path (#855) ... * fixes in-consistency with prometheus behaviour for scrape targets url path. https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835	2020-10-29 08:39:42 +03:00
Aliaksandr Valialkin	0729cc36b2	lib/memory: do not print trailing zeroes in logs for -memory.allowedPercent command-line flag	2020-10-20 14:32:07 +03:00
Aliaksandr Valialkin	5bfd4e6218	app/vmstorage: support for -retentionPeriod smaller than one month ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17	2020-10-20 14:31:44 +03:00
faceair	84e3881c0b	disable response compression on websocket (#841)	2020-10-17 13:32:34 +03:00
Aliaksandr Valialkin	01987f8c77	lib/storage: small code adjustements after d2960a20e0 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-10-17 01:16:54 +03:00
faceair	d2960a20e0	evaluate the execution cost of all tag filters (#824) ... * evaluate the execution cost of all tag filters * fix suffixes typo	2020-10-17 00:46:55 +03:00
Nikolay Khramchikhin	e6ab69dd88	fixes openstack api endpoint with suffix trim adds openstack (#840) ... api v2.0 check	2020-10-16 21:20:57 +03:00
Aliaksandr Valialkin	63c4999e06	lib/promscrape: code prettifying after 9bd9f67718	2020-10-12 16:12:36 +03:00
Nikolay Khramchikhin	9bd9f67718	Adds dockerswarm sd (#818) ... * adds dockerswarm service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656 Following roles supported: services, tasks and nodes. Basic, token and tls auth supported. Added tests for labels generation. * added unix socket support to discovery utils Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-12 13:38:21 +03:00
Aliaksandr Valialkin	b449607181	lib/backup: add MustStop() method for all remote filesystems	2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin	cf5f2874cd	lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter	2020-10-09 15:12:03 +03:00
Aliaksandr Valialkin	68f0e00761	app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start	2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin	561a7619a5	lib/promscrape: fix tests after 71ea4935de	2020-10-08 19:32:36 +03:00
Aliaksandr Valialkin	71ea4935de	lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors ... Show also original labels for duplicate targets in error message in order to simplify debugging the issue. Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target. This may simplify debugging for target relabeling. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651	2020-10-08 18:58:30 +03:00
Aliaksandr Valialkin	9b0a5c1028	lib/backup/actions: improve logging to be more clear to humans	2020-10-08 14:23:07 +03:00
Aliaksandr Valialkin	af90b3121c	app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it	2020-10-07 17:35:44 +03:00
Aliaksandr Valialkin	e19d400230	lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does ... Such timestamps are rounded to seconds like Carbon does. See b0ba62a62d/lib/carbon/protocols.py (L197)	2020-10-06 11:38:29 +03:00
Aliaksandr Valialkin	90aa2a8ffd	lib/promscrape/discovery/openstack: show expiration time for refreshed OpenStack token in seconds - this is easier to interpret by human	2020-10-06 11:34:09 +03:00
Aliaksandr Valialkin	aba899c298	lib/promscrape/discovery/openstack: code prettifying after cbe3cf683b ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728	2020-10-05 18:11:55 +03:00
Nikolay Khramchikhin	cbe3cf683b	Adds openstack sd (#811) ... * adds openstack service discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728 implemented hypervisors and instance discovery with openstack v3 api. Added tests for labeling and data parsing. Added token refresh. * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2020-10-05 16:45:33 +03:00
Aliaksandr Valialkin	f42194d817	lib/promrelabel: make a copy of label with new name for action: labelmap in the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/812	2020-10-05 16:19:19 +03:00
Aliaksandr Valialkin	bbeac0ba46	lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807	2020-10-05 15:19:05 +03:00
Aliaksandr Valialkin	47db9bb24a	lib/decimal: add tests for negative values passed to maxUpExponent	2020-10-05 14:56:45 +03:00
Aliaksandr Valialkin	bc7d67cee2	lib/decimal: properly calibrate scale for blocks with Inf values ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/805	2020-10-05 14:52:44 +03:00
Aliaksandr Valialkin	764dc2499f	lib/storage: code cleanup after 10f2eedee0 ... Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search, since this code became unused after implementing per-day inverted index almost a year ago. While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names such as `foo.bar.baz`).	2020-10-01 19:06:23 +03:00
Aliaksandr Valialkin	10f2eedee0	lib/storage: imrpove cache effectiveness for time series ids matching the given filters ... Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day. This should reduce CPU usage in the following cases: * when querying recently added data with small churn rate for time series * when querying historical data	2020-10-01 14:38:25 +03:00
Aliaksandr Valialkin	bec9b31b81	lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space if there are multiple partitions with deferred merges due to disk space shortage	2020-09-29 22:51:43 +03:00
Aliaksandr Valialkin	a9db81c4ab	app/vmstorage: add metrics for determining whether background merges need additional disk space to complete ... These metrics are: * vm_small_merge_need_free_disk_space * vm_big_merge_need_free_disk_space Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-29 21:48:33 +03:00
Aliaksandr Valialkin	127537d631	app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-29 19:49:12 +03:00
Aliaksandr Valialkin	76b244cfcf	lib/cgroup: do not adjust the number of detected CPU cores via /sys/devices/system/cpu/online ... The adjustement increases the resulting GOMAXPROC by 1, which looks confusing to users as outlined at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-698595309	2020-09-29 13:55:26 +03:00
Aliaksandr Valialkin	1659135752	lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm	2020-09-29 13:10:22 +03:00
Aliaksandr Valialkin	38789e4aa0	lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm	2020-09-29 12:40:35 +03:00
Aliaksandr Valialkin	19c0b6f3ef	lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic	2020-09-29 12:30:04 +03:00
Aliaksandr Valialkin	7cde336b33	lib/protoparser/prometheus: fix TestParseStream after 124f78857b	2020-09-29 12:11:17 +03:00
Aliaksandr Valialkin	7673839228	lib/{fs,filestream}: small consistency-related updates after cc90a548b1	2020-09-29 00:42:43 +03:00
Nikolay Khramchikhin	cc90a548b1	added openbsd implementations (#790) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:29:04 +03:00
Aliaksandr Valialkin	7500146321	lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer	2020-09-28 17:19:16 +03:00
Aliaksandr Valialkin	124f78857b	app/{vminsert,vmagent}: improve data ingestion speed over a single connection ... Process data obtianed from a single connection on all the available CPU cores.	2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin	5cdad60a6f	lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server ... This should reduce syscall overhead when reading big amounts of data	2020-09-28 02:07:10 +03:00
Aliaksandr Valialkin	95688cbfc5	all: add native format for data export and import ... The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.	2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin	b4bf722d8f	lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream ... Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.	2020-09-26 04:21:32 +03:00
Aliaksandr Valialkin	82973f8ae7	Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" ... This reverts commit bab6a15ae0. Reason for revert: the `fetchData` arg is used in cluster branch. Leaving this arg in master branch makes smaller the diff with cluster branch.	2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin	bab6a15ae0	lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock ... This arg became unused after 23bdc1f107	2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin	24ca30bf66	lib/storage: correctly use maxBlockSize in various checks ... Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.	2020-09-24 18:12:56 +03:00
Aliaksandr Valialkin	2985077c35	all: consistently use "%w" formatting in fmt.Errorf for wrapped errors	2020-09-23 22:46:34 +03:00
Aliaksandr Valialkin	c5ef0e6327	lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue	2020-09-23 02:17:47 +03:00
Aliaksandr Valialkin	5c42965853	lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728	2020-09-22 23:27:19 +03:00
Aliaksandr Valialkin	a69234ed18	lib/storage: code prettifying after be5e1222f3 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781	2020-09-22 00:36:45 +03:00
faceair	be5e1222f3	add filter to getMetricIDs (#783) ... * add getMetricIDs filter * check nil filter before use	2020-09-22 00:33:43 +03:00
Aliaksandr Valialkin	29108cc53e	lib/logger: add -loggerDisableTimestamps command-line flag for disabling timestamps in logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778	2020-09-21 19:28:04 +03:00
Aliaksandr Valialkin	964bc7595c	lib/promscrape/discovery/ec2: code prettifying after 312fead9a2 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771	2020-09-21 18:43:34 +03:00
Nikolay Khramchikhin	312fead9a2	Add improvements to ec2_sd_discovery (#775) ... * Add improvements to ec2 discovery https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771 role_arn support with aws sts instance iam_role support refreshing temporary tokens * Apply suggestions from code review Co-authored-by: Roman Khavronenko <hagen1778@gmail.com> * changed implementation, removed tests, clean up code * moved endpoint builder into getEC2APIResponse Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>	2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin	9739283dad	lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges	2020-09-21 15:54:11 +03:00
Aliaksandr Valialkin	3e8569f456	lib/decimal: optimize maxUpExponent() by eliminating division from hot path	2020-09-19 13:50:09 +03:00
Aliaksandr Valialkin	f00e0e0103	lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush	2020-09-19 12:51:41 +03:00
Aliaksandr Valialkin	26115891db	lib/decimal: properly store Inf values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-18 19:07:07 +03:00
Aliaksandr Valialkin	1a9ee39b0e	lib/promscrape: avoid copying response body when scraping targets. ... This should reduce memory usage when scraping targets with millions of metrics.	2020-09-18 13:05:43 +03:00
Aliaksandr Valialkin	70c721c01b	lib/persistentqueue: flush data to disk every second ... Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-18 13:05:40 +03:00
Aliaksandr Valialkin	1f33dd717f	lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions ... This may be useful for freeing up storage space after time series deletion. See https://victoriametrics.github.io/#force-merge for more details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686	2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin	8beb0da6ad	lib/{mergeset,storage}: compare errors with errors.Is()	2020-09-17 03:03:02 +03:00
Aliaksandr Valialkin	067d7c1ea1	lib/{mergeset,storage}: code prettifying	2020-09-17 02:06:31 +03:00
Aliaksandr Valialkin	020bd8685e	lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions	2020-09-17 01:49:03 +03:00
Aliaksandr Valialkin	ab53cb6f7b	app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens ... Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773	2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin	39dee12ed7	lib/persistentqueue: code simplification after d455764a6f	2020-09-16 21:14:19 +03:00
Aliaksandr Valialkin	d455764a6f	lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset) ... The strategy is: - Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown. - Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data. - Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown. - Skipping of broken chunk file if it cannot be read. - Fsyncing finalized chunk files. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687	2020-09-16 18:13:44 +03:00
Aliaksandr Valialkin	ffadf035fa	lib/protoparser/vmimport: add more testcases for invalid timestamps and values ... Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:22:06 +03:00
Aliaksandr Valialkin	d8183c3124	lib/protoparser: report more errors for incorrect timestamps and/or values ... Previously certain errors in timestamps and/or values could be silently skipped, which could lead to samples with zero values stored in the database. Updates https://github.com/VictoriaMetrics/vmctl/issues/25	2020-09-16 02:14:18 +03:00
Aliaksandr Valialkin	9bc8484ab6	lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99	2020-09-16 01:35:12 +03:00
Aliaksandr Valialkin	fa01169c3d	lib/promscrape: add a link to troubleshooting docs to error message when duplicate scrape target with identical labels is skipped	2020-09-15 14:16:05 +03:00
Aliaksandr Valialkin	ba74d0c14c	lib/promscrape: typo fix	2020-09-12 00:14:21 +03:00
Aliaksandr Valialkin	7d893a234c	lib/promscrape: do not reset the remaining rows when pushing a part of data to remote storage during big scrapes ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/753 Thanks to @PerGon and @clmssz for help with debugging.	2020-09-11 23:39:13 +03:00
Aliaksandr Valialkin	0e19f35af5	lib/promscrape/discovery/dns: add __meta_dns_srv_record_target and __meta_dns_srv_record_port labels ... This syncs dns service discovery with Prometheus 2.21 - see https://github.com/prometheus/prometheus/releases and https://github.com/prometheus/prometheus/pull/7678 .	2020-09-11 23:39:13 +03:00
Aliaksandr Valialkin	62fde80490	lib/protoparser/common: do not read request body when parsing timestamp query arg ... This was preventing from reading data via /api/v1/prometheus/import . Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 14:44:58 +03:00
Aliaksandr Valialkin	5a90a92378	lib/storage: do not store inf values, since they may lead to significant precision loss for previously stored values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752	2020-09-11 14:44:53 +03:00
Aliaksandr Valialkin	f95eea60d1	lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus ... This improves consistency with timestamps in Prometheus text exposition format	2020-09-11 14:04:46 +03:00
Aliaksandr Valialkin	2380e9b017	app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750	2020-09-11 13:27:14 +03:00
Nikolay Khramchikhin	6c80ae0da8	Added endpointslices discovery to k8s api (#760) ... This is similar to https://github.com/prometheus/prometheus/pull/6838 , which will be added in Prometheus v2.21. See https://github.com/prometheus/prometheus/releases/tag/v2.21.0-rc.1 * Added endpointslices discovery to k8s api Started from 1.17 k8s version endpointslices is beta, it allows to query k8s api for endpoints more efficient. It presents at scrape_config.yaml as separate role for kubernetes_sd_config. kubernetes_sd_config: - role: endpointslices * fixed typos, changed EndpointConditions signature - with values instead of pointers	2020-09-11 12:16:45 +03:00
Aliaksandr Valialkin	f6bc608e86	app/vmselect: initial implementation of Graphite Metrics API ... See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api	2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin	9d8fdff6c5	lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps ... This should reduce disk space usage when scraping targets containing metrics with identical names such as `node_cpu_seconds_total`, histograms, quantiles, etc. Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring the effectiveness of timestamp blocks merging.	2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin	df169b1ebd	lib/httpserver: add a jitter to connection timeouts in order to protect from Thundering herd problem	2020-09-08 19:55:09 +03:00
Nikolay Khramchikhin	8d71a60a76	Changed s3 configProfile flag default, (#749) ... aws sdk has complicated logic for chosing profile name and we shouldn't set it to `default` value. It leads to bugs and improper configuration. Set it to empty value by default is safe. It will be automatically set to `default` by sdk.	2020-09-07 21:53:24 +03:00
Aliaksandr Valialkin	be6ae4b5e7	lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699	2020-09-04 00:05:05 +03:00
Aliaksandr Valialkin	d387da142e	lib/httpserver: add -http.connTimeout command-line flag for limiting the lifetime for incoming http connections ... This can be useful for balancing incoming connections among multiple services.	2020-09-03 22:23:29 +03:00
Aliaksandr Valialkin	f41b36bb9a	app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats ... Extra labels may be added to the imported data by passing `extra_label=name=value` query args. Multiple query args may be passed in order to add multiple extra labels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719	2020-09-02 19:43:21 +03:00
Aliaksandr Valialkin	038358b777	lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling ... The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc , because `sw.labels` may be reset multiple times when processing big number of rows.	2020-09-02 18:46:01 +03:00
Aliaksandr Valialkin	c2ff8de456	lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections	2020-09-01 15:33:24 +03:00
Aliaksandr Valialkin	b059f194e4	lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics ... This has been broken at 7785869ccc	2020-09-01 11:08:13 +03:00
Aliaksandr Valialkin	7785869ccc	lib/promscrape: reduce memory usage when scraping targets with millions of metrics ... This should help when scraping /federate endpoints from Prometheus instances, which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/	2020-09-01 10:57:07 +03:00
Aliaksandr Valialkin	acbcad1ece	lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy	2020-08-28 09:49:32 +03:00
Aliaksandr Valialkin	f4c4ab811b	lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores ... There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.	2020-08-28 09:49:32 +03:00
Roman Khavronenko	f2c004d1ae	lib/flagutil: avoid int overflow for arch 386 (#710) ... Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32, whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with `int overflow` error.	2020-08-20 22:27:37 +01:00
Aliaksandr Valialkin	efc730863b	lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels ... Previously targets with big number of metrics and/or labels could generated too big buffers, which then could be re-used when scraping targets with small number of metrics. This resulted in memory waste. Now big buffers are used only for targets with big number of metrics / labels, while small buffers are used for targets with small number of metrics / labels.	2020-08-16 22:29:51 +03:00
Aliaksandr Valialkin	d6967319b6	lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket ... This should reduce the number of resizings for the returned byte buffers.	2020-08-16 22:13:30 +03:00
Roman Khavronenko	f5f59896ec	lib/decimal: rename significant decimal digits to significant figures (#698) ... The previous notion was inconsistent with what `decimal.Round` does. According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding applied to all significant figures, not just decimal ones.	2020-08-16 17:21:35 +03:00
Aliaksandr Valialkin	147c35ebd4	all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates	2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin	7c0d6a8b88	lib/memory: improve log message about the memory allowed to use by VictoriaMetrics	2020-08-16 16:04:11 +03:00
Aliaksandr Valialkin	ed00eb3f33	lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data ... The OS should return any buffered data in the stream without the need to set the read timeout. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-15 15:38:08 +03:00
Aliaksandr Valialkin	00b1659dde	lib: dump compressed block contents on error during decompression ... This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222	2020-08-15 14:44:33 +03:00
Aliaksandr Valialkin	b3849a90fd	lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty ... This should reduce memory allocations and copying when the byte slice is growing.	2020-08-15 01:40:54 +03:00
Aliaksandr Valialkin	7554be172d	lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:40:15 +03:00
Aliaksandr Valialkin	4beab7ad39	lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696	2020-08-14 20:14:11 +03:00
Aliaksandr Valialkin	52791fd1c0	lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches	2020-08-14 19:13:38 +03:00
Aliaksandr Valialkin	576da0fe46	app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag	2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin	215967437d	lib/protoparser/prometheus: typo fix in error message	2020-08-14 11:04:23 +03:00
Aliaksandr Valialkin	e646674b23	lib/promscrape: use a hint on body length instead of body capacity ... This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:17:52 +03:00
Aliaksandr Valialkin	4628deecd1	lib/promscrape: reduce memory usage when scraping big number of targets ... Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689	2020-08-14 01:04:53 +03:00
Aliaksandr Valialkin	eead3ee8ec	lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes	2020-08-13 22:31:52 +03:00
Aliaksandr Valialkin	c402265e88	all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components ... Such placeholders are substituted by the corresponding environment variable values. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583	2020-08-13 17:15:25 +03:00
Aliaksandr Valialkin	ef7e2af8f5	app: respect CPU limits set via cgroups ... Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage for cases when VictoriaMetrics components run in containers with CPU limits. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685	2020-08-11 22:59:19 +03:00
Aliaksandr Valialkin	15aa6142ef	lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use	2020-08-11 17:04:39 +03:00
Aliaksandr Valialkin	c098988a18	lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint	2020-08-10 20:23:26 +03:00
Aliaksandr Valialkin	1bdfa29ef7	lib/promscrape: optimize per-metric hash calculations ... This increases vmagent performance by up to 10% when scraping big number of metrics	2020-08-10 19:49:03 +03:00
Aliaksandr Valialkin	8d9eb5f808	lib/storage: mention time range used in the query that led to error message ... This should improve detecting slow queries with too big time ranges	2020-08-10 13:46:36 +03:00
Aliaksandr Valialkin	582c74cd93	lib/storage: mention tag filters used in the query that led to error message ... This should improve detecting invalid or heavy queries that lead to errors.	2020-08-10 13:36:49 +03:00
Aliaksandr Valialkin	f3d33e23c9	app/vmstorage: improve error logging when the request times out	2020-08-10 13:23:26 +03:00
Aliaksandr Valialkin	455bf50a91	lib/promscrape: show real timestamp and real duration for the scape on /targets page ... Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp	2020-08-10 12:40:25 +03:00
Aliaksandr Valialkin	a499de45cc	lib/promscrape: make errcheck happy	2020-08-09 13:17:18 +03:00
Aliaksandr Valialkin	23c9e6b727	lib/promscrape: export scrape_samples_added per-target metric like Prometheus does ... This metric may be useful for detecting targets with high churn rate for the exported metrics. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683	2020-08-09 12:45:39 +03:00
Aliaksandr Valialkin	9d32fb1d9e	lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails ... this is expected condition, so it is better to use WARN log level for it	2020-08-09 12:07:30 +03:00
Aliaksandr Valialkin	d4b6d22987	lib/promscrape: add a test for scrape config for blackbox exporter ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684	2020-08-09 12:02:48 +03:00
Aliaksandr Valialkin	84fd8af6d3	lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit ... This should improve data ingestion performance when heavy searches are executed See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-08-07 08:49:40 +03:00
Aliaksandr Valialkin	9043a509a3	lib/storage: properly check timeouts and pace limits ... Previously they were checked on every iteration for small number of iterations	2020-08-07 08:40:37 +03:00
Aliaksandr Valialkin	bc8381613d	app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names	2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin	8e44fba76d	lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x	2020-08-06 18:45:55 +03:00
Aliaksandr Valialkin	7dbe335426	lib/pacelimiter: increase scalability for multi-CPU system	2020-08-06 18:32:59 +03:00
Aliaksandr Valialkin	ad730d8a17	lib/storage: optimize prefetching metric names for the given metricIDs	2020-08-06 16:53:10 +03:00
Aliaksandr Valialkin	639b26b40c	lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock	2020-08-06 15:31:34 +03:00
Aliaksandr Valialkin	8f16388428	lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 ... This should limit the maximum memory usage and reduce CPU trashing on vmstorage when multiple heavy queries are executed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin	aaa497ff0b	Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer	2020-08-05 11:55:58 +03:00
Aliaksandr Valialkin	a034f02fb2	lib/backup: allow using ~/.aws/config without region ... Use us-west-2 for determining bucket region.	2020-08-04 13:07:59 +03:00
Aliaksandr Valialkin	509d12643b	app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin	5e71fab8a6	lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 ... Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this increases query latencies since more CPUs are busy with merges. While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.	2020-07-31 17:46:56 +03:00
Aliaksandr Valialkin	e7959094f6	lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected ... The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts. Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon, which should return CPU and disk IO usage to normal levels. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618	2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin	922d9aadf2	lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once ... Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series, while only a single increment is needed when inserting the first data point for this time series.	2020-07-30 16:17:24 +03:00
Aliaksandr Valialkin	67a64c142d	lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659	2020-07-29 13:12:42 +03:00
Sasasu	5b81bdde39	lib/storage: metaindexRow use memroy more efficiently (#655) ... due to memory align the metaindexRow structure use 64-byte pre object. this commit changes the order of field, make metaindexRow use 56-byte pre object. Signed-off-by: Sasasu <su@sasasu.me>	2020-07-27 19:02:53 +03:00
Aliaksandr Valialkin	865610a7c8	lib/protoparser/prometheus: add a test for cassandra-exporter ... Thanks to Seva	2020-07-27 18:37:11 +03:00
Aliaksandr Valialkin	54ef2d8112	lib/storage: slightly reduce code difference between single-node and cluster versions	2020-07-24 00:31:16 +03:00
Aliaksandr Valialkin	039c9d2441	lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index ... Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`. This commit stops searching in inverted index on query timeout.	2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin	2a45871823	lib/storage: add more fine-grained pace limiting for search	2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin	6f05c4d351	lib/storage: improve prioritizing of data ingestion over querying ... Prioritize also small merges over big merges. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648	2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin	61c611f5ad	lib/storage: properly calculate global metrics in UpdateStats()	2020-07-23 00:35:15 +03:00
Aliaksandr Valialkin	9224ede54f	lib/mergeset: properly calculate global metrics in UpdateStats() ... Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.	2020-07-23 00:35:13 +03:00
Aliaksandr Valialkin	228d137936	lib/storage: reorder mergeBlockStreams() args in order to make them more consistent	2020-07-22 21:58:10 +03:00
Aliaksandr Valialkin	e4303d3d21	lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs ... This condition may occur after the following sequence of events: 1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs. 2) All the goroutines return from Storage.AddRows. 3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body. The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal(). This may take indefinite time.	2020-07-22 21:52:34 +03:00
Aliaksandr Valialkin	31ae5911a8	app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage	2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin	d3442b40b2	lib/uint64set: optimize adding items to the set via Set.AddMulti	2020-07-21 20:56:59 +03:00
Aliaksandr Valialkin	b35cb293f5	lib/httpserver: log remote address in error message from httpserver.Errorf ... This should improve detection of the root cause of errors. Thanks to Anant for the idea.	2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin	e1107fec10	lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted ... This is a follow-up commit after 12b16077c4 , which didn't reset the `tsidCache` in all the required places. This could result in indefinite errors like: missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time Fix this by resetting the cache inside deleteMetricIDs function.	2020-07-14 14:06:32 +03:00
Aliaksandr Valialkin	cde18d1f43	lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"} metric	2020-07-14 12:16:35 +03:00
Seva Poliakov	457e61900d	add vm_protoparser_rows_read_total metrics to promscrape (#624) ... * add vm_protoparser_rows_read_total metrics to promscrape move vm_protoparser_rows_read_total for promscrape to better place move vm_protoparser_rows_read_total for promscrape to better place * remove possibility of infinity loop at prometheus parser	2020-07-14 12:16:34 +03:00
Roman Khavronenko	7e347972c4	lib/flagutil: specify additional description for all Array type flags (#620) ... Array type flag is now defined as `value` type in flag description when printed. This change adds additional description to every Array type flag so it would be clear what exact type is used: ``` -remoteWrite.urlRelabelConfig array Optional path to relabel config for the corresponding -remoteWrite.url Supports array of values separated by comma or specified via multiple flags. ```	2020-07-13 21:56:37 +03:00
Roman Khavronenko	19dd121968	lib/persistentqueue: add vm_persistentqueue_bytes_pending metric (#619) ... Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount of bytes in persistentqueue flushed on disk as a difference between write and read offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes` except of accounting for bytes in-memory.	2020-07-13 21:54:09 +03:00
Roman Khavronenko	829ec4f9cf	Extend metric vm_promscrape_targets with status label (#615) ... The change to `vm_promscrape_targets` metric suppose to improve observability for `vmagent` so it will be possible to track how many targets are up or down for every specific scrape group: ``` vm_promscrape_targets{type="static_configs", status="down"} 1 vm_promscrape_targets{type="static_configs", status="up"} 2 ```	2020-07-13 21:52:03 +03:00
Aliaksandr Valialkin	cba820e390	app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus	2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin	6fe3c48a6e	properly calculate readCalls	2020-07-10 12:00:58 +03:00
Aliaksandr Valialkin	3e1d7d8489	lib/promscrape: send Accept header similar to Prometheus when scraping targets ... This should fix scraping Spring Boot servers, which return incorrect response unless `Accept: text/plain` request header is set. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608	2020-07-08 19:48:22 +03:00
Aliaksandr Valialkin	cb92113632	lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS ... Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense, since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs, i.e. to GOMAXPROCS.	2020-07-08 17:32:18 +03:00
Roman Khavronenko	e7557e0252	lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607) ... The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to respect the naming standard.	2020-07-08 14:18:41 +03:00
Aliaksandr Valialkin	e59b9916aa	lib/protoparser/graphite: go fmt	2020-07-08 14:12:10 +03:00
Aliaksandr Valialkin	d0b694c5c8	lib/protoparser/graphite: add more tests after eb45185eef ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610	2020-07-08 14:10:35 +03:00
Seva Poliakov	eb45185eef	Fix graphite minus one timestamp (#609) ... * fix graphite -1 timestamp * format the graphite fix -1 timestamp	2020-07-08 13:59:19 +03:00
Aliaksandr Valialkin	32b9fb58b8	lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag	2020-07-08 13:54:26 +03:00
Aliaksandr Valialkin	12b16077c4	lib/storage: reset MetricName->TSID cache after deleting time series ... This should prevent from adding new data points to deleted time series without the need to check for the deleted time series. This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set contains big number of time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596 Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604	2020-07-06 22:01:08 +03:00
Aliaksandr Valialkin	a23806f486	lib/fs: clarify description for -fs.disableMmap command-line flag	2020-07-06 14:28:34 +03:00
Aliaksandr Valialkin	6daa5f7500	lib/storage: prioritize data ingestion over heavy queries ... Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream. Prevent this by delaying queries' execution until free resources are available for data ingestion. Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources for data ingestion and/or for executing heavy queries. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291	2020-07-05 19:42:05 +03:00
Aliaksandr Valialkin	ff6a0955eb	lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts	2020-07-03 21:33:22 +03:00
Aliaksandr Valialkin	8b133e40d5	lib/promscrape: prevent from too big deadline misses on scrape retries ... The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case. By default it is limited to scrape_interval configured for the given scrape target.	2020-07-03 20:41:36 +03:00
Aliaksandr Valialkin	44a54b8b3d	lib/promscrape: check for nil error before checking for the returned status code when scraping targets	2020-07-03 18:37:14 +03:00
Aliaksandr Valialkin	0b2086b7a5	app/vminsert: prevent from adding and/or selecting labels with empty values ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600	2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin	91b3482894	app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries ... See https://victoriametrics.github.io/#relabeling	2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin	4dd3de9286	lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-02 14:19:14 +03:00
Aliaksandr Valialkin	8da3f773ae	lib/promscrape: add -promscrape.disableKeepAlive command-line flag for disabling http keep-alive connections when scraping targets ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-07-01 02:20:20 +03:00
Aliaksandr Valialkin	b277ba8121	lib/httpserver: add Unwrap method to ErrorWithStatusCode, so As and Is functions in standard errors package may properly unwrap the error inside ErrorWithStatusCode	2020-07-01 00:54:01 +03:00
Aliaksandr Valialkin	56ccfa5218	all: use errors.As instead of type assertion for detecting net.Error	2020-07-01 00:15:34 +03:00
Aliaksandr Valialkin	7c2c8b2981	all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode	2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin	586c5be404	lib/promscrape: add missing label sorting for autogenerated metrics ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/592	2020-06-29 22:36:12 +03:00
Ween	1cd01b5359	Fix Auto metrics relabeled errors (#593) ... * Fix Auto metrics relabeled errors * Finalize auto-genenated Labels * Fix Test Errors Co-authored-by: xinyulong <xinyulong@kuaishou.com>	2020-06-29 22:29:29 +03:00
Aliaksandr Valialkin	7eb171182b	lib/promrelabel: properly apply ^ and $ anchors to regex value in Prometheus relabeling rules	2020-06-25 17:19:19 +03:00
Aliaksandr Valialkin	a560b4788e	lib/fs: go fmt	2020-06-23 23:02:39 +03:00
Aliaksandr Valialkin	8141541e61	lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data ... This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 22:55:22 +03:00
Aliaksandr Valialkin	2a39ba639d	lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs ... These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values. For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port equals __meta_kubernetes_pod_container_port_number: - action: keep_if_equal source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]	2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin	8f0bcec6cc	lib/promscrape: preserve the previously discovered targets on discovery errors per each job_name ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582	2020-06-23 15:40:40 +03:00
Aliaksandr Valialkin	c970cb912c	lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 13:39:49 +03:00
Aliaksandr Valialkin	b5206ce33f	lib/logger: add -loggerErrorsPerSecondLimit for limiting the rate of ERROR messages	2020-06-23 12:41:36 +03:00
Aliaksandr Valialkin	4c7f216dfe	lib/promscrape: retry performing the request to the server for up to 3 times before giving up when it closes keep-alive connections ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580	2020-06-23 12:33:54 +03:00
Aliaksandr Valialkin	7532dbcdf5	app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577	2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin	e149019c00	lib/promscrape/discovery/consul: reduce load on Consul when discovering big number of targets by using background caching ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 18:20:01 +03:00
Aliaksandr Valialkin	7bf2cbad32	lib/promscrape: reduce default value for -promscrape.discovery.concurrency from 500 to 100 ... This should reduce load on Kubernetes API server and Consul when big number of targets are discovered Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574	2020-06-20 17:53:42 +03:00
Aliaksandr Valialkin	6ff821c70d	lib/promscrape/discovery/ec2: expose __meta_ec2_ami like the next Prometheus release will do ... See b5d61fb66c for details	2020-06-20 17:45:23 +03:00
Aliaksandr Valialkin	5820c0ffb7	lib/promrelabel: allows regex capture groups in target_label like Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569	2020-06-19 02:21:24 +03:00
Tristan Su	ac3700ed1e	lib/storage: set big/small merge concurrency (#568) ... fixed #567 Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>	2020-06-19 01:25:48 +03:00
Aliaksandr Valialkin	b542e50680	app/vminsert: export metrics for determining ingested rows with dropped or truncated labels ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565	2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin	08495360b0	lib/storage: add key!=".+" filter additionally to negative filter matching empty value such as key!~"|foo" ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-18 20:03:48 +03:00
Aliaksandr Valialkin	ae1cc0fc4b	lib/storage: properly match {tag!="|foo"} filters ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546	2020-06-10 19:35:56 +03:00
Aliaksandr Valialkin	69a647b0d2	lib/httpserver: do not flush and do not close gzip writer if response compression is disabled ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535	2020-06-05 21:37:28 +03:00
Aliaksandr Valialkin	4e8d6b80e0	lib/backup: properly create missing parent directories in fs.CreateFile	2020-06-05 19:28:19 +03:00
Aliaksandr Valialkin	d120197676	lib/fs: optimize queries that read recent samples for big number of time series ... Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data. This improves read performance by up to 4x.	2020-06-05 19:10:04 +03:00
Aliaksandr Valialkin	4cb3af1a36	lib/fs: add a benchmark for ReaderAt.MustReadAt	2020-06-05 19:10:03 +03:00
Aliaksandr Valialkin	4fd3f6f991	lib/backup/fsremote: create all the parent directories before creating file in CreateFile	2020-06-05 10:25:10 +03:00
Aliaksandr Valialkin	3d4008263f	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:15:47 +03:00
Vyacheslav Mitrofanov	72ff05255f	allow to use values lower than 10 with the flag -memory.allowedPercent (#531) ... Co-authored-by: Vyacheslav Mitrofanov <vmitrofanov@mfms.ru>	2020-06-03 23:39:29 +03:00
Aliaksandr Valialkin	937338abdf	lib/bytesutil: prevent from garbage collecting s before returning from ToUnsafeBytes	2020-06-03 00:23:40 +03:00
Aliaksandr Valialkin	a7797dae09	lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0 ... v1.36.0 always returns empty responses for Graphite wildcards like the following {__name__=~"foo\\.[^.]*\\.bar\\.baz"} Temporary workaround for v1.36.0 is to add `[^.]*` to the end of the regexp.	2020-05-28 12:03:49 +03:00
Aliaksandr Valialkin	d186472081	lib/storage: improve search speed for time series matching Graphite whildcards such as foo.*.bar.baz ... Add index for reverse Graphite-like metric names with dots. Use this index during search for filters like `__name__=~"foo\\.[^.]*\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case. This change may "hide" historical time series during queries. The workaround is to add `[.]*` to the end of regexp label filter, i.e. "foo\\.[^.]*\\.bar\\.baz" should be substituted with "foo\\.[^.]*\\.bar\\.baz[.]*".	2020-05-27 21:45:52 +03:00
Aliaksandr Valialkin	f5a4731412	lib/httpserver: properly set status code for empty response	2020-05-24 23:55:28 +03:00
Aliaksandr Valialkin	947009f459	lib/httpserver: fix compression for static files	2020-05-24 22:17:21 +03:00
Aliaksandr Valialkin	c602284a99	lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response	2020-05-24 14:41:14 +03:00
Aliaksandr Valialkin	b3fcd726e3	lib/httpserver: do not recompress already compressed response ... This shoud help with vmauth issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/514	2020-05-22 16:45:04 +03:00
Aliaksandr Valialkin	f9847352b4	app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin	d1a9d8aa1c	lib/promscrape: add -promscrape.config.dryRun flag for checking -promscrape.config for errors or unsupported options ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362	2020-05-21 14:55:11 +03:00
Aliaksandr Valialkin	cf87b810b7	lib/promscrape: add -promscrape.discovery.concurrency and -promscrape.discovery.concurrentWaitTime flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes	2020-05-19 17:35:53 +03:00
Aliaksandr Valialkin	b19ca3eb5f	lib/storage: do not increment vm_slow_metric_name_loads_total counter for metric_ids which shouldnt be prefetched, since this may mislead users	2020-05-16 10:21:17 +03:00
Aliaksandr Valialkin	4e850cd6a7	lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484	2020-05-16 09:31:46 +03:00
肖贝贝	a0380a0a91	fix: fix vmagent multi queue may become one because sync bug (#484) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-05-16 09:19:52 +03:00
Aliaksandr Valialkin	3412d5d138	lib/backup: remove misleading -dst mention in error message ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482	2020-05-15 17:13:37 +03:00
Aliaksandr Valialkin	27f7cca7ff	lib/backup: donload only the remaining parts for partially downloaded files after vmrestore restart ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487	2020-05-15 17:03:33 +03:00
Aliaksandr Valialkin	82ffbcb9a6	app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance	2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin	82ccdfaa91	app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series	2020-05-15 13:44:32 +03:00
Aliaksandr Valialkin	0eacea1de1	lib/{storage,mergeset}: further tuning of compression levels depending on block size ... This should improve performance for querying newly added data, since it can be unpacked faster.	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	737d641920	lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	4fc33163c4	lib/storage: optimize ingestion pefrormance for new time series	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	f9f3afb6af	lib/mergeset: tune compression levels in order to improve ingestion performance a bit	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	8b32e7c3a0	lib/storage: reduce indentation in Storage.add	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	1573ececb2	lib/storage: return the first error instead of the last error, since the first error usually points to the root cause	2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin	0afd48d2ee	lib: extract common code for returning fast unix timestamp into lib/fasttime	2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin	42866fa754	lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused	2020-05-14 15:32:07 +03:00
Aliaksandr Valialkin	827a3a7866	lib/storage: document that getnerateUniqueMetricID should return dense ids	2020-05-14 14:08:45 +03:00
Aliaksandr Valialkin	606585f7be	lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse	2020-05-14 14:03:03 +03:00
Aliaksandr Valialkin	4fe67504f9	lib/storage: optimize label matching for regexp ending with literal suffix ... For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`, so it should work faster now.	2020-05-13 11:47:07 +03:00
Aliaksandr Valialkin	96e001d254	app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set ... This bug could result in incorrect relabeling and metrics' drop. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467	2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin	a6f16dcc11	lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM	2020-05-12 20:22:09 +03:00
Aliaksandr Valialkin	0a134ace63	app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470	2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin	8300cc17af	app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set	2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin	3232605524	lib/storage: properly initialize part struct before trying to close it on error ... This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .	2020-05-12 14:54:31 +03:00
Aliaksandr Valialkin	dbd0c552d5	lib/storage: gradually pre-populate per-day inverted index for the next day ... This should prevent from CPU usage spikes at 00:00 UTC every day when inverted index for new day must be quickly created for all the active time series. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430	2020-05-12 12:13:05 +03:00
Aliaksandr Valialkin	cc00a2c453	lib/storage: typo fixes in error messages: or -> of	2020-05-12 12:12:42 +03:00
Aliaksandr Valialkin	ce2107bc52	lib/storage: speed up matching for common regexps in label filters ... The following regexps have been optimized: * 'foo.+bar' * 'foo.+bar.+baz' This should improve performance for matching Graphite-like metrics.	2020-05-11 22:40:55 +03:00
Aliaksandr Valialkin	12a1a71cc1	lib/storage: add a benchmark for Graphite-like regexps for metric names	2020-05-11 22:37:32 +03:00
Aliaksandr Valialkin	099e44005b	lib/httpserver: add -http.shutdownDelay flag for a grace period before http server shutdown ... The http server returns 503 non-OK error at `/health` page during grace period, so load balancers in front of the http server could re-route incoming requests to other servers. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/463	2020-05-07 15:30:35 +03:00
Aliaksandr Valialkin	787fcfba0c	lib/httpserver: reduce typical duration for http server graceful shutdown ... Previously the duration for graceful shutdown for http server could take more than a minute because of imporperly set timeouts in setNetworkTimeout. Now typical duration for graceful shutdown should be reduced to less than 5 seconds.	2020-05-07 14:12:39 +03:00
Aliaksandr Valialkin	91a49eecea	lib/flagutil: make errcheck happy by explicitly ignoring Array.Set result in tests	2020-05-06 22:37:39 +03:00
Aliaksandr Valialkin	c4c447507d	lib/flagutil: properly parse quoted flag values for flagutil.Array	2020-05-06 22:27:21 +03:00
Aliaksandr Valialkin	8a00807f60	app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url	2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin	68928bf3df	lib/promscrape/discovery/gce: discover per-zone instances for gce_sd_config in parallel. This should reduce discovery latency	2020-05-06 15:00:09 +03:00
Aliaksandr Valialkin	3f52a97f9b	lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs	2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin	364789c24c	lib/promscrape: properly connect to TCP6 addresses if -enableTCP6 is set	2020-05-06 00:01:57 +03:00
Aliaksandr Valialkin	7b5ef63384	lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP	2020-05-05 10:54:09 +03:00
Aliaksandr Valialkin	4fa817be10	lib/promscrape: allow explicitly setting empty token via token: "" in consul_sd_config	2020-05-05 07:50:15 +03:00
Aliaksandr Valialkin	40c3ffb359	lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330	2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin	83f0e35b7b	lib/promauth: properly set up client certificate in tls.Config ... Previously the client certificate has been mistakenly set up as a server certificate	2020-05-04 20:51:08 +03:00
Aliaksandr Valialkin	218e566647	lib/promscrape: move common code for discovery api config map handling into discoveryutils	2020-05-04 20:51:01 +03:00
Aliaksandr Valialkin	6310b20e72	lib/promscrape/discovery/kubernetes/: unify apiConfig creation	2020-05-04 20:50:49 +03:00
Aliaksandr Valialkin	66b0ae79a5	lib/promscrape: remove debug line left after the commit e4aac6ea40	2020-05-03 17:15:32 +03:00
Aliaksandr Valialkin	69004a5f67	lib/promscrape: fix tests after the commit 658a8742ac ... The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does. See https://www.robustperception.io/life-of-a-label for details. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453	2020-05-03 16:56:15 +03:00
DexterZhang	658a8742ac	fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] (#454)	2020-05-03 16:51:03 +03:00
Aliaksandr Valialkin	e4aac6ea40	lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels ... This should make consistent intervals between data points for scrape targets across reloads. Previously these intervals were random.	2020-05-03 14:30:21 +03:00
Aliaksandr Valialkin	d9f1b4d6a3	lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after 3b234d82e5	2020-05-03 14:28:31 +03:00
Aliaksandr Valialkin	3b234d82e5	lib/promscrape: reload only modified scrapers on config changes ... This should improve scrape stability when big number of targets are scraped and these targets are frequently changed. Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests: - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459 - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460	2020-05-03 12:45:40 +03:00
Aliaksandr Valialkin	ee810e5f3a	lib/httpserver: rename http.externalURL to http.pathPrefix and improve help message for this flag ... The `http.externalURL` flag name was slightly misleading, so it has been renamed to `http.pathPrefix`.	2020-05-02 13:07:34 +03:00
DexterZhang	34743974d5	feat(httpserver): add http.externalUrl config to http server, it adds prefix to http path automatically (#452)	2020-05-02 12:42:53 +03:00
Aliaksandr Valialkin	432187ac3b	app/vminsert: add /-/reload handler in the same way as for vmagent	2020-04-30 02:15:39 +03:00
Aliaksandr Valialkin	825a2dd554	lib/procutil: prevent from app termination on SIGHUP signal, since this signal is frequently used for config reload	2020-04-30 02:09:27 +03:00
Aliaksandr Valialkin	01c17092e1	lib/httpserver: mention that -http.maxGracefulShutdownDuration command-line flag value can be increased on shutdown timeout	2020-04-30 01:38:06 +03:00
Aliaksandr Valialkin	5ec036439d	lib/promscrape: set 30 seconds timeout for discovery api requests ... Previously such requests could hang for long time. This could make debugging harder.	2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin	43c39dc36c	vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp ... The upstream fasthttp may contain issues like 996610f021 , plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.	2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin	4e4f57b121	lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics	2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin	83aca79137	lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown ... Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet. Log about this and delete the corresponding metricID, so it could be re-created next time.	2020-04-28 12:00:33 +03:00
Aliaksandr Valialkin	521df0e2fc	lib/promscrape: handle connection reset when targets responds with http redirect	2020-04-28 02:13:02 +03:00
肖贝贝	2b16c188e8	fix: vmagent not follow 301/302 redirect bug (#445) ... Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>	2020-04-28 01:29:37 +03:00
Aliaksandr Valialkin	303905cd84	lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting	2020-04-28 00:01:55 +03:00
Aliaksandr Valialkin	36fa3078c2	lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize	2020-04-28 00:01:54 +03:00
Aliaksandr Valialkin	95942f1ac6	lib/promscrape/discovery/gce: make golangci-lint happy	2020-04-27 19:28:10 +03:00
Aliaksandr Valialkin	b768bc9a6a	lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs	2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin	de59703a16	lib/promscrape/discovery/gce: properly set filter query arg in api url	2020-04-27 16:01:17 +03:00
Aliaksandr Valialkin	b4afe562c1	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin	0224071ebe	lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config	2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin	6954d0edb7	lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered	2020-04-26 14:34:11 +03:00
kreedom	fb967ae6c8	happy fmt	2020-04-26 14:16:32 +03:00
Aliaksandr Valialkin	d7c1ff8b0c	lib/storage: improve deduplication algorithm ... Now it leaves only the first data point on each `-dedup.minScrapeInterval` interval. Previously it may leave two data points on the interval. This could lead to unexpected results for `histogram_quantile(phi, sum(rate(buckets)) by (le))` query.	2020-04-26 13:10:02 +03:00
Aliaksandr Valialkin	491b31b369	lib/storage: postpone label filters matching too many time series instead of giving up with error ... This should reduce the frequency of the following errors: cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range	2020-04-24 21:13:50 +03:00
Aliaksandr Valialkin	7b8008e0bd	lib/promscrape/discovery/gce: make golint happy by ignoring resp.Body.Close() result	2020-04-24 18:13:09 +03:00
Aliaksandr Valialkin	9ef5935552	lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine	2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin	24461153bf	lib/promscrape: query /api/v1/namespaces/* for the configured namespaces in kubernetes_sd_config ... This should fix authroization issues described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432	2020-04-24 14:33:50 +03:00
Aliaksandr Valialkin	00e897119f	lib/promscrape: add -promscrape.configCheckInterval command-line flag for automating config checking ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/431	2020-04-23 23:41:08 +03:00
Aliaksandr Valialkin	a9a7a7175e	lib/promscrape: access Config entries by reference, so they can be compared by addresses	2020-04-23 14:38:20 +03:00
Aliaksandr Valialkin	1c5d14a2eb	lib/promscrape: move KubernetesSDConfig to lib/promscrape/discovery/kubernetes	2020-04-23 11:34:22 +03:00
Aliaksandr Valialkin	a714568374	lib/promscrape/discovery/kubernetes: hide role switch logic behind GetLabels function	2020-04-22 22:16:11 +03:00
Aliaksandr Valialkin	364db13c9c	app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues ... See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268	2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin	2dc5593b75	lib/writeconcurrencylimiter: improve docs for -maxConcurrentInserts command-line flag	2020-04-20 21:03:00 +03:00
Aliaksandr Valialkin	5454b518a6	lib/promscrape/discovery/kubernetes: reuse a client for empty api_server inside different jobs	2020-04-20 17:07:11 +03:00
Aliaksandr Valialkin	43375df923	lib/promscrape/discovery/kubernetes: update stale comments	2020-04-17 14:06:20 +03:00
Aliaksandr Valialkin	5d1537a395	lib/promscrape: suppress scrape errors if -promscrape.suppressScrapeErrors flag is set	2020-04-16 23:41:30 +03:00
Aliaksandr Valialkin	600490131f	lib/promscrape: print all the labels for the target on error message for failed scrape ... This should improve debuggability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/420	2020-04-16 23:35:05 +03:00
Aliaksandr Valialkin	bd4c6d21dd	lib/promscrape: retry target scraping when the target closes previously established keep-alive connection to it ... This should fix the following error: the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection	2020-04-16 23:25:29 +03:00
Aliaksandr Valialkin	2fd2dec5eb	lib/logger: typo fix	2020-04-16 00:19:10 +03:00
Aliaksandr Valialkin	071fdf5518	lib/logger: add WARN level for logging expected errors such as invalid user queries	2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin	6f7f64f757	app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415	2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin	426a0567c4	lib/promscrape: code cleanup in runScraper func	2020-04-15 11:36:24 +03:00
Aliaksandr Valialkin	c1de3f67b4	lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search ... This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet. This should fix the following error: error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF	2020-04-15 00:06:43 +03:00
Aliaksandr Valialkin	067c7afebc	lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page ... This is a common error whith improperly configured target autodiscovery and/or relabeling. This error leads to duplicate scraping of the same targets with the same set of labels, which leads to duplicate samples in time series.	2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin	ac35635b71	lib/promscrape/discovery/kubernetes: remove only unused client for API server during cleaning	2020-04-14 14:19:21 +03:00
Aliaksandr Valialkin	78863d7066	lib/promscrape: add promrelabel.GetLabelValueByName helper function	2020-04-14 14:12:01 +03:00
Aliaksandr Valialkin	c64f003cfb	lib/promscrape: mention job name in error messages when target cannot be scraped ... This should improve debuggability	2020-04-14 13:33:13 +03:00
Aliaksandr Valialkin	4718a5d951	lib/promscrape: reset ScrapeWork.ID in tests	2020-04-14 13:31:31 +03:00
Aliaksandr Valialkin	257521a634	lib/promscrape: properly expose statuses for targets with duplicate scrape urls at /targets page ... Previously targets with duplicate scrape urls were merged into a single line on the page. Now each target with duplicate scrape url is displayed on a separate line.	2020-04-14 13:10:01 +03:00
Aliaksandr Valialkin	6a75c95194	lib/promscrape: remove labels starting with __meta_ after applying relabel_configs as Prometheus does ... This should reduce CPU load during scraping when target discovery generates big number of `__meta_*` labels (for instance, k8s discovery). See https://www.robustperception.io/life-of-a-label for details.	2020-04-14 12:23:22 +03:00
Aliaksandr Valialkin	01d7d799dc	lib/promscrape: rename 'scrape_config->scrape_limit' to 'scrape_config->sample_limit' ... `scrape_config` block from Prometheus config contains `sample_limit` field, while in `vmagent` this field was mistakenly named as `scrape_limit`.	2020-04-14 11:59:57 +03:00
Aliaksandr Valialkin	2e4e202c2b	lib/promscrape: add initial support for kubernetes_sd_config ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334	2020-04-13 21:03:28 +03:00
Aliaksandr Valialkin	2814b1490f	lib/promscrape: add -promscrape.config.strictParse flag for detecting errors in -promscrape.config file	2020-04-13 13:15:44 +03:00
Aliaksandr Valialkin	90b4a6dd12	lib/promscrape: extract common auth code to lib/promauth	2020-04-13 12:59:10 +03:00
Aliaksandr Valialkin	4de6c6bbf0	lib/storage: disable deduplication after dedup tests are complete ... The rest of tests expect that the de-duplication is disabled.	2020-04-10 17:28:31 +03:00
Aliaksandr Valialkin	ded0c0d3c7	lib/storage: correctly handle -dedup.minScrapeInterval values smaller than 8ms ... Such small values may be used for removing samples with duplicate timestamps. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.	2020-04-10 16:36:41 +03:00
Aliaksandr Valialkin	7d73623c69	lib/{storage,mergeset}: make sure that requests and misses cache counters never go down	2020-04-10 14:45:01 +03:00
Aliaksandr Valialkin	e62afc7366	lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data ... These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols	2020-04-10 12:44:39 +03:00
Aliaksandr Valialkin	0681b4c27a	lib/workingsetcache: accumulate stat counters on cache rotation ... This should prevent from cache stats counters going down after cache rotation, which may corrupt `cache hit ratio` graph on the official Grafan dasbhoards when using the following query: 1 - (sum(rate(vm_cache_misses_total[5m])) by (type) / sum(rate(vm_cache_requests_total[5m])) by (type))	2020-04-10 11:51:40 +03:00
Aliaksandr Valialkin	f86947d55c	lib/memory: add more details to -memory.allowedPercent help message	2020-04-09 15:28:53 +03:00
kreedom	298eb0a0f8	[vmalert] improve external url handling	2020-04-01 22:29:11 +03:00
Aliaksandr Valialkin	cdf0a4cf8f	lib/httpserver: remove unnecessary http.HandlerFunc wrapper in gzipHandler	2020-04-01 18:14:17 +03:00
Aliaksandr Valialkin	e0d0348f36	lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init	2020-04-01 17:42:44 +03:00
Aliaksandr Valialkin	3e55c7e069	lib/promscrape: reduce timestamp jitter when scraping targets ... This should improve compression for timestamps	2020-04-01 16:11:35 +03:00
Aliaksandr Valialkin	c4acd20d2a	lib/storage: remove duplicate data points on 7/8*minScrapeInterval interval instead of 1/2*minScrapeInterval ... This should reduce storage usage and should improve deduplication accuracy	2020-04-01 15:48:48 +03:00
Aliaksandr Valialkin	b699c46046	lib/storage: handle errors returned from TagFilters.Add when cloning TagFilters with negative filter	2020-03-31 16:18:02 +03:00
Aliaksandr Valialkin	972713bd79	lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet	2020-03-31 12:51:21 +03:00
Aliaksandr Valialkin	5d99ca6cfc	lib/storage: optimize per-day inverted index search for tag filters matching big number of time series ... - Sort tag filters in the ascending number of matching time series in order to apply the most specific filters first. - Fall back to metricName search for filters matching big number of time series (usually this are negative filters or regexp filters).	2020-03-31 00:48:35 +03:00
Aliaksandr Valialkin	318326c309	lib/storage: properly handle {label=~"foo|"} filters as Prometheus does ... Such filters must match all the time series with `label="foo"` plus all the time series without `label` Previously only time series with `label="foo"` were matched. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395	2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin	b98ca56d94	lib/envflag: add -envflag.prefix for setting optional prefix for environment vars	2020-03-30 15:51:19 +03:00
kreedom	bf6c24d0f4	[vmalert] config parser (#393) ... * [vmalert] config parser * make linter be happy * fix test * fix sprintf add test for rule validation	2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin	149f365f74	lib/httpserver: add -http.maxGracefulShutdownDuration command-line flag for tuning the maximum duration required for graceful shutdown of http server	2020-03-27 21:23:30 +02:00
Aliaksandr Valialkin	047849e855	lib/uint64set: remove zero buckets after Set.Intersect	2020-03-27 01:15:58 +02:00
Aliaksandr Valialkin	f3ec424e7d	lib/uint64set: small code cleanup and perf tuning ... * Remember the last accessed bucket on Has() call. * Inline fast paths inside Add() and Has() calls. * Remove fragile code with maxUnsortedBuckets inside bucket32.	2020-03-25 15:30:25 +02:00
Aliaksandr Valialkin	dde4a97534	lib/uint64set: go fmt	2020-03-24 22:28:43 +02:00
Aliaksandr Valialkin	f3e0c55ea1	lib/storage: serialize snapshot creation process with mutex ... This guarantees that the snapshot contains all the recently added data from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.	2020-03-24 22:27:05 +02:00
Aliaksandr Valialkin	97fb0edd07	lib/uint64set: added more tests	2020-03-24 22:27:04 +02:00
Aliaksandr Valialkin	df91d2d91f	lib/storage: remove obsolete code	2020-03-13 22:48:17 +02:00
Aliaksandr Valialkin	499594f421	lib/promscrape: allow overriding external_labels as Prometheus does ... Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say: > In communication with external systems, they are always applied only > when a time series does not have a given label yet and are ignored otherwise. Though this may result in consistency chaos when scrape targets override `external_labels`, let's stick with Prometheus behavior for the sake of backwards compatibility. There is last resort in vmagent with `-remoteWrite.label`, which consistently sets the configured labels to all the metrics before sending them to remote storage. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366	2020-03-12 20:24:42 +02:00
Aliaksandr Valialkin	fdc2a9d1d7	app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369	2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin	c8dc1cd218	lib/protoparser/csvimport: add missing metric vm_rows_invalid_total{type="csvimport"}	2020-03-12 15:27:45 +02:00
Aliaksandr Valialkin	d4beb17ebe	lib/promscrape: remove possible races when registering and de-registering scrape workers for /targets page	2020-03-11 16:30:21 +02:00
Aliaksandr Valialkin	cdf70b7944	lib/promscrape: consistently update /targets page after SIGHUP	2020-03-11 03:20:03 +02:00
Aliaksandr Valialkin	1fe66fb3cc	app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv	2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin	49d7cb1a3f	all: fix golangci-lint issues	2020-03-10 19:41:46 +02:00
Aliaksandr Valialkin	7c432da788	lib/promscrape: do not retry idempotent requests when scraping targets ... This should prevent from the following unexpected side-effects of idempotent request retries: - increased actual timeout when scraping the target comparing to the configured scrape_timeout - increased load on the target Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357	2020-03-09 13:31:52 +02:00
Aliaksandr Valialkin	986dba5ab3	app/vmagent: do not allow non-supported fields in -remoteWrite.relabelConfig and file_sd_configs ... This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363	2020-03-06 20:19:13 +02:00
Aliaksandr Valialkin	c5f894b361	Makefile: add build and test rules with enabled race detector. These rules have -race suffix ... Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .	2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin	9a944fd169	lib/promscrape: consistency renaming: stopCh -> globalStopCh	2020-03-03 20:08:08 +02:00
Aliaksandr Valialkin	76036c1897	app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352	2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin	1d7ab78b55	lib/protoparser/prometheus: allow trailing comma in tags list ... The trailing comma is generated by cloudwatch exporter. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/350	2020-03-02 22:22:09 +02:00
Aliaksandr Valialkin	b785429ddb	lib/protoparser: metrics renaming: vm_protoparser_<type>_* -> vm_protoparser_*{type="<type>"} ... This should improve composability of these metrics in PromQL queries	2020-02-28 20:20:10 +02:00
Aliaksandr Valialkin	e22fdc1073	lib/persistentqueue: reset chunk file when the persistent queue is empty	2020-02-28 20:05:53 +02:00
Aliaksandr Valialkin	18af31a4c2	all: properly split vm_deduplicated_samples_total among cluster components ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345	2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin	6819db5686	lib/envflag: typo fix in docs to -envflag.enable: envoronment->environment	2020-02-27 21:47:58 +02:00
Aliaksandr Valialkin	b63e4464f4	lib/promscrape: properly reload new configs on SIGHUP ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/335	2020-02-26 13:54:00 +02:00
Aliaksandr Valialkin	6739c2749d	lib/promscrape: go fmt	2020-02-25 20:56:44 +02:00
Aliaksandr Valialkin	7a33da8fea	lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331	2020-02-25 20:49:50 +02:00
Aliaksandr Valialkin	4e24839a2c	app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize	2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin	6386aeb1e0	app/vmagent: add ability to accept Influx line protocol data via TCP and UDP ... Just set `-influxListenAddr` command-line flag Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333	2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin	7ef7c9368e	lib/fs: typo fix: read blocks bigger than 8KB via pread() call instead of using mmap	2020-02-25 18:05:06 +02:00
Aliaksandr Valialkin	fed2959658	lib/envflag: substitute dots with underscores in env var names if -envflag.enable is set ... See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-24 21:14:44 +02:00
Aliaksandr Valialkin	04762344c6	app/vmagent: initial implementation for vmagent	2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin	d21cb43e48	lib/storage: add vm_ prefix to deduplicated_samples_total metric to be conistent with other metrics	2020-02-21 19:33:59 +02:00
Aliaksandr Valialkin	71a52f5f90	lib/protoparser/prometheus: skip leading whitespace from tag names	2020-02-16 19:06:33 +02:00
Aliaksandr Valialkin	ce15cecae4	lib/storage: typo fix	2020-02-16 15:53:44 +02:00
Aliaksandr Valialkin	32e153e834	lib/storage: prevent from clobbering nin-nil lastError in Storage.add	2020-02-16 15:51:26 +02:00
Aliaksandr Valialkin	7b1c7051a3	app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions ... This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.	2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin	7836ad8907	lib/mergeset: skip createing temporary part objects when merging source inmemory parts ... This should reduce CPU usage when adding new entries to inverted index. This should alos prevent from creating stalled cleaner goroutines for the created temporary parts, since they were never closed. This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 14:09:32 +02:00
Aliaksandr Valialkin	eceaf13e5e	lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate ... It has been appeared that time.Timer was used in places where time.Ticker must be used instead. This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .	2020-02-13 13:10:07 +02:00
Aliaksandr Valialkin	4cc0163c7c	docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent	2020-02-10 23:02:43 +02:00
Aliaksandr Valialkin	025297f15d	lib/envflag: check for incorrect flag values read from environment vars	2020-02-10 16:08:10 +02:00
Aliaksandr Valialkin	5d207b2025	lib/envflag: add -envflag.enable command-line flag for enabling reading flags from environment vars ... By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 16:02:37 +02:00
Aliaksandr Valialkin	8466ab0034	all: allow setting flags via environment vars ... Now flags can be set via environment vars with the same names as flags. Command-line flags override flags set via env vars. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311	2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin	e210cd9da1	lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version	2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin	40c6ae2952	lib/logger: initialize output to os.Stderr by default	2020-02-04 22:40:44 +02:00
Aliaksandr Valialkin	e0a4c37fc1	lib/logger: add -loggerOutput command-line flag ... This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306	2020-02-04 21:47:56 +02:00
Aliaksandr Valialkin	7f3e3a6034	lib/logger: do not clutter -loggerFormat=json output with stack trace ... This should improve json parsing	2020-02-04 21:37:25 +02:00
Aliaksandr Valialkin	bd4698bb7a	lib/storage: do not deduplicate blocks with less than 32 samples during merge ... This should improve deduplication accuracy for blocks with higher number of samples.	2020-02-04 18:41:54 +02:00
Aliaksandr Valialkin	42864bb52f	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin	011a79da85	lib/fs: remove unused readerAt interface	2020-01-31 15:12:43 +02:00
Aliaksandr Valialkin	c3d86eef96	all: add -dedup.minScrapeInterval command-line flag for data de-duplication ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278	2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin	2152f6f0cd	lib/storage: re-use indexSearch inside Storage.prefetchMetricNames	2020-01-31 01:16:53 +02:00
Aliaksandr Valialkin	d70ba7eb37	lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor ... This should improve performance when reading many small blocks.	2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin	ad8af629bb	all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt	2020-01-30 15:08:58 +02:00
Aliaksandr Valialkin	d68546aa4a	lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init ... This should speed up Search.NextMetricBlock loop for big number of found time series.	2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin	5bb9ccb6bf	lib/mergeset: properly update lastAccesstime in indexBlockCache entries ... This is a follow-up for 6665f10e7b	2020-01-29 21:20:47 +02:00
Aliaksandr Valialkin	a462355b2f	app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value	2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin	371e86194d	app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps	2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin	6caa9bb51b	lib/logger: fix improperly set skipframes for all the logging functions ... The bug has been introduced in the previous commit f6baee6efe	2020-01-26 18:34:27 +02:00
Aliaksandr Valialkin	f6baee6efe	lib/httpserver: log the caller of httpserver.Errorf ... Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.	2020-01-25 20:17:59 +02:00
Aliaksandr Valialkin	9df5b2d1c3	app/victoria-metrics: add -selfScrapeInterval flag for self-scraping /metrics page ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180	2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin	2a0a0ed14d	lib/protoparser: add parser for Prometheus exposition text format ... This parser will be used by vmagent	2020-01-24 20:11:02 +02:00
Aliaksandr Valialkin	6456c93dbb	app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent	2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin	680080887d	all: consistently log durations in seconds with millisecond precision ... This should improve logs readability	2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin	582dd01f42	app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d	2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin	36973ee975	app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp	2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin	6665f10e7b	lib/{mergeset,storage}: properly update lastAccessTime in index and data block cache entries	2020-01-20 14:59:47 +02:00
Aliaksandr Valialkin	257e61195a	lib/uint64set: add missing bucket32.b16his values	2020-01-18 14:26:04 +02:00
Aliaksandr Valialkin	4cc0c44b9e	lib/uint64set: optimize Set.Union ... This should improve performance for queries over big number of time series	2020-01-18 13:47:03 +02:00
Aliaksandr Valialkin	1b5f02e293	lib/uint64set: add benchmarks for Set.Union	2020-01-18 13:47:02 +02:00
Aliaksandr Valialkin	3748fb24b6	lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64)	2020-01-18 00:09:33 +02:00
Aliaksandr Valialkin	c9472e4f3a	all: use github.com/klauspost/compress/gzip instead of compress/gzip ... `github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`. This gives better gzip compression and decompression speeds.	2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin	bc0f897fcb	lib/uint64set: reduce memory allocations in Set.AppendTo	2020-01-17 22:33:09 +02:00
Aliaksandr Valialkin	f9289b804a	lib/storage: reduce memory allocations when merging metricID sets	2020-01-17 22:10:44 +02:00
Aliaksandr Valialkin	0c8ad08578	lib/uint64set: typo fix in Set.Intersect	2020-01-17 18:10:58 +02:00
Aliaksandr Valialkin	9f027ec176	lib/uint64set: optimize Intersect, Subtract and Union functions ... This should improve performance for queries over big number of time series.	2020-01-17 16:11:49 +02:00
Aliaksandr Valialkin	cd53f7d177	lib/uint64set: improve benchmark for Set.Intersect	2020-01-17 16:08:17 +02:00
Aliaksandr Valialkin	8dbf430469	lib/uint64set: add benchmark for Set.Intersect	2020-01-17 00:31:07 +02:00
Aliaksandr Valialkin	0d7505b00e	all: mention command-line flags used for limiting the incoming request size in error messages ... This should improve error logs usability. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287	2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin	605d588ba6	lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods ... Iterate items with newly added Set.ForEach method instead of allocating `[]uint64` slice for all the items before the iteration.	2020-01-15 12:12:49 +02:00
Aliaksandr Valialkin	893b62c682	lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)	2020-01-14 22:47:04 +02:00
Aliaksandr Valialkin	7830c10eb2	lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches ... This should reduce memory usage in the long run when old blocks and indexes aren't accessed anymore.	2020-01-14 21:38:44 +02:00
Aliaksandr Valialkin	8b14572f70	app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions ... These functions can be used for calculating Hoeffding bounds for `m` over `d` time range and for the given `phi` in the range `[0..1]`. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283	2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin	164278151f	app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function ... This function can be used for simultaneous calculating of multiple `aggr_func*` functions that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])` would calculate `min_over_time` and `max_over_time` for `m[d]`.	2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin	c4632faa9d	app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions ... These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`	2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin	b8038a14e7	lib/backup/s3remote: check whether the file exists before deleting it ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284	2020-01-09 23:20:31 +02:00
Aliaksandr Valialkin	f358fb72d1	app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup ... This should prevent from restoring from incomplete backups. Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.	2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin	fc71602039	lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition ... This should reduce write amplification for high ingestion rate on multi-CPU systems	2020-01-04 23:57:31 +02:00
Aliaksandr Valialkin	d9c4ac9978	lib/metricsql: export IsRollupFunc and IsTransformFunc, since they can be used by package users	2020-01-04 13:25:05 +02:00
Aliaksandr Valialkin	f409f2d050	app/vmselect/promql: add histogram_share(le, buckets) function	2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin	b1ded7cf9a	app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 ... See https://github.com/prometheus/prometheus/issues/2882	2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin	a8360d04c0	app/vmselect/promql: add histogram_over_time(m[d]) rollup function	2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin	695682232f	lib/uint64set: reduce memory usage when storing big number of sparse metric_id values	2020-01-03 18:16:44 +02:00
Aliaksandr Valialkin	cb3a342882	app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations	2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin	0038365206	docs: refer to standalone MetricsQL package	2020-01-02 23:43:35 +02:00
Aliaksandr Valialkin	a21d786d3c	lib/metricsql: add example for ExpandWithExprs	2019-12-26 21:32:11 +02:00
Aliaksandr Valialkin	6f67e0b56b	lib/metricsq: add ExpandWithExprs	2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin	1925ee038d	Rename lib/promql to lib/metricsql and apply small fixes	2019-12-25 22:03:59 +02:00
Mike Poindexter	bec62e4e43	Split Extended PromQL parsing to a separate library	2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin	c18802af59	lib/fs: typo fix in fadvise_unix.go	2019-12-24 20:59:28 +02:00
Aliaksandr Valialkin	4ba4abe666	lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled ... This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:48:31 +02:00
Aliaksandr Valialkin	5bb39e757b	lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* ... This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:41:52 +02:00
Aliaksandr Valialkin	d5c9841220	lib/encoding: mention unpacked block size in the error message if unparsed tail left ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281	2019-12-24 20:35:13 +02:00
Aliaksandr Valialkin	31d04fb5df	Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner" ... This reverts commit e3c462f08a. Reason to revert: this leaves incomplete logs on app shutdown.	2019-12-24 12:21:39 +02:00
Aliaksandr Valialkin	5b75984aa9	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin	6a1499efa3	lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel ... Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.	2019-12-23 18:05:41 +02:00
Aliaksandr Valialkin	e3c462f08a	lib/logger: prevent from blocking when log output isn't consumed in timely manner ... Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.	2019-12-20 11:49:34 +02:00
Aliaksandr Valialkin	1825893eef	lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores	2019-12-19 18:18:29 +02:00
Aliaksandr Valialkin	97f70ccda7	lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series ... This should speed up `/api/v1/import` and make it more scalable on multi-core systems.	2019-12-19 18:18:29 +02:00
Aliaksandr Valialkin	bb530a0591	lib/httpserver: inline checkAuth code to make it more clear	2019-12-18 23:06:25 +02:00
koalaty-code	aea4c80dd7	Ignore /health endpoint when checking auth	2019-12-18 23:04:31 +02:00
Aliaksandr Valialkin	bc645152cb	app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266	2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin	f5ac9b0721	lib/logger: add -loggerFormat for choosing log message formats ... Supported formats: default, json Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265	2019-12-13 15:10:05 +02:00
Aliaksandr Valialkin	3c076544bf	app/vmselect/promql: allow negative offsets ... Updates https://github.com/prometheus/prometheus/issues/6282	2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin	0ed9258545	lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup ... This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.	2019-12-09 15:44:53 +02:00
Aliaksandr Valialkin	72345eb5bd	lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions call. ... `runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction deletions are finished before returning from `runTransactions`.	2019-12-04 21:40:30 +02:00
Aliaksandr Valialkin	1244ad810d	lib/httpserver: add /ping handler for compatibility with Influx agents ... Certain Influx agents check for `/ping` endpoint before starting to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint	2019-12-04 19:15:52 +02:00
Aliaksandr Valialkin	a247236f61	lib/storage: fall back to global inverted index if a filter match too many time series in per-day index ... Previously this resulted to error message. The query may succeed via search in global index.	2019-12-03 14:48:31 +02:00
Aliaksandr Valialkin	54741ee578	lib/storage: fix printing tag filters in TagFilters.String	2019-12-03 14:25:13 +02:00
Aliaksandr Valialkin	efbc83a13e	lib/storage: print __name__ instead of empty string in user-visible tag filters	2019-12-03 14:18:28 +02:00
Aliaksandr Valialkin	f52874dab4	lib/storage: optimize regexp filter search	2019-12-03 00:43:12 +02:00
Aliaksandr Valialkin	638a5cbb16	lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed ... This should fix the issue on NFS when incompletely removed dirs may be left after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction files are already removed. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-12-02 21:36:31 +02:00
Aliaksandr Valialkin	20812008a7	lib/storage: remove metricID with missing metricID->metricName entry ... The metricID->metricName entry can be missing in the indexdb after unclean shutdown when only a part of entries for new time series is written into indexdb. Recover from such a situation by removing the broken metricID. New metricID will be automatically created for time series with the given metricName when new data point will arive to it.	2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin	62a915f2b2	lib/storage: protect from time drift during indexdb rotation ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248	2019-12-02 14:44:42 +02:00
Aliaksandr Valialkin	42da569bcd	lib/logger: merge file and line labels into location="file:line" ... This should improve the usability for `vm_log_messages_total` metric during practical queries	2019-12-02 14:44:40 +02:00
Aliaksandr Valialkin	70b8191fab	lib/storage: generate more human-friendly result in TagFilters.String	2019-12-02 13:52:22 +02:00
Aliaksandr Valialkin	542b9c2043	lib/logger: consistency renaming from vm_log_messages_count to vm_log_messages_total, since this is a counter	2019-12-02 00:49:00 +02:00
Aliaksandr Valialkin	c567919f80	lib/logger: track the number of log messages by (level, file, line) in the vm_log_messages_count metric	2019-12-01 18:37:49 +02:00
Aliaksandr Valialkin	761645b20a	lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6 is set ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244	2019-12-01 02:57:13 +02:00
Aliaksandr Valialkin	f733cb2186	lib/backup: cosmetic fixes after #243	2019-11-29 18:07:04 +02:00
glebsam	15b7406f7b	Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243) ... * Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile * make fmt	2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin	9010c6a1d6	lib/netutil: add -enableTCP6 command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports	2019-11-29 17:32:47 +02:00
Aliaksandr Valialkin	a7125a5b7b	lib/backup: remove flock.lock file in empty dirs ... This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations: 1. Stop VictoriaMetrics. 2. Delete `<-storageDataPath>` dir. 3. Start VictoriaMetrics, then stop it. 4. Restore data from backup with `vmrestore`. 5. Start VictoriaMetrics. `vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.	2019-11-28 13:38:58 +02:00
Aliaksandr Valialkin	14bcff6015	lib/httpserver: improve docs for -tls* flags to be more clear ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/242	2019-11-26 18:08:35 +02:00
Aliaksandr Valialkin	da98703748	app/vmselect/promql: optimize binary search over big number of samples during rollup calculations	2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin	5285acae3e	lib/decimal: calculate ln2/ln10 constant during compile time	2019-11-23 15:52:58 +02:00
Aliaksandr Valialkin	9d1fcb2be6	vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms	2019-11-23 00:20:27 +02:00
Aliaksandr Valialkin	7a4635f853	all: remove the remaining mentions of cluster version	2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin	3e9b7addb1	lib/httpserver: typo fix in -httpAuth.password command-line description	2019-11-21 21:54:26 +02:00
Aliaksandr Valialkin	f652c0f40f	lib/storage: move non-matching tag filters to the top at matchTagFilters ... This should reduce the amount of useless work needed for matching the next metricNames.	2019-11-21 21:35:13 +02:00
Aliaksandr Valialkin	b8cde6cce1	lib/storage: speed up time series search for queries with multiple filters ... Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.	2019-11-21 18:43:17 +02:00
Aliaksandr Valialkin	5c1e4143e9	lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers	2019-11-20 15:39:28 +02:00
Aliaksandr Valialkin	52d7ca6bf0	lib/decimal: increase decimal->float speed conversion for integer numbers	2019-11-20 13:04:34 +02:00
Aliaksandr Valialkin	75eeea21ee	lib/decimal: reduce rounding error when converting from decimal to float with negative exponent ... While at it, slightly increase the conversion performance by moving fast path to the top of the loop.	2019-11-19 23:35:33 +02:00
Aliaksandr Valialkin	cfb9fa2100	lib/backup: retrieve only the required metadata when reading GCS objects	2019-11-19 21:06:34 +02:00
Aliaksandr Valialkin	443189fb0a	app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore	2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin	2db06f0ef8	lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore	2019-11-19 18:36:23 +02:00
Aliaksandr Valialkin	b6f22a62cb	lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues ... The previous commit was accidentally creating 10x smaller number of time series than Prometheus and this led to invalid benchmark results. The updated benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 6194893 -97.73% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 10781372 -92.19% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 10632834 -92.11% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 10679975 -94.55% BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 100118510 -98.74% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 154955671 -97.96% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 258003769 -77.42% BenchmarkHeadPostingForMatchers/i!="" 9964150263 159783895 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 10937895 -94.96% BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 10990027 -94.57% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 87004349 -82.11% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 53342793 -84.79% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 54256156 -85.76% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 21823279 -75.62% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 46671359 -87.70% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 53915842 -87.30% VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)	2019-11-18 19:50:58 +02:00
Aliaksandr Valialkin	8a0dfc6220	lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus ... See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52) The benchmark allows performing apples-to-apples comparison of time series search in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness - contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix this. Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots: - Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers - VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers Benchmark results: benchmark old ns/op new ns/op delta BenchmarkHeadPostingForMatchers/n="1" 272756688 364977 -99.87% BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 1181636 -99.14% BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 1141578 -99.15% BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 1148056 -99.41% BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 8716755 -99.89% BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 12096587 -99.84% BenchmarkHeadPostingForMatchers/i=~"" 1142371741 16164560 -98.59% BenchmarkHeadPostingForMatchers/i!="" 9964150263 12230021 -99.88% BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 1173476 -99.46% BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 1299743 -99.36% BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 11555193 -97.62% BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 5607506 -98.40% BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 6380335 -98.32% BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 2078970 -97.68% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 6561368 -98.27% BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 6757132 -98.41% The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics. As you can see, VictoriaMetrics outperforms Prometheus by more than 100x in almost all the test cases of this benchmark. Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.	2019-11-18 18:45:06 +02:00
Aliaksandr Valialkin	2ab4cea5e5	lib/storage: always start using per-day inverted index on the next day after its creation ... The current day could miss entries for already stopped time series before enabling per-day index. This fixes the issue when queries return empty results during the first hour after upgrading to v1.29.*	2019-11-16 12:11:25 +02:00
Aliaksandr Valialkin	119dfd01bb	lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"} metric	2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin	86a1cd700b	lib/storage: remove inmemory index for recent hour, since it uses too much memory ... Production workload shows that the index requires ~4Kb of RAM per active time series. This is too much for high number of active time series, so let's delete this index. Now the queries should fall back to the index for the current day instead of the index for the recent hour. The query performance for the current day index should be good enough given the 100M rows/sec scan speed per CPU core.	2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin	33895d4a0f	lib/storage: add missing increment for recentHourInvertedIndexSearchCalls	2019-11-13 15:13:51 +02:00
Aliaksandr Valialkin	c57eb0ff83	lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour ... This may be useful for saving RAM on high number of time series aka high cardinality	2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin	e14ab14e54	lib/storage: verify marshaling for iidx.pendingMetricIDs in TestInmemoryInvertedIndexMarshalUnmarshal	2019-11-13 13:35:30 +02:00
Aliaksandr Valialkin	ca259864e2	lib/storage: return back inmemory inverted index for recent hour ... Issues fixed: - Slow startup times. Now the index is loaded from cache during start. - High memory usage related to superflouos index copies every 10 seconds.	2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin	01bb3c06c7	lib/storage: remove inmemory inverted index for recent hours ... Production load with >10M active time series showed it could slow down VictoriaMetrics startup times and could eat all the memory leading to OOM. Remove inmemory inverted index for recent hours until thorough testing on production data shows it works OK.	2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin	6c2303764e	Revert "lib/fs: do not postpone directory removal on NFS error" ... This reverts commit 4c02e496f7. Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234	2019-11-12 16:18:09 +02:00
Mike Poindexter	f3ad330635	Add test for invalid caching of tsids (#232) ... * Add test for invalid caching of tsids * Clean up error handling	2019-11-12 15:09:33 +02:00
Oleg Kovalov	b4f44befa3	fix misspelled words (#229)	2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin	8e8f98f712	lib/storage: add tests for dateMetricIDCache	2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin	c342f5e37e	lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has	2019-11-10 22:04:01 +02:00
Aliaksandr Valialkin	4c02e496f7	lib/fs: do not postpone directory removal on NFS error ... Continue trying to remove NFS directory on temporary errors for up to a minute. The previous async removal process breaks in the following case during VictoriaMetrics start - VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them. - The transaction instructs removing old directories for parts, which were already merged into bigger part. - VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors. - VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished and finds directories, which should be removed, but weren't still removed due to NFS errors. - VictoriaMetrics panics when it finds unexpected empty directory. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin	3956003dd0	lib/storage: reorganize the code in getStartDateForPerDayInvertedIndex according to golangci-lint	2019-11-10 00:38:59 +02:00
Aliaksandr Valialkin	ee7765b10d	lib/storage: implement per-day inverted index	2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin	5810ba57c2	lib/storage: use specialized cache for (date, metricID) entries ... This improves ingestion performance.	2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin	e573ef2126	lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero	2019-11-09 19:03:34 +02:00
Aliaksandr Valialkin	823fa085ef	lib/storage: properly set time range when deleting time series	2019-11-09 18:49:49 +02:00
Aliaksandr Valialkin	695c1dc5eb	lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster	2019-11-09 18:04:33 +02:00
Aliaksandr Valialkin	cdbe848102	lib/storage: small code prettifying	2019-11-09 14:19:52 +02:00
Aliaksandr Valialkin	5c25070556	lib/uint64set: remove superflouos check for item existence before deleting it in Set.Subtract	2019-11-09 14:19:47 +02:00
Aliaksandr Valialkin	bb08bab263	lib/storage: inmemoryInvertedIndex prettifying	2019-11-09 14:19:41 +02:00
Aliaksandr Valialkin	6ad7fe8eeb	lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	9ea549ed24	lib/storage: sync with cluster changes	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	d888b21657	lib/storage: add inmemory inverted index for the last hour ... It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	1e46961d68	app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38	2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin	543dc8d337	lib/storage: populate partition names from both small and big directories ... Certain partition directories may be missing after restoring from backups if they had no data. Re-create such directories on start.	2019-11-06 19:49:34 +02:00
Aliaksandr Valialkin	e472f0b23b	lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter ... The origin of the error has been detected and documented in the code, so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`, so it could be monitored and alerted on high error rates. Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`, so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.	2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin	c51ca04a43	lib/storage: take into account the requested time range when caching TSIDs for the given tag filters	2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin	e37f06dc52	lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting	2019-11-05 18:44:22 +02:00
Aliaksandr Valialkin	5c2099ecfe	lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions	2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin	885ba17905	lib/storage: separate the max inverted index scan loops per metric into fast and slow loops ... Slow loops could require seeks and expensive regexp matching, while fast loops just scans all the metricIDs for the given `tag=value` prefix. So these operations must have separate max loops multiplier.	2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin	b9a06e8e74	lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive ... This should improve performance for query filters over big number of time series.	2019-11-05 14:19:13 +02:00
Aliaksandr Valialkin	30c8301b11	lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name ... The new value reduces the amount of wasted work during index scans over big number of time series.	2019-11-05 14:19:06 +02:00
Aliaksandr Valialkin	e53f9e553d	lib/storage: try potentially faster tag filters at first, then apply slower tag filters ... The fastest tag filters are non-negative non-regexp, since they are the most specific. The slowest tag filters are negative regexp, since they require scanning all the entries for the given label.	2019-11-05 14:19:01 +02:00
Aliaksandr Valialkin	3c90d77858	lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method ... This fixes lint errors	2019-11-04 01:07:19 +02:00
Artem Navoiev	478767d0ed	add unittests for bytesutil and storage (#221)	2019-11-04 00:54:46 +02:00
Aliaksandr Valialkin	02e0b19a62	lib/storage: tune the returned value from adjustMaxMetricsAdaptive	2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin	6be4456d88	lib/{storage,uint64set}: add Set.Union() function and use it	2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin	9becc26f4b	lib/storage: remove interface conversion in hot path during block merging ... This should improve merge speed a bit for parts with big number of small blocks.	2019-11-03 12:33:34 +02:00
Aliaksandr Valialkin	c62399eb3e	lib/{storage,mergeset}: create missing partition directories after restoring from backups ... Backup tools could skip empty directories. So re-create such directories on the first run.	2019-11-02 02:27:11 +02:00
Aliaksandr Valialkin	55d728c849	lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones ... Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.	2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin	808fc0971f	lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones ... Time series with such values are quite common in monitoring space, so it would be great to have benchmarks for them.	2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin	370cfbb365	lib/uint64set: return an emptry set instead of nil set from Set.Clone, since the caller may add data to the cloned set ... This fixes the following panic in v1.28.1: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e] goroutine 1155 [running]: github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec) github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000) github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260 github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)	2019-11-01 16:12:44 +02:00
Aliaksandr Valialkin	d18ea0c95b	app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges	2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin	e0b292c6de	lib/storage: small cleanup in Storage.add	2019-10-31 14:30:34 +02:00
Aliaksandr Valialkin	e76e21e4c7	lib/decimal: speed up FromFloat for common case with integers	2019-10-31 13:24:59 +02:00
Aliaksandr Valialkin	cfa5e279c2	lib/decimal: increase float64->decimal conversion precision a bit ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin	26d570bb3a	lib/storage: get parts to merge after applying the limit on the number of concurrent merges ... This should reduce write amplification under high ingestion rate.	2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin	2e2eff90d5	lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge	2019-10-29 12:45:28 +02:00
Aliaksandr Valialkin	04e48ef064	lib/fs: typo fix in comment to WriteFileAtomically	2019-10-29 11:31:26 +02:00
Aliaksandr Valialkin	a42b5db39f	lib/decimal: increase float->decimal conversion precision for big numbers ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213	2019-10-28 13:23:44 +02:00
hanzai	b3c946e35a	warns during rows addition (#214)	2019-10-20 23:41:07 +03:00
Aliaksandr Valialkin	e83fe938c8	all: make fmt	2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin	97ce4e03a5	all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin	6ebf537153	lib/memory: properly handle int overflow in sysTotalMemory ... This should fix builds on 32-bit architectures such as arm. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212	2019-10-17 00:50:48 +03:00
Aliaksandr Valialkin	195cefd81a	lib/prompb: removed outdated README.md	2019-10-14 22:12:57 +03:00
Aliaksandr Valialkin	f6334bffa1	lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order	2019-10-09 12:13:17 +03:00
Aliaksandr Valialkin	2abd5154e0	lib/storage: typo fix in comment to maxRowsPerSmallPart.	2019-10-08 18:51:20 +03:00
Aliaksandr Valialkin	c1cf7d9f93	lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. ... This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.	2019-10-08 16:27:35 +03:00
Aliaksandr Valialkin	fdc5cfd838	lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195	2019-09-30 12:25:40 +03:00
Aliaksandr Valialkin	0fa6df94a2	lib/storage: optimize TSID comparison	2019-09-26 14:16:02 +03:00
Aliaksandr Valialkin	c39355921e	lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows ... This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.	2019-09-26 13:13:41 +03:00
Aliaksandr Valialkin	0db9fcedd5	lib/storage: properly match labels against regexp with (?i) flag ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-09-26 11:03:10 +03:00
Aliaksandr Valialkin	2444433d83	lib/storage: add missing break in removeDuplicateMetricIDs	2019-09-25 18:23:43 +03:00
Aliaksandr Valialkin	ea4c828bae	lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index	2019-09-25 17:55:13 +03:00
Aliaksandr Valialkin	aebc45ad26	lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items ... This should reduce the amounts of used RAM during queries with filters over big number of time series.	2019-09-25 14:02:15 +03:00
Aliaksandr Valialkin	2cb811b42f	lib/uint64set: optimize Set.AppendTo	2019-09-25 00:34:17 +03:00
Aliaksandr Valialkin	b986516fbe	lib/storage: create and use lib/uint64set instead of map[uint64]struct{} ... This should improve inverted index search performance for filters matching big number of time series, since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls. See the corresponding benchmarks in `lib/uint64set`.	2019-09-24 21:17:55 +03:00
Aliaksandr Valialkin	ef2296e420	lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows	2019-09-24 19:32:34 +03:00
Aliaksandr Valialkin	a6086cde78	lib/storage: limit the number of metricIDs in tag->metricIDs row ... This reduces the overhead on index and metaindex in lib/mergeset	2019-09-24 00:49:51 +03:00
Aliaksandr Valialkin	c9063ece66	lib/storage: share tsids across all the partSearch instances ... This should reduce memory usage when big number of time series matches the given query.	2019-09-23 22:35:15 +03:00
Aliaksandr Valialkin	4e26ad869b	lib/{storage,mergeset}: verify PrepareBlock callback results ... Do not touch the first and the last item passed to PrepareBlock in order to preserve sort order of mergeset blocks.	2019-09-23 20:43:13 +03:00
Aliaksandr Valialkin	0772191975	lib/mergeset: detect whether we are in test by executable suffix	2019-09-22 23:12:15 +03:00
Aliaksandr Valialkin	48999e5396	lib/workingsetcache: remove data race when resetting c.misses	2019-09-22 19:36:49 +03:00
Aliaksandr Valialkin	0adebae1f8	lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID ... The first item from each mergeset block goes into index (lib/mergeset.blockHeader), so it must be short in order to reduce index size.	2019-09-22 19:21:33 +03:00
Aliaksandr Valialkin	0686ac52c3	lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values ... This should improve lookup performance if the same `label=value` pair exists in big number of time series. This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows occupy less space than the original `tag->metricID` rows.	2019-09-20 22:06:41 +03:00
Aliaksandr Valialkin	68722c3c74	lib/encoding: optimize UnmarshalUint* and UnmarshalInt*	2019-09-20 13:08:16 +03:00
Aliaksandr Valialkin	a544f49c2b	lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries ... The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries and MetricID->TSID entries are usually shorter than tag->MetricID entries. This should improve performance when selecting all the metricIDs.	2019-09-20 11:54:10 +03:00
Aliaksandr Valialkin	00cfb2d2b9	lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts	2019-09-19 21:48:20 +03:00
Aliaksandr Valialkin	37dc223e25	lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations	2019-09-19 20:13:08 +03:00
Aliaksandr Valialkin	a84fe76677	lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs	2019-09-19 20:07:22 +03:00
Aliaksandr Valialkin	3a697a935a	lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects	2019-09-19 17:49:56 +03:00
Aliaksandr Valialkin	51a21c7d4b	lib/mergeset: fill partHeader.firstItem on first block flush	2019-09-19 17:48:09 +03:00
Aliaksandr Valialkin	3d83f5d334	lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless ... This will save CPU on subsequent calls for this filter	2019-09-18 19:10:32 +03:00
Aliaksandr Valialkin	8d35718dc6	lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there	2019-09-17 23:20:27 +03:00
Aliaksandr Valialkin	0c8057924f	lib/mergeset: properly check for sorted block headers ... Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181	2019-09-13 21:59:29 +03:00
hanzai	e2274714b1	lib/workingsetcache: adjust switching from mode=split to mode=whole smoothly and load cachefile successfully	2019-09-13 19:13:01 +03:00
Aliaksandr Valialkin	bad53e4207	lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache	2019-09-11 14:53:45 +03:00
Aliaksandr Valialkin	88f8670ede	lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown ... This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin	9eb5de334f	lib/storage: typo fix	2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin	16dd145586	lib/storage: remove duplicate tag keys on MetricName.Marshal call ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172	2019-09-04 18:13:45 +03:00
Aliaksandr Valialkin	4862e93024	lib/fs: try harder with directory removal on NFS in the event of temporary lock ... Do not give up after 11 attempts of directory removal on laggy NFS. Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts on directory removal. Log failed attempts on directory removal after long sleep times. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162	2019-09-04 12:24:50 +03:00
Aliaksandr Valialkin	e1d76ec1f3	lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163	2019-08-29 15:08:35 +03:00
Aliaksandr Valialkin	aeaa5de5fe	lib/prombp: apply ba06b47c16 ... The following commands used: gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/ gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/	2019-08-29 13:35:27 +03:00
Aliaksandr Valialkin	9196c085a7	all: port to FreeBSD on GOARCH=amd64	2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin	2655220c58	lib/storage: go fmt	2019-08-27 14:29:51 +03:00
Aliaksandr Valialkin	bf915fc0db	lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters	2019-08-27 14:21:42 +03:00
Aliaksandr Valialkin	2fc157ff7a	lib/storage: properly handle (?i) in the tag filter regexp ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161	2019-08-26 00:44:45 +03:00
Aliaksandr Valialkin	0dc0006f34	lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159 This simplifies error detection additionally to the `vm_rows_ignored_total` counters.	2019-08-25 15:31:47 +03:00
Aliaksandr Valialkin	4b688fffee	lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent ... This should improve query speed over recent data on machines with big amounts of RAM	2019-08-25 14:41:12 +03:00
Aliaksandr Valialkin	1402a6b981	lib/storage: properly limit the number of output rows in small and big parts storage ... Previously small parts storage didn't take into account the available disk space for big parts.	2019-08-25 14:41:12 +03:00
Aliaksandr Valialkin	3308279c4e	lib/storage: remove outdated comment on maxRowsPerSmallPart ... The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a, which stops merging of small parts on graceful shutdown instead of waiting for their completion.	2019-08-25 13:47:32 +03:00
Aliaksandr Valialkin	ac004665b5	all: return 503 http error if service is temporarily unavailable ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156	2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin	8c03a8c4b4	app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries	2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin	380cae23a0	lib/storage: add benchmarks for regexp filter match / mismatch ... These benchmarks allow estimate the performance of regexp filters in promql	2019-08-22 16:36:42 +03:00
Aliaksandr Valialkin	4f738c8a15	lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error	2019-08-19 16:04:21 +03:00
Aliaksandr Valialkin	c23b66a1ad	lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders ... This reduces memory usage and memory fragmentation when working with big number of time series	2019-08-19 12:46:33 +03:00
Aliaksandr Valialkin	483de1cc06	lib/workingsetcache: automatically detect when it is better to double cache capacity	2019-08-15 22:57:55 +03:00
Aliaksandr Valialkin	5b41122292	lib/storage: properly cache tagFilters -> TSIDs entries from historical index	2019-08-14 02:29:58 +03:00
Aliaksandr Valialkin	964c296f96	lib/storage: compress contents of cache for tagFilters -> TSIDs ... This should increase cache capacity	2019-08-14 02:29:52 +03:00
Aliaksandr Valialkin	09fc6e22e5	all: use workingsetcache instead of fastcache ... This should reduce the amount of RAM required for processing time series with non-zero churn rate. The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.	2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin	99c37c2c96	lib/fs: add test for IsTemporaryFileName	2019-08-13 21:33:45 +03:00
Aliaksandr Valialkin	ec1b185991	lib/storage: remove broken BenchmarkIndexDBSearchTSIDs	2019-08-13 20:22:08 +03:00
Aliaksandr Valialkin	0967683ae9	lib: move common code for creating flock.lock file into fs.CreateFlockFile	2019-08-13 01:45:46 +03:00
Aliaksandr Valialkin	5d8d110010	lib/fs: atomically create file with the given contents on WriteFileAtomically ... This should prevent from `transaction` and `metadata.json` files corruption on unclean shutdown such as OOM, `kill -9`, power loss, etc. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148	2019-08-12 15:02:55 +03:00
Aliaksandr Valialkin	0b488f1e37	lib/storage: do not change timestamps to constant rate if values are constant or have constant delta ... This breaks the original timestamps, which results in issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .	2019-08-06 15:40:07 +03:00
Aliaksandr Valialkin	b8bb74ffc6	app/vmstorage: add vm_concurrent_addrows_* metrics for tracking concurrency for Storage.AddRows calls ... Track also the number of dropped rows due to the exceeded timeout on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`	2019-08-06 15:08:33 +03:00
Aliaksandr Valialkin	8822079b77	lib/storage: properly reset partSearch.fetchData in partSearch.reset	2019-08-05 09:56:06 +03:00
Aliaksandr Valialkin	47e4b50112	app/vmselect: optimize /api/v1/series by skipping storage data ... Fetch and process only time series metainfo.	2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin	10caad4728	lib/decimal: modernize tests a bit	2019-07-31 21:10:03 +03:00
Aliaksandr Valialkin	c14fd6c43f	lib/storage: typo fixes after a77e88db7d	2019-07-30 15:38:52 +03:00
Aliaksandr Valialkin	a77e88db7d	lib/storage: fix matching against tag filter with empty name ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137	2019-07-30 15:15:09 +03:00
Aliaksandr Valialkin	f586e1f83c	lib/storage: add metrics for calculating skipped rows outside the retention ... The metrics are: - vm_too_big_timestamp_rows_total - vm_too_small_timestamp_rows_total	2019-07-26 14:11:01 +03:00
Aliaksandr Valialkin	b339890dca	lib/encoding/zstd: go fmt	2019-07-25 01:37:16 +03:00
Aliaksandr Valialkin	6c4ca89d75	lib/encoding/zstd: disable CRC checks in pure Go build ... This should give slightly better compression and decompressions performance. Additionally this shaves off 4 bytes per each compressed block.	2019-07-24 19:17:16 +03:00
Aliaksandr Valialkin	d07f616609	lib/encoding: small fixes in tests after the PR #114	2019-07-23 19:37:51 +03:00
Roman Khavronenko	5bf4e5ffb5	all: add Pure Go build (pull request #114) ... Updates #94	2019-07-23 19:26:39 +03:00
Aliaksandr Valialkin	0f2ecde0e6	lib/encoding: improve gauge series detection ... - Series with negative values are always gauges - Counters may only have increasing values with possible counter resets This should improve compression ratio for gauge series which were previously mistakenly detected as counters.	2019-07-20 14:05:09 +03:00
Aliaksandr Valialkin	daba0cdb05	lib/netutil: do not count timeouts as network errors	2019-07-15 23:05:35 +03:00
Aliaksandr Valialkin	e75d5f47c4	lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs	2019-07-12 02:28:23 +03:00
Aliaksandr Valialkin	fc90ebf43c	lib/storage: do not reduce maxMetrics on time ranges exceeding maxDaysForDateMetricIDs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95	2019-07-12 02:20:34 +03:00
Aliaksandr Valialkin	2bd1a01d1a	lib/storage: do not pollute inverted index with data for samples outside the retention period	2019-07-11 17:04:56 +03:00
Aliaksandr Valialkin	d031e04023	lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value	2019-07-11 14:48:37 +03:00
Aliaksandr Valialkin	43ea4ce428	lib/storage: remember and skip individual tag filters matching too many metrics ... This saves CPU time by skipping useless matching for individual tag filters.	2019-07-11 14:48:30 +03:00
Aliaksandr Valialkin	1fe6d784d8	all: consistency renaming: bytesSize -> sizeBytes	2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin	56c154f45b	all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size	2019-07-04 19:42:30 +03:00
Aliaksandr Valialkin	c7034fc51b	lib/memory: attempt #3 to determine memory limit for LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-07-01 14:01:13 +03:00
Aliaksandr Valialkin	a41955863a	lib/mergeset: make fmt	2019-06-29 14:25:26 +03:00
Aliaksandr Valialkin	2ecb117082	lib/storage: skip non-matching metricIDs in sortedFilter ... This should improve performance for big sorteFilter lists.	2019-06-29 13:48:32 +03:00
Aliaksandr Valialkin	0c88afa386	lib/mergeset: speed up binarySearchKey by skipping the first item during binary search	2019-06-29 13:45:49 +03:00
Aliaksandr Valialkin	828078eb45	lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis	2019-06-28 18:22:46 +03:00
Aliaksandr Valialkin	7b59466667	lib/memory: attempt #2 to determine memory limit inside LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 18:08:15 +03:00
Aliaksandr Valialkin	593bd35aaa	lib/memory: an attempt to read proper memory limit inside LXC container ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84	2019-06-28 15:34:30 +03:00
Aliaksandr Valialkin	c1be1e4342	lib/storage: optimize time series search by regexp filter ... This should improve search speed on label filters like `{foo=~"bar.+baz"}`	2019-06-27 16:17:43 +03:00
Aliaksandr Valialkin	683bf2a11f	lib/storage: make sure non-nil args are passed to openIndexDB	2019-06-25 20:10:04 +03:00
Aliaksandr Valialkin	eb2283a029	lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive ... This should improve performance on inverted index search for big amount of unique time series when big -search.maxUniqueTimeseries is set.	2019-06-25 19:55:27 +03:00
Aliaksandr Valialkin	e8377011ab	lib/storage: free up memory from caches owned by indexDB when it is deleted	2019-06-25 14:42:44 +03:00
Aliaksandr Valialkin	33ea2120c3	lib/storage: use unversioned keys for tag cache in extDB ... Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache. This should improve performance for index lookups over big amount of time series.	2019-06-25 13:08:58 +03:00
Aliaksandr Valialkin	cf63669303	lib/storage: skip searching in extDB if it doesn't contain items for the given time range ... This should improve inverted index search performance for big amount of unique time series when the search is performed only on recent data.	2019-06-25 13:00:37 +03:00
Aliaksandr Valialkin	af2ceaaa0b	lib/storage: mention source parts on merge error ... This should improve determining broken source part. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76	2019-06-24 14:08:43 +03:00
Aliaksandr Valialkin	e88a03323a	all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70	2019-06-20 20:07:10 +03:00
Aliaksandr Valialkin	80db24386e	lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:37:55 +03:00
Aliaksandr Valialkin	296c14317f	lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69	2019-06-20 14:36:15 +03:00
Aliaksandr Valialkin	a78b3dba7f	app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"} metric for tracking active time series count	2019-06-19 18:36:47 +03:00
Aliaksandr Valialkin	710d6c33ea	lib/prompb: remove superflouos bytes copying in ReadSnappy	2019-06-18 20:37:51 +03:00
Aliaksandr Valialkin	6d6c9eb1f8	lib/flagutil: remove unused package	2019-06-18 10:43:55 +03:00
Aliaksandr Valialkin	d2c801029b	lib/storage: persist metric ids for the current and the previous hour on graceful shutdown ... This should improve performance after restart when the db contains a lot of time series with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)	2019-06-14 07:55:14 +03:00
Aliaksandr Valialkin	26f8d7ea1b	lib/fs: sync parent dir in MustRemoveAll only if it exists ... The parent directory may be non-existing when the deleted directory didn't exist before the MustRemoveAll call	2019-06-12 02:14:44 +03:00
Aliaksandr Valialkin	419197ba08	lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func ... The func syncs parent dir in order to persist directory removal in the event of power loss	2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin	c1276edab5	lib/fs: panic with fatal error when directories cannot be removed ... Unremoved directories may lead to inconsistent data directory, so VictoriaMetrics will fail to start next time. So panic on the first error when trying to remove directory in order to simplify recover process.	2019-06-12 01:20:54 +03:00
Aliaksandr Valialkin	2322c9a45a	lib/fs: attempt #2 to work around NFS issue with directory removal ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-12 01:07:05 +03:00
Aliaksandr Valialkin	935bfd7a18	lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error	2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin	3dd36b8088	lib/fs: make sure the created directory remains visible in the fs in the event of power loss ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 23:08:09 +03:00
Aliaksandr Valialkin	afb964670a	lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused	2019-06-11 22:54:26 +03:00
Aliaksandr Valialkin	20fc0e0e54	lib/{storage,mergeset}: sync filenames inside part when finalizing the part ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63	2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin	ac7b186f13	all: try hard removing directory with contents ... Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61	2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin	cbe692f0e2	app/vmselect: add /api/v1/labels/count handler for quick detection of labels with the maximum number of distinct values	2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin	7b6623558f	lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter ... This should improve speed for searching metrics among high number of time series with high churn rate like in big Kubernetes clusters with frequent deployments.	2019-06-10 14:07:39 +03:00
Aliaksandr Valialkin	a1351bbaee	lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters	2019-06-10 13:26:44 +03:00
Aliaksandr Valialkin	b4d707d9bb	lib/storage: give clearer names to more functions	2019-06-10 13:01:23 +03:00
Aliaksandr Valialkin	bee7298f81	lib/storage: give more clear names to functions	2019-06-10 12:50:44 +03:00
Aliaksandr Valialkin	dbd217b8f0	lib/storage: test GetSeriesCount	2019-06-10 12:43:34 +03:00
Aliaksandr Valialkin	4d936b1524	lib/storage: make getSeriesCount func indexSearch method	2019-06-10 12:29:11 +03:00
Aliaksandr Valialkin	d37924900b	lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)	2019-06-09 19:13:56 +03:00
Aliaksandr Valialkin	a3e26de45e	lib/procutil: typo fix in comment to WaitForSigterm	2019-06-04 17:31:47 +03:00
Aliaksandr Valialkin	28f6c36ab4	lib/storage: tune updating a map with today`s metric ids ... - Increase update iterval from 1s to 10s. This should reduce CPU usage for large amounts of metric ids with constant churn. - Reduce pendingTodayMetricIDsLock lock duration during the update.	2019-06-02 21:58:16 +03:00
Aliaksandr Valialkin	4794f894a4	lib/storage: speed up checking metricID existence in the list for the current date	2019-06-02 18:34:08 +03:00
Aliaksandr Valialkin	e307a4d92c	lib/timerpool: use timer pool in concurrency limiters ... This should reduce the number of memory allocations in highly loaded system	2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin	54fb8b21f9	all: fix misspellings	2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin	d6523ffe90	Makefile: add -s flag to go fmt in make fmt command	2019-05-25 21:43:35 +03:00
Aliaksandr Valialkin	6b0ae332f8	lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls	2019-05-24 13:01:02 +03:00
Aliaksandr Valialkin	6fb9dd09f5	lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio	2019-05-24 12:34:32 +03:00
Aliaksandr Valialkin	19b6643e5c	lib/encoding: substitute CompressZSTD with CompressZSTDLevel	2019-05-24 12:32:55 +03:00
Aliaksandr Valialkin	08b889ef09	lib/httpserver: add -http.disableResponseCompression flag, which may help saving CPU resources at the cost of higher network bandwidth usage	2019-05-24 12:18:40 +03:00
Aliaksandr Valialkin	674888fdc9	lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29	2019-05-23 17:18:35 +03:00
Aliaksandr Valialkin	1836c415e6	all: open-sourcing single-node version	2019-05-23 00:18:06 +03:00