Aliaksandr Valialkin
937f382938
lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format
...
Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.
2021-03-02 13:21:43 +02:00
Aliaksandr Valialkin
019d8e88d8
lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as foo; 1 2
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100
2021-03-01 17:16:35 +02:00
Aliaksandr Valialkin
1a3689af9a
lib/promscrape/discovery/kubernetes: deflake tests; a follow-up for 05fb08713c
2021-03-01 14:32:12 +02:00
Aliaksandr Valialkin
62ebf5c88e
lib/promscrape: explicitly stop and cleanup service discovery routines when new config is read from -promscrape.config
...
This should reduce memory usage when `-promscrape.config` file frequently changes
2021-03-01 14:14:00 +02:00
Aliaksandr Valialkin
e32ad9e923
lib/promscrape: use target arg in ScrapeWork cache
2021-03-01 12:29:09 +02:00
Aliaksandr Valialkin
f5d77a7081
lib/promscrape: typo fix, which prevented from caching ScrapeWork entries
2021-03-01 12:12:56 +02:00
Aliaksandr Valialkin
e84153d5ca
lib/promscrape: add vm_promscrape_scrapework_cache_* metrics for tracking ScrapeWork cache effectiveness
2021-03-01 12:05:45 +02:00
Aliaksandr Valialkin
7f15cd7161
lib/httpserver: make make errcheck
happy after the commit 9fc7726d84
2021-03-01 00:34:43 +02:00
Aliaksandr Valialkin
530e9904af
lib/promscrape: reduce CPU usage an memory allocations when constructing scrapeWorkKey
2021-02-28 22:29:58 +02:00
Aliaksandr Valialkin
9fc7726d84
lib/httpserver: make sure the gzipResponseWriter.Write() is called on Flush() and Close() calls
...
This should fix the `http: superfluous response.WriteHeader call` issue
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1078
2021-02-28 19:22:50 +02:00
Aliaksandr Valialkin
e5ca8ac0db
lib/promscrape: add ability to spread scrape targets among multiple vmagent instances
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
2021-02-28 18:41:08 +02:00
Aliaksandr Valialkin
e02d1ef93c
lib/promscrape/discovery/kubernetes: properly account the number of objects when watcher is stopped
...
A follow-up for b21b110b7a
2021-02-28 17:06:02 +02:00
Aliaksandr Valialkin
b21b110b7a
lib/promscrape/discovery/kubernetes: add vm_promscrape_discovery_kubernetes_*
metrics for monitoring internal state of k8s service discovery
2021-02-28 16:57:40 +02:00
Aliaksandr Valialkin
c459600346
lib/promscrape/discovery/kubernetes: remove resourceVersionMatch=NotOlderThan
query arg when watching for k8s object changes, since it cannot be used when watch=1
query arg is passed
2021-02-28 16:07:14 +02:00
Aliaksandr Valialkin
59a31171e3
lib/promscrape: fix possible deadlock in parallel execution of target relabeling
2021-02-28 16:05:13 +02:00
Aliaksandr Valialkin
68a0f5ce12
lib/promscrape/discovery/kubernetes: fix deadlock in startWatcherForURL
...
reloadObjects must be called without holding aw.mu lock
2021-02-28 15:26:30 +02:00
Aliaksandr Valialkin
b523e0369c
lib/promscrape/discovery/kubernetes: typo fix after 241ffd1f3b
2021-02-28 15:12:17 +02:00
Aliaksandr Valialkin
241ffd1f3b
lib/promscrape/discovery/kubernetes: pre-populate labelsByKey in reloadObject()
2021-02-28 15:09:49 +02:00
Aliaksandr Valialkin
05fb08713c
lib/promscrape/discovery/kubernetes: compare sorted sets of labels in tests
...
This should deflake tests where the order of labels isn't stable
2021-02-28 14:10:19 +02:00
Aliaksandr Valialkin
af8b7e8391
lib/promscrape: add missing startWatchersForRole()
call at the beginning of apiWatcher.getLabelsForRole
2021-02-28 14:00:17 +02:00
Aliaksandr Valialkin
966e9c227a
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-27 01:48:33 +02:00
Aliaksandr Valialkin
422b31de40
lib/promscrape/discovery/kubernetes: reload k8s resources on every error
...
This is needed for obtaining fresh resourceVersion
2021-02-27 01:47:27 +02:00
Aliaksandr Valialkin
7cc3d96a41
lib/fs: follow-up after f3a03c4164
2021-02-27 01:01:47 +02:00
Nikolay
f3a03c4164
Adds windows build ( #1040 )
...
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036
NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall
* rollback separator
* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil
* check for invlaid param
* Fixes posix delete semantic
* refactored a bit
* fixes openbsd build
* removed windows api call
* Fixes code after windows add
* Update lib/procutil/signal_windows.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-02-27 00:37:07 +02:00
Aliaksandr Valialkin
edb2ab7d8e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-27 00:25:01 +02:00
Aliaksandr Valialkin
a78948ae8b
lib/promscrape: yet another typo fix after ed8441ec52
2021-02-26 23:35:47 +02:00
Aliaksandr Valialkin
8683ea85e6
lib/fs: properly handle stale NFS file handle
error during file deletion
...
This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.
2021-02-26 23:25:14 +02:00
Aliaksandr Valialkin
9fa2632ac3
lib/promscrape: typo fix after ed8441ec52
2021-02-26 23:04:05 +02:00
Aliaksandr Valialkin
0ad887fd4d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-26 23:00:40 +02:00
Aliaksandr Valialkin
ed8441ec52
lib/promscrape: cache ScrapeWork
...
This should reduce the time needed for updating big number of scrape targets.
2021-02-26 21:43:22 +02:00
Aliaksandr Valialkin
815666e6a6
lib/promscrape/discovery/kubernetes: cache target labels
...
This should reduce CPU usage on repeated SDConfig.GetLabels() calls.
2021-02-26 20:23:28 +02:00
Aliaksandr Valialkin
19712fc2bd
lib/promscrape/discovery/kubernetes: errcheck fix
2021-02-26 17:00:08 +02:00
Aliaksandr Valialkin
c8f2f9b2e8
lib/promscrape: cleanup after 9b2246c29b
...
Main points:
* Revert changes outside lib/promscrape/discovery/kuberntes . These changes can be applied later in a separate commit
* Minimize changes in lib/promscrape/discovery/kubernetes compared to a93e644001
* Corner case fixes.
2021-02-26 16:54:05 +02:00
Nikolay
9b2246c29b
vmagent kubernetes watch stream discovery. ( #1082 )
...
* started work on sd for k8s
* continue work on watch sd
* fixes
* continue work
* continue work on sd k8s
* disable gzip
* fixes typos
* log errror
* minor fix
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-02-26 16:46:13 +02:00
Aliaksandr Valialkin
a93e644001
lib/promscrape: remove duplicate code a bit
2021-02-26 16:39:56 +02:00
Aliaksandr Valialkin
f7b242540b
lib/promscrape: reduce processing time for big number of discovered targets by processing them in parallel
2021-02-26 16:39:56 +02:00
Aliaksandr Valialkin
f7049e2af7
lib/promrelabel: optimize labeldrop
and labelkeep
relabeling for prefix.*
and prefix.+
regexps
2021-02-24 17:58:28 +02:00
Aliaksandr Valialkin
2c44178645
lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache
2021-02-23 15:47:19 +02:00
faceair
15d61c4879
lib/storage: correct tagfilter match cost ( #1079 )
2021-02-22 21:46:56 +02:00
Aliaksandr Valialkin
d136081040
lib/promrelabel: add more optimizations for relabeling for common cases
2021-02-22 16:33:55 +02:00
Aliaksandr Valialkin
dd1e53b119
lib/promrelabel: optimize relabeling performance for common cases
2021-02-22 00:51:13 +02:00
Aliaksandr Valialkin
ff5bbc4b88
lib/promscrape: export vm_promscrape_target_relabel_duration_seconds metric
2021-02-21 23:21:42 +02:00
Aliaksandr Valialkin
636c55b526
lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation
...
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
2021-02-21 22:06:47 +02:00
Aliaksandr Valialkin
388cdb1980
lib/storage: do not re-calculate stats for heavy tag filters
...
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
2021-02-21 21:39:01 +02:00
Aliaksandr Valialkin
48656dcc38
lib/{mergeset,storage}: allow merging smaller number of small parts
...
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.
This partially reverts ebf8da3730
2021-02-21 21:28:36 +02:00
Aliaksandr Valialkin
cb311bb156
lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains
2021-02-21 21:18:59 +02:00
Aliaksandr Valialkin
2cfb376945
lib/promscrape: typo fix after the commit f26162ec99
2021-02-19 00:33:37 +02:00
Aliaksandr Valialkin
c2678754e4
app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-19 00:31:34 +02:00
Aliaksandr Valialkin
f26162ec99
lib/promscrape: add scrape_align_interval config option into scrape config
...
This option allows aligning scrapes to a particular intervals.
2021-02-18 23:53:44 +02:00
Aliaksandr Valialkin
d5dde7f6b1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-18 19:14:41 +02:00
Aliaksandr Valialkin
f9084611bd
lib/storage: use composite index for a query with a name filter and negative filters
2021-02-18 18:57:23 +02:00
Aliaksandr Valialkin
a537c4f602
lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters
...
Example: `node_cpu_seconds_total{mode!="idle"}`
2021-02-18 18:46:36 +02:00
Aliaksandr Valialkin
3588687f84
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-18 14:53:19 +02:00
Aliaksandr Valialkin
e540c02014
lib/storage: prevent from running identical heavy tag filters in concurrent queries when measuring the number of loops for such tag filter.
...
This should reduce CPU usage spikes when measuring the number of loops needed for heavy tag filters
2021-02-18 13:58:18 +02:00
Aliaksandr Valialkin
711f8a5b8d
lib/storage: sort tag filters by the number of loops they need for the execution
...
This metric should work better than the filter execution duration, since it cannot be distorted
by concurrently running queries.
2021-02-18 12:47:38 +02:00
Aliaksandr Valialkin
ce99b48a9a
Revert "lib/mergeset: tune lifetime for entries inside block caches"
...
This reverts commit 458c89324d
.
Production testing revealed zero improvements for memory usage with reduced lifetime for entries in block caches.
2021-02-17 20:42:21 +02:00
Aliaksandr Valialkin
939d5ffc2b
lib/storage: move composite filters to the top during sorting
2021-02-17 20:26:51 +02:00
Aliaksandr Valialkin
faad6f84a4
lib/storage: return back filter arg to getMetricIDsForTagFilter function
...
The filter arg has been removed in the commit c7ee2fabb8
because it was preventing from caching the number of matching time series per each tf.
Now the cache contains duration for tf execution, so the filter shouldn't break such caching.
2021-02-17 19:33:22 +02:00
Aliaksandr Valialkin
d4849561ef
app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics
2021-02-17 19:13:38 +02:00
Aliaksandr Valialkin
33806264ec
lib/storage: revert ecf132933e
, since negative filters require the same amount of work as non-negative filters
2021-02-17 18:55:04 +02:00
Aliaksandr Valialkin
63fc140624
lib/storage: tag filters sorting...
2021-02-17 17:55:29 +02:00
Aliaksandr Valialkin
74424b55ee
lib/storage: further tune tag filters sorting
2021-02-17 17:28:15 +02:00
Aliaksandr Valialkin
442fcfec5a
lib/storage: tune the logic for sorting tag filters according the their exeuction times
2021-02-17 15:00:08 +02:00
Aliaksandr Valialkin
4a07820048
lib/storage: make sure that nobody uses partitions when closing the table
2021-02-17 14:59:04 +02:00
Aliaksandr Valialkin
687eb4ab00
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-16 22:29:45 +02:00
Aliaksandr Valialkin
1256931aee
lib/httpserver: make errcheck happy
2021-02-16 22:05:32 +02:00
Aliaksandr Valialkin
d61f7b7279
lib/storage: more tuning for tag filters sorting according the time they take
2021-02-16 21:22:23 +02:00
Aliaksandr Valialkin
458c89324d
lib/mergeset: tune lifetime for entries inside block caches
...
This should reduce memory usage in general case without significant CPU usage increase
2021-02-16 18:11:51 +02:00
Aliaksandr Valialkin
2824856691
lib/mergeset: clarify comments in the code a bit
2021-02-16 18:02:57 +02:00
Aliaksandr Valialkin
1bf6cd814d
lib/uint64set: remove memory allocation in bucket16.appendTo when sorting smallPool
2021-02-16 15:31:49 +02:00
Aliaksandr Valialkin
8ec45ff335
lib/httpserver: cache /metrics
output for a second
...
This should reduce CPU load when `/metrics` output is scraped with a frequency exceeding a request per second
2021-02-16 14:56:36 +02:00
Aliaksandr Valialkin
b861a64510
lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names
2021-02-16 13:59:18 +02:00
Aliaksandr Valialkin
7faa762021
lib/mergeset: remove unused code after a4140de9e6
2021-02-16 13:40:09 +02:00
Aliaksandr Valialkin
ca191696fe
lib/storage: tune sorting for tag filters
2021-02-16 13:04:49 +02:00
Aliaksandr Valialkin
ecf132933e
lib/storage: increase match cost for negative tag filters, since they need to scan all the label pairs
2021-02-15 16:34:23 +02:00
Aliaksandr Valialkin
4e39bf148c
vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0
...
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
9f5ac603a7
lib/storage: reduce the minimum supported retention for inverted index from one month to one day
2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
2e30202dc7
lib/flagutil: prevent from integer overflow when parsing duration
2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
38d7e96602
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_*
and __meta_kuberntes_endpoints_annotation_*
labels to role: endpoints
...
This syncs kubernetes SD with Prometheus 2.25
See 617c56f55a
2021-02-15 02:51:16 +02:00
Aliaksandr Valialkin
b04fece006
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpoints_label_*
and __meta_kuberntes_endpoints_annotation_*
labels to role: endpoints
...
This syncs kubernetes SD with Prometheus 2.25
See 617c56f55a
2021-02-15 02:50:46 +02:00
Aliaksandr Valialkin
d0c364d93d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-15 01:45:39 +02:00
Aliaksandr Valialkin
74963f71c6
lib/logger: explicitly import "time/tzdata" package for embedding tzdata into the app
...
The approach with `timetzdata` build tag didn't work for GOARCH=arm and GOARCH=ppc64le
due to the issue https://github.com/golang/go/issues/44073#issuecomment-778854298
2021-02-15 01:00:01 +02:00
Aliaksandr Valialkin
71c417427c
lib/storage: sort tag filters by actual execution time instead of by the number of matching time series
...
This should improve query speed for queries with regexp filters matching small number of time series
on a label with big number of unique values.
2021-02-15 00:18:13 +02:00
Aliaksandr Valialkin
c727d2219b
lib/storage: properly hanle regexp tag filters with dots, which can be converted to full string match filters.
...
For example `{label=~"foo\.bar"}` should be converted to `{label="foo.bar"}`. Previously it has was mistakenly conveted to `{label="foo\.bar"}` .
This could result in missing time series for such tag filters.
2021-02-14 23:38:14 +02:00
Aliaksandr Valialkin
80dc74dbc1
lib/promscrape: remove vm_promscrape_scrapes_failed_per_url_total and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total metrics
...
These metrics may result in big number of time series when vmagent scrapes thousands of targets and these targets constantly changes.
* It is better using `up == 0` query for determining failing targets.
* It is better using the following query for determining targets with exceeded limit on the number of metrics:
scrape_samples_scraped > 0 if up == 0
2021-02-12 05:26:04 +02:00
Aliaksandr Valialkin
0e26b7168a
lib/storage: return back in-order applying of tag filters, since concurrently executing tag filters can result in CPU and RAM waste in common case
2021-02-10 22:41:04 +02:00
Aliaksandr Valialkin
b51c23dc5b
lib/storage: parallelize tag filters execution a bit
...
This should reduce execution time when a query contains multiple tag filters and each such filter matches big number of time series.
2021-02-10 18:12:25 +02:00
Aliaksandr Valialkin
c7ee2fabb8
lib/storage: remove filter arg from getMetricIDsForDateTagFilter function
...
The `filter` arg breaks the logic for sorting tag filters by the matching metrics,
which may result in non-optimal performance during time series search.
2021-02-10 18:12:20 +02:00
Aliaksandr Valialkin
57cac289e0
lib/storage: fix inconsistencies in error logs
2021-02-10 18:12:16 +02:00
Aliaksandr Valialkin
5d5f0b0627
lib/storage: load metadata before loading indexdb, since indexdb depends on the metadata
2021-02-10 17:55:40 +02:00
Aliaksandr Valialkin
cdecf83ce5
app/vmstorage: export vm_composite_index_min_timestamp metric
2021-02-10 17:14:08 +02:00
Aliaksandr Valialkin
553016ea99
lib/storage: disable composite index usage when querying old data
2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
fcb7655d1e
lib/storage: fix metric name match for composite filter
2021-02-10 01:27:45 +02:00
Aliaksandr Valialkin
c7dccebaef
lib/storage: optimize search by label filters matching big number of time series
2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
6b4e6c229c
lib/storage: reduce lock contention in dateMetricIDCache when registering new time series for the current day
...
This should help systems with multiple CPU cores
2021-02-10 00:01:13 +02:00
Aliaksandr Valialkin
31f6b9c977
lib/fs: remove the code for tracking whether the given memory region is in page cache
...
This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.
2021-02-09 16:49:03 +02:00
Aliaksandr Valialkin
0a69122d81
lib/mergeset: remove dead code left after a4140de9e6
2021-02-09 16:33:52 +02:00
Aliaksandr Valialkin
d56390b925
optimize Storage.updatePerDateData()
2021-02-09 02:55:36 +02:00
Aliaksandr Valialkin
fda61e8e96
lib/storage: skip deduplication when creating inmemory data blocks
...
The deduplication will be performed later during merging such blocks.
2021-02-09 02:25:32 +02:00
Aliaksandr Valialkin
a4140de9e6
lib/mergeset: unconditionally cache indexdb blocks
...
Production workloads show that indexdb blocks must be cached unconditionally for reducing CPU usage.
This shouldn't increase memory usage too much, since unused blocks are removed from the cache every two minutes.
2021-02-09 00:47:50 +02:00
Aliaksandr Valialkin
cb96a1865b
app/vmstorage: export missing vm_cache_size_bytes
metrics for indexdb and data caches
2021-02-09 00:47:00 +02:00
Aliaksandr Valialkin
c5770600a2
lib/cgroup: follow-up after b9bf3cbe3e
2021-02-08 15:54:38 +02:00
Nikolay
b9bf3cbe3e
refactored cgroups limits, ( #1061 )
...
adds tests, remove os.Exec
2021-02-08 15:46:22 +02:00
Aliaksandr Valialkin
2242647a04
lib/storage: optimize data ingestion in the beginning of every hour
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
2021-02-08 12:01:12 +02:00
Aliaksandr Valialkin
8f28a578d3
lib/logger: exit the app if unsupported timezone value has been passed to -loggerTimezone
...
While at it, clarify descrption for `-loggerTimezone` command-line flag.
2021-02-07 23:35:37 +02:00
Aliaksandr Valialkin
83d3e582ab
lib/storage: check for prevHourMetricIDs cache before falling back to checking for (date, metricID) entries during data ingestion
...
This should reduce possible CPU usage spikes at the beginning of every hour.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1046
2021-02-04 18:48:13 +02:00
Aliaksandr Valialkin
9fb38569eb
lib/httpserver: expose process_open_fds
and process_max_fds
metrics
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037
2021-02-04 16:40:50 +02:00
Nikolay
48c8c5093b
fixes dockerswarm ( #1053 )
...
fixes improper usage of host network services
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1028
2021-02-04 15:56:42 +02:00
Aliaksandr Valialkin
63c88d8ea2
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-03 23:52:43 +02:00
Aliaksandr Valialkin
d16f22f3a1
app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards
...
Example: foo{bar{x,yz},a[b-c],*de}
2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
a5a1b9bd66
lib/storage: fix a bug, which breaks searching by Graphite wildcard filters
2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
6123aa3e75
sort orSuffixes in tagFilter.InitFromGraphiteQuery for faster seeks
2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
dc6636e2b2
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-03 12:30:44 +02:00
Aliaksandr Valialkin
157c02622b
app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"}
syntax
2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
4068f8d590
lib/promscrape: add vm_promscrape_service_discovery_duration_seconds metric
2021-02-02 16:15:25 +02:00
Aliaksandr Valialkin
c13f1d99e0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-01 20:19:50 +02:00
Aliaksandr Valialkin
bd11fd8f1d
lib/promscrape: add vm_promscrape_scrape_retries_total
, vm_promscrape_discovery_retries_total
and vm_promscrape_discovery_requests_total
metrics
2021-02-01 20:06:27 +02:00
Aliaksandr Valialkin
f0087f0dbb
lib/flagutil: typo fix in comment to ArrayInt.GetOptionalArgOrDefault() func
2021-02-01 14:35:39 +02:00
Aliaksandr Valialkin
b2aa80e74b
app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values
...
This commit also adds --vm-round-digits command-line option to vmctl tool.
2021-02-01 14:27:09 +02:00
Aliaksandr Valialkin
079888f719
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-27 01:12:45 +02:00
Aliaksandr Valialkin
d6347a3e56
lib/logger: initialize timezone by UTC in order to fix failing tests
2021-01-27 00:59:12 +02:00
Aliaksandr Valialkin
fc5b26d856
lib/promscrape: export vm_promscrape_scrapes_failed_per_url_total
and vm_promscrape_scrapes_skipped_by_sample_limit_per_url_total
metrics
...
These metrics could be useful for determining imporperly working scrape targets.
Note that these metrics are exported only for failing scrape targets. They aren't exposed for normally working targets.
2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
de3c662e8a
all: consistently use timers from timerpool
2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
3149ac7a7e
lib/fs: properly initialize cleaner for pageCache bitmaps
...
Previously it wasnt working because the timer was fired only once
2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
3fe848cdd7
lib/logger: add -loggerTimezone
command-line flag for adjusting timezone for timestamps in log messages
2021-01-26 22:51:54 +02:00
Aliaksandr Valialkin
8cea3c3cc4
lib/promscrape: retry scrape and service discovery requests when the remote server closes http keep-alive connection
2021-01-22 13:22:33 +02:00
Aliaksandr Valialkin
b68264b4f5
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-22 12:10:57 +02:00
faceair
b638c1eed5
lib/mergeset: add missing shouldCacheBlock ( #1019 )
2021-01-15 11:46:01 +02:00
Aliaksandr Valialkin
bdd0a1cdb2
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:16:35 +02:00
Aliaksandr Valialkin
48951073c4
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:15:38 +02:00
Aliaksandr Valialkin
d72fc60108
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-13 11:41:15 +02:00
Nikolay
7bf5d48315
bumps minimal tls version ( #1012 )
2021-01-13 00:35:47 +02:00
Aliaksandr Valialkin
31ec79eaf6
lib/storage: inline marshalTags function and remove the code for handling duplicate tags from here
...
This is a follow-up commit after c8ea697db8
2021-01-12 15:13:30 +02:00
Aliaksandr Valialkin
c8ea697db8
lib/storage: de-duplicate tags in MetricName.sortTags
...
Leave only the last tag among tags with duplicate keys. This is needed for reliable addition of extra_labels
during data ingestion. See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1007 for details.
2021-01-12 15:03:42 +02:00
Nikolay
7976c22797
Fixes error handling for promscrape.streamParse ( #1009 )
...
properly return error if client cannot read data,
properly suppress scraper errors
2021-01-12 13:31:47 +02:00
Aliaksandr Valialkin
0b92e18047
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-11 21:16:32 +02:00
Aliaksandr Valialkin
2c44f9989a
lib/promscrape: properly show scrape duration on /targets
page
...
Previously it has been shown as 0.000s for any scrape duration.
2021-01-11 21:14:46 +02:00
Aliaksandr Valialkin
9e10d5083e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-11 13:04:58 +02:00
Aliaksandr Valialkin
24ffad74c1
all: use net.Dial
instead of fasthttp.Dial
, because fasthttp.Dial
limits the number of concurrent dials to 1000
2021-01-11 12:53:30 +02:00
Aliaksandr Valialkin
30c2d75815
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-01-08 00:26:00 +02:00
Aliaksandr Valialkin
9dcb18e03d
app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage
2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin
4cdbc4642d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-25 17:41:24 +02:00
Aliaksandr Valialkin
490c69c64e
lib/storage: wait for pending transactions before closing and dropping the partition
...
This deflakes `make test-full-386` test
2020-12-25 11:45:53 +02:00
Aliaksandr Valialkin
23c0fb1efc
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-24 17:21:19 +02:00
Aliaksandr Valialkin
cab7e936a3
lib/storage: physically remove stale parts
...
Previously they were removed from partition struct, but the corresponding directories weren't removed.
This is a follow-up for 46dba00756
2020-12-24 16:51:36 +02:00
Aliaksandr Valialkin
a0ea5777f0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-24 11:49:03 +02:00
Nikolay
b21e16ad0c
fixes kubernetes_sd ( #983 )
...
* fixes kubernetes_sd,
adds missing service metadata for pod ports without endpoint
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/982
* fix test
2020-12-24 11:26:14 +02:00
Aliaksandr Valialkin
820669da69
lib/promscrape: code prettifying for 8dd03ecf19
2020-12-24 10:56:10 +02:00
Nikolay
8dd03ecf19
adds proxy_url support, ( #980 )
...
* adds proxy_url support,
adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery,
adds proxy_url to the scrape_config for targets scrapping,
http based proxy is supported atm,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503
* fixes imports
2020-12-24 10:52:37 +02:00
Aliaksandr Valialkin
9e4ed5e591
lib/storage: do not remove parts outside the configured retention if they are currently merged
...
These parts are automatically removed after the merge is complete.
2020-12-24 08:51:28 +02:00
Aliaksandr Valialkin
46dba00756
lib/storage: remove stale parts as soon as they go outside the configured retention
...
Previously such parts could remain undeleted for long durations until they are merged with other parts.
This should help for `-retentionPeriod` values smaller than one month.
2020-12-22 19:54:31 +02:00
Aliaksandr Valialkin
8593358965
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-19 16:43:03 +02:00
Aliaksandr Valialkin
d65c03c004
lib/storage: properly determine max rows for output part when merging small parts
2020-12-18 23:14:38 +02:00
Aliaksandr Valialkin
ebf8da3730
lib/{storage,mergeset}: tune background merge process in order to reduce CPU usage and disk IO usage
2020-12-18 20:01:08 +02:00
Aliaksandr Valialkin
2dfa746c91
lib/promscrape: remove ID
field from ScrapeWork
struct. Use a pointer to ScrapeWork as a key in targetStatusMap
...
This simplifies the code a bit.
2020-12-17 14:32:56 +02:00
Aliaksandr Valialkin
9abb2d6c74
lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2
...
Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.
2020-12-16 23:40:17 +02:00
BigFish
27f0261257
lib/protoparser/prometheus/parser.go ( #970 )
...
fix parse timestamp error if there are some whitespaces after timestamp
2020-12-16 23:36:20 +02:00
Aliaksandr Valialkin
d0311b7fe5
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-15 22:44:35 +02:00
Aliaksandr Valialkin
a4c7fcb5e1
lib/promscrape: properly remove deleted target from /targets
page
...
Previously `sw` variable wasn't captured correctly by the started goroutine.
2020-12-15 20:57:09 +02:00
Aliaksandr Valialkin
56054f4eb7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-15 14:33:39 +02:00
Aliaksandr Valialkin
b730fc2667
lib/promscrape: properly handle scrape errors when stream parsing is enabled
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/967
2020-12-15 14:08:28 +02:00
Nikolay
95c9b630cc
adds new Array Flags ( #965 )
...
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url
* added backward compatibility testcases for ArrayDuration and ArrayBool
* fixes bool flag
* fixes test cases
2020-12-15 12:51:12 +02:00
Aliaksandr Valialkin
f9c706e186
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-15 12:43:44 +02:00
Aliaksandr Valialkin
60fcac4878
lib/promscrape: add bootstrap styles to /targets
html page
2020-12-15 12:37:56 +02:00
Aliaksandr Valialkin
5af2a9ca0e
lib/promscrape: formatting fixes for /tarets
page
2020-12-15 11:59:04 +02:00
Aliaksandr Valialkin
020917949b
lib/promscrape: formatting fixes for /targets
page
2020-12-15 11:24:18 +02:00
Aliaksandr Valialkin
d1193c87a8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-14 20:21:19 +02:00
Aliaksandr Valialkin
d006b41eff
lib/persistentqueue: verify that ReaderOffset doesnt exceed WriterOffset when opening the persistent queue
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964
2020-12-14 19:25:25 +02:00
Aliaksandr Valialkin
4f311e5827
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-14 14:20:52 +02:00
Aliaksandr Valialkin
ae972429c7
lib/promscrape: add missing whitespace between duration and ago
word at /targets
page
2020-12-14 14:19:58 +02:00
Aliaksandr Valialkin
069c9ade52
app/{vmagent,vminsert}: follow-up for ce8c2dd1f1
: return /targets
page in HTML when requested via web browser
2020-12-14 14:06:00 +02:00
Nikolay
ce8c2dd1f1
Changes targets api ( #961 )
...
* changes /targets api
adds html response if requester accepts text/html,
adds quick template for /targets api,
fixes pathPrefix for / requests
* changes namings
* renamed targets file
* Update app/victoria-metrics/main.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
* adds trimspace to qtpl,
moves content-type for targets response closer to writer
* fixes bug with prefix
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-12-14 13:36:48 +02:00
Aliaksandr Valialkin
8beb1f9519
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-11 21:29:22 +02:00
Aliaksandr Valialkin
5f9d88a3cb
lib/promscrape/discovery/consul: reduce load on Consul API server by increasing timeout for blocking requests from 50 seconds to 9 minutes
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
2020-12-11 17:24:13 +02:00
Aliaksandr Valialkin
501fd8efd9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-11 12:10:21 +02:00
Aliaksandr Valialkin
fd9fd191b9
lib/promscrape/discovery/consul: properly pass Datacenter filter to Consul API server
...
Previously it has been passed as `sdc` query arg, while it should be passed as `dc` query arg.
See https://www.consul.io/api-docs/health#list-nodes-for-service for details.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574#issuecomment-740454170
2020-12-08 21:52:42 +02:00
Aliaksandr Valialkin
4146fc4668
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
364f30a6e7
lib/promscrape: store ScrapeWork items by pointer in the slice returned from get*ScrapeWork()
...
This should prevent from possible 'memory leaks' when a pointer to ScrapeWork item stored in the slice
could prevent from releasing memory occupied by all the ScrapeWork items stored in the slice when they
are no longer used.
See the related commit e205975716
and the related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-12-08 17:50:05 +02:00
Aliaksandr Valialkin
08b71d2067
lib/promscrape: re-use strings for labels stored in ScrapeWork
...
This should reduce memory usage when working with big number of scrape targets.
2020-12-08 12:22:59 +02:00
Aliaksandr Valialkin
0f1b969aa6
lib/promscrape: export vm_promscrape_scrapers_{started|stopped}_total
metrics for monitoring target churn rate
2020-12-08 11:57:52 +02:00
Aliaksandr Valialkin
c7ac7c1807
lib/promscrape: store targetStatus entries in targetStatusMap by pointer instead of by value
...
This guarantees that GC frees memory occupied by targetStatus after it is unregistered from targetStatusMap.
2020-12-08 11:50:48 +02:00
Aliaksandr Valialkin
05813259dc
lib/promscrape: export vm_promscrape_active_scrapers{type="<sd_type>"}
metric for tracking the number of active scrapers per each service discovery type
2020-12-08 01:54:23 +02:00
Aliaksandr Valialkin
9c1c9d8e76
lib/promscrape: do not enable strict config parsing when -promscrape.config.dryRun
command-line flag is passed
...
Strict parsing for -promscrape.config can be enabled by passing `-promscrape.config.strictParse` command-line flag.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944
2020-12-07 13:18:50 +02:00
Aliaksandr Valialkin
82972a8f2a
lib/promscrape: mention in scrape error message that scrape errors can be disabled by -promscrape.suppressScrapeErrors
command-line flag
2020-12-06 23:27:58 +02:00
Aliaksandr Valialkin
299a35948c
lib/promscrape: clarify error message on failed connection to scrape target when -enableTCP6 command-line flag isn't set
2020-12-06 13:18:39 +02:00
Aliaksandr Valialkin
b0e4b234cb
lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol
2020-12-06 12:01:27 +02:00
Aliaksandr Valialkin
cb2342029e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-12-05 13:24:28 +02:00
Aliaksandr Valialkin
52915c8f7e
lib/promscrape/discoveryutils: remove limit on the number of concurrently running blocking queries
...
Too low limit could result in unexpected errors when performing big number of blocking queries.
2020-12-05 12:15:52 +02:00
Aliaksandr Valialkin
eb27dbde13
lib/flagutil: make golangci-lint happy by using strings.TrimPrefix instead of manual prefix removal via strings.HasPrefix
2020-12-03 22:07:57 +02:00
Aliaksandr Valialkin
9d787f9edd
all: do not print usage info for all the flags when incorrect command-line flag is passed
...
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
2020-12-03 21:47:37 +02:00
Aliaksandr Valialkin
de0643fab5
lib/promscrape/discovery/consul: log the time needed for stoppig Consul service watcher
2020-12-03 20:14:55 +02:00
Aliaksandr Valialkin
9cd8eb92f1
lib/promscrape/discovery/consul: make sure that block response contains X-Consul-Index header
2020-12-03 20:05:23 +02:00
Aliaksandr Valialkin
5009b25a03
lib/promscrape: code cleanup after c6dee6c52d
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
2020-12-03 19:50:53 +02:00
Nikolay
c6dee6c52d
Changes consul discovery api ( #921 )
...
* adds consul watch api,
it must reduce load on consul service with blocking wait requests,
changed discoveryClient api with fetchResponseMeta callback.
* small fix
* fix after master merge
* adds watch client at discovery utils
* fixes consul watcher,
changes namings,
fixes data race
* small typo fix
* sanity fix
* fix naming and service node update
2020-12-03 19:47:40 +02:00
Aliaksandr Valialkin
357f886f97
Revert "lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test"
...
This reverts commit f3e80eb70e493620e42a0cc22a62c9af75076c77, since it isn't needed for single-node version of VictoriaMetrics
2020-11-29 01:27:25 +02:00
Aliaksandr Valialkin
ace969d595
lib/storage: add missing (AccountID, ProjectID) in MetricName.String() test
2020-11-29 01:26:04 +02:00
Aliaksandr Valialkin
32869e4c0f
lib/promscrape: fix failing tests after a906b3862f
2020-11-29 01:26:03 +02:00
Aliaksandr Valialkin
a906b3862f
lib/protoparser/prometheus: properly parse OpenMetrics timestamps
...
OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds.
This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.
2020-11-27 14:54:29 +02:00
Aliaksandr Valialkin
3a32789352
lib/promscrape: reduce memory allocations when unpacking gzipped responses received from scrape targets
2020-11-26 18:32:06 +02:00
Aliaksandr Valialkin
2cea4d403f
all: typo fix: thouthand->thousand
2020-11-26 13:33:46 +02:00
Aliaksandr Valialkin
b0a5c382ee
lib/promscrape: release http response non-200 status code
2020-11-26 13:25:17 +02:00
Aliaksandr Valialkin
afe6d2e736
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-25 23:03:44 +02:00
Aliaksandr Valialkin
b7fcdb528d
app/{vmagent,victoria-metrics}: add -dryRun
option and make more clear handling for -promscrape.config.dryRun
2020-11-25 22:59:13 +02:00
Aliaksandr Valialkin
1c669a69a8
lib/mergeset: tune the number of rawItemsBlocks to merge at once
...
512 blocks give higher ingestion performance and slightly lower memory usage
2020-11-25 21:52:52 +02:00
Aliaksandr Valialkin
7119f294f3
lib/mergeset: help GC by removing refereces to slices in inmemoryBlock.Reset
2020-11-25 21:19:43 +02:00
Aliaksandr Valialkin
8a057e705a
lib/storage: log metric name plus all its labels when the metric timestamp is outside the configured retention
...
This should simplify debugging when the source of the metric with unexpected timestamp must be found.
2020-11-25 14:41:37 +02:00
Aliaksandr Valialkin
b65236530c
lib/storage: typo fix in error message: allowd->allowed
2020-11-25 14:15:42 +02:00
Aliaksandr Valialkin
ae04378424
lib/protoparser/prometheus: properly parse "infinity" values in OpenMetrics format
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924
2020-11-24 19:03:38 +02:00
Aliaksandr Valialkin
bf95fbfc1d
lib/logger: disable rate limiting for error and warn logs by default
2020-11-24 12:42:25 +02:00
Aliaksandr Valialkin
78d2715d04
all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9
2020-11-24 12:42:22 +02:00
Aliaksandr Valialkin
b7f4fc6e0d
lib/protoparser/prometheus: properly parse metrics with exemplars
...
Examplars have been introduced in OpenMetrics - see https://github.com/OpenObservability/OpenMetrics/blob/master/OpenMetrics.md#exemplars-1
Previously VictoriaMetrics couldn't parse the following metric
foo{bar="baz"} 123 # exemplar here
This commit fixes this. Note that VictoriaMetrics ignores the exemplar as for now.
2020-11-24 12:34:56 +02:00
Aliaksandr Valialkin
aa90b93778
lib/promscrape: expose __meta_ec2_ipv6_addresses
label for ec2_sd_config
like Prometheus will do in the next release
2020-11-23 16:56:42 +02:00
Aliaksandr Valialkin
de523c81b9
lib/promscrape: add filters
option to dockerswarm_sd_config
like Prometheus did in v2.23.0
2020-11-23 16:27:40 +02:00
Aliaksandr Valialkin
f765985947
lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt
...
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin
e1a6262302
lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt
...
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
2020-11-23 09:55:41 +02:00
Aliaksandr Valialkin
f000a10cd0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-23 09:41:15 +02:00
Aliaksandr Valialkin
9d160f9048
lib/promscrape: hint that -enableTCP6 command-line flag can be used for connecting to IPv6 addresses
2020-11-21 14:39:00 +02:00
Aliaksandr Valialkin
d7932775cc
lib/promscrape/discovery/eureka: follow-up after eec76718e9
2020-11-20 14:00:12 +02:00
Nikolay
eec76718e9
Adds eureka service discovery ( #913 )
...
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-20 13:38:12 +02:00
Aliaksandr Valialkin
fb2d4e56ce
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-19 18:56:02 +02:00
Aliaksandr Valialkin
97a0c80904
lib/logger: follow-up for 09105ff49c
2020-11-19 12:37:00 +02:00
Nikolay
09105ff49c
Adds log suppression per caller ( #908 )
...
* Adds log suppression per caller
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905
* fixes style and report message
2020-11-19 12:17:23 +02:00
Aliaksandr Valialkin
a6b2b2c005
lib/logger: add -loggerWarnsPerSecondLimit
command-line flag for rate limiting of WARN log messages
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/905
2020-11-18 03:43:37 +02:00
Aliaksandr Valialkin
36b748dfc7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-16 21:06:21 +02:00
Aliaksandr Valialkin
266788be14
app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place
...
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
0208d8c103
lib/storage: add a test for Storage.SearchMetricNames
2020-11-16 13:15:16 +02:00
Aliaksandr Valialkin
465923b181
app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
48d033a198
app/vminsert: add /tags/tagSeries
and /tags/tagMultiSeries
handlers from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin
4aaee33860
lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page
2020-11-16 00:44:35 +02:00
Aliaksandr Valialkin
0deabbbb4a
lib/protoparser/promremotewrite: log the time spent on unsuccessful data read from the network
...
This should help with debugging `connection timed out` errors.
2020-11-13 17:49:12 +02:00
Vasily
6fcbd17bdd
Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig ( #796 )
...
* Add omitempty for DisableCompression and DisableKeepAlive fields in ScrapeConfig
* Add omitempty annotation to all the default/optional values
* Fix annotations after review
2020-11-13 16:19:05 +02:00
Aliaksandr Valialkin
cdf1e6684b
lib/protoparser/opentsdbhttp: increment errors counter on unmarshal errors
...
This is a follow-up for 149c0c4a6d
2020-11-13 13:23:17 +02:00
Aliaksandr Valialkin
149c0c4a6d
lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols
...
The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-13 13:05:24 +02:00
Aliaksandr Valialkin
4f8a3af061
lib/protoparser/promremotewrite: synchronously process Prometheus remote_write requests
...
There is no reason in processing these requests asynchronously in the face of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
Synchronous processing code is easier to read and understand than the previous async code
2020-11-13 12:17:25 +02:00
Aliaksandr Valialkin
57a4af98fa
lib/protoparser/promremotewrite: forward errors, which can occur during data ingestion, to the caller of ParseStream, so it could properly return HTTP 503 status code on non-nil error
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-13 11:01:07 +02:00
Aliaksandr Valialkin
47a038401b
all: consistently return text-based HTTP responses with charset=utf-8
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
2020-11-13 10:35:41 +02:00
immerrr again
51c529a2b6
app/vmstorage: add "/internal/force_flush" endpoint ( #893 )
2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin
e205975716
lib/promscrape: make a copy of ScrapeWork from discovered []ScrapeWork slice instead of referring to an item in this slice
...
This should prevent from holding previously discovered []ScrapeWork slices when a part of discovered targets changes over time.
This should reduce memory usage for the case when big number of discovered scrape targets changes over time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-10 16:13:05 +02:00
Aliaksandr Valialkin
6e668fd480
lib/promscrape: pre-allocate slice for discovered targets based on previously discovered targets
...
This should reduce load on GC a bit when discovering big number of scrape targets
2020-11-10 15:56:51 +02:00
Aliaksandr Valialkin
e32620afa1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-10 00:21:55 +02:00
Aliaksandr Valialkin
bcd139362b
lib/promscrape: add -promscrape.dropOriginalLabels
command-line flag for reducing memory usage when discovering big number of scrape targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-10 00:19:57 +02:00
Aliaksandr Valialkin
6c24c5caa3
lib/promscrape: further reduce memory usage for per-scrape target labels by making a copy of actually used labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-09 10:54:42 +02:00
Aliaksandr Valialkin
3f298272a8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-08 13:41:41 +02:00
Aliaksandr Valialkin
83df20b5b5
lib/promscrape: clean references to label name and label value strings after applying per-target relabeling
...
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 16:19:59 +02:00
Aliaksandr Valialkin
9e83335ca9
lib/promscrape/discovery/kubernetes: go fmt
2020-11-07 13:03:49 +02:00
Aliaksandr Valialkin
5407eed2f6
lib/promscrape/discovery/kubernetes: reduce memory usage for labels when discovering big number of scrape targets by using string concatenation instead of fmt.Sprintf
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 13:03:08 +02:00
Aliaksandr Valialkin
188325f0fc
lib/promscrape: eliminate data race in stream parse
mode
...
Previously `-promscrape.streamParse` mode could result in garbage labels for the scraped metrics because of data race.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825#issuecomment-723198247
2020-11-07 12:45:17 +02:00
Aliaksandr Valialkin
56303aee5b
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-05 02:51:08 +02:00
Aliaksandr Valialkin
368b69b4c4
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin
1cb78ba1a0
lib/storage: remove data race when updating rowsDeleted
2020-11-05 01:12:21 +02:00