Aliaksandr Valialkin
db16759c68
lib/storage: optimize matching speed for non-trivial regexp filters
...
Wrap re.Match into bytesutil.FastStringMatcher.
This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:06:06 +03:00
Aliaksandr Valialkin
e8a64f6e7a
lib/promrelabel: remove redundant memory allocations by using interned strings
2022-10-01 11:50:21 +03:00
Aliaksandr Valialkin
73dc17ef64
lib/promrelabel: add a benchmark for realistic Kubernetes relabeling
...
The benchmark name is BenchmarkApplyRelabelConfigs/kubernetes
This benchmark has been copied from d521933053/model/relabel/relabel_test.go (L505)
See also https://github.com/prometheus/prometheus/pull/11147
2022-10-01 10:38:22 +03:00
Aliaksandr Valialkin
c54e14cdec
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
...
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:48:32 +03:00
Aliaksandr Valialkin
6ad7b0619c
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-30 18:42:04 +03:00
Nikolay
33f40f4a5f
app/vminsert: allows parsing tenant id from labels ( #3009 )
...
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
* Update docs/Cluster-VictoriaMetrics.md
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* wip
* app/vminsert/netstorage: clean remaining labels in order to free up GC
* docs/Cluster-VictoriaMetrics.md: typo fix
* wip
* wip
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 18:35:53 +03:00
Aliaksandr Valialkin
3a15bc761b
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-30 13:21:21 +03:00
Aliaksandr Valialkin
171dd14aa3
lib/promrelabel: go fmt
2022-09-30 12:28:55 +03:00
Aliaksandr Valialkin
a18d6d5ccc
lib/promrelabel: optimize action: replace
for non-trivial regex values
...
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.
Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:25:05 +03:00
Aliaksandr Valialkin
146021a076
lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481
2022-09-30 10:49:18 +03:00
Aliaksandr Valialkin
899d2c40fb
lib/promrelabel: optimize action: labelmap
for non-trivial regexs
2022-09-30 10:43:31 +03:00
Aliaksandr Valialkin
17289ff481
lib/regexutil: cache MatchString results for unoptimized regexps
...
This increases relabeling performance by 3x for unoptimized regexs
2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin
fda60b3d4d
lib/promrelabel: properly parse regex with escaped $ at the end
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131
Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil
...
lib/bytesutil is more appropriate place for InternString() function
2022-09-30 07:44:35 +03:00
Nikolay
f61b8cec69
lib/awsapi: fixes sign encoding ( #3183 )
...
* lib/awsapi: fixes sign encoding
previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171
* Update lib/awsapi/sign.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:43:44 +03:00
Aliaksandr Valialkin
6a32a64073
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed
2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin
92b3622253
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
...
This is a follow-up for 7c2474dac7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin
ef435f8cc4
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
...
Optimize this function by using results cache for input strings.
Use this function all over the code.
This is a follow-up for fcffdba9dc
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin
bd79706eb3
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-26 18:03:30 +03:00
Aliaksandr Valialkin
6411bbcce7
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
...
This should simplify further code maintenance and refactoring
This is a follow-up after 6ab1cede62
2022-09-26 17:58:01 +03:00
Dmytro Kozlov
6ab1cede62
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server ( #3109 )
...
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server
* lib/httpserver: added descriptions about tls supported versions
* lib/netutil: check minimal tls version, added supported tls versions to error
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:35:45 +03:00
Roman Khavronenko
e96ccf3f71
lib/mergeset: follow-up after a0e7432e42
( #3145 )
...
* lib/mergeset: follow-up after a0e7432e42
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Apply suggestions from code review
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:39:56 +03:00
Aliaksandr Valialkin
1a9cb85647
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-26 16:36:50 +03:00
Zakhar Bessarab
f022296d96
vmbackup: configure retries for GCS remote FS ( #3156 )
2022-09-26 16:28:20 +03:00
Aliaksandr Valialkin
41f8c2987d
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102
See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:17:25 +03:00
Aliaksandr Valialkin
7c2474dac7
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
...
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:57:23 +03:00
匠心零度
3d5509a720
lib/querytracer: fix comment ( #3135 )
2022-09-22 19:19:48 +03:00
Aliaksandr Valialkin
1c9f5b3580
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-14 17:54:06 +03:00
Aliaksandr Valialkin
56ce7ce85b
lib/promscrape: typo fix after 74c00a8762
2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin
74c00a8762
lib/promscrape: read response body into memory in stream parsing mode before parsing it
...
This reduces scrape duration for targets returning big responses.
The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin
ccad651a61
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error
...
This should improve debugging issues with Kubernetes API server
2022-09-13 19:36:55 +03:00
Aliaksandr Valialkin
ce2c07c5a7
lib/mergeset: atomically remove part dirs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
042a532f70
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
68e32b0764
lib/storage: atomically remove parts inside partitions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
340ada871d
lib/storage: atomically remove partitions, which went outside the configured retention
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin
978dcb4574
lib/storage: properly remove cache directory contents if reset_cache_on_startup
file is located there
...
Previously the cache directory was removed. This could result in error when the cache directory
is mounted to a separate filesystem.
2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin
5f28ca1f42
lib/storage: atomically remove snapshot directories
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin
9682c23786
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-09-08 19:02:16 +03:00
Aliaksandr Valialkin
5fa9525498
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:08:09 +03:00
Aliaksandr Valialkin
96160000e0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-30 12:40:52 +03:00
Bryce Lampe
74f8e12e87
Support "HTTP" and "HTTPS" schemes ( #3019 )
...
* Support "HTTP" and "HTTPS" schemes
* Update lib/promscrape/config.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin
30b8d91727
lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache
...
This is a follow-up for c06e7a142c
2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName()
...
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
a2cd79576f
lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix
...
This should improve slow path speed for regexps without literal prefixes
2022-08-26 21:48:30 +03:00
Aliaksandr Valialkin
f49c9bb700
lib/promrelabel: optimize common regex mismatch cases for action: replace
and action: labelmap
2022-08-26 15:45:31 +03:00
Aliaksandr Valialkin
4c6916f32a
lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop
,labelkeep
,drop
and keep
...
This makes possible optimizing additional cases inside regexutil.PromRegex
2022-08-26 15:23:45 +03:00
Aliaksandr Valialkin
7afe8450fc
lib/promrelabel: optimize matching for commonly used regex patterns in if
option
...
The following regex patterns are optimized:
- literal string match, e.g. "foo"
- prefix match, e.g. "foo.*" and "foo.+"
- substring match, e.g. ".*foo.*" and ".+foo.+"
- alternate values match, e.g. "foo|bar|baz"
2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin
0ad3bbadd3
lib/regexutil: add Simplify() function for simplifying the regular expression
2022-08-26 11:57:12 +03:00
Aliaksandr Valialkin
b373661988
lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep}
with anchored regex
prefix
...
The following commonly used relabeling rules must work faster now:
- action: labeldrop
regex: "^foo.+$"
- action: labeldrop
regex: "^bar.*"
2022-08-25 23:23:55 +03:00
Aliaksandr Valialkin
0d4ea03a73
lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop}
with regex
containing alternate values
...
For example, the following relabeling rule must work much faster now:
- action: labeldrop
regex: "foo|bar|baz"
2022-08-24 17:54:29 +03:00
Aliaksandr Valialkin
0d46e24af5
lib/storage: increase the maximum possible or
values extracted from regexp from 20 to 100
...
This should improve time series search speed for regexp filters with big number of `or` values.
2022-08-24 17:15:25 +03:00
Aliaksandr Valialkin
fdbf5b5795
lib/storage: ignore start text
and end text
anchors in getOrValues(regexp) function
...
This is OK, since the anchors are implicitly applied to the whole regexp.
This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors.
For example, `{label="^(foo|bar)$"}`
2022-08-24 17:12:52 +03:00
Aliaksandr Valialkin
796aa310c2
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series
metrics if -storage.max{Hourly,Daily}Series
limits are set
...
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:
vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:44:04 +03:00
Aliaksandr Valialkin
628e87e727
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-22 00:40:04 +03:00
Aliaksandr Valialkin
1f89278d88
all: subsitute ioutil.ReadAll with io.ReadAll
...
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.
This is a follow-up for 02ca2342ab
2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin
2c3a89339d
all: use os.ReadDir instead of ioutil.ReadDir
...
The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir
This is a follow-up for 02ca2342ab
2022-08-22 00:02:25 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:52:35 +03:00
Roman Khavronenko
d59d829cdb
lib/storage: bump max merge concurrency for small parts to 15 ( #2997 )
...
* lib/storage: bump max merge concurrency for small parts to 15
The change is based on the feedback from users on github.
Thier examples show, that limit of 8 sometimes become a
bottleneck. Users report that without limit concurrency
can climb up to 15-20 merges at once.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Update lib/storage/partition.go
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-21 23:32:08 +03:00
Aliaksandr Valialkin
8550c44e31
app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement
field during relabeling
...
For example:
- target_label: composite-label
replacement: "{{source_label1}}-{{source_label2}}"
2022-08-21 22:50:01 +03:00
Aliaksandr Valialkin
bb154f8829
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-18 01:31:49 +03:00
Roman Khavronenko
31f922944e
lib/storage: fix the search for empty label name ( #2991 )
...
* lib/storage: fix the search for empty label name
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Apply suggestions from code review
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-17 21:32:25 +03:00
Aliaksandr Valialkin
7d26414b2e
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit
...
The following metrics are generated:
- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped
These metrics simplify alerting on targets, which expose too many time series
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin
bb68ab99fa
lib/promscrape: retry http requests if the server returns 429 status code
...
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin
b0e1bb517e
lib/storage: typo fix in comments after f830edc0bc
2022-08-16 13:44:45 +03:00
Aliaksandr Valialkin
f830edc0bc
lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[]
filter matches small number of time series
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
2022-08-16 13:32:40 +03:00
Aliaksandr Valialkin
d2e293b5c9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-15 01:43:51 +03:00
Aliaksandr Valialkin
c3f8481011
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html
2022-08-15 01:40:20 +03:00
Aliaksandr Valialkin
95d36da358
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image
label in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11034
2022-08-15 01:18:23 +03:00
Aliaksandr Valialkin
c4fcd9f1c5
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number
label to role: service
in the same way as Prometheus 2.38 does
...
See https://github.com/prometheus/prometheus/pull/11002
2022-08-15 01:06:34 +03:00
Aliaksandr Valialkin
511805d88d
lib/promscrape/discovery/dns: add support for resolving MX records
...
See https://github.com/prometheus/prometheus/pull/10099
2022-08-15 00:32:34 +03:00
Roman Khavronenko
a0e7432e42
lib/storage: prevent excessive loops when storage is in RO ( #2962 )
...
* lib/storage: prevent excessive loops when storage is in RO
Returning nil error when storage is in RO mode results
into excessive loops and function calls which could
result into CPU exhaustion. Returning an err instead
will trigger delays in the for loop and save some resources.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* document the change
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-09 12:17:00 +03:00
Aliaksandr Valialkin
e80ddbebd4
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-08 19:59:28 +03:00
Aliaksandr Valialkin
46d7792b72
lib/promscrape: follow-up after 2c553d5a2f
...
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
2022-08-08 14:46:26 +03:00
Fury
2c553d5a2f
add support to scrape multi tenant metrics ( #2950 )
...
* add support to scrape multi tenant metrics
* add support to scrape multi tenant metrics
Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
2022-08-08 14:10:18 +03:00
Aliaksandr Valialkin
a8fee2d9b6
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-08 13:56:56 +03:00
Roman Khavronenko
d3f13ab85b
lib/promrelabel: fix expected test result ( #2957 )
...
follow-up after 68c4ec9472
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-08 13:47:29 +03:00
Aliaksandr Valialkin
68c4ec9472
lib/promrelabel: do not split regex into multiple lines if it contains groups
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928
2022-08-08 03:15:26 +03:00
Aliaksandr Valialkin
892c97e350
lib/auth: follow-up after b6a6a659f4
2022-08-07 23:14:39 +03:00
Dmytro Kozlov
b6a6a659f4
lib/auth: add tests for NewToken function ( #2921 )
...
* lib/auth: add tests from NewToken function
* lib/auth: update test, fix problem with type conversion
* lib/auth: update test description
* lib/auth: simplify failure tests
2022-08-07 23:07:57 +03:00
Aliaksandr Valialkin
9fa6b25fb2
lib/logger: prettify logging the defined command-line flags
2022-08-07 22:58:29 +03:00
Aliaksandr Valialkin
2dbbf51ea9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-07 22:49:37 +03:00
Aliaksandr Valialkin
0ef29ceb14
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name
label for role: ingress
...
See 7e65ad3e43
and 7e1111ff14
2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin
f2816ef031
lib/promscrape/discovery/ec2: properly handle custom endpoint
option in ec2_sd_configs
...
This option was ignored since d289ecded1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin
3e8890e71b
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels
...
See https://github.com/prometheus/prometheus/issues/9187
2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin
68de1f4e4a
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default
...
This aligns with Prometheus behaviour.
See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin
02de848c88
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f
2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin
83a4abda3f
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
2022-08-04 22:26:43 +03:00
Igor Tiunov
6e5ac32fba
YC service discovery ( #2923 )
...
* YC service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
* Fixed linter suggestions
* fixed golint errors
2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin
d5df08e9c2
lib/mergeset: cleanup after de6dd1cd5a
...
Remove unused getInmemoryPart and putInmemoryPart functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-08-04 18:23:01 +03:00
Aliaksandr Valialkin
cd5cc4ec81
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-04 18:04:51 +03:00
Aliaksandr Valialkin
7c99b9eaad
lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function
2022-08-04 17:42:43 +03:00
Aliaksandr Valialkin
549d430907
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-02 13:33:59 +03:00
Aliaksandr Valialkin
6b0550c023
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin
5a4c58f9a2
lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index
2022-08-02 09:12:29 +03:00
Aliaksandr Valialkin
78520f2702
lib/storage: do not compress small number of tsids when storing them in tagFiltersCache
...
This speeds up tsids retreival from the cache for 0-2 tsids
2022-07-30 00:08:51 +03:00
Aliaksandr Valialkin
de6dd1cd5a
lib/mergeset: optimize mergeInmemoryBlocks() function
...
Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs.
Just merge inmemoryBlock structs directly.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-07-27 23:58:05 +03:00
Aliaksandr Valialkin
a3f5822dc2
lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge
...
Just read the current item directly from blockStreamReader.Block.Items
with the helper method - blockStreamReader.CurrItem()
2022-07-27 23:05:02 +03:00
Aliaksandr Valialkin
be1c82beb1
benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 22:20:27 +03:00
Aliaksandr Valialkin
5c84f09762
lib/mergeset: add tests and benchmarks for commonPrefixLen function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 21:24:51 +03:00
Aliaksandr Valialkin
f5676123cc
lib/pushmetrics: make fmt
2022-07-26 20:40:19 +03:00
Aliaksandr Valialkin
da11056d85
all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency
2022-07-26 19:24:24 +03:00
Aliaksandr Valialkin
ad6b3cd47d
lib/pushmetrics: properly handle errors when initializing pushmetrics
2022-07-22 13:36:06 +03:00
Aliaksandr Valialkin
4c2f9a1a2e
lib/promscrape: set up=0
for partially failed scrape in stream parsing mode
...
This behaviour aligns with Prometheus behavior
2022-07-22 13:29:44 +03:00
Roman Khavronenko
2914ce5ca5
vmalert: remove dependency on datasource pkg from config ( #2905 )
...
* vmalert: remove dependency on datasource pkg from config
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-22 10:44:55 +02:00
Aliaksandr Valialkin
69aef55ae7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-21 21:21:04 +03:00
Aliaksandr Valialkin
4ce5875fa8
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url
2022-07-21 20:36:27 +03:00
Aliaksandr Valialkin
274145af2d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-21 20:23:41 +03:00
Aliaksandr Valialkin
46f803fa7a
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url
2022-07-21 19:49:52 +03:00
Roman Khavronenko
88edb3f6cf
vmalert: allow configuring custom headers per group ( #2901 )
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 15:59:55 +02:00
Aliaksandr Valialkin
0fd86e2364
lib/promscrape: reload all the scrape configs when the global
section is changed inside -promscrape.config
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
2022-07-18 17:15:07 +03:00
Boris Petersen
2f9668eba5
fix assume role when running in ECS. ( #2876 )
...
This fixes #2875
Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
2022-07-18 12:33:52 +03:00
Aliaksandr Valialkin
814bb1685f
all: fix other typos in the same way as 6f4d9b2a48
does
2022-07-18 12:08:15 +03:00
zhenyuxie
f3ea7823f3
fix inmemoryBlock's Less method ( #2881 )
2022-07-18 11:56:17 +03:00
Aliaksandr Valialkin
77a1af4f7f
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-14 00:56:50 +03:00
Nikolay
7301aa678c
lib/promscrape: adds azure service discovery ( #2743 )
...
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364
* makes linter happy
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* wip
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:43:18 +03:00
Aliaksandr Valialkin
c83ff99e0d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-13 18:07:46 +03:00
guidao
91faa152a5
add next retention metric ( #2863 )
...
Co-authored-by: wangfeng <wangfeng@zhihu.com>
2022-07-13 12:37:04 +03:00
Dmytro Kozlov
306ec10c39
lib/mergeset: fix linter error ( #2864 )
2022-07-13 12:31:35 +03:00
Aliaksandr Valialkin
17b5ac1608
lib/mergeset: optimize merge speed a bit
...
Use heap.Fix instead of heap.Pop + heap.Push when merging blocks
2022-07-12 12:50:26 +03:00
Aliaksandr Valialkin
5c8eee26bf
all: make fmt
via the upcoming Go1.19
2022-07-11 19:22:15 +03:00
Aliaksandr Valialkin
b42e5627fb
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-07 02:37:06 +03:00
Aliaksandr Valialkin
f97355d9fb
lib/promscrape: properly set Host header when sending requests via http proxy
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin
10cb67adb5
app/{vmagent,vminsert}: follow-up after d19e46de55
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839
2022-07-07 01:30:58 +03:00
Aliaksandr Valialkin
01f55bc66b
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice
targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin
b186b63e07
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints
and role: endpointlice
targets in the same way as Prometheus does
...
See https://github.com/prometheus/prometheus/pull/10759
2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin
0746766d95
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-06 13:31:28 +03:00
Aliaksandr Valialkin
e6ba2af7a1
lib/promscrape: fix a test after c66f676f3b
2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin
6712a8269c
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-06 13:04:08 +03:00
Aliaksandr Valialkin
c66f676f3b
lib/promscrape: push scrape_samples_limit
metric to remote storage if sample_limit
option is set in scrape_config
for this target
...
See https://github.com/VictoriaMetrics/operator/issues/497
2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin
77cbbacfdb
lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics
...
This reduces the number of args to vmselectapi calls
2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
e1b8059086
lib/vmselectapi: rename deleteMetrics to more correct deleteSeries
2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
a60e03b3a7
lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes()
...
This improves the API consistency
2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin
edc76286ac
lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index
...
Previously the time series could be put into dateMetricIDCache without
registering in the per-day inverted index if GetOrCreateTSIDByName
finds TSID entry in the global index. This could lead to missing
series in query results.
The issue has been introduced in the commit 55e7afae3a
,
which has been included in VictoriaMetrics v1.78.0
2022-07-05 14:54:03 +03:00
Aliaksandr Valialkin
855436efd2
lib/promauth: refactor NewConfig in order to improve maintainability
...
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin
44dfb2ec0d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-30 20:20:19 +03:00
Aliaksandr Valialkin
c392d6d173
app/vmagent/remotewrite: add -remoteWrite.header
command-line flag for setting additional http headers to send to -remoteWrite.url
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805
2022-06-30 20:00:23 +03:00
Aliaksandr Valialkin
e40b40afe6
Revert "lib/promscrape, vmagent: fix path to files ( #2801 )"
...
This reverts commit 0a8e35835c
.
Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005
2022-06-30 18:23:56 +03:00
Aliaksandr Valialkin
3e2dd85f7d
all: readability improvements for query traces
...
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
2022-06-30 18:20:33 +03:00
Dmytro Kozlov
0a8e35835c
lib/promscrape, vmagent: fix path to files ( #2801 )
...
vmagent: respect `-pathPrefix` flag for static files and links
2022-06-30 16:22:54 +02:00
ttyv
bdf9f4669a
lib/promscrape: fix vmagent tickerCh reload behaviour ( #2786 )
...
Co-authored-by: Dmitriy <dab@ttyv.ru>
2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin
e7b4e657a1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-28 20:22:11 +03:00
Aliaksandr Valialkin
a350d1e81c
lib/storage: return marshaled metric names from SearchMetricNames
...
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.
While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.
Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin
2c836bd398
lib/storage: put into query trace the number of found entries in SearchMetricNames
2022-06-28 14:50:53 +03:00
Aliaksandr Valialkin
cd91c29243
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-28 13:26:58 +03:00
Aliaksandr Valialkin
e578549b8a
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day
2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin
a963b2a0aa
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds
2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
ba514284f1
lib/storage: add querytracer to more contexts
...
querytracer has been added to the following storage.Storage methods:
- RegisterMetricNames
- DeleteMetrics
- SearchTagValueSuffixes
- SearchGraphitePaths
2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
134751e43e
all: locate throttled loggers via logger.WithThrottler() only once and then use them
...
This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler()
is called frequently from concurrent goroutines.
2022-06-27 13:45:50 +03:00
Aliaksandr Valialkin
8b8e547dc8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-23 19:33:36 +03:00
Aliaksandr Valialkin
52eadb729e
lib/promscrape: always send stale markers with the real scrape timestamp
...
This guarantees that query won't return data just after the series is disappeared.
2022-06-23 11:34:18 +03:00