Aliaksandr Valialkin
a18d6d5ccc
lib/promrelabel: optimize action: replace for non-trivial regex values
...
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.
Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:25:05 +03:00
Aliaksandr Valialkin
146021a076
lib/promrelabel: there is no need in calling regex.HasPrefix() after the optimization at 17289ff481
2022-09-30 10:49:18 +03:00
Aliaksandr Valialkin
899d2c40fb
lib/promrelabel: optimize action: labelmap for non-trivial regexs
2022-09-30 10:43:31 +03:00
Aliaksandr Valialkin
17289ff481
lib/regexutil: cache MatchString results for unoptimized regexps
...
This increases relabeling performance by 3x for unoptimized regexs
2022-09-30 10:41:29 +03:00
Aliaksandr Valialkin
fda60b3d4d
lib/promrelabel: properly parse regex with escaped $ at the end
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131
Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:15:43 +03:00
Aliaksandr Valialkin
593da3603e
lib/bytesutil: move InternString() from lib/promscrape/discoverytutils to lib/bytesutil
...
lib/bytesutil is more appropriate place for InternString() function
2022-09-30 07:44:35 +03:00
Nikolay
f61b8cec69
lib/awsapi: fixes sign encoding ( #3183 )
...
* lib/awsapi: fixes sign encoding
previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171
* Update lib/awsapi/sign.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:43:44 +03:00
Aliaksandr Valialkin
6a32a64073
lib/bytesutil: add FastStringTransformer and use it in the rest of the code where needed
2022-09-28 10:41:00 +03:00
Aliaksandr Valialkin
92b3622253
lib/protoparser/datadog: optimize sanitizeName() function by using result cache for input strings
...
This is a follow-up for 7c2474dac7https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin
ef435f8cc4
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
...
Optimize this function by using results cache for input strings.
Use this function all over the code.
This is a follow-up for fcffdba9dchttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:40:59 +03:00
Aliaksandr Valialkin
6411bbcce7
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
...
This should simplify further code maintenance and refactoring
This is a follow-up after 6ab1cede62
2022-09-26 17:58:01 +03:00
Dmytro Kozlov
6ab1cede62
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server ( #3109 )
...
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server
* lib/httpserver: added descriptions about tls supported versions
* lib/netutil: check minimal tls version, added supported tls versions to error
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:35:45 +03:00
Roman Khavronenko
e96ccf3f71
lib/mergeset: follow-up after a0e7432e42 ( #3145 )
...
* lib/mergeset: follow-up after a0e7432e42
2022-09-26 16:39:56 +03:00
Zakhar Bessarab
f022296d96
vmbackup: configure retries for GCS remote FS ( #3156 )
2022-09-26 16:28:20 +03:00
Aliaksandr Valialkin
41f8c2987d
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102
See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:17:25 +03:00
Aliaksandr Valialkin
7c2474dac7
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
...
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105
Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:57:23 +03:00
匠心零度
3d5509a720
lib/querytracer: fix comment ( #3135 )
2022-09-22 19:19:48 +03:00
Aliaksandr Valialkin
56ce7ce85b
lib/promscrape: typo fix after 74c00a8762
2022-09-14 15:06:50 +03:00
Aliaksandr Valialkin
74c00a8762
lib/promscrape: read response body into memory in stream parsing mode before parsing it
...
This reduces scrape duration for targets returning big responses.
The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
2022-09-14 13:15:29 +03:00
Aliaksandr Valialkin
ccad651a61
lib/promscrape/discovery/kubernetes: add more context on WatchEvent parse error
...
This should improve debugging issues with Kubernetes API server
2022-09-13 19:36:55 +03:00
Aliaksandr Valialkin
ce2c07c5a7
lib/mergeset: atomically remove part dirs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
042a532f70
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
68e32b0764
lib/storage: atomically remove parts inside partitions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin
340ada871d
lib/storage: atomically remove partitions, which went outside the configured retention
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin
978dcb4574
lib/storage: properly remove cache directory contents if reset_cache_on_startup file is located there
...
Previously the cache directory was removed. This could result in error when the cache directory
is mounted to a separate filesystem.
2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin
5f28ca1f42
lib/storage: atomically remove snapshot directories
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin
5fa9525498
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:08:09 +03:00
Bryce Lampe
74f8e12e87
Support "HTTP" and "HTTPS" schemes ( #3019 )
...
* Support "HTTP" and "HTTPS" schemes
* Update lib/promscrape/config.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2022-08-27 02:22:37 +03:00
Aliaksandr Valialkin
30b8d91727
lib/promscrape/discoveryutils: always store just allocated string to sanitized label names cache
...
This is a follow-up for c06e7a142c
2022-08-27 00:28:39 +03:00
Aliaksandr Valialkin
c06e7a142c
lib/promscrape: optimize discoveryutils.SanitizeLabelName()
...
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
2022-08-27 00:17:45 +03:00
Aliaksandr Valialkin
a2cd79576f
lib/promrelabel: call PromRegex.MatchString() on a slow path only if it contains non-empty literal prefix
...
This should improve slow path speed for regexps without literal prefixes
2022-08-26 21:48:30 +03:00
Aliaksandr Valialkin
f49c9bb700
lib/promrelabel: optimize common regex mismatch cases for action: replace and action: labelmap
2022-08-26 15:45:31 +03:00
Aliaksandr Valialkin
4c6916f32a
lib/promrelabel: use regexutil.PromRegex for regex matching in actions labeldrop,labelkeep,drop and keep
...
This makes possible optimizing additional cases inside regexutil.PromRegex
2022-08-26 15:23:45 +03:00
Aliaksandr Valialkin
7afe8450fc
lib/promrelabel: optimize matching for commonly used regex patterns in if option
...
The following regex patterns are optimized:
- literal string match, e.g. "foo"
- prefix match, e.g. "foo.*" and "foo.+"
- substring match, e.g. ".*foo.*" and ".+foo.+"
- alternate values match, e.g. "foo|bar|baz"
2022-08-26 14:53:06 +03:00
Aliaksandr Valialkin
0ad3bbadd3
lib/regexutil: add Simplify() function for simplifying the regular expression
2022-08-26 11:57:12 +03:00
Aliaksandr Valialkin
b373661988
lib/promrelabel: optimize action: {drop,keep,labeldrop,labelkeep} with anchored regex prefix
...
The following commonly used relabeling rules must work faster now:
- action: labeldrop
regex: "^foo.+$"
- action: labeldrop
regex: "^bar.*"
2022-08-25 23:23:55 +03:00
Aliaksandr Valialkin
0d4ea03a73
lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values
...
For example, the following relabeling rule must work much faster now:
- action: labeldrop
regex: "foo|bar|baz"
2022-08-24 17:54:29 +03:00
Aliaksandr Valialkin
0d46e24af5
lib/storage: increase the maximum possible or values extracted from regexp from 20 to 100
...
This should improve time series search speed for regexp filters with big number of `or` values.
2022-08-24 17:15:25 +03:00
Aliaksandr Valialkin
fdbf5b5795
lib/storage: ignore start text and end text anchors in getOrValues(regexp) function
...
This is OK, since the anchors are implicitly applied to the whole regexp.
This optimization should improve the speed for regexp series filters with explicit $ and ^ anchors.
For example, `{label="^(foo|bar)$"}`
2022-08-24 17:12:52 +03:00
Aliaksandr Valialkin
796aa310c2
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
...
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:
vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:44:04 +03:00
Aliaksandr Valialkin
1f89278d88
all: subsitute ioutil.ReadAll with io.ReadAll
...
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.
This is a follow-up for 02ca2342ab
2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin
2c3a89339d
all: use os.ReadDir instead of ioutil.ReadDir
...
The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir
This is a follow-up for 02ca2342ab
2022-08-22 00:02:25 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:52:35 +03:00
Roman Khavronenko
d59d829cdb
lib/storage: bump max merge concurrency for small parts to 15 ( #2997 )
...
* lib/storage: bump max merge concurrency for small parts to 15
The change is based on the feedback from users on github.
Thier examples show, that limit of 8 sometimes become a
bottleneck. Users report that without limit concurrency
can climb up to 15-20 merges at once.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Update lib/storage/partition.go
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-21 23:32:08 +03:00
Aliaksandr Valialkin
8550c44e31
app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling
...
For example:
- target_label: composite-label
replacement: "{{source_label1}}-{{source_label2}}"
2022-08-21 22:50:01 +03:00
Roman Khavronenko
31f922944e
lib/storage: fix the search for empty label name ( #2991 )
...
* lib/storage: fix the search for empty label name
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* Apply suggestions from code review
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-17 21:32:25 +03:00
Aliaksandr Valialkin
7d26414b2e
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit
...
The following metrics are generated:
- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped
These metrics simplify alerting on targets, which expose too many time series
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
2022-08-17 13:19:33 +03:00
Aliaksandr Valialkin
bb68ab99fa
lib/promscrape: retry http requests if the server returns 429 status code
...
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-16 15:01:08 +03:00
Aliaksandr Valialkin
b0e1bb517e
lib/storage: typo fix in comments after f830edc0bc
2022-08-16 13:44:45 +03:00
Aliaksandr Valialkin
f830edc0bc
lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
2022-08-16 13:32:40 +03:00
