Dmytro Kozlov
b6a6a659f4
lib/auth: add tests for NewToken function ( #2921 )
...
* lib/auth: add tests from NewToken function
* lib/auth: update test, fix problem with type conversion
* lib/auth: update test description
* lib/auth: simplify failure tests
2022-08-07 23:07:57 +03:00
Aliaksandr Valialkin
9fa6b25fb2
lib/logger: prettify logging the defined command-line flags
2022-08-07 22:58:29 +03:00
Aliaksandr Valialkin
2dbbf51ea9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-07 22:49:37 +03:00
Aliaksandr Valialkin
0ef29ceb14
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name
label for role: ingress
...
See 7e65ad3e43
and 7e1111ff14
2022-08-05 20:55:00 +03:00
Aliaksandr Valialkin
f2816ef031
lib/promscrape/discovery/ec2: properly handle custom endpoint
option in ec2_sd_configs
...
This option was ignored since d289ecded1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
2022-08-05 18:50:02 +03:00
Aliaksandr Valialkin
3e8890e71b
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels
...
See https://github.com/prometheus/prometheus/issues/9187
2022-08-05 16:11:28 +03:00
Aliaksandr Valialkin
68de1f4e4a
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default
...
This aligns with Prometheus behaviour.
See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
2022-08-05 14:41:40 +03:00
Aliaksandr Valialkin
02de848c88
lib/promscrape/discovery/yandexcloud: further code cleanup after 83a4abda3f
2022-08-05 10:30:47 +03:00
Aliaksandr Valialkin
83a4abda3f
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
2022-08-04 22:26:43 +03:00
Igor Tiunov
6e5ac32fba
YC service discovery ( #2923 )
...
* YC service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
* Fixed linter suggestions
* fixed golint errors
2022-08-04 20:44:16 +03:00
Aliaksandr Valialkin
d5df08e9c2
lib/mergeset: cleanup after de6dd1cd5a
...
Remove unused getInmemoryPart and putInmemoryPart functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-08-04 18:23:01 +03:00
Aliaksandr Valialkin
cd5cc4ec81
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-04 18:04:51 +03:00
Aliaksandr Valialkin
7c99b9eaad
lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function
2022-08-04 17:42:43 +03:00
Aliaksandr Valialkin
549d430907
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-08-02 13:33:59 +03:00
Aliaksandr Valialkin
6b0550c023
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 12:59:07 +03:00
Aliaksandr Valialkin
5a4c58f9a2
lib/storage: explain why the GetOrCreateTSIDByName function doesnt check whether the per-day entry for the given date exists if TSID is found in global index
2022-08-02 09:12:29 +03:00
Aliaksandr Valialkin
78520f2702
lib/storage: do not compress small number of tsids when storing them in tagFiltersCache
...
This speeds up tsids retreival from the cache for 0-2 tsids
2022-07-30 00:08:51 +03:00
Aliaksandr Valialkin
de6dd1cd5a
lib/mergeset: optimize mergeInmemoryBlocks() function
...
Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs.
Just merge inmemoryBlock structs directly.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-07-27 23:58:05 +03:00
Aliaksandr Valialkin
a3f5822dc2
lib/mergeset: do not update blockStreamReader.bh.firstItem during the merge
...
Just read the current item directly from blockStreamReader.Block.Items
with the helper method - blockStreamReader.CurrItem()
2022-07-27 23:05:02 +03:00
Aliaksandr Valialkin
be1c82beb1
benchmark inmemoryBlock.{Marshal,Unmarshal} for different prefix length
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 22:20:27 +03:00
Aliaksandr Valialkin
5c84f09762
lib/mergeset: add tests and benchmarks for commonPrefixLen function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2254
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2913
2022-07-27 21:24:51 +03:00
Aliaksandr Valialkin
f5676123cc
lib/pushmetrics: make fmt
2022-07-26 20:40:19 +03:00
Aliaksandr Valialkin
da11056d85
all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency
2022-07-26 19:24:24 +03:00
Aliaksandr Valialkin
ad6b3cd47d
lib/pushmetrics: properly handle errors when initializing pushmetrics
2022-07-22 13:36:06 +03:00
Aliaksandr Valialkin
4c2f9a1a2e
lib/promscrape: set up=0
for partially failed scrape in stream parsing mode
...
This behaviour aligns with Prometheus behavior
2022-07-22 13:29:44 +03:00
Roman Khavronenko
2914ce5ca5
vmalert: remove dependency on datasource pkg from config ( #2905 )
...
* vmalert: remove dependency on datasource pkg from config
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-22 10:44:55 +02:00
Aliaksandr Valialkin
69aef55ae7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-21 21:21:04 +03:00
Aliaksandr Valialkin
4ce5875fa8
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url
2022-07-21 20:36:27 +03:00
Aliaksandr Valialkin
274145af2d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-21 20:23:41 +03:00
Aliaksandr Valialkin
46f803fa7a
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url
2022-07-21 19:49:52 +03:00
Roman Khavronenko
88edb3f6cf
vmalert: allow configuring custom headers per group ( #2901 )
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 15:59:55 +02:00
Aliaksandr Valialkin
0fd86e2364
lib/promscrape: reload all the scrape configs when the global
section is changed inside -promscrape.config
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
2022-07-18 17:15:07 +03:00
Boris Petersen
2f9668eba5
fix assume role when running in ECS. ( #2876 )
...
This fixes #2875
Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
2022-07-18 12:33:52 +03:00
Aliaksandr Valialkin
814bb1685f
all: fix other typos in the same way as 6f4d9b2a48
does
2022-07-18 12:08:15 +03:00
zhenyuxie
f3ea7823f3
fix inmemoryBlock's Less method ( #2881 )
2022-07-18 11:56:17 +03:00
Aliaksandr Valialkin
77a1af4f7f
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-14 00:56:50 +03:00
Nikolay
7301aa678c
lib/promscrape: adds azure service discovery ( #2743 )
...
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364
* makes linter happy
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* wip
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:43:18 +03:00
Aliaksandr Valialkin
c83ff99e0d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-13 18:07:46 +03:00
guidao
91faa152a5
add next retention metric ( #2863 )
...
Co-authored-by: wangfeng <wangfeng@zhihu.com>
2022-07-13 12:37:04 +03:00
Dmytro Kozlov
306ec10c39
lib/mergeset: fix linter error ( #2864 )
2022-07-13 12:31:35 +03:00
Aliaksandr Valialkin
17b5ac1608
lib/mergeset: optimize merge speed a bit
...
Use heap.Fix instead of heap.Pop + heap.Push when merging blocks
2022-07-12 12:50:26 +03:00
Aliaksandr Valialkin
5c8eee26bf
all: make fmt
via the upcoming Go1.19
2022-07-11 19:22:15 +03:00
Aliaksandr Valialkin
b42e5627fb
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-07 02:37:06 +03:00
Aliaksandr Valialkin
f97355d9fb
lib/promscrape: properly set Host header when sending requests via http proxy
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
2022-07-07 02:27:52 +03:00
Aliaksandr Valialkin
10cb67adb5
app/{vmagent,vminsert}: follow-up after d19e46de55
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839
2022-07-07 01:30:58 +03:00
Aliaksandr Valialkin
01f55bc66b
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice
targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
2022-07-07 00:32:26 +03:00
Aliaksandr Valialkin
b186b63e07
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints
and role: endpointlice
targets in the same way as Prometheus does
...
See https://github.com/prometheus/prometheus/pull/10759
2022-07-06 23:18:59 +03:00
Aliaksandr Valialkin
0746766d95
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-06 13:31:28 +03:00
Aliaksandr Valialkin
e6ba2af7a1
lib/promscrape: fix a test after c66f676f3b
2022-07-06 13:26:35 +03:00
Aliaksandr Valialkin
6712a8269c
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-07-06 13:04:08 +03:00
Aliaksandr Valialkin
c66f676f3b
lib/promscrape: push scrape_samples_limit
metric to remote storage if sample_limit
option is set in scrape_config
for this target
...
See https://github.com/VictoriaMetrics/operator/issues/497
2022-07-06 12:37:55 +03:00
Aliaksandr Valialkin
77cbbacfdb
lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics
...
This reduces the number of args to vmselectapi calls
2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
e1b8059086
lib/vmselectapi: rename deleteMetrics to more correct deleteSeries
2022-07-06 12:37:54 +03:00
Aliaksandr Valialkin
a60e03b3a7
lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes()
...
This improves the API consistency
2022-07-06 12:37:53 +03:00
Aliaksandr Valialkin
edc76286ac
lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index
...
Previously the time series could be put into dateMetricIDCache without
registering in the per-day inverted index if GetOrCreateTSIDByName
finds TSID entry in the global index. This could lead to missing
series in query results.
The issue has been introduced in the commit 55e7afae3a
,
which has been included in VictoriaMetrics v1.78.0
2022-07-05 14:54:03 +03:00
Aliaksandr Valialkin
855436efd2
lib/promauth: refactor NewConfig in order to improve maintainability
...
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
2022-07-04 14:31:12 +03:00
Aliaksandr Valialkin
44dfb2ec0d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-30 20:20:19 +03:00
Aliaksandr Valialkin
c392d6d173
app/vmagent/remotewrite: add -remoteWrite.header
command-line flag for setting additional http headers to send to -remoteWrite.url
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805
2022-06-30 20:00:23 +03:00
Aliaksandr Valialkin
e40b40afe6
Revert "lib/promscrape, vmagent: fix path to files ( #2801 )"
...
This reverts commit 0a8e35835c
.
Reason for revert: it incorrectly fixes the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2799#issuecomment-1171392005
2022-06-30 18:23:56 +03:00
Aliaksandr Valialkin
3e2dd85f7d
all: readability improvements for query traces
...
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
2022-06-30 18:20:33 +03:00
Dmytro Kozlov
0a8e35835c
lib/promscrape, vmagent: fix path to files ( #2801 )
...
vmagent: respect `-pathPrefix` flag for static files and links
2022-06-30 16:22:54 +02:00
ttyv
bdf9f4669a
lib/promscrape: fix vmagent tickerCh reload behaviour ( #2786 )
...
Co-authored-by: Dmitriy <dab@ttyv.ru>
2022-06-30 12:33:01 +02:00
Aliaksandr Valialkin
e7b4e657a1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-28 20:22:11 +03:00
Aliaksandr Valialkin
a350d1e81c
lib/storage: return marshaled metric names from SearchMetricNames
...
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.
While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.
Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
2022-06-28 18:17:15 +03:00
Aliaksandr Valialkin
2c836bd398
lib/storage: put into query trace the number of found entries in SearchMetricNames
2022-06-28 14:50:53 +03:00
Aliaksandr Valialkin
cd91c29243
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-28 13:26:58 +03:00
Aliaksandr Valialkin
e578549b8a
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day
2022-06-28 13:02:47 +03:00
Aliaksandr Valialkin
a963b2a0aa
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds
2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
ba514284f1
lib/storage: add querytracer to more contexts
...
querytracer has been added to the following storage.Storage methods:
- RegisterMetricNames
- DeleteMetrics
- SearchTagValueSuffixes
- SearchGraphitePaths
2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
134751e43e
all: locate throttled loggers via logger.WithThrottler() only once and then use them
...
This reduces the contention on logThrottlerRegistryMu mutex when logger.WithThrottler()
is called frequently from concurrent goroutines.
2022-06-27 13:45:50 +03:00
Aliaksandr Valialkin
8b8e547dc8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-23 19:33:36 +03:00
Aliaksandr Valialkin
52eadb729e
lib/promscrape: always send stale markers with the real scrape timestamp
...
This guarantees that query won't return data just after the series is disappeared.
2022-06-23 11:34:18 +03:00
Aliaksandr Valialkin
1c4f67c5d2
lib/promauth: add ability to send additional http headers in requests to scrape targets
...
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
2022-06-22 20:39:43 +03:00
Aliaksandr Valialkin
34a6b1fa3b
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-22 13:20:58 +03:00
Aliaksandr Valialkin
e6ed92529b
all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package
...
This package already has the same name, so there is no need in explicit name
2022-06-21 20:23:32 +03:00
Loki's Wager
ac411be904
BugFix part_header.go ( #2763 )
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2757
Co-authored-by: haotingyi <haotingyi@corp.netease.com>
2022-06-21 15:56:41 +03:00
Aliaksandr Valialkin
1b24afec36
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-20 17:42:27 +03:00
Aliaksandr Valialkin
49586566a3
docs: follow-up after e4d6b750f6
2022-06-20 17:14:43 +03:00
Nikolay
e4d6b750f6
lib/httpserver: adds flagsAuthKey command-line flag ( #2758 )
...
* lib/httpserver: adds flagsAuthKey command-line flag
It protects /flags endpoint with authKey.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-20 17:09:32 +03:00
Aliaksandr Valialkin
b958fc7846
lib/storage: properly take into account already registered series when -storage.maxHourlySeries
or -storage.maxDailySeries
limits are enabled
...
The commit 5fb45173ae
takes into account only newly registered series
when applying cardinality limits. This means that the cardinality limit could be exceeded with already registered series.
This commit returns back accounting for already registered series when applying cardinality limits.
2022-06-20 13:47:47 +03:00
Aliaksandr Valialkin
cacd3d6f6d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-19 23:05:31 +03:00
Aliaksandr Valialkin
55e7afae3a
lib/storage: create per-day indexes together with global indexes when registering new time series
...
Previously the creation of per-day indexes and global indexes
for the newly registered time series was decoupled.
Now global indexes and per-day indexes for the current day are created toghether for new time series.
This should speed up registering new time series a bit.
2022-06-19 22:42:10 +03:00
Aliaksandr Valialkin
5fb45173ae
lib/storage: do not register new series if -storage.maxHourlySeries
or -storage.maxDailySeries
limits are exceeded
...
Previously samples for new series weren't added as expected when series limits were reached,
but new series were still registered in indexdb.
2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin
62e2371a67
lib/storage: reset metric id caches for the previous and the current hour
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698
2022-06-19 22:42:09 +03:00
Aliaksandr Valialkin
c18f8cccfa
lib/promrelabel: support action: graphite
relabeling
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737
2022-06-16 20:24:22 +03:00
Aliaksandr Valialkin
8632b8200e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-15 18:42:09 +03:00
Aliaksandr Valialkin
ec7963208d
app/vmselect: accept focusLabel
query arg at /api/v1/status/tsdb
...
This allows filling the seriesCountByFocusLabelValue list in the /api/v1/status/tsdb response
with label values for the specified focusLabel, which contain the highest number of time series.
TODO: add this to Cardinality explorer at VMUI - https://docs.victoriametrics.com/#cardinality-explorer
2022-06-14 18:36:54 +03:00
Aliaksandr Valialkin
b6c1ca12b7
lib/storage: show top labels with the highest number of series in cardinality explorer
2022-06-14 16:32:38 +03:00
Aliaksandr Valialkin
0445ad59db
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-14 13:29:48 +03:00
Aliaksandr Valialkin
a75e59700f
lib/storage: improve error message when -search.max* command-line flag values are exceeded
2022-06-14 13:27:59 +03:00
Aliaksandr Valialkin
f7b52b64a3
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-14 12:26:32 +03:00
Aliaksandr Valialkin
52cf05c6d2
lib/storage: test GetTSDBStatusWithFiltersForDate on a global time range
2022-06-12 14:27:40 +03:00
Aliaksandr Valialkin
374beb350e
app/vmselect: optimize /api/v1/labels
and /api/v1/label/.../values
handlers when match[]
query arg is passed to them
2022-06-12 04:32:13 +03:00
Aliaksandr Valialkin
2bcb960f17
all: improve query tracing coverage for indexdb search
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
2022-06-09 20:07:07 +03:00
Aliaksandr Valialkin
7fc62feddc
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-09 13:33:07 +03:00
Howie
76f05f8670
feat: rule limit ( #2676 )
...
vmalert: support `limit` param in groups definition
`limit` param limits number of time series samples produced by a single rule
during execution.
On reaching the limit rule will return an err.
Signed-off-by: lihaowei <haoweili35@gmail.com>
2022-06-09 08:21:30 +02:00
Aliaksandr Valialkin
12ac255dae
lib/querytracer: make it easier to use by passing trace context message to New and NewChild
...
The context message can be extended by calling Donef.
If there is no need to extend the message, then just call Done.
2022-06-08 21:06:52 +03:00
Dmytro Kozlov
018d2303c4
Cardinality explorer ( #2625 )
...
* Cardinality explorer
* vmui, vmselect: updated field name, added description to spinner
* make vmui-update
* updated const name, make vmui-update
* lib/storage: changes calculation for totalSeries values
* added static files
* wip
* wip
* wip
* wip
* docs/CHANGELOG.md: document cardinality explorer feature
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-08 18:43:05 +03:00
Roman Khavronenko
63b538ecd1
vmagent: update SD duration histogram metric if SD is active ( #2677 )
...
The change updates histogram for registering SD update duration
only SD is considered as `active`. SD is active if at least
one scraper for this SD has started.
This change supposed to reduce metrics cardinality produced
by duration histogram which gets updated even if SD isn't configured.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2671
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-07 15:46:44 +03:00
Roman Khavronenko
1ee1e986da
lib/storage: limit max mergeConcurrency value for systems with high number of CPUs ( #2673 )
...
Workers count for merges affects the max part size during merges. Such behaviour
protects storage from running out of disk space for scenario when all workers
are merging parts with the max size.
This works very well for most cases. But for systems where high number of CPUs
is allocated for vmstorage components this could significantly impact the max
part size and result in more unmerged parts than expected.
While checking multiple production highly loaded setups it was discovered that
`max_over_time(vm_active_merges{type="storage/big}[1h]}"` rarely exceeds 2,
and `max_over_time(vm_active_merges{type="storage/small}[1h]}"` rarely exceeds 4.
The change in this commit limits the max value for concurrency accordingly.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-07 14:55:09 +03:00
Aliaksandr Valialkin
a5814fe16a
lib/promscrape/discovery/kubernetes: use unsupportedFieldError() function instead of errContext string
...
This improves code readability and maintainability a bit, since the format string
is passed as string literal into fmt.Errorf.
2022-06-07 01:22:07 +03:00
Aliaksandr Valialkin
8608dd093c
all: follow-up after 8edb390e21
...
- Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb
- Add /service-discovery page for API compatibility with Prometheus
- Properly load bootstrap.min.css from /prometheus/targets
- Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there
2022-06-07 00:57:09 +03:00
Aliaksandr Valialkin
6f0a0e3072
lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534
...
- make more clear error logs
- simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct
2022-06-06 14:40:52 +03:00
Aliaksandr Valialkin
0ea0168d98
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-06-04 01:13:48 +03:00
Aliaksandr Valialkin
cfefdde042
lib/promauth: follow-up after 006b8c7534
...
- Take into account `ca`, `key` and `cert` values when generating string representation of TLSConfig.
Print hashes instead of real values because of security considerations.
- Properly update Config.tlsCertDigets when `key` and `cert` values are set.
This allows properly updating scrape targets after these values are updated in configs.
- Do not re-generate certificate from `key` and `cert` values per each call to getTLSCert,
because these values are immutable.
- Do not set `ca` value from `ca_file` value, so it isn't exposed at `/config` page.
- Generate proper error messages on incorrect `key`, `cert` or `ca` values.
2022-06-04 01:01:16 +03:00
Aliaksandr Valialkin
0922ed2b7e
lib/promscrape: add -promscrape.cluster.name
command-line flag
...
This flag is used for proper data de-duplication when the same target is scraped
from multiple vmagent clusters.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679
2022-06-04 00:37:01 +03:00
Dmytro Kozlov
8edb390e21
lib/promscrape: adds service discovery visualization for /targets page( #2675 )
...
* lib/promscrape: updated template
* lib/promscrape: fixed click on unhealthy and all btns
* app/vmselect: jquery scripts into static folder
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2022-06-03 15:38:45 +02:00
Nikolay
a18914abee
lib/promscrape/discovery/kubernetes: follow-up after 0b5c874911
( #2672 )
2022-06-01 20:44:45 +02:00
hadesy
006b8c7534
promscrape/discovery: support kubeconfig ( #2533 )
2022-06-01 20:34:00 +02:00
Aliaksandr Valialkin
ca689fec54
docs/CHANGELOG.md: follow-up after 2177089f94
2022-06-01 14:51:26 +03:00
Aliaksandr Valialkin
ea06d2fd3c
lib/storage: stop background merge when storage enters read-only mode
...
This should prevent from `no space left on device` errors when VictoriaMetrics
under-estimates the additional disk space needed for background merge.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2603
2022-06-01 14:36:45 +03:00
Roman Khavronenko
642eb1c534
lib/storage: make indexdb/tagFilters
cache size configurable ( #2667 )
...
The default size of `indexdb/tagFilters` now can be overridden via
`storage.cacheSizeIndexDBTagFilters` flag.
Please, be careful with changing default size since it may
lead to inefficient work of the vmstorage or OOM exceptions.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2663
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2022-06-01 10:07:53 +02:00
Roman Khavronenko
2177089f94
promrelabel: add support of lowercase
and uppercase
relabeling actions ( #2665 )
...
* promrelabel: add support of `lowercase` and `uppercase` relabeling actions
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2664
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* lib/storage: make golangci-lint happy
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2022-06-01 10:02:37 +02:00
Aliaksandr Valialkin
41958ed5dd
all: add initial support for query tracing
...
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin
d2567ccdd6
lib/promscrape: use strconv.Atoi instead of strconv.ParseInt for parsing -promscrape.cluster.memberNum
...
In this case there is no need in converting int64 to int
2022-06-01 01:42:34 +03:00
Aliaksandr Valialkin
a1add5c2c7
lib/storage: make fmt
2022-05-31 12:54:37 +03:00
Aliaksandr Valialkin
bac75ea8a2
lib/storage: do not take into account series from the next day when match[]
filter is passed to /api/v1/status/tsdb
2022-05-31 12:15:26 +03:00
Dmytro Kozlov
11f91532c5
issue-2594: use embedded for static files ( #2650 )
...
embed static js and css files from CDN into vmalert, vmagent and vmsingle binaries.
Co-authored-by: f41gh7 <nik@victoriametrics.com>
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2594
2022-05-31 01:55:28 +02:00
Dmytro Kozlov
1eb29794e6
removed redundant return (fixed linter) ( #2647 )
...
* removed redundant return
* updated lint package version
2022-05-26 16:24:01 +02:00
Aliaksandr Valialkin
796804e4b0
lib/promscrape: add -promscrape.suppressScrapeErrorsDelay command-line flag
...
This flag can be used for reducing the amounts of logs when scraping unreliable scrape targets.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575
The patch is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576 .
Thanks to @jelmd .
2022-05-25 22:59:36 +03:00
Aliaksandr Valialkin
f6d11a49aa
lib/storage: add ability to change the indexdb rotation time offset with -retentionTimezoneOffset command-line flag
...
This is a follow-up for 0fbf59199a
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2574
2022-05-25 16:05:29 +03:00
阳明
0fbf59199a
lib/storage: Remove the effect of time zone on next retention period ( #2568 ) ( #2574 )
2022-05-25 15:08:24 +03:00
Aliaksandr Valialkin
993ecbb141
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-21 02:27:04 +03:00
Roman Khavronenko
d5eb6afe26
lib/promscrape/discovery/kubernetes: fixes kubernetes service discovery ( #2615 )
...
* lib/promscrape/discovery/kubernetes: properly updates discovered scrape works
previously, added or updated scrapeworks may override previuosly
discovered.
it happens because swosByKey may contain small subset of kubernetes
objects with it's labels.
It happens for objectsUpdated and objectsAdded maps, which include only changed elements
* Properly calculate vm_promscrape_discovery_kubernetes_scrape_works
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-21 01:01:37 +03:00
Aliaksandr Valialkin
5a60387eea
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-20 14:34:02 +03:00
Boris Petersen
3df8caca15
Add ability to sign requests for all AWS services ( #2604 )
...
This adds the ability to utilize sigv4 signing for all AWS services not
just "aps". When the newly introduced property "service" is not set it
will default to "aps".
Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
2022-05-18 14:58:31 +02:00
Aliaksandr Valialkin
a0727ab1b1
docs/vmagent.md: typo fix in the description for -promscrape.cluster.replicationFactor
command-line flag
2022-05-12 18:50:29 +03:00
Aliaksandr Valialkin
9ea3f0c0d3
lib/awsapi: remove whitelist arg from GetFiltersQueryString(), since it may break new filters in the future
...
Let users decide which filters to use. If users start using disallowed filters, then AWS will return an error.
2022-05-09 15:33:22 +03:00
Aliaksandr Valialkin
2685992ca9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-07 02:02:31 +03:00
Aliaksandr Valialkin
123aa4c79e
lib/promscrape: properly implement ScrapeConfig.clone()
...
Previously ScrapeConfig.clone() was improperly copying promauth.Secret fields -
their contents was replaced with `<secret>` value.
This led to inability to use passwords and secrets in `-promscrape.config` file.
The bug has been introduced in v1.77.0 in the commit 67b10896d2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2551
2022-05-07 00:05:40 +03:00
Aliaksandr Valialkin
1dc4cc243b
lib/promscrape: rename promscrape_stale_samples_created_total
metric to vm_promscrape_stale_samples_created_total
, so its name is consistent with the rest of vm_promscrape_
metrics
2022-05-06 15:33:13 +03:00
Aliaksandr Valialkin
d5b55fe22d
lib/promscrape/discovery/ec2: add ability to filter Availability Zones in ec2_sd_config
via az_filters
section
2022-05-06 12:43:29 +03:00
Aliaksandr Valialkin
ee63748753
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-05 11:01:55 +03:00
Aliaksandr Valialkin
97f9c2f667
lib/promscrape/discovery/ec2: properly pass filters to DescribeAvailabilityZones API call
...
Previously filters wheren't passed to this call after the commit 0e09fdb8b0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
2022-05-05 11:00:23 +03:00
Aliaksandr Valialkin
620b0d11b7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-05 10:33:08 +03:00
Aliaksandr Valialkin
d285c2fea7
lib/awsapi: pass filtersQueryString
arg to GetEC2APIResponse() function, so the caller could decide whether to use the filters during the AWS API query
...
The filters shouldn't be passed to DescribeAvailabilityZones API call.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
Related commits:
0e09fdb8b0
d289ecded1
2022-05-05 10:29:34 +03:00
Aliaksandr Valialkin
9eb61e67af
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-05 00:02:14 +03:00
Dmytro Kozlov
7dd9f3b98e
{vmbackup, vmbackup/snapshot}: fixed problem with snapshot backup in another snapshot folder ( #2535 )
...
* {vmbackup, vmbackup/snapshot}: validate snapshot name
* vmbackup/snapshot: added another checks
* backup/actions: added check that we ignore backup_complete.ignore file
* vmbackup: moved snapshot to lib directory
* lib/snapshot: added functions description
* lib/snapshot: fixed typo
* vmbackup: code cleanup
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-04 22:12:03 +03:00
Nikolay
d289ecded1
{lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite ( #2458 )
...
* {lib/promscrape,app/vmagent}: adds sigv4 support for vmagent remoteWrite
moves aws related code into separate lib from lib/promscrape
it allows to write data from vmagent to the AWS managed prometheus (cortex)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
* Apply suggestions from code review
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-04 20:24:19 +03:00
Nikolay
3575aabeaf
lib/promscrape: adds correct http status codes for redirect ( #2530 )
...
standard http client accepts multiple http status codes as redirect
it should fix issue with incorrect redirects
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2482
2022-05-03 13:31:31 +03:00
Aliaksandr Valialkin
a7333a7380
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-05-02 22:06:17 +03:00
Aliaksandr Valialkin
0d86644d65
lib/storage: leave the last sample per each discrete interval during the deduplicaton
...
This aligns better with staleness logic in Prometheus - https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness
2022-05-02 21:50:45 +03:00
Artem Navoiev
37cf509c3a
lib/{storage,flagutil} - Add option for snapshot autoremoval ( #2487 )
...
* lib/{storage,flagutil} - Add option for snapshot autoremoval
- add prometheus-like duration as command flag
- add option to delete stale snapshots
- update duration.go flag to re-use own code
* wip
* lib/flagutil: re-use Duration.Set() call in NewDuration
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-05-02 11:00:15 +03:00
Aliaksandr Valialkin
20bc2a2c44
lib/flagutil: re-use Duration.Set() call in NewDuration
2022-05-02 10:56:39 +03:00
Aliaksandr Valialkin
d713bdec20
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2022-04-29 14:23:04 +03:00
Dima Lazerka
84683b8569
Fix targetstatus qtpl paths ( #2517 )
...
Ran `make quicktemplate-gen` from the root directory
2022-04-29 10:36:03 +03:00
Aliaksandr Valialkin
6be07e8c25
lib/promscrape/discovery/kubernetes: do not drop pod meta-labels even if the corresponding node objects are missing
...
This reflects the logic used in Prometheus.
See https://github.com/prometheus/prometheus/pull/10080
2022-04-26 15:26:01 +03:00
Aliaksandr Valialkin
9fe1bf5d53
lib/promauth: take into account tls_config and proxy_url when serializing OAuth2Config to string
2022-04-23 00:23:19 +03:00
Aliaksandr Valialkin
eb5d7ad089
lib/promauth: add support for min_version
option at tls_config
section in the same way as Prometheus does
2022-04-23 00:16:39 +03:00
Aliaksandr Valialkin
174431e31b
lib/promauth: add support for proxy_url
option at oauth2
section in the same way as Prometheus does
2022-04-23 00:00:44 +03:00