Commit graph

2610 commits

Author SHA1 Message Date
Yury Molodov
00a0816ab1
vmui: predefined dashboards docs (#3895)
* fix: correct display predefined panels

* docs: update the documentation for predefined dashboards
2023-03-12 17:16:26 -07:00
Aliaksandr Valialkin
0af9e2b693
app/vmselect/promql: prevent from SIGBUS crash on architecures, which deny unaligned access to 8-byte words (e.g. ARM)
Thanks to @oliverpool for nailing down the root cause of the issue and for the initial attempt to fix it
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927
2023-03-12 16:32:08 -07:00
Yury Molodov
01367faa39
vmui: remove send step param for instant queries (#3931)
* fix: remove step param for instant queries (#3896)

* vmui: remove send step param for instant queries

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-12 03:09:56 -07:00
Aliaksandr Valialkin
b5db69fe05
app/vmselect/netstorage: do not intern string representation of MetricName for time series received from vmstorage
It has been appeared that this interning may lead to increased memory usage and increased CPU usage
when vmselect performs queries, which select big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863
2023-03-12 00:52:35 -08:00
Aliaksandr Valialkin
d95037a175
app/vmctl/README.md: remove trailing space from the line added at 4c3bc04efa 2023-03-12 00:11:51 -08:00
Zakhar Bessarab
f7a7eb8f3e
docs: add a note about cache reset for vmalert backfilling docs (#3940)
docs: add a note about cache reset for vmalert backfilling docs
2023-03-10 13:45:11 +01:00
Dmytro Kozlov
4c3bc04efa
app/vmctl: update importing tips when migrating data with overlapping time range (#3941)
app/vmctl: update importing tips when migrating data with overlapping time range
2023-03-10 11:29:08 +01:00
Dmytro Kozlov
3c9058c168
app/vmctl: add support of basic auth and barer token (#3921)
app/vmctl: add support of basic auth and bearer token
2023-03-09 14:53:29 +01:00
Roman Khavronenko
d66bae212b
app/vmalert: log number of configration files found for each specified -rule (#3936)
The change also introduces `List` method to `FS` interface.
The `List` method can be used for wildcard support in object storage FS.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-03-09 14:46:19 +01:00
Dmytro Kozlov
7f54c181bb
app/vmctl: follow up after 09e3742a82 (#3937)
app/vmctl: follow up after 09e3742a82
2023-03-09 13:28:55 +01:00
Roman Khavronenko
3de7fc5c71
security: bump go version to 1.20.2 (#3935)
upgrade Go builder from Go1.20.1 to Go1.20.2
See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-09 13:20:54 +01:00
Gowtam Lal
09e3742a82
app/vmctl: Allow vmnative exports to skip HTTP keepalive. (#3909)
app/vmctl: support HTTP keepalive disabling for vm-native mode
2023-03-09 09:47:46 +01:00
Aliaksandr Valialkin
1b5dc9f91d
all: follow-up for 7a3e16e774
- Sync the description for -httpListenAddr.useProxyProtocol command-line flag at vmagent and vmauth,
  so it is consistent with the description at vmauth and victoria-metrics
- Add a sample of panic text to docs/CHANGELOG.md, so it could be googled
- Mention the -httpListenAddr.useProxyProtocol command-line flag in the description for the bugfix

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:26:55 -08:00
Aliaksandr Valialkin
05709bdfae
app/vmselect/vmui: make vmui-update after bbf8e459a0 2023-03-08 01:15:52 -08:00
Aliaksandr Valialkin
138757629b
app/vmctl/README.md: remove trailing space after cc5b916237 2023-03-08 00:32:00 -08:00
Aliaksandr Valialkin
9e71462cee
all: typo fixes of the same type as in the d056be710b 2023-03-08 00:30:21 -08:00
Aliaksandr Valialkin
1bb529e23e
app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c
- Document the fix
- Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient()
  This simplifies the fix to the following diff:

diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go
index 099899c19..70b904af4 100644
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
        }
        c.sendBlock = c.sendBlockHTTP

-       return c
-}
-
-func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        useVMProto := forceVMProto.GetOptionalArg(argIdx)
        usePromProto := forcePromProto.GetOptionalArg(argIdx)
        if useVMProto && usePromProto {
@@ -173,6 +169,10 @@ func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        }
        c.useVMProto = useVMProto

+       return c
+}
+
+func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
2023-03-07 23:54:24 -08:00
Dmytro Kozlov
66bf1987bf
app/vmagent: fix panic if auth config not defined (#530) 2023-03-07 23:51:30 -08:00
Nikolay
7a3e16e774
lib/netutil: fixes panic at proxy protocol (#3905)
it may occur if non proxy protocol message received by tcp server.
Listener Accept method must return only non-recoverable errors.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-07 08:50:18 -08:00
Yury Molodov
bbf8e459a0
vmui: fix display of selected value in the selector (#3919)
vmui: fix selected value in dropdowns for Explore page
2023-03-07 16:23:02 +01:00
Roman Khavronenko
2472baa934
app/vmalert: do not wait for group start on removal (#3891)
Each group in vmalert starts with an artifical delay to avoid
thundering herd problem. For some groups with high evaluation
intervals, the delay could be significant.
If during this delay user will remove the group from the config
and hot-reload it - vmalert will have to wait until the delay
ends. This results into slow config reloading and UI hang.

The change moves the start-delay logic back to the group's
`start` method. Now, group can immediately exit from the
delay when `group.close()` method is called.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-06 14:04:43 +01:00
Dmytro Kozlov
cc5b916237
docs: follow up after 4b136abff8 (#3918)
docs: follow up after 4b136abff8
2023-03-06 12:41:48 +01:00
Gowtam Lal
4b136abff8
app/vmctl: Add ability to set headers for vm-native HTTP requests. (#3906)
app/vmctl: Add ability to set headers for vm-native HTTP requests
2023-03-06 11:22:31 +01:00
Roman Khavronenko
4e8de26fec
docs: follow-up e781e22c9c (#3902)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-03 16:15:15 +01:00
Yury Molodov
62beea23f7
vmui: show query error (#3890)
* add links support with old query params
* show error after execute query
2023-03-03 16:07:47 +01:00
Haleygo
d056be710b
fix some typo (#3898) 2023-03-03 11:02:13 +01:00
Dmytro Kozlov
8bab50dc29
app/vmctl: add backoff retries to native protocol (#3859)
app/vmctl: vm-native - split migration on per-metric basis

`vm-native` mode now splits the migration process on per-metric basis. 
This allows to migrate metrics one-by-one according to the specified filter. 
This change allows to retry export/import requests for a specific metric and provides a better 
understanding of the migration progress.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-03-02 13:19:45 +01:00
Roman Khavronenko
d6fa4da712
vmalert: cancel in-flight requests on group's update or close (#3886)
When group's update() or close() method is called, the group
still need to wait for its current evaluation to finish.
Sometimes, evaluation could take a significant amount of time
which slows configuration update or vmalert's graceful shutdown.

The change interrupts current evaluation in order to speed up
the graceful shutdown or config update procedures.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-01 15:48:20 +01:00
Dmytro Kozlov
ec2abf9b69
app/vmctl: Increase http request timeout made by remote read client, add importing tips (#3879)
app/vmctl: Increase default http request timeout made by remote read client
2023-02-28 09:50:33 +01:00
Aliaksandr Valialkin
46127b432d
lib/bytesutil: add -internStringDisableCache and -internStringCacheExpireDuration command-line flags
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872
2023-02-27 14:16:49 -08:00
Aliaksandr Valialkin
0d3f31f60e
lib/storage: follow-up for 39cdc546dd
- Use flag.Duration instead of flagutil.Duration for -snapshotCreateTimeout,
  since the flagutil.Duration is intended mostly for big durations, e.g. days, months and years,
  while the -snapshotCreateTimeout is usually smaller than one hour.
- Add links to https://docs.victoriametrics.com/#how-to-work-with-snapshots in docs/CHANGELOG.md,
  so readers could easily find the corresponding docs when reading the changelog.
- Properly remove all the created directories on unsuccessful attempt to create
  snapshot in Storage.CreateSnapshot().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
2023-02-27 13:07:38 -08:00
Zakhar Bessarab
39cdc546dd
lib/storage: enhancements for snapshots process (#3873)
* lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858)

* lib/{mergeset,storage}: add timeout configuration for snapshots creation, remove incomplete snapshots from storage

* docs: fix formatting

* app/vmstorage: add metrics to track status of snapshots

* app/vmstorage: use `vm_http_requests_total` metric for snapshot endpoints metrics, rename new flag to make name more clear

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update flag name in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: reflect new metrics names change in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 12:12:03 -08:00
Aliaksandr Valialkin
b3bb18d674
app/vmselect/promql: fix panic when calculating aggr_func(rollup*())
The panic has been introduced in dac21d874b
2023-02-27 11:48:27 -08:00
Zakhar Bessarab
ac4c7adec6
app/vmgateway: add new flag doc
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:34 -08:00
Zakhar Bessarab
5d7da8479f
app/vmgateway: fix typo in docs
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:08 -08:00
Zakhar Bessarab
4ee73f54a6
app/vmgateway: add OpenID discovery of JWKS endpoints
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:17:39 -08:00
Aliaksandr Valialkin
23871fb0bf
app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol 2023-02-27 11:03:49 -08:00
Dmytro Kozlov
27c9446520
app/vmctl: skip series if measurement not found (#3869)
app/vmctl: skip measurements with no fields for influxdb mode
2023-02-27 14:28:47 +01:00
Dmytro Kozlov
bcbf73225e
app/vmctl: enable version flag (#3868) 2023-02-27 10:30:53 +01:00
Aliaksandr Valialkin
255a0cf635
all: add makefile rules for GOARCH=s390x for all the VictoriaMetrics components
This is a follow-up for 007530f882
2023-02-26 12:36:51 -08:00
v1gnesh
007530f882
Option to build for s390x (Linux on IBM Z) (#3870)
Added `victoria-metrics-linux-s390x` to allow single node builds for `s390x` platform.

Leaving other packaging options at the moment, as on this platform, they're mostly going to be built from/with container images hosted within the company as a base, and not alpine.
2023-02-26 12:29:50 -08:00
Aliaksandr Valialkin
f7ef80aaad
.golangci.yml: properly enable revive linter and fix all the warnings it detects 2023-02-26 12:18:59 -08:00
Aliaksandr Valialkin
ffa327d6d1
app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-26 12:07:47 -08:00
Roman Khavronenko
e1c3267e34
vmselect/promql: check for deadline in count_values fn (#3806)
* vmselect/promql: check for deadline in `count_values` fn

`count_values` could be very slow during the data processing.
Checking for deadline between iterations supposed to reduce
probability of exceeding `search.maxQueryDuration`.

The change also adds a new trace record, which captures the time
spent in aggregation function. Before that, the trace for aggr funcs
could be confusing since it doesn't account for all the places where
time was spent.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 16:59:26 -08:00
Roman Khavronenko
dac21d874b
metricsql: support optional 2nd argument for rollup functions (#3841)
* metricsql: support optional 2nd argument for rollup functions

Support optional 2nd argument `min`, `max` or `avg` for rollup functions:
 * rollup
 * rollup_delta
 * rollup_deriv
 * rollup_increase
 * rollup_rate
 * rollup_scrape_interval

 If second argument is passed, then rollup function will return only the selected aggregation type.
 This change can be useful for situations where only one type of rollup calculation is needed.
 For example, `rollup_rate(requests_total[5m], "max")`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 13:47:52 -08:00
Aliaksandr Valialkin
c87c7d1e29
app/vmselect/promql: measure the time required for calculating the aggregate function from the prepared source time series 2023-02-23 20:05:14 -08:00
Aliaksandr Valialkin
8b7a828c65
app/vmselect/vmui: make vmui-update after d4fc0ed874 2023-02-23 19:25:52 -08:00
Yury Molodov
d4fc0ed874
vmui: improve mobile ui (#3848)
* feat: improve mobile ui

* feat: improve mobile ui

* fix: change style server url

* fix: improve ExploreMetrics mobile

* fix: display global settings on all pages
2023-02-23 19:18:49 -08:00
Aliaksandr Valialkin
a02cf92fd1
docs: update --help descriptions after recent changes 2023-02-23 19:02:27 -08:00
Aliaksandr Valialkin
b285207aa7
app/vmselect: add -search.logQueryMemoryUsage command-line flag for logging queries, which take big amounts of memory
Thanks to @michal-kralik for initial attempts for this feature:

- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3651
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3715

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553
2023-02-23 18:47:08 -08:00
Aliaksandr Valialkin
c080443fef
app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol
Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag,
which can be used for forcing Prometheus remote write protocol in cases when the remote storage
supports VictoriaMetrics remote write protocol.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-23 17:36:55 -08:00
Aliaksandr Valialkin
510f78a96b
all: consistently use http.Method{Get,Post,Put} across the codebase
This is a follow-up after 9dec3c8f80
2023-02-22 18:58:46 -08:00
my-git9
9dec3c8f80
chore: Use http constants to replace numbers (#3846)
Signed-off-by: xin.li <xin.li@daocloud.io>
2023-02-22 18:53:05 -08:00
Alexander Marshalov
aaef0bac00
fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-22 18:38:43 -08:00
Yury Molodov
fc720a5a78
fix: change query params update (#3860) 2023-02-22 18:25:32 -08:00
Aliaksandr Valialkin
9fbd45a22f
lib/promscrape/discovery/kuma: follow-up for 317fef95f9
- Do not generate __meta_server label, since it is unavailable in Prometheus.
- Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md,
  so users could click it and read the docs without the need to search the corresponding docs.
- Remove kumaTarget struct, since it is easier generating labels for discovered targets
  directly from the response returned by Kuma. This simplifies the code.
- Store the generated labels for discovered targets inside atomic.Value. This allows reading them
  from concurrent goroutines without the need to use mutex.
- Use synchronouse requests to Kuma instead of long polling, since there is a little sense
  in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval.
- Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used.
- Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus.
- Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability.
- Remove unused fields from discoveryRequest and discoveryResponse.
- Update tests.
- Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config.
- Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback,
  since these are public types.

Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label,
since usually this label is set by the Prometheus itself to __address__ after the relabeling phase.
See https://www.robustperception.io/life-of-a-label/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389

See https://github.com/prometheus/prometheus/issues/7919
and https://github.com/prometheus/prometheus/pull/8844
as a reference implementation in Prometheus
2023-02-22 17:51:51 -08:00
Aliaksandr Valialkin
f8811411fd
app/vmagent/remotewrite: removed unneeded code in testPushWriteRequest after 57801660ab 2023-02-22 17:51:50 -08:00
Alexander Marshalov
317fef95f9
add kuma_sd_config for Kuma Control Plane targets discovery (#3389) (#3840) 2023-02-22 13:59:56 +01:00
Dmytro Kozlov
110c3896e7
app/vmctl: add retry backoff policy (#3844)
app/vmctl: move retries logic into a separate pkg
2023-02-22 13:06:55 +01:00
Alexander Marshalov
57801660ab
Fix TestPushWriteRequest for remotewriteprotocol for pure-test (with native zstd) (#3850)
app/vmagent: fix TestPushWriteRequest for pure-test (with native zstd)

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-02-22 11:26:19 +01:00
Artem Navoiev
6b4ccc17c2
docs: fix typo OpentTSDB -> OpenTSDB (#3854)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 11:24:26 +01:00
Aliaksandr Valialkin
10cf6c9781
app/vmselect: allow zero value for -search.latencyOffset command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2023-02-21 18:06:55 -08:00
Roman Khavronenko
5446ce0018
docs: mention rules replay blogpost in vmalert docs (#3851)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-21 22:16:15 +01:00
Zakhar Bessarab
9d91d8fc91
vmgateway: add support of JWKS endpoint usage for JWT keys verification (#521) 2023-02-20 19:22:55 -08:00
Aliaksandr Valialkin
f4f1f2f976
docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol
The 10x savings are reproduced only on artificial data.
The savings on production data are usually in the range 2x-4x.
2023-02-20 19:11:31 -08:00
Aliaksandr Valialkin
a678cbe62e
docs/vmagent.md: mention that Mimir doesnt support backfilling 2023-02-20 19:11:30 -08:00
Aliaksandr Valialkin
76f2c70be3
app/vmagent: add support for VictoriaMetrics remote write protocol, which allows saving up to 10x on network bandwidth costs under high load
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-20 19:11:30 -08:00
Roman Khavronenko
74f122294d
docs: update vmalert docs (#3843)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-20 16:39:31 +01:00
Aliaksandr Valialkin
2cea923ff7
app/vmselect/promql: add share(q) aggregate function for normalizing results across multiple time series in [0..1] value range per each timestamp and aggregation group 2023-02-18 22:42:01 -08:00
Aliaksandr Valialkin
c86f1f1d1b
app/vmselect/promql: add range_zscore(q) and range_trim_zscore(z, q) functions
These functions may be useful for dropping outliers at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 22:41:59 -08:00
Aliaksandr Valialkin
8bd2eee8e0
app/vmalert/README.md: sync with docs/vmalert.md after 6ef6f3a771 2023-02-18 15:21:24 -08:00
Aliaksandr Valialkin
406822a16c
app/vmselect/promql: add range_mad(q) and range_trim_outliers(k, q) functions
These functions may help trimming outliers during query time
for the use case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 15:19:10 -08:00
Haleygo
6ef6f3a771
vmalert: fix maxResolveDuration flag note (#3827)
Signed-off-by: Haleygo <hui.wang@daocloud.io>
2023-02-16 19:26:17 +01:00
Aliaksandr Valialkin
87f1ed5d87
app/vmui: tooltip formatting enhancements according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706#issuecomment-1429980038 2023-02-14 23:37:26 -08:00
Aliaksandr Valialkin
11ce30820b
all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:05:16 -08:00
Aliaksandr Valialkin
5c4f5b83fc
all: rename ParseStream -> stream.Parse
This is a follow-up for 057698f7fb
2023-02-13 10:52:05 -08:00
Aliaksandr Valialkin
ccdddf7996
lib/protoparser/promremotewrite: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:46:54 -08:00
Aliaksandr Valialkin
9be1398b92
lib/protoparser/native: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:43:05 -08:00
Aliaksandr Valialkin
8830607021
lib/protoparser/graphite: extract stream parsing code into a separate stream package 2023-02-13 10:32:36 -08:00
Aliaksandr Valialkin
a646841c07
lib/protoparser/csvimport: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:25:46 -08:00
Aliaksandr Valialkin
7568658c19
lib/protoparser/vmimport: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:20:19 -08:00
Aliaksandr Valialkin
af37717108
lib/protoparser/opentsdbhttp: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:16:03 -08:00
Aliaksandr Valialkin
7720d403c0
lib/protoparser/opentsdb: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 10:03:16 -08:00
Aliaksandr Valialkin
fe196e0b7a
lib/protoparser/influx: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 09:58:52 -08:00
Aliaksandr Valialkin
f83d6d69b2
lib/protoparser/datadog: extract stream parsing code into a separate stream package
This is a follow-up for 057698f7fb
2023-02-13 09:51:47 -08:00
Roman Khavronenko
057698f7fb
lib/protoparser/prometheus: move streamparser to subpackage (#3814)
`lib/protoparser/prometheus` is used by various applications,
such as `app/vmalert`. The recent change to the
`lib/protoparser/prometheus` package introduced a new dependency
of `lib/writeconcurrencylimiter` which exposes some metrics.
Because of the dependency, now all applications which have this
dependency also expose these metrics.

Creating a new `lib/protoparser/prometheus/stream` package helps
to remove these metrics from apps which use `lib/protoparser/prometheus`
as dependency.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 09:26:07 -08:00
Roman Khavronenko
a645a95bd6
docs: improve troubleshooting docs for vmalert (#3812)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 17:29:30 +01:00
Oleksandr Redko
9fff48c3e3
app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
Aliaksandr Valialkin
438b2e11bd
app/vmauth: allow specifying max_concurrent_requests value on a per-user basis bigger than the -maxConcurrentPerUserRequests value 2023-02-11 20:53:08 -08:00
Aliaksandr Valialkin
25a9017a72
app/vmui: show median instead of avg on graph tooltip and line legend
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
2023-02-11 12:51:12 -08:00
Aliaksandr Valialkin
b11bdc46be
app/vmselect/promql: add mad_over_time(m[d]) function
See https://github.com/prometheus/prometheus/issues/5514
2023-02-11 01:06:20 -08:00
Aliaksandr Valialkin
ed4492ddd5
all: update alpine base docker image from 1.17.1 to 1.17.2
See https://alpinelinux.org/posts/Alpine-3.17.2-released.html
2023-02-11 00:37:17 -08:00
Aliaksandr Valialkin
776391917f
app/vmauth: improve load balancing by sending incoming requests to backends with the lowest number of concurrent requests
While at it, stop sending requests to unavailable backend for 3 seconds
before the next attempt. This should reduce the amounts of useless work
and the number of useless network packets when the backend is temporarily unavailable.
2023-02-11 00:30:31 -08:00
Aliaksandr Valialkin
f3625e4f3f
app/vmauth: add -maxConcurrentPerUserRequests command-line option for limiting the number of concurrent requests on a per-user basis
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346
2023-02-10 21:58:21 -08:00
Aliaksandr Valialkin
70f8911ca7
app/vmauth: automatically retry failing GET requests on the remaining backends 2023-02-09 21:05:55 -08:00
Dmytro Kozlov
f582f9e8ab
app/vmauth: add concurrent requests limit per auth record (#3749)
* app/vmauth: add concurent requests limit per auth record

* app/vmauth: added clarification comment

* app/vmauth: remove unused code

* app/vmauth: move read from limiter

* app/vmauth: fix text

* app/vmauth: fix comments

* - Clarify the docs for the max_concurrent_requests option at docs/vmauth.md
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that the -maxConcurrentRequests takes precedence over per-user max_concurrent_requests
- Update tests for verifying that the max_concurrent_requests option is parsed properly

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 20:03:01 -08:00
Aliaksandr Valialkin
73358571ee
app/vmalert: follow-up after d3c64aae8768d58781ee7e358bd7f3d8e0eb836d
- Document the change at docs/CHANGELOG.md
- Add `Reading rules from object storage` section to docs/vmalert.md
- Add `s3` prefix to command-line flags related to the configuration of s3 and gcs clients
- Explicitly mention that reading rules from object storage is supported only in enterprise version
2023-02-09 18:52:00 -08:00
Roman Khavronenko
14c20c1843
vmalert: support object storage for rules (#519)
* vmalert: support object storage for rules

Support loading of alerting and recording rules from object
storages `gcs://`, `gs://`, `s3://`.

* review fixes
2023-02-09 18:50:48 -08:00
Aliaksandr Valialkin
0e0095d350
all: run apk update && apk upgrade in base Alpine Docker image in order to get all the recent security fixes 2023-02-09 14:01:32 -08:00
Aliaksandr Valialkin
513707a8c7
app/vmui: UX enhancements for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
- Display `min` value additionally to `avg`, `max` and `last`
- Allow copy-n-pasting metric name with its labels from both legend and tooltup
2023-02-09 11:04:51 -08:00
Aliaksandr Valialkin
f40661e7b7
docs/vmagent.md: clarify that automatically generated metrics contain all the target-specific labels, including instance and job 2023-02-09 11:04:51 -08:00
Yury Molodov
bff18cb5dd
vmui: lazy loading predefined panels (#3795)
* fix: change logic lazy loading predefined panels

* app/vmselect/vmui: `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 00:11:55 -08:00
Yury Molodov
e1063ce3c1
vmui: improve tenant selector (#3794)
* fix: change styles tenant selector (#3792)

* docs/CHANGELOG.md: document the change

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 00:08:59 -08:00
Yury Molodov
8afc0aef8d
vmui: add last/max/avg values (#3789)
* feat: add last/max/avg values (#3706)

* fix: change filter exclude values

* app/vmui: wip

- improve the visualization for avg/max/last values
- make getAvgFromArray() function resilient against inf/undefined/nil
- export getLastFromArray() function, which is resilient against inf/undefined/nil
- run `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-08 22:41:20 -08:00
Yury Molodov
75bcf86a31
fix: turn off the local dashboards(#3740) (#3793) 2023-02-08 11:13:15 -08:00
Roman Khavronenko
e83f14210d
Vmalert fixes (#3788)
* vmalert: use group's ID in UI to avoid collisions

Identical group names are allowed. So we should used IDs
for various groupings and aggregations in UI.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: prevent disabling state updates tracking

The minimum number of update states to track is now set to 1.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly update `debug` and `update_entries_limit` params on hot-reload

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: display `debug` field for rule in UI

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: exclude `updates` field from json marhsaling

This field isn't correctly marshaled right now.
And implementing the correct marshaling for it doesn't
seem right, since json representation is mostly used
by systems like Grafana. And Grafana doesn't expect this
field to be present.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-08 14:34:03 +01:00
Roman Khavronenko
c32d8ea29e
vmalert: update docs (#3770)
vmalert: update flags description

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-06 09:51:30 +01:00
Roman Khavronenko
6fd10e8871
vmalert: speed up state restore procedure on start (#3758)
* vmalert: speed up state restore procedure on start

Alerts state restore procedure has been changed to become asynchronous.
It doesn't block groups start anymore which significantly improves vmalert's startup time.
Instead, state restore is called by each group in their goroutines after the first rules
evaluation.

While previously state restore attempt was made for all loaded alerting rules,
now it is called only for alerts which became active after the first evaluation.
This reduces the amount of API calls to the configured remote read URL.

This also means that `remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now
and will have no effect if configured.

See relevant issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2608

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* make lint happy

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-03 19:46:13 -08:00
Aliaksandr Valialkin
0a824d9490
app/vmselect/vmui: make vmui-update after e4c04b6dbe 2023-02-03 19:34:01 -08:00
Yury Molodov
e4c04b6dbe
vmui: set light theme for app mode (#3748)
* fix: set light theme for app mode

* fix: check inputTenantID flag

* fix: rename inputTenantID to useTenantID
2023-02-03 19:31:37 -08:00
Yury Molodov
f63f487787
vmui: mobile view (#3742)
* feat: add detect the system theme

* fix: change logic fetch tenants

* feat: add docs and info to cardinality page

* feat: add mobile view #3707
2023-02-03 19:27:57 -08:00
Aliaksandr Valialkin
607b542222
vendor: make vendor-update 2023-02-01 12:23:23 -08:00
Zakhar Bessarab
4cf37c5e70
app/vmbackup: fix deleting snapshot after backup completion (#3735) (#3736)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-01 11:23:58 +01:00
Aliaksandr Valialkin
3d331e4c5d
app/vmselect/vmui: make vmui-update after dcc5616126 2023-01-31 13:24:43 -08:00
Yury Molodov
dcc5616126
vmui: improvement the theme (#3731)
* feat: add detect the system theme

* fix: change logic fetch tenants

* feat: add docs and info to cardinality page

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-31 12:54:59 -08:00
Roman Khavronenko
1cbdcd391c
docs: mention -vmalert.proxyURL in vmalert docs (#3730)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-30 16:28:33 +01:00
Yury Molodov
ac14d50c18
vmui: add select of Tenant ID (#3673)
* feat: add select of tenantID

* feat: replace tenantID to default url

* fix: move the tenantID selector to the top header

* fix: hide tenantID selector by condition

* fix: correct z-index

* app/vmselect/vmui: `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-27 15:53:14 -08:00
Aliaksandr Valialkin
06f6b76521
app/vmagent: properly return 200 response code when importing data via Prometheus PushGateway protocol
This is the same fix as has been already applied to app/vminsert at cdb6d651e9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2023-01-27 14:40:02 -08:00
Aliaksandr Valialkin
a0c8b86eab
docs/vmauth.md: update docs after ff39a91147
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346
2023-01-27 14:10:19 -08:00
Aliaksandr Valialkin
ff39a91147
app/vmauth: limit the number of concurrent requests served by vmauth with the -maxConcurrentRequests command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346

This commit is based on the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486
2023-01-27 14:07:30 -08:00
Aliaksandr Valialkin
372b1688d7
app/vmauth: do not use net/http/httputil.ReverseProxy
This allows better controlling requests to backends and providing better error logging.
For example, if the backend was unavailable, then the ReverseProxy was logging the error
message without client ip and the initial request uri. This could harden debugging.

This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486
2023-01-27 13:40:05 -08:00
Aliaksandr Valialkin
1b81d8f542
lib/netutil: move IsTrivialNetworkError() function there, since it is used in multiple places across the code 2023-01-27 13:24:30 -08:00
Aliaksandr Valialkin
7e355080ce
app/vmauth: pass the target url to reverse proxy via context.Value instead of request header
This is less hacky way, since it doesn't clash with request headers
2023-01-27 12:15:52 -08:00
Aliaksandr Valialkin
36941d6d75
app/vmauth: consistency renaming: UserInfo.URLMap -> UserInfo.URLMaps
This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486
2023-01-27 00:19:02 -08:00
Aliaksandr Valialkin
0890adde67
docs: update command-line descriptions after 73256fe438 2023-01-27 00:00:37 -08:00
Aliaksandr Valialkin
cb374677a9
app/vmagent/prometheusimport: delete the temporary directory created by vmagent after the test is complete
This is a follow-up for 1cfa183c2b
2023-01-26 23:21:24 -08:00
Nikolay
73256fe438
lib/netutil: init implimentation of proxy protocol (#3687)
* lib/netutil: init implimentation of proxy protocol
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-26 23:08:35 -08:00
Roman Khavronenko
95ee86b600
docs: specify the time window for series_limit (#3708)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-25 09:30:20 -08:00
Aliaksandr Valialkin
28f66f0079
docs: update the list of command-line flags according to the latest changes 2023-01-25 09:20:24 -08:00
Aliaksandr Valialkin
d655d6b047
lib/streamaggr: add ability to de-duplicate input samples before aggregation 2023-01-25 09:14:49 -08:00
Yury Molodov
99d49e3ceb
vmui: include fonts in its bundle (#3705)
* feat: include fonts in the build

* fix: reduce size fonts

* wip

- Document the change at docs/CHANGELOG.md
- Run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-24 09:30:56 -08:00
Yury Molodov
20ad848c5d
vmui: improvements to the UI styles (#3704)
* feat: add dark theme

* update packages

* feat: add multilevel menu (#3678)

* fix: correct styles

* fix: update link to cardinality-explorer

* fix: remove unused scss variables

* docs/CHANGELOG.md: document the changes

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-24 09:20:31 -08:00
Aliaksandr Valialkin
c496f06ca3
app/vmagent/{promremotewrite,vmimport}: remove unused functions InsertHandlerForReader()
Thanks to 1cfa183c2b , where the first such function has been removed
2023-01-23 22:31:29 -08:00
Aliaksandr Valialkin
f7acdb13db
app/{vmagent,vminsert}: follow-up for 1cfa183c2b
- Call httpserver.GetQuotedRemoteAddr() and httpserver.GetRequestURI() only when the error occurs.
  This saves CPU time on fast path when there are no parsing errors.
- Create a helper function - httpserver.LogError() - for logging the error with the request uri and remote addr context.
2023-01-23 22:26:53 -08:00
Artem Navoiev
1cfa183c2b
add error handler for parsing prometheus text format to vmagent and v… (#3693)
* add error handler for parsing prometheus text format to vmagent and vminsert

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* fix typo

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* typo

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* fix variables naming and error message

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-23 22:14:34 -08:00
Yury Molodov
3536bef36e
vmui: add open graph and twitter card tags (#3697)
* feat: add open graph and twitter card tags

* app/vmui: spelling fixes

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-23 22:04:46 -08:00
Aliaksandr Valialkin
30e968df6d
app/vmselect: use consistent randomizer in tests
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
2023-01-23 19:27:25 -08:00
Aliaksandr Valialkin
f2b40dbe9a
app/vmalert: use consistent randomizer in tests
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
2023-01-23 19:25:10 -08:00
Artem Navoiev
7c1daade15
tests: use DebugFlush instead of vmstorage stop. This simplifies the logic and allows to remove test-only methodds (#3694)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-23 14:45:59 +01:00
Aliaksandr Valialkin
74df30456b
app/vmselect: make vmui-update after df7b81b44d 2023-01-20 12:07:13 -08:00
Yury Molodov
df7b81b44d
vmui: add support for time zone selection for older versions of browsers (#3680)
* fix: add check for support of getting time zones

* vmui: add support for time zone selection for older versions of browsers
2023-01-20 11:47:53 -08:00
Tobias Jungel
777038fe44
app/vmbackup: prevent password leaks (#3672)
This prevents vmbackup from leaking passwords into logs like shown below.

2023-01-11T15:00:01.050Z        info    VictoriaMetrics/lib/logger/flag.go:12   build version: vmbackup-20221214-211706-tags-v1.85.1-0-g09a70d3e9
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/lib/logger/flag.go:13   command-line flags
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/lib/logger/flag.go:20     -dst="fs:///vm-backups/latest"
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/lib/logger/flag.go:20     -snapshot.createURL="http://user:super_sercret123@victoriametricspshot/create"
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/lib/logger/flag.go:20     -storageDataPath="/storage"
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/app/vmbackup/main.go:53 Snapshot create url http://user:super_sercret123@victoriametrics:8428/snapshot/create
2023-01-11T15:00:01.050Z        info    VictoriaMetrics/app/vmbackup/main.go:60 Snapshot delete url http://user:super_sercret123@victoriametrics:8428/snapshot/delete
2023-01-18 11:35:21 -08:00
Aliaksandr Valialkin
e867df5ef5
app/vmui: increase perceived performance by 2.5x by reducing the delay before the query execution from 0.8s to 0.3s
The delay cannot be removed, since it is used for limiting the rate of queries sent to VictoriaMetrics during graph scrolling.
2023-01-18 01:33:48 -08:00
Aliaksandr Valialkin
0c625185cb
app/vmselect/promql: updates tests for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664 2023-01-17 23:25:45 -08:00
Yury Molodov
fcef2ff6b2
vmui: correctly display range results in Table view (#3657)
* fix: properly display range results

* fix: set range values to empty array

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-17 21:03:28 -08:00
Yury Molodov
e58921aa8f
vmui: give more visually different colors to graph lines (#3656)
* feat: make more different colors of graph lines

* docs/CHANGELOG.md: give more visually different colors to graph lines
2023-01-17 20:25:37 -08:00
Aliaksandr Valialkin
c94020f7dc
app/vmui: do not round the number in formatPrettyNumber() if the range isn't set 2023-01-17 19:50:44 -08:00
Aliaksandr Valialkin
09d7fa2737
lib/{mergeset,storage}: do not slow down concurrently executed queries during assisted merges
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2023-01-16 14:31:52 -08:00
Yury Molodov
04c408e986
vmui: make the step input field global across all the tabs and views (#3644)
* feat: make the step input field global

* fix: correct get step from url

* fix: set minimumSignificantDigits to 1

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-15 13:47:08 -08:00
Aliaksandr Valialkin
cdb6d651e9
app/vminsert: return 200 OK status code when importing data in pushgateway format
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636
2023-01-15 13:29:53 -08:00
Aliaksandr Valialkin
207a62a3c2
app/vmselect/promql: reduce memory allocations when searching for time series pairs with identical labelsets in q1 op q2 queries
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641
2023-01-15 13:03:23 -08:00
Aliaksandr Valialkin
27afe7bc38
app/vmselect/promql: reduce the number of memory allocations inside getCommonLabelFilters()
This should improve performance a bit for `q1 op q2` queries
2023-01-15 13:03:23 -08:00
Yury Molodov
7fd82c0d3a
feat: make nav menu as links (#3646) 2023-01-13 09:45:07 +01:00
Aliaksandr Valialkin
be8fba9b6a
app/vmselect/netstorage: tune the number of blocks per series which should be unpacked by a single goroutine instead of spinning up multiple goroutines
This reduces overhead on time series data unpacking for typical cases,
this reducing CPU usage at vmselect
2023-01-12 09:31:44 -08:00
Aliaksandr Valialkin
7067e8206c
app/vmselect/promql: reduce memory allocations at getCommonLabelFilters() function
Intern tag keys and values there
2023-01-12 01:27:41 -08:00
Aliaksandr Valialkin
9f5b5708ff
app/vmselect: handle the /custom-dashboards request from /graph/ page in the same way as from the /vmui/ page
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322
2023-01-11 23:41:04 -08:00
Aliaksandr Valialkin
a194982117
app/vmselect: follow-up after 820312a2b1
- Move the feature description at the correct place at docs/CHANGELOG.md
- Run `make vmui-update`
- Various cosmetic fixes

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322
2023-01-11 23:28:00 -08:00
Dmytro Kozlov
820312a2b1
app/vmui: define custom path for dashboards json file (#3545)
* app/vmui: define custom path for dashboards json file

* app/vmui: remove unneeded code

* app/vmui: move handler to own file, fix show dashboards,

* app/vmui: move flag to handler, add flag description

* app/vmauth: fix part of the comments

* feat: add store for dashboards

* fix: prevent fetch dashboards for app mode

* app/vmauth: use simple cache for predefined dashboards

* app/vmauth: update dashboards doc

* app/vmauth: fix ci

* app/vmui: decrease timeout

* app/vmselect: removed cache, fix comments

* app/vmselect: remove unused const

* app/vmselect: fix error log, use slice byte instead of struct

Co-authored-by: Yury Moladau <yurymolodov@gmail.com>
2023-01-11 23:06:07 -08:00
Dmytro Kozlov
20af29294e
app/vmctl: add remote read protocol integration tests (#3626) 2023-01-11 23:00:10 -08:00
Yury Molodov
5c8bb029b5
vmui: small changes on explore metrics page (#3634)
* fix: change issue link

* fix: remove legend toggle

* fix: move select graph size

* feat: save url params on explore metrics page

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-11 22:16:10 -08:00
Aliaksandr Valialkin
dff47c73b7
app/vmselect: improve logging when the incoming query cannot be executed because of timeout in the wait queue 2023-01-11 01:06:05 -08:00
Aliaksandr Valialkin
65b9dcfcca
app/vmselect/promql: typo fix after 0771d57860 2023-01-11 01:05:31 -08:00
Aliaksandr Valialkin
0771d57860
app/vmselect/promql: make a copy of per-series timestamps before their modification
The per-series timestamps are usually shared among series, so it is unsafe modifying them.

The issue has been appeared after the optimization at 2f3ddd4884
2023-01-11 00:59:13 -08:00
Aliaksandr Valialkin
31fc29599f
app/vmselect/promql: move the eval function args in parallel query trace outside the loop 2023-01-10 22:23:30 -08:00
Aliaksandr Valialkin
c449714c0a
deployment/docker: update Go builder from v1.19.4 to v1.19.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.5+label%3ACherryPickApproved
2023-01-10 18:43:04 -08:00
Aliaksandr Valialkin
2d294cca59
deployment/docker: update Alpine base image from v3.17.0 to v3.17.1
See https://alpinelinux.org/posts/Alpine-3.17.1-released.html
2023-01-10 16:14:40 -08:00
Aliaksandr Valialkin
95ce1ba6ce
lib/httpserver: directly pass flag value to CheckAuthFlag()
There is no sense in passing a pointer to flag value there.

This is a follow-up for 4225a0bd75
2023-01-10 15:52:23 -08:00
Zakhar Bessarab
4225a0bd75
Use httpAuth.* flags as a fallback for endpoints protected by *AuthKey flags (#3582)
* {lib/server, app/}: use `httpAuth.*` flag as fallback for `*AuthKey` if it is not set

* lib/ingestserver/opentsdbhttp: fix opentdb HTTP handler not respecting `httpAuth.*` flags

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-10 15:46:13 -08:00
Dmytro Kozlov
0811000bb0
app/vmctl: Add insecure skip verify flag for remote read protocol (#3611)
* app/vmctl: Add insecure skip verify flag for remote read protocol
2023-01-10 23:18:49 +01:00
Artem Navoiev
37c52ccaf4
vmagent: add minimal scrape file exampe for vmagent quick start (#3627)
* vmagent: add minimal scrape file exampe for vmagent quick start

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* replace example with link to your prometheus.yml in docker

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-10 23:16:10 +01:00
Aliaksandr Valialkin
53d871d0b1
app/vmselect/netstorage: reduce tail latency during query processing
Previously the selected time series were split evenly among available CPU cores
for further processing - e.g unpacking the data and applying the given rollup
function to the unpacked data.
Some time series could be processed slower than others.
This could result in uneven work distribution among available CPU cores,
e.g. some CPU cores could complete their work sooner than others.
This could slow down query execution.

The new algorithm allows stealing time series to process from other CPU cores
when all the local work is done. This should reduce the maximum time
needed for query execution (aka tail latency).

The new algorithm should also scale better on systems with many CPU cores,
since every CPU processes locally assigned time series without inter-CPU communications.

The inter-CPU communications are used only when all the local work is finished
and the pending work from other CPUs needs to be stealed.
2023-01-10 13:43:14 -08:00
Aliaksandr Valialkin
e640ff72f1
app/vmselect/netstorage: reduce memory allocations when unpacking time series
Unpack time series with less than 400K samples in the currently running goroutine.
Previously a new goroutine was being started for unpacking the samples.
This was requiring additional memory allocations.
2023-01-09 23:18:17 -08:00
Aliaksandr Valialkin
9a563a6aef
app/vmselect/promql: eliminate memory allocation when sorting values inside float64s 2023-01-09 23:06:46 -08:00
Aliaksandr Valialkin
30ed33fae0
app/vmselect/promql: pre-allocate memory for values to be merged in mergeTimeseries()
This should reduce the number of memory re-allocations
2023-01-09 22:51:17 -08:00
Aliaksandr Valialkin
645c24dc5f
app/vmselect/promql: consistently intern series names obtained from marshalMetricNameSorted
This reduces memory allocations when the returned series names are used as map keys later
2023-01-09 22:45:40 -08:00
Aliaksandr Valialkin
2f3ddd4884
app/vmselect/promql: avoid memory allocations and copying from source timeseries to the returned result at timeseriesToResult() 2023-01-09 22:38:59 -08:00
Aliaksandr Valialkin
26cf680468
app/vmselect/promql: remove memory allocations from sortMetricTags() 2023-01-09 22:22:15 -08:00
Aliaksandr Valialkin
4f0c11ee93
app/vmselect/promql: intern output series names inside timeseriesToResult()
This reduces the number of memory allocations for repeated queries,
which return (almost) the same set of time series.
2023-01-09 22:19:56 -08:00
Aliaksandr Valialkin
562d6bca08
app/vmselect/promql: intern output series names during normal aggregation 2023-01-09 22:15:24 -08:00
Aliaksandr Valialkin
21ee9a1fab
app/vmselect/promql: intern output series names during incremental aggregation
This should reduce the number of memory allocations for repeated queries
2023-01-09 22:11:36 -08:00
Aliaksandr Valialkin
df2a494a7c
app/vmselect/netstorage: pre-allocate 4 block references per each time series during querying
Usually the number of blocks returned per each time series during queries is around 4.
So it is a good idea to pre-allocate 4 block references per time series
in order to reduce the number of memory allocations.
2023-01-09 22:03:23 -08:00
Aliaksandr Valialkin
c5e0f527bc
app/vmselect/netstorage: cache canonical MetricName for time series returned from the storage
This reduces memory allocations for repeated queries, which return (almost) the same set of time series.
2023-01-09 21:53:10 -08:00
Aliaksandr Valialkin
7afcca0c51
all: use metricsql.CompileRegexp instead of regexp.Compile for compiling regexps used in graphite queries
This should speed up repeated queries, since metricsql.CompileRegexp returns regexps from the cache
on subsequent calls for the same input regexp.
2023-01-09 21:43:08 -08:00
Aliaksandr Valialkin
c38a10e143
app/vmselect/netstorage: eliminate memory allocation for sortBlocksHeap arg when calling mergeSortBlocks() 2023-01-09 21:08:51 -08:00
Aliaksandr Valialkin
1f9d605988
app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333

This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 ,
but doesn't slow down the common case with merging replicated data blocks so significantly.

Benchmark results:

Before the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13968	     85643 ns/op	 956.53 MB/s	    1700 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	   10806	    109171 ns/op	1500.77 MB/s	    2191 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    8887	    130623 ns/op	1881.45 MB/s	    2660 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    7440	    157348 ns/op	2082.52 MB/s	    3174 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    6534	    184473 ns/op	2220.38 MB/s	    3612 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   13419	     85205 ns/op	 961.44 MB/s	    2213 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     579	   1894900 ns/op	  43.23 MB/s	   46760 B/op	       1 allocs/op

After the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13832	     85298 ns/op	 960.40 MB/s	    1716 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	    8833	    134222 ns/op	1220.66 MB/s	    2675 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    6487	    184830 ns/op	1329.65 MB/s	    3636 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    4977	    236318 ns/op	1386.61 MB/s	    4733 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    4088	    296734 ns/op	1380.36 MB/s	    5761 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   14083	     84067 ns/op	 974.47 MB/s	    2110 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     536	   2043534 ns/op	  40.09 MB/s	   50511 B/op	       1 allocs/op
2023-01-09 13:01:48 -08:00
Aliaksandr Valialkin
af263fe881
all: small improvements in error messages and command-line flag descriptions related to concurrency limiters 2023-01-07 00:11:44 -08:00
Aliaksandr Valialkin
293e4dc77b
app/{vminsert,vmstorage}: add comments on why storage.AddRows() is called without limiting the number of concurrent calls 2023-01-06 22:40:07 -08:00
Aliaksandr Valialkin
c63755c316
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:20:19 -08:00
Aliaksandr Valialkin
e7637885a6
app/vmselect: improve error message when the request cannot be started because too many concurrent requests are already executed 2023-01-06 22:10:42 -08:00
Karan Sharma
48f371a46c
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549)
Add nomad_sd_config support for service discovery
2023-01-05 23:03:58 +01:00
Zakhar Bessarab
01bc0c94ab
doc: add vmbackupmanager monitoring section (#3605)
* doc: add vmbackupmanager monitoring section

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-01-05 16:03:06 +01:00
Aliaksandr Valialkin
0dea3b71da
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2023-01-05 03:26:49 -08:00
Yury Molodov
2460e0f51e
vmui: improve Explore metrics (#3598)
* feat: add multiple select

* feat: improve explore interface

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-05 02:23:04 -08:00
Aliaksandr Valialkin
04dff34de4
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-01-05 01:50:10 -08:00
Aliaksandr Valialkin
fd175ad80b
docs: update -help outputs for vm* tools 2023-01-03 23:27:06 -08:00
Aliaksandr Valialkin
fa13bbc48a
app/{vmagent,vminsert}: add support for streaming aggregation
See https://docs.victoriametrics.com/stream-aggregation.html

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
2023-01-03 22:19:21 -08:00
Aliaksandr Valialkin
f33e687723
app/vmagent/remotewrite: improve descriptions for -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig
- Cross-reference these command-line flags.
- Add a link to https://docs.victoriametrics.com/vmagent.html#relabeling
2023-01-03 22:04:49 -08:00
Aliaksandr Valialkin
6db5c3801e
app/vmbackup: remove superflouos whitespace after 8fe21ec707 2023-01-03 21:52:40 -08:00
yanggang
8fe21ec707
Fix flag help message for the backups types. (#3577)
Signed-off-by: yanggang <gang.yang@daocloud.io>
2023-01-03 10:56:42 +01:00
yanggang
fa842d6534
fix typo for json values. (#3576)
Signed-off-by: yanggang <gang.yang@daocloud.io>
2023-01-03 10:55:38 +01:00
yanggang
93e935bcaa
Fix vmctl command hint for vm-native-step-interval (#3575)
Signed-off-by: yanggang <gang.yang@daocloud.io>
2023-01-03 10:54:53 +01:00
Roman Khavronenko
4b3d8eb573
vmalert: mention specifics of Alertmanager HA mode (#3573)
Stress the importance of specifying of all Alertmanager
URLs in vmalert's `-notifier.url` or `notifier.config`
if it runs in cluster mode.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3547

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-30 09:54:03 +01:00
Aliaksandr Valialkin
05fa11b296
app/vmui: reuse the series color in line tooltip 2022-12-29 15:32:41 -08:00
Aliaksandr Valialkin
1794f3d46e
app/vmui: small usability improvements
- Show in the line tooltip the number of the query which generates the given line.
  This simplifies comparison of lines generated by multiple queries.

- Show metric name as __name__ label in the line tooltip in the same way as other labels are shown there.
  This makes the label information in the tooltip more consistent.

- Properly quote label values with JSON.stringify(). This prevents from improper formatting
  when label values contain doublequote chars.

- Remove double curly braces artifact at graph legend for lines without names and labels.

- Properly use modifier for regular expressions across the code.
2022-12-29 14:52:51 -08:00
Aliaksandr Valialkin
b8bc62431a
app/vmselect/vmui: make vmui-update after 1720bddb4f 2022-12-29 12:20:06 -08:00
Yury Molodov
1720bddb4f
fix: display correct graph tooltip title (#3562) 2022-12-29 12:06:48 -08:00
Roman Khavronenko
83870aeb8d
tests: attempt to fix flaky graphite test (#3567)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-29 11:48:47 -08:00
ChenyuanHu
0a9e1d64fe
app/vmselect/prometheus: no need manually call queryDuration.UpdateDuration (#3564)
There is no need to manually call `queryDuration.UpdateDuration(startTime)`, because `defer queryDuration.UpdateDuration(startTime)` is executed at the beginning of the function(L660).
2022-12-29 14:18:00 +01:00
Roman Khavronenko
6588fcbfca
vmalert: allow configuring the default number of stored rule's update states (#3556)
Allow configuring the default number of stored rule's update states in memory
 via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's
 `update_entries_limit` configuration param.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-29 12:36:44 +01:00
Aliaksandr Valialkin
3dc684634e
app/vmselect/searchutils: accept partial RFC3339 values at time, start and end query args
This simplifies manual usage of the APIs. For example, the following query
would return the results over the 2022 year.

  /api/v1/query_range?start=2022&end=2023&step=1d&query=...

This is equivalent to:

  /api/v1/query_range?start=2022-01-01T00:00:00Z&end=2023-01-01T00:00:00Z&step=1d&query=...
2022-12-28 19:41:54 -08:00
Yury Molodov
d2f89b55b7
vmui: fix step field (#3561)
* feat: use a unit next to the step value

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-28 16:00:51 -08:00
Artem Navoiev
7d9c4bebc0
update links to grafana dashboards (#3534)
docs: update links to grafana dashboards

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-25 17:36:20 +01:00
Aliaksandr Valialkin
b11c806d1c
app/vmui: show min, max and avg lines at Explore metrics graphs when instance is selected in the same way as when only the job is selected
This improves consistency of the graphs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-23 23:21:11 -08:00
Aliaksandr Valialkin
2a7e392bb3
app/vmselect/vmui: make vmui-update after 0dca224ec3 2022-12-23 22:25:04 -08:00
Aliaksandr Valialkin
0dca224ec3
app/vmui: small improvements for header panel
- Rename `Custom panel` tab to more clear `Query` tab
- Rename `Cardinality` tab to `Explore cardinality`, so it becomes consistent with `Explore metrics` tab
- Move `Dashboards` tab to the end, since it isn't used too much
2022-12-23 22:24:31 -08:00
Aliaksandr Valialkin
8ef1fe2047
app/vmui: move the Explore metrics tab closer to Custom panel tab
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-23 22:16:21 -08:00
Aliaksandr Valialkin
f599e1bd34
app/vmui: show less lines at metrics explorer when the instance isn't selected
Show min, max and avg graphs across instances for the selected job.
This should improve usability of such a graphs when the job contains many instances.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-23 22:15:01 -08:00
Aliaksandr Valialkin
4deea604bf
app/vmui: follow-up after f6d31f5216
- Document the feature at docs/CHANGELOG.md.
- Document the metrics explorer at https://docs.victoriametrics.com/#metrics-explorer .
- Properly set `start` and `end` args for the selected time range
  when performing the request, which returns metric names.
- Improve queries, so they return lower number of lines and labels.
  This should improve metrics' exploration.
- Properly encode label filters and query args before passing them to VictoriaMetrics.
- Various cosmetic fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-22 17:17:01 -08:00
Yury Molodov
f6d31f5216
vmui: add explore tab for exploration of metrics, which belong to a particular job/instance (#3470)
* feat: add "Explore" page

* feat: add graphs for explore page

* vmui: add explore tab for exploration of metrics, which belong to a particular job/instance

* refactor: rename variables

* refactor: extract graph to ExploreMetricItemGraph.tsx

* feat: add searchable for Select.tsx

* feat: improve metrics explorer

* feat: set document title by page

* feat: add page to view icons

* fix: improve styles

* fix: add encodeURIComponent to query
2022-12-22 15:24:40 -08:00
Yury Molodov
1df87807fd
vmui: step (#3521)
* feat: add step rounding

* fix: change step in URL parameters

* refactor: change comment for roundStep
2022-12-22 14:54:28 -08:00
Aliaksandr Valialkin
f1441a598f
app/vmselect/promql: add tests for d3de110070 2022-12-21 20:25:21 -08:00
Aliaksandr Valialkin
d3de110070
app/vmselect/promql: make sure that label_replace() doesn't create an empty dst_label if the src_label doesn't match regex 2022-12-21 20:20:01 -08:00
Yury Molodov
0f8ffc7df9
vmui: fix change timezone (#3519)
vmui: fix time picker with changed time zone
2022-12-21 17:22:37 +01:00
Aliaksandr Valialkin
77874d6055
app/vmselect/vmui: make vmui-update after 731d189fa9 2022-12-20 14:51:07 -08:00
Roman Khavronenko
e4e9dfb785
vmagent: respect -usePromCompatibleNaming if no relabeling is set (#3511)
* vmagent: respect `-usePromCompatibleNaming` if no relabeling is set

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: upd test

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-20 14:32:45 -08:00
Yury Molodov
731d189fa9
fix: change the logic for hide query (#3514) 2022-12-20 14:29:46 -08:00
Zakhar Bessarab
4be4645142
app/vmbackupmanager: add metrics for better observability (#488)
* app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response

* app/vmbackupmanager: drop old metrics before creating new ones

* app/vmbackupmanager: use `_total` postfix for counter metrics

* app/vmbackupmanager: remove `_total` postfix for gauge-like metrics

* app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention

* app/vmbackupmanager: address review feedback

* app/vmbackupmanager: fix metric name

* app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage

* app/vmbackupmanager: improve performance for backup size calculation

* app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic

* {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package

* app/vmbackupmanager: fix creating new metrics instead of reusing existing ones

* lit/formatutil: add comment to make linter happy

* app/vmbackupmanager: address review feedback
2022-12-20 14:18:06 -08:00
Zakhar Bessarab
18e55d14c6
app/vmbackupmanager: update doc to include cluster to cluster restore example (#3506) 2022-12-20 14:54:56 +01:00
Aliaksandr Valialkin
9fa3f1dc57
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:20 -08:00
Aliaksandr Valialkin
56a9ea3753
docs/vmalert.md: mention latency_offset query arg, which has been added in 86dae56bd0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:20:37 -08:00
Aliaksandr Valialkin
285841bcce
app/vmselect/prometheus: follow-up after 86dae56bd0
Return error if the provided latency_offset query arg cannot be parsed.
This should simplify debugging in production.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:13:20 -08:00
Roman Khavronenko
86dae56bd0
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 16:54:57 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
5d30080555
lib/flagutil: support for TB and TiB suffixes for command-line flags, which accept byte sizes 2022-12-14 17:52:32 -08:00
Aliaksandr Valialkin
38341802c2
app/vmselect: add /expand-with-exprs page 2022-12-14 16:18:55 -08:00
Aliaksandr Valialkin
231569f89b
docs/vmagent.md: small formatting fix 2022-12-14 14:25:35 -08:00
Aliaksandr Valialkin
ac5948b3f3
app/vmselect/vmui: make vmui-update 2022-12-14 12:01:48 -08:00
Aliaksandr Valialkin
7a61bafe59
docs/vmagent.md: clarify that relabeling is actually a debugging at relabel debug section 2022-12-13 15:46:21 -08:00
Yury Molodov
d3418bafc0
fix: prevent run query when selecting autocomplete option (#3480) 2022-12-13 09:30:05 -08:00
Aliaksandr Valialkin
33bff00890
app/vmselect/vmui: make vmui-update 2022-12-12 17:46:30 -08:00
Yury Molodov
cca3bc756b
vmui: minor enhancements (#3471)
* update package-lock.json

* fix: correct handle click by "action" on cardinality page

* fix: correct styles for icons width

* feat: add layout with copyright

* feat: add website and issue to footer
2022-12-12 17:44:13 -08:00
Aliaksandr Valialkin
d1af6046c7
lib/{mergeset,storage}: do not block small merges by pending big merges - assist with small merges instead
Blocked small merges may result into big number of small parts, which, in turn,
may result in increased CPU and memory usage during queries, since queries need to inspect
all the existing small parts.

The issue has been introduced in 8189770c50

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-12 17:00:50 -08:00
Zakhar Bessarab
ebcb4ab617
{app/{vmbackup/vmrestore}: update path example to use Azure terminology for consistency (#3475) 2022-12-12 22:17:22 +03:00
Roman Khavronenko
8286f9608f
vmalert: support $for or .For template variables (#3474)
support `$for` or `.For` template variables  in alert's annotations.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3246

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-12 22:16:10 +03:00
Aliaksandr Valialkin
88597f187b
app/{vmagent,vminsert}/datadog: make the host label optional in DataDog data ingestion protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3432
2022-12-10 23:32:31 -08:00
Aliaksandr Valialkin
e272a0ec78
app/vmselect/promql: allow passing inf arg into functions, which accept numeric limit on the number of output time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3461
2022-12-10 22:47:47 -08:00
Aliaksandr Valialkin
b01607e3fb
docs: clarify that single-node VictoriaMetrics also provides functionality for relabel debugging
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2022-12-10 20:49:52 -08:00
Aliaksandr Valialkin
a30ae502ef
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2022-12-10 12:44:45 -08:00
Aliaksandr Valialkin
3e7276639e
app/{vminsert,vmselect}: move the handler for /metric-relabel-debug from vminsert to vmselect to be consistent with the cluster version 2022-12-10 02:45:40 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
6f0179405a
app/vmalert: properly handle nil req passed to requestToCurl()
This fixes a panic in the TestAlertingRule_Exec_Negative test.
The panic has been introduced in the commit b97bd01605
2022-12-10 02:04:17 -08:00
Aliaksandr Valialkin
ca59d3de59
app/vmalert: do not show system links at http://vmalert:8880/ page when it is requested via proxy
The system links are absolute, e.g. they start from `/`, so there are high chances
they won't work as expected when requested via proxy such as vmselect with -vmalert.proxyURL
command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3424
2022-12-09 11:46:28 -08:00
Roman Khavronenko
b97bd01605
vmalert: do not hold pointer to http.Request (#3467)
http.Request was used as a part of state struct
for generating the curl command when viewing the rule's
state changes.
It appears, that holding a referencing is far more expensive
than generating the curl command immediately.
On the test with 40k rules, this change reduces memory
and CPU usage by 50%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-09 18:13:29 +03:00
Roman Khavronenko
7c0ae3a86a
lib/storage: keep sample with the biggest value on timestamp conflict (#3421)
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:06:11 -08:00
Aliaksandr Valialkin
eeacbaf0b6
all: update Go builder from v1.19.3 to v1.19.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved
2022-12-08 16:41:24 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
f9730676d9
app/vmselect/searchutils: do not print flag name responsible for query timeout if the timeout isn't reached
This should make the log message more clear
2022-12-08 13:07:33 -08:00
Aliaksandr Valialkin
3f82e3fa36
docs: follow-up after e1bf2a85d0559d112908ce81597f3261d3a085c0
- Document the change at docs/CHANGELOG.md
- Run `make docs-sync` for copying app/vmgateway/README.md to docs/vmgateway.md
  in order to propagate docs' changes to https://docs.victoriametrics.com/vmgateway.html
2022-12-06 21:05:22 -08:00
Aliaksandr Valialkin
758e8a15fd
app/vmselect: typo fixes in code comments 2022-12-06 20:58:16 -08:00
Roman Khavronenko
3dec847c93
vmalert: correctly return error for RW failures (#3452)
* vmalert: correctly return error for RW failures

By mistake, in 0989649ad0 the error
for remote write failures weren't return to user.
This change fixes it.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-06 15:36:46 +01:00
Aliaksandr Valialkin
e2e341da9f
app/vmselect/vmui: make vmui-update after 7645d9ae00 2022-12-05 23:07:08 -08:00
Pedro Gonçalves
1e0666abb4
Datadog - Add device as a tag if it's present as a field in the series object (#3431)
* Datadog - Add device as a tag if it's present as a field in the series object

* address PR comments
2022-12-05 23:06:03 -08:00
Yury Molodov
7645d9ae00
feat: add toggle query display by Ctrl (#3449) 2022-12-05 22:45:15 -08:00
Yury Molodov
01a9b36a95
vmui: timezone select (#3414)
* feat: add timezone selection

* vmui: provide feature timezone select

* fix: correct timezone with relative time

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:44:31 -08:00
Yury Molodov
fd43b5bad0
vmui: fix multi-line query (#3448)
* fix: remove prevent nav by up/down keys for multi-line query

* fix: add query params encode in URL
2022-12-05 21:56:54 -08:00
Aliaksandr Valialkin
5eae9a9914
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:01 -08:00
Zakhar Bessarab
86c31f2955
app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450) 2022-12-05 17:18:09 -08:00
Aliaksandr Valialkin
8189770c50
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:16:14 -08:00
Yury Molodov
461158a437
fix: add word-break for tooltip (#3437) 2022-12-05 08:50:34 +01:00
Roman Khavronenko
91a8afa172
vmalert: reduce allocations for Prometheus resp parse (#3435)
Method `metrics()` now pre-allocates slices for labels
and results from query responses. This reduces the number 
of allocations on the hot path for instant requests.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-05 08:34:54 +01:00
Aliaksandr Valialkin
14660d4df5
all: typo fix: the the -> the 2022-12-03 21:53:01 -08:00
Roman Khavronenko
bcba5d2a78
vmalert: fix replay step param (#3428)
The recent change in modifying default value
of `datasource.queryStep` flag resulted in situation
where replay mode was always running queries with
step=`datasource.queryStep`. When it should always
use rule's evaluation interval.

The fix is related not to replay mode only, but
for all Range requests. Now step param is set
individually for each mode.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 13:57:53 +01:00
Zakhar Bessarab
30fea30685
app/vmalert: add remoteWrite.sendTimeout command-line flag to configure timeout for sending data to remoteWrite.url (#3423)
* app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url`

* vmalert: remove WriteTimeout from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the WriteTimeout.

* vmalert: remove DisablePathAppend from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the DisablePathAppend.

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:57:19 +01:00
Roman Khavronenko
8cc4f7eac6
vmalert: properly pass headers during the restore procedure (#3420)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-01 09:27:39 +01:00
Aliaksandr Valialkin
3cdff3de23
docs/vmagent.md: update after 959f06d175 2022-11-29 21:33:49 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
c7ce4979ec
all: follow-up after 05cf8a6ecc 2022-11-29 21:03:59 -08:00
Aliaksandr Valialkin
4822406b64
app/vmalert: substitute -datasource.disablePathAppend with -remoteRead.disablePathAppend in the description for -datasource.url command-line flag
This is a follow-up for 959f06d175
2022-11-29 20:36:41 -08:00
Dmytro Kozlov
05cf8a6ecc
vmctl: support of the remote read protocol (#3232)
vmctl: support of the remote read protocol

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 22:53:28 +01:00
Max Golionko
959f06d175
vmalert: flag reference update (#3415)
* flag reference update

there is no flag `-datasource.disablePathAppend` and datasource actually checking for `-remoteRead.disablePathAppend`

* update source for doc as well
2022-11-29 19:22:57 +01:00
Aliaksandr Valialkin
654e94f420
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
2022-11-28 18:37:09 -08:00
Aliaksandr Valialkin
ad105147dd
docs/CHANGELOG.md: cut v1.84.0 2022-11-25 19:53:29 -08:00
Aliaksandr Valialkin
58d459e8a8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:48:14 -08:00
Pedro Gonçalves
53a63c6c4c
Adding pushgateway basic capabilities to vmagent (#3360)
* init pushgateway implementation

* Initial implementation of pushgateway in vmagent

* Initial implementation of pushgateway in vmagent
2022-11-25 16:35:01 -08:00
Aliaksandr Valialkin
4ca44cfe9c
app/vmselect/vmui: make vmui-update after 37cda9abd0 2022-11-25 07:33:09 -08:00
Yury Molodov
37cda9abd0
fix: change header settings (#3391) 2022-11-25 07:26:35 -08:00
Yury Molodov
1ab66186ca
refactor: create Autocomplete component (#3390) 2022-11-25 07:25:35 -08:00
Aliaksandr Valialkin
da13d36af9
app/vmselect/vmui: make vmui-update after eb772aa50e 2022-11-24 17:37:18 -08:00
Yury Molodov
eb772aa50e
vmui: improve table view (#3377)
* vmui: add compact table view (#3365)

* feat: add compact table view

* fix: add overflow table

* fix: change table styles

* vmui: compact table view

* Update docs/CHANGELOG.md

Co-authored-by: Michal Kralik <michal.kralik@percona.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-24 17:33:07 -08:00
Aliaksandr Valialkin
045fec631b
docs/vmagent.md: typo fix 2022-11-24 17:19:45 -08:00
Aliaksandr Valialkin
ccf9bb32ac
app/vmselect/vmui: make vmui-update after 7dc2349913 2022-11-22 15:36:03 +02:00
Yury Molodov
7dc2349913
vmui: add set up series custom limits (#3368)
* feat: add set up series custom limits

* feat: add button for show series without limits

* fix: resolve merge conflicts
2022-11-22 15:31:17 +02:00
Aliaksandr Valialkin
9498f871e7
app/vminsert: add missing vm_relabel_config_* metrics after 03d88bc066
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
2022-11-22 00:47:49 +02:00
Roman Khavronenko
03d88bc066
vmagent: expose metrics for tracking config state (#3375)
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:38:43 +02:00
Aliaksandr Valialkin
2ddfde78c3
app/vmselect/vmui: make vmui-update after 7d1b3e7e14 2022-11-22 00:34:33 +02:00
Yury Molodov
7d1b3e7e14
vmui: add copy button to row on Table view (#3363)
* feat: add copy button to row on Table view

* vmui: add copy button to row on Table view

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-22 00:31:33 +02:00
Yury Molodov
f81072f9a7
vmui: minor fixes (#3361)
* fix: reset the value of the switches trace and cache

* fix: add cursor text for inputs

* fix: solve the Infinite loop of useFetchQuery.ts

* fix: change condition for show/hide autocomplete

* fix: add limit error length for input
2022-11-22 00:28:03 +02:00
Yury Molodov
82d254af08
vmui: sticky tooltip (#3376)
* feat: add ability to make tooltip "sticky"

* vmui: add ability to make tooltip "sticky"
2022-11-22 00:26:53 +02:00
Aliaksandr Valialkin
d9c3a2b605
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range
This may be useful for analyzing correlation between time series with different value ranges
2022-11-21 23:25:00 +02:00
Aliaksandr Valialkin
2c9e403d5f
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
2022-11-21 16:47:37 +02:00
Aliaksandr Valialkin
b796a0dc3f
app/vmselect/promql: optimize e1 op e2 when e1 returns an empty result
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
2022-11-21 16:09:10 +02:00
Roman Khavronenko
84742f229a
vmalert: add default list of alerting rules (#3373)
The default list of alerting rules contains the basic
rules for checking vmalert's health state and is recommended
to use for monitoring vmalert deployments.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-21 14:45:45 +01:00
Aliaksandr Valialkin
20d758e3e4
all: add a link to https://docs.victoriametrics.com/enterprise.html into description for enterprise flags 2022-11-21 15:42:01 +02:00
Aliaksandr Valialkin
cae0f37edd
app/vmselect/netstorage: remove superflouos map lookup at ProcessSearchQuery
This should reduce CPU usage a bit during querying
2022-11-18 13:40:04 +02:00
Yury Molodov
519bd2af7b
vmui: add trace analyzer (#3310)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add page with trace analyzer

* fix: change detect trace data

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* fix: change descriptions on trace page

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* feat: add base components

* feat: add reset styles

* docs: add description about trace analyzer

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

* feat: add paste json for trace analyzer

* vmui: update trace analyzer docs

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 22:22:01 +02:00
Aliaksandr Valialkin
578bb58ea9
app/vmselect/vmui: make vmui-update after 51bfd1ab80 2022-11-17 18:54:55 +02:00
Yury Molodov
51bfd1ab80
vmui: add ability hide query (#3359)
* feat: add ability hide query

* fix: change logic hide query

* fix: remove console.log
2022-11-17 18:42:33 +02:00
Dmytro Kozlov
fb65fb39d2
app/vmstorage: fix potential file inclusion via variable (#3339)
* app/vmstorage: fix potential file inclusion via variable

* app/vmstorage: cleanup
2022-11-17 01:29:43 +02:00
Aliaksandr Valialkin
a21c8e7b9a
app/vmselect/vmui: make vmui-update after bc8a782f74 2022-11-17 01:15:45 +02:00
Yury Molodov
bc8a782f74
vmui/refactor (#3298)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add base components

* feat: add reset styles

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 01:09:14 +02:00
Aliaksandr Valialkin
a260e2659e
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:40 +02:00
Aliaksandr Valialkin
c1a3192d8b
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:38:48 +02:00
Dmytro Kozlov
5ff6e0fb02
vmui: fix vmui vulnerability (#3336)
* vmui: fix vmui vulnerability

* vmui: code cleanup
2022-11-10 02:28:37 +01:00
Aliaksandr Valialkin
6c7361b1c5
app/vmselect/vmui: make vmui-update after 7130af7fd2 2022-11-09 16:43:18 +02:00
Michal Kralik
7130af7fd2
vmui: show tracing in json view (#3316)
* vmui: show tracing in json view

* vmui: refactor tracing view
2022-11-09 12:12:28 +02:00
dependabot[bot]
10791bf224
build(deps): bump loader-utils in /app/vmui/packages/vmui (#3328)
Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/webpack/loader-utils/releases)
- [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.3/CHANGELOG.md)
- [Commits](https://github.com/webpack/loader-utils/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: loader-utils
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 12:11:13 +02:00
Aliaksandr Valialkin
20046dab6e
app/vmui/packages/vmui: return back accidental changes at 9f8bf524ad 2022-11-09 11:55:34 +02:00
Roman Khavronenko
9f8bf524ad
bump go version to 1.19.3 (#3327)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-08 16:43:59 +01:00
Michal Kralik
9b540bba6f
vmui: change graph legend label format (#3315) 2022-11-08 15:16:24 +01:00
Aliaksandr Valialkin
d61691d5fa
deployment/docker: update Go builder from v1.19.2 to v1.19.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved
2022-11-05 10:19:54 +02:00
Roman Khavronenko
ac4e23de39
vmctl: fix panic on start (#3300)
The change disables initing the `-version` flag in new
`urfave/cli/v2` update. The `-version` flag conflicts
with the identical flag from `lib/buildinfo` and causes panic.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3299

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-01 19:50:22 +01:00
Aliaksandr Valialkin
cdd3443806
app/vmbackupmanager: add functionality for automated restore from backup 2022-10-29 02:30:52 +03:00
Aliaksandr Valialkin
9565dbed34
app/vmselect/vmui: make vmui-update after 54e1865d17 2022-10-28 14:51:23 +03:00
Yury Molodov
54e1865d17
vmui: minor fixes (#3276)
* feat: apply serverURL on down Enter

* fix: change method of set time range

* fix: remove prevent run fetch without changes

* fix: prevent reset timerange when autorefresh
2022-10-28 14:47:50 +03:00
Aliaksandr Valialkin
a018b1d75e
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-28 00:01:16 +03:00
Aliaksandr Valialkin
4bd0244599
Revert "vmalert: escape query params if external alert source defined (#3267)"
This reverts commit 00c838353d.

Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 .
Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana:

https://grafana.example.com/explore%3ForgId%3D1%26left%3D...

The next commit will contain the correct fix of the issue - the `quotesEscape` function must
properly escape the string, so it could be embedded into JSON string. This function must
properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary.
Actually, the next commit makes the `crlfEscape` function deprecated.
2022-10-27 22:30:27 +03:00
Dmytro Kozlov
00c838353d
vmalert: escape query params if external alert source defined (#3267)
vmalert: escape query args if external alert source defined
2022-10-26 10:00:14 -04:00
Aliaksandr Valialkin
518c340ae3
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:49:33 +03:00
Aliaksandr Valialkin
3c66e45ef0
app/vmselect/vmui: make vmui-update after eae6063450 2022-10-26 02:50:46 +03:00
Yury Molodov
eae6063450
fix: change step setting field (#3270)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-26 02:46:46 +03:00
Yury Molodov
794bd8b424
vmui: limit the number of decimal places to 10 characters (#3258)
* fix: limit the number of decimal places to 10 characters

* fix: add float numbers stabilizer
2022-10-26 02:43:13 +03:00
Yury Molodov
bc7456841f
vmui: add responsive styles for small screens (#3256)
* fix: add responsive styles for small screens

* fix: correct additional settings margins

* docs/CHANGELOG.md: add responsive styles
2022-10-26 02:39:54 +03:00
Aliaksandr Valialkin
c4265322f4
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin
b7882dc9af
app/vmselect/vmui: make vmui-update after 274e235bf7 2022-10-24 21:29:13 +03:00
Yury Molodov
274e235bf7
vmui: optimize memory (#3255)
* fix: change series limit logic

* fix: remove spread operators
2022-10-24 20:51:24 +03:00
Yury Molodov
59199a98dd
vmui: extend options for app mode (#3252)
* feat: add vmui customization for dbaas

* feat: extends vmui customization for dbaas

* fix: move input tenandId after query

* fix: change serverURL when changing tenantID

* fix: remove options

* docs: add options description
2022-10-24 20:45:41 +03:00
Aliaksandr Valialkin
c52c23c272
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:02:03 +03:00
Aliaksandr Valialkin
8e998aa1a1
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:40:20 +03:00
Aliaksandr Valialkin
5e4dfe50c6
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:23:47 +03:00
Aliaksandr Valialkin
a10647e0bf
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:13:47 +03:00
Roman Khavronenko
4e0ea95f26
vmalert: lower severity level for RW retries (#3237)
The message about dropped data still remains at `error` level.
The change supposed to make log message more clear about how
serious it is.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-18 14:27:20 +02:00
Aliaksandr Valialkin
3bb3893b2d
app/vmselect/vmui: make vmui-update after 080562030d 2022-10-18 10:50:35 +03:00
Yury Molodov
080562030d
fix: remove rounding of axis limits (#3238) 2022-10-18 10:47:58 +03:00
Aliaksandr Valialkin
069401a304
all: log error when environment variables referred from -promscrape.config are missing
This should prevent from using incorrect config files
2022-10-18 10:47:16 +03:00
Yury Molodov
ff6151fa49
vmui: limit number of plotted series (#3229)
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:13:47 +03:00
Aliaksandr Valialkin
92f7fe306e
app/vmagent/remotewrite: typo fix after c914e4dace 2022-10-13 12:04:10 +03:00
Aliaksandr Valialkin
e6fd33044f
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:03:08 +03:00
Siqi Liu
930f1ee153
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225)
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
2022-10-13 11:57:16 +03:00
Aliaksandr Valialkin
c914e4dace
app/vmagent/remotewrite: typo fix after 50f5eae0e0 2022-10-13 10:19:02 +03:00
Roman Khavronenko
96a106eab2
vmalert: update troubleshooting docs (#3228)
The default value of `-datasource.queryStep` has changed, so we update
the troubleshooting docs accordingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-12 12:12:37 +02:00