Commit graph

2359 commits

Author SHA1 Message Date
Aliaksandr Valialkin
61736e4a1d
app/vmselect/netstorage: remove superflouos map lookup at ProcessSearchQuery
This should reduce CPU usage a bit during querying
2022-11-18 13:49:59 +02:00
Aliaksandr Valialkin
eb784ff399
app/vmselect/netstorage: emit more useful information in query traces when some of vmstorage nodes return errors or if there is no need to wait for their responses 2022-11-18 13:01:42 +02:00
Yury Molodov
60aaf4b2c2
vmui: add trace analyzer (#3310)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add page with trace analyzer

* fix: change detect trace data

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* fix: change descriptions on trace page

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* feat: add base components

* feat: add reset styles

* docs: add description about trace analyzer

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

* feat: add paste json for trace analyzer

* vmui: update trace analyzer docs

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 22:23:05 +02:00
Aliaksandr Valialkin
7969ca6c80
app/vmselect/vmui: make vmui-update after 51bfd1ab80 2022-11-17 18:57:44 +02:00
Yury Molodov
734172d2d0
vmui: add ability hide query (#3359)
* feat: add ability hide query

* fix: change logic hide query

* fix: remove console.log
2022-11-17 18:57:42 +02:00
Dmytro Kozlov
6f7956503f
app/vmstorage: fix potential file inclusion via variable (#3339)
* app/vmstorage: fix potential file inclusion via variable

* app/vmstorage: cleanup
2022-11-17 01:31:37 +02:00
Aliaksandr Valialkin
f67c2a2f8f
app/vmselect/vmui: make vmui-update after bc8a782f74 2022-11-17 01:21:56 +02:00
Yury Molodov
423eb73051
vmui/refactor (#3298)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add base components

* feat: add reset styles

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 01:21:54 +02:00
Aliaksandr Valialkin
0cc59b9f95
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:02 +02:00
Aliaksandr Valialkin
facf7efa3a
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:39:17 +02:00
Dmytro Kozlov
8b6bfae5ca
vmui: fix vmui vulnerability (#3336)
* vmui: fix vmui vulnerability

* vmui: code cleanup
2022-11-10 13:32:50 +02:00
Aliaksandr Valialkin
58e7963828
app/vmselect/vmui: make vmui-update after 7130af7fd2 2022-11-09 16:43:09 +02:00
Michal Kralik
051aa079cb
vmui: show tracing in json view (#3316)
* vmui: show tracing in json view

* vmui: refactor tracing view
2022-11-09 12:13:05 +02:00
dependabot[bot]
9c5d01d8c3
build(deps): bump loader-utils in /app/vmui/packages/vmui (#3328)
Bumps [loader-utils](https://github.com/webpack/loader-utils) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/webpack/loader-utils/releases)
- [Changelog](https://github.com/webpack/loader-utils/blob/v2.0.3/CHANGELOG.md)
- [Commits](https://github.com/webpack/loader-utils/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: loader-utils
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 12:11:52 +02:00
Aliaksandr Valialkin
001b4da314
app/vmui/packages/vmui: return back accidental changes at 9f8bf524ad 2022-11-09 11:58:13 +02:00
Roman Khavronenko
8ee464b22b
bump go version to 1.19.3 (#3327)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 11:56:38 +02:00
Michal Kralik
651a3aa18b
vmui: change graph legend label format (#3315) 2022-11-09 11:52:48 +02:00
Aliaksandr Valialkin
fe8d40f12c
app/{vminsert,vmselect}: test initialization with different number of storage nodes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
2022-11-09 11:48:39 +02:00
Aliaksandr Valialkin
8540dd669b
app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket
This should prevent from panics when the list of discovered vmstorage nodes changes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
2022-11-09 11:45:24 +02:00
Aliaksandr Valialkin
61bcbfd697
deployment/docker: update Go builder from v1.19.2 to v1.19.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved
2022-11-05 10:19:24 +02:00
Roman Khavronenko
803ab6ed30
vmctl: fix panic on start (#3300)
The change disables initing the `-version` flag in new
`urfave/cli/v2` update. The `-version` flag conflicts
with the identical flag from `lib/buildinfo` and causes panic.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3299

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-05 10:09:11 +02:00
Zakhar Bessarab
fd36c9b3b1
docs/{vminsert,vmselect}: clarify flag description for clusternativeListenAddr (#3314)
docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`
2022-11-04 09:49:53 +01:00
Zakhar Bessarab
7afa67dba0
{app/vmselect,app/vminsert}: ensure -storageNodes flag has no empty values (#3291) 2022-11-01 14:54:55 +02:00
Aliaksandr Valialkin
cf5efb93d9
app/vmbackupmanager: add functionality for automated restore from backup 2022-10-29 02:31:06 +03:00
Aliaksandr Valialkin
a02a5f646f
app/vmselect/vmui: make vmui-update after 54e1865d17 2022-10-28 14:52:16 +03:00
Yury Molodov
9732790935
vmui: minor fixes (#3276)
* feat: apply serverURL on down Enter

* fix: change method of set time range

* fix: remove prevent run fetch without changes

* fix: prevent reset timerange when autorefresh
2022-10-28 14:52:15 +03:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
976bbe3677
app/{vminsert,vmselect}: limit the access to storageNodes to getStorageNodesBucket and setStorageNodesBucket functions
This makes the code more maintainable and earier to test.
2022-10-28 11:41:55 +03:00
Aliaksandr Valialkin
7ae038766c
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
8a6898b625
Revert "vmalert: escape query params if external alert source defined (#3267)"
This reverts commit 00c838353d.

Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 .
Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana:

https://grafana.example.com/explore%3ForgId%3D1%26left%3D...

The next commit will contain the correct fix of the issue - the `quotesEscape` function must
properly escape the string, so it could be embedded into JSON string. This function must
properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary.
Actually, the next commit makes the `crlfEscape` function deprecated.
2022-10-28 00:08:50 +03:00
Dmytro Kozlov
3123059407
vmalert: escape query params if external alert source defined (#3267)
vmalert: escape query args if external alert source defined
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Aliaksandr Valialkin
46450995ae
app/vmselect/vmui: make vmui-update after eae6063450 2022-10-26 02:51:09 +03:00
Yury Molodov
1847a207f8
fix: change step setting field (#3270)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-26 02:47:41 +03:00
Yury Molodov
346d49ae61
vmui: limit the number of decimal places to 10 characters (#3258)
* fix: limit the number of decimal places to 10 characters

* fix: add float numbers stabilizer
2022-10-26 02:43:43 +03:00
Yury Molodov
ab880afa50
vmui: add responsive styles for small screens (#3256)
* fix: add responsive styles for small screens

* fix: correct additional settings margins

* docs/CHANGELOG.md: add responsive styles
2022-10-26 02:40:41 +03:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
596251eb87
app/vmselect/vmui: make vmui-update after 274e235bf7 2022-10-24 21:29:57 +03:00
Yury Molodov
96a579c13b
vmui: optimize memory (#3255)
* fix: change series limit logic

* fix: remove spread operators
2022-10-24 21:03:02 +03:00
Yury Molodov
5de090ab43
vmui: extend options for app mode (#3252)
* feat: add vmui customization for dbaas

* feat: extends vmui customization for dbaas

* fix: move input tenandId after query

* fix: change serverURL when changing tenantID

* fix: remove options

* docs: add options description
2022-10-24 20:49:46 +03:00
Aliaksandr Valialkin
8ea84432ef
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
c92aef39b5
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:14:07 +03:00
Roman Khavronenko
f7d69c1735
vmalert: lower severity level for RW retries (#3237)
The message about dropped data still remains at `error` level.
The change supposed to make log message more clear about how
serious it is.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin
574b9f4f78
app/vmselect/vmui: make vmui-update after 080562030d 2022-10-18 10:51:05 +03:00
Yury Molodov
14c7223dc4
fix: remove rounding of axis limits (#3238) 2022-10-18 10:51:03 +03:00
Aliaksandr Valialkin
d0288ea417
all: log error when environment variables referred from -promscrape.config are missing
This should prevent from using incorrect config files
2022-10-18 10:29:59 +03:00
Yury Molodov
a50f2e141c
vmui: limit number of plotted series (#3229)
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:14:53 +03:00
Aliaksandr Valialkin
c7360e1fa0
app/vmagent/remotewrite: typo fix after c914e4dace 2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
9bf8d64605
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:06:24 +03:00
Siqi Liu
c0fc716426
BUGFIX: properly calculate histogram_quantile with the same value and different string le (#3225)
Co-authored-by: 647(siki.liu) <siki.liu@huolala.cn>
2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
207b4a6c58
app/vmagent/remotewrite: typo fix after 50f5eae0e0 2022-10-13 10:19:46 +03:00
Roman Khavronenko
895cb3e7c6
vmalert: update troubleshooting docs (#3228)
The default value of `-datasource.queryStep` has changed, so we update
the troubleshooting docs accordingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-13 10:14:40 +03:00
Zakhar Bessarab
4e1c12a10d
doc: describe usage of env variables for obtaining credentials (#3219) 2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin
a0f3247b14
app/vmselect/promql: properly handle zero and negative values for -search.maxMemoryPerQuery
This is a follow-up for 04a05f161c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-12 09:33:16 +03:00
Roman Khavronenko
7fc812d3c4
vmalert: revert unexpected fileds rename during refactoring (#3222)
Due to auto-refactoring, the filed `state` was automatically
renamed to `ruleState` when the entity with the same name
was renamed in other file. Reverting the change.

https://github.com/VictoriaMetrics/helm-charts/issues/391
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin
098c9bda27
app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set
This is needed for preserving backwards compatibility with the previous releases of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-10 21:54:18 +03:00
Howie
d9abdc57d4
fix issue#3053 (#3182)
vmalert: prevent duplicating label `alertname` for notifications

The issue has no impact on alerting procedure. But still needs to be fixed
for clarity. 

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3053

Signed-off-by: lihaowei <haoweili35@gmail.com>
2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin
087393bcef
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.

This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin
d8057fca0b
docs: mention -search.maxMemoryPerQuery in the description to -search.maxConcurrentQueries command-line flag
This is a follow-up for 5138eaeea0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-09 13:58:18 +03:00
Aliaksandr Valialkin
62e9dfeee6
docs/vmbackupmanager.md: update docs after adding the support to make backups to Azure blob storage
This is a follow-up for 262ce77e2d
2022-10-08 10:30:58 +03:00
Aliaksandr Valialkin
938ff7bba6
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-08 01:14:01 +03:00
Aliaksandr Valialkin
3b828535f0
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
2022-10-07 23:37:31 +03:00
Yury Molodov
9e4e5abac6
vmui: auto-update chart after query field removed (#3210)
* feat: run query after query field removed

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-07 23:08:22 +03:00
Aliaksandr Valialkin
98a4ab796c
all: update the minimum required Go verson from 1.19.1 to 1.19.2
This is needed because of security vulnerabilities found in Go 1.19.1
See https://go.dev/doc/devel/release#go1.19.2
2022-10-07 22:46:44 +03:00
Aliaksandr Valialkin
98bd843f3c
app/{vminsert,vmselect,vmstorage}: add package-{vminsert,vmselect,vmstorage}-{goarch} Makefile rules for building arch-specific docker images 2022-10-07 20:17:01 +03:00
Aliaksandr Valialkin
58ee1b9edb
app/vmselect/vmui: make vmui-update after a54987f671 2022-10-07 03:31:00 +03:00
Aliaksandr Valialkin
916f1ab86c
app/vmstorage/servers: properly copy MetricName into MetricBlock inside blockIterator.NextBlock
This should fix the issue at cf36bfa189
2022-10-07 02:57:54 +03:00
Aliaksandr Valialkin
b05ff477c8
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:23:17 +03:00
Aliaksandr Valialkin
baa11a778d
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:38 +03:00
Roman Khavronenko
de92a8375c
vmalert: fix misleading line regarding multitenancy (#3206)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-06 15:10:52 +03:00
Aliaksandr Valialkin
958c1f291c
app: follow-up after ec04fcac93
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:54:15 +03:00
Dmytro Kozlov
4064db27a8
Properly parse json when export import metric (#3180)
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API

* app/vmselect: update convert function

* app/vmselect: export null if `math.IsNaN(v)`

* app/vmselect: get float from json

* lib/protoparser: add test

* docs: add change log

* lib/protoparser: make export import api compatible
2022-10-06 14:54:14 +03:00
Yury Molodov
112ae2978d
vmui: maximum queries (#3196)
vmui: allow using up 4 queries at the same time

The change also introduces UI updates to make using 
multiple queries more conveniently.
2022-10-06 13:30:47 +03:00
Aliaksandr Valialkin
e6d866b37c
docs: follow-up after 262ce77e2d
* Document the addition of Azure blob storage support in vmbackup / vmrestore
* List the supported storage system types at docs/vmrestore.md
* Mention about azblob storage system support at -src and -dst command-line flags
  for vmbackup / vmrestore tools.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029
2022-10-06 00:36:20 +03:00
Zakhar Bessarab
6a6dcc059b
lib/backup: add support of Azure Blob Storage (#460)
* lib/backup: add support of Azure Blob Storage

* lib/backup: add enterprise support of Azure Blob Storage
2022-10-06 00:36:19 +03:00
Aliaksandr Valialkin
9b1443bde5
app/vmalert: follow-up after f8ac55d70ada9ef8490b322abefb05f28f75e2e9
* Use vm_account_id and vm_project_id labels to be consistent with https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels
* Document the feature that vmalert now exposes vm_account_id and vm_project_id
  labels if -clusterMode is set.
* Use literal strings instead of string constants for vm_account_id and vm_project_id.
  This improves code readability.
2022-10-06 00:06:06 +03:00
Aliaksandr Valialkin
98d58fdb57
app/vmalert: update -external.alert.source command-line flag description after 61544e13ad 2022-10-05 22:54:23 +03:00
Roman Khavronenko
6f6f6afae0
vmalert: allow using {{$labels}} for templating in -external.alert.source (#3194)
The change is supposed to provide additional flexibility for generating alert's
source link based on label values.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-05 22:53:02 +03:00
Aliaksandr Valialkin
43bdd96a6e
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:16:07 +03:00
Aliaksandr Valialkin
5497997b72
app/vmselect/promql: increase scalability of incremental aggregate calculations on systems with many CPU cores
Use sync.Map instead of a global mutex there. This should lift scalability limits
on systems with many CPU cores.
2022-10-01 20:14:14 +03:00
Aliaksandr Valialkin
972fcfcd17
app/vmselect/prometheus: improve scalability of /federate endpoint on systems with many CPU cores
Minimize usage of global lock inside bufferedwriter.Write() when processing `/federate` data
on systems with many CPU cores
2022-10-01 20:14:13 +03:00
Aliaksandr Valialkin
ae6c7edf04
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-01 19:48:21 +03:00
Aliaksandr Valialkin
e9f1213cc9
docs/vmagent.md: fix incorrect url for multitenant writes to VictoriaMetrics cluster 2022-10-01 18:51:57 +03:00
Aliaksandr Valialkin
b857365b84
app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin
6f9ce3f6d6
lib/flagutil: rename Array to ArrayString
This makes the ArrayString more consistent with other Array* types.

While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
2022-10-01 18:28:19 +03:00
Aliaksandr Valialkin
93e84a1c57
lib/httpserver: use 302 redirects instead of 301 redirects
Incorrect 301 redirects can be cached by user agents such as web browsers.
This can complicate recovery procedure after the incorrect redirect is fixed,
e.g. web browser cache must be reset.

The related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:56:43 +03:00
Aliaksandr Valialkin
e03a924236
app/vmauth: do not remove trailing slash from the proxied path
This should fix the issue with opening VMUI at /vmui/ page.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:56:42 +03:00
Aliaksandr Valialkin
d81285c814
app/vmagent: accept requests with /prometheus and /influx path prefixes in the same way as VictoriaMetrics does
This allows using vmagent as a drop-in replacement for VictoriaMetrics
for push protocols.
2022-10-01 14:11:09 +03:00
Aliaksandr Valialkin
b5fe42514d
docs/vmagent.md: formatting fixes 2022-10-01 13:36:48 +03:00
Aliaksandr Valialkin
9676519830
app/vmagent/README.md: apply typo fix from 1fda517af9 2022-10-01 12:31:42 +03:00
Roman Khavronenko
408d7043a1
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 18:38:11 +03:00
Aliaksandr Valialkin
c8000c029a
app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints
These labels are substituted by documented vm_account_id and vm_project_id labels.

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels

This is a follow up for 505d359b39
2022-09-30 18:06:00 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
Dmytro Kozlov
d0a9fff70c
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:56:43 +03:00
Aliaksandr Valialkin
98175493d7
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:44:06 +03:00
Roman Khavronenko
273472dc20
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:44:05 +03:00
Roman Khavronenko
a2ded58600
vmalert: allow using extra labels in annotations (#3181)
According to Ruler specification, only labels returned within time series
should be available for use in annotations.

For long time, vmalert didn't respect this rule. And in PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2403
this was fixed for the sake of compatibility. However, this resulted
into users confusion, as they expected all configured and extra labels
to be available - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013

This fix allows to use extra labels in Annotations. But in the case of conflicts
the original labels (extracted from time series) are preferred.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 07:48:59 +03:00
panguicai
156e7035c7
docs: fix typo for vmalert docs (#3173)
Signed-off-by: panguicai008 <1121906548@qq.com>
2022-09-28 10:42:13 +03:00
Aliaksandr Valialkin
7f0b95b50a
lib/promrelabel: add SanitizeName() function for sanitizing Prometheus metric names and label names
Optimize this function by using results cache for input strings.
Use this function all over the code.

This is a follow-up for fcffdba9dc

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113
2022-09-28 10:02:11 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Roman Khavronenko
4acbf6e7d4
vmselect/rollup: rm workaround for slow-changing counters (#3163)
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962.
However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating
`delta` or `increase` functions over slow-changing counters with vary intervals
between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 15:39:49 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin
6a6cf9c590
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:13:31 +03:00
Aliaksandr Valialkin
594a4ab345
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-09-26 08:49:47 +03:00
Aliaksandr Valialkin
a6a869c365
docs/vmalert.md: follow-up after 7748a9d629
- Consistently use single dash in front of command-line flags instead of double dashes.
- Add a warning that too small -search.latencyOffset may lead to incomplete query results.
2022-09-26 08:49:47 +03:00
Roman Khavronenko
cffceba0f5
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:40:17 +03:00
Roman Khavronenko
b86cf7d707
vmalert: add info about search.latencyOffset to Troubleshooting (#3151)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:38:23 +03:00
Dmytro Kozlov
ed842e7d3a
app/{vmctl,vmalert}: update progress bar library (make vendor-update) (#3138)
* app/{vmctl,vmalert}: update progress bar library (make vendor-update)

* app/{vmctl,vmalert}: make vendor-update
2022-09-21 11:11:40 +03:00
Roman Khavronenko
74e81d31a7
vmalert: prodvide more details on duplicates (#3136)
Now vmalert will print the following messages on dupliсates:
```
"recording rule \"record\"; expr: \"up == 1\"; labels: summary={{ value|query }}" is a duplicate within the group "test"
"alerting rule \"alert\"; expr: \"up == 1\"; labels: description={{ value|query }}" is a duplicate within the group "test"
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3127
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-21 11:11:40 +03:00
Aliaksandr Valialkin
336007182c
app/vmagent/remotewrite: go fmt after 2b55d167d7 2022-09-19 15:14:03 +03:00
Roman Khavronenko
360b022603
vmalert: always re-evaluate Annotations (#3119)
* vmalert: always re-evaluate Annotations

Previously, Annotations were evaluated only:
1. On alert creating.
2. On alert's value change.

This is premature optimization. It was assumed that since annotations
could contain only text with alert's labels or value - there is no need
in spending resources to re-compile Annotations.

Later, template function `query` was added, which can execute
arbitrary queries and return different results on every evaluation.
So if it was used in annotations, it would be executed only on init
or value change.

Another case when optimization caused an issue - annotations hot reload.
In this case, annotations of the active alert won't change even if Rule's
annotations were changed.

This fix enables Annotations re-evaluation on each iteration to resolve
issues above. It would have some impact on performance, but it is unlikely
it will be noticeable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add tp Changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-19 15:04:37 +03:00
Roman Khavronenko
1c13cce5ed
vmalert: add Troubleshooting section to docs (#3115)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-19 15:04:37 +03:00
Roman Khavronenko
09e211a05f
vmalert: print example of curl command for rule's state (#3112)
The change adds an example of `curl` command to the Rule's page.
The command is generated for each recorded state. It is supposed
user can just copy&execute the command to see what was returned
to vmalert.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-19 15:04:37 +03:00
Aliaksandr Valialkin
34db3fdd3f
app/vmagent/remotewrite: add benchmarks for comparing the performance of standard Snappy encoder with github.com/klauspost/compress/s2 encoder
The standard Snappy encoder from github.com/golang/snappy shows quite good performance number
for compressing the Prometheus remote_write proto messages according to the added benchmarks,
so there is no need in switching to github.com/klauspost/compress/s2 yet.
2022-09-19 14:27:56 +03:00
Dmytro Kozlov
52a79a901e
app/vmctl: add description about influx-skip-database-label flag (#3111) 2022-09-15 13:24:16 +03:00
Aliaksandr Valialkin
713ebb8297
docs: update -help output after explicit mentioning of enterprise flags 2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin
d77eb5170c
docs/vmauth.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin
424bcfc17b
docs/vmalert.md: update -help output after explicit marking of enterprise flags 2022-09-15 13:19:02 +03:00
Aliaksandr Valialkin
1275e0580e
docs: update docs with explicitly marked enterprise command-line flags for VictoriaMetrics and vmagent 2022-09-15 13:19:01 +03:00
Dmytro Kozlov
36ea8537d9
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:42:07 +03:00
Roman Khavronenko
6ae4f3526b
vmalert: add experimental feature of storing Rule's evaluation state (#3106)
vmalert: add experimental feature of storing Rule's evaluation state

The new feature keeps last 20 state changes of each Rule
in memory. The state are available for view on the Rule's
view page. The page can be opened by clicking on `Details`
link next to Rule's name on the `/groups` page.

States change suppose to help in investigating cases when Rule
doesn't generate alerts or records.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 15:08:40 +03:00
Roman Khavronenko
d071e39694
bump Go version to 1.19.1 (#3108)
The reason is to cover vulnerability GO-2022-0969
Found in: net/http@go1.18.5
Fixed in: net/http@go1.19.1
More info: https://pkg.go.dev/vuln/GO-2022-0969

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 13:43:27 +03:00
Aliaksandr Valialkin
b6bdbb4787
app/vmselect/vmui: make vmui-update after 1304824201 2022-09-14 13:32:43 +03:00
Yury Molodov
46e8bd34f7
vmui: fix query params saving in URL (#3104) 2022-09-13 19:19:57 +03:00
Aliaksandr Valialkin
20834c1757
app/vmalert: follow-up after 8441375da2
- Rename logDebug() to logDebugf() and pass format string together
  with format args directly to logDebugf(). This eliminates fmt.Sprintf()
  overhead at logDebug() call site when debugging is disabled.

- Format labels in debug message in Prometheus format, e.g. {label1="value1",...labelN="valueN"}

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025
2022-09-13 16:36:31 +03:00
Roman Khavronenko
a887c1bc07
vmalert: add debug mode for alerting rules (#3055)
* vmalert: add `debug` mode for alerting rules

Debug information includes alerts state changes and requests
sent to the datasource. Debug can be enabled only on rule's
level. It might be useful for debugging unexpected
behaviour of alerting rule.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmalert/alerting.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

* vmalert: go fmt

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-13 16:36:30 +03:00
Yury Molodov
465894d720
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:56:39 +03:00
Yury Molodov
6494808a24
fix: change columns for Top Queries (#3093) 2022-09-12 10:46:48 +03:00
Yury Molodov
c8de98e03f
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:44:43 +03:00
Dmytro Kozlov
ec273eafef
app/{vmselect, vminsert}: fails with error when user defines equal addresses in the -storageNodes flag (#3082) 2022-09-08 21:17:58 +03:00
Aliaksandr Valialkin
e619fa8b38
deployment/docker: update Go builder for prod binaries from Go1.19.0 to Go1.19.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.1+label%3ACherryPickApproved
2022-09-08 18:27:55 +03:00
Aliaksandr Valialkin
f0eea5b02d
app/vmselect/netstorage: fix a typo, which leads to incorrect query results in VictoriaMetrics cluster
The typo has been introduced in the commit 1a254ea20c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067
2022-09-08 13:46:40 +03:00
Aliaksandr Valialkin
6ebe4f5392
docs/Single-server-VictoriaMetrics.md: add a note that cardinality explorer at cluster version of VictoriaMetrics may return lower than expected number of unique label values
See the corresponding comment in the code:

5a6e617b5e/app/vmselect/netstorage/netstorage.go (L1039-L1045)
2022-09-06 14:48:21 +03:00
Dmytro Kozlov
96ecec877d
vmselect/{promql, prometheus}: show flag names which user can update in error message (#3049)
* vmselect/{promql, prometheus}: show flag names which user can update in error message

* vmselect/{promql, prometheus}: fix typo
2022-09-06 14:48:20 +03:00
Aliaksandr Valialkin
f6990871a3
docs/vmctl.md: make docs-sync after c5261d5f56
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:20:16 +03:00
Zakhar Bessarab
ef01f01b6c
vmctl: implement support of chunking data for vm-native export (#3044)
vmctl: implement support of chunking data for vm-native export process

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:14:48 +03:00
Aliaksandr Valialkin
b7f3569522
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:57:08 +03:00
匠心零度
65d1124dc4
api prometheus/api/v1/label/../values time not specified, (#3052)
modify default start values
2022-09-05 11:51:32 +03:00
Aliaksandr Valialkin
c6b74148cf
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:19:05 +03:00
Aliaksandr Valialkin
9cca3a0a1b
app/vmselect/netstorage: fix potential panic under high load
The panic may trigger during data blocks' processing received
from vmstorage nodes when some of vmstorage nodes return an error
or when `-replicationFactor` is set to values higher than 2 at `vmselect`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058
2022-09-02 21:36:15 +03:00
Aliaksandr Valialkin
024e2f18da
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 21:01:04 +03:00
匠心零度
80067fd03a
reduce unnecessary vmstorage query (#3031)
* reduce unnecessary vmstorage query

* reduce unnecessary vmstorage query

* rollback limit logic /api/v1/label/*
2022-08-30 12:34:23 +03:00
Aliaksandr Valialkin
529bcc0761
docs/CHANGELOG.md: document the 044d51b668 2022-08-30 09:42:51 +03:00
Aliaksandr Valialkin
8aaaf221cc
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:38:17 +03:00
Dmytro Kozlov
2cf29cc4d1
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:34:54 +03:00
Zakhar Bessarab
1c7db1c8fd
vmctl: fix progress bar not being stopped on error during import process (#3023)
vmctl: fix progress bar not being stopped on error during import process
2022-08-26 21:51:20 +03:00
Aliaksandr Valialkin
d4cf52f092
app/vmagent: follow-up after 2b22aa1537
- Document the change at docs/CHANGELOG.md
- Move auth token parsing from app/vmagent/opentsdbhttp/ to app/vmagent/main.go,
  since it must be parsed only when multitenancy support is enabled at vmagent side.
  See https://docs.victoriametrics.com/vmagent.html#multitenancy
2022-08-24 16:21:23 +03:00
Jianyun Cheng
eccae22522
[vmagent] make opentsdb insert url support multitenant (#3015) 2022-08-24 16:21:22 +03:00
Dmytro Kozlov
d32a6359b0
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:27:41 +03:00
Aliaksandr Valialkin
e2d8916935
docs: mention that it is safe sharing the collected profiles from security PoV
The collected profiles do not contain sensitive information
2022-08-24 14:08:30 +03:00
Aliaksandr Valialkin
7b9ba456ff
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:

    vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:41:57 +03:00
Aliaksandr Valialkin
547d655d29
docs/vmagent.md: fix alerting query when scraped samples are dropped because of exceeded series limit
This is a follow-up after 7d26414b2e
2022-08-24 01:23:37 +03:00
Roman Khavronenko
555a202d80
docs: follow-up after 88425bb285 (#3007)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-24 01:23:37 +03:00
laixintao
76a291a95b
vmalert: add $activeAt into template variables. (#3000)
vmalert: add `$activeAt` template variable for annotations

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2999
2022-08-24 01:23:37 +03:00
Aliaksandr Valialkin
08b8467e97
app/vmselect/netstorage: make golangci-lint happy by naming the unused padding field as _ 2022-08-22 00:32:37 +03:00
Aliaksandr Valialkin
9ddd2699fd
all: remove the remaining bits of io/ioutil
The io/ioutil package is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics requires at least Go1.18, so it is time to remove the io/ioutil from source code

This is a follow-up for 02ca2342ab
2022-08-22 00:22:41 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
1c7f402598
app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling
For example:

- target_label: composite-label
  replacement: {{source_label1}}-{{source_label2}}
2022-08-21 22:49:24 +03:00
Aliaksandr Valialkin
6d2354e7a4
app/vmalert/README.md: sync with docs/vmalert.md after a229182dbe 2022-08-20 08:54:55 +03:00
Roman Khavronenko
2256d51418
docs: fix docs formatting related to vmalert (#2994)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-19 11:05:08 +03:00
Aliaksandr Valialkin
29eb9d4602
docs/relabeling.md: add a cookbook for common relabeling tasks 2022-08-18 01:14:30 +03:00
Aliaksandr Valialkin
bbfa52bd75
docs: follow-up after 68e56b6fc5 2022-08-17 21:27:24 +03:00
Roman Khavronenko
cfcb5ab15b
vmalert: set alert's source link to UI instead of JSON source (#2986)
We switch default alert's source link to redirect user
to vmalert's UI instead of previous JSON object. While it breaks
compatibility, it also supposed to improve user's experience.
The old behavior can be achieved by updating `-external.alert.source`
command-line flag.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-17 21:27:24 +03:00
Aliaksandr Valialkin
87e0d69bf4
app/vmselect/netstorage: fix a bug introduced in 1a254ea20c
The bug results in `duplicate output time series` error
because the same time series is added two times into the orderedMetricNames list
inside the tmpBlocksFileWrapper.Finalize().

While at it, properly release all the tmpBlocksFile structs on tbf.Finalize() error.
Previously only the remaining tbf entries were released. This could result in resource leak.
2022-08-17 14:07:51 +03:00
Aliaksandr Valialkin
1812d33a2d
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit
The following metrics are generated:

- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped

These metrics simplify alerting on targets, which expose too many time series

See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
2022-08-17 13:22:02 +03:00
Roman Khavronenko
436b4d90af
docs: update vmalert docs (#2987)
* mention recently added `$alertID` and `$groupID` variables in the changelog
* properly escape template examples in the vmalert's README

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-16 12:10:08 +03:00
Roman Khavronenko
0d5c403b6e
vmalert: support $alertID and $groupID in template variables (#2983)
Support of these two variables allows building custom URLs with
alert's ID and group ID params.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/517#issuecomment-1207141432

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-16 11:24:01 +03:00
Aliaksandr Valialkin
7d7cf2b6fd
app/vmselect: follow-up after 63e0f16062
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
2022-08-15 13:53:19 +03:00
Roman Khavronenko
8a26ec435d
vmselect: introduce UserReadableError type of error (#2894)
When read query fails, VM returns rich error message with
all the details. While these details might be useful
for debugging specific cases, they're usually too verbose
for users.
Introducing a new error type `UserReadableError` is supposed
to allow to return to user only the most important parts
of the error trace. This supposed to improve error readability
in web interfaces such as VMUI or Grafana.

The full error trace is still logged with the full context
and can be found in vmselect logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-15 13:53:18 +03:00
Yury Molodov
dc52b283a3
vmui: shortcut keys legend (#2971)
* feat: add shortcut modal

* feat: add shortcut descriptions

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-15 12:04:30 +03:00
Aliaksandr Valialkin
c27bd63f6c
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html 2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin
198d8eeeaa
app/vmalert/templates: add toTime() template function in the same way as Prometheus 2.38 does
See https://github.com/prometheus/prometheus/pull/10993
2022-08-15 00:49:52 +03:00
Roman Khavronenko
36660bcfe2
vmalert: follow-up after 28441711e6 (#2972)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-11 23:52:18 +03:00
Matthew Blewitt
b1b8c99b17
vmalert: mark some url flags as sensitive (#2965)
Other components, such as `vmagent`, mark these flags as sensitive and
hide them from the `/metrics` endpoint by default. This commit adds
similar handling to the `vmalert` component, hiding them by default, to
prevent logging of secrets inappropriately.

Showing of these values is controlled by an additional flag.

Follow up to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2947
2022-08-11 23:51:43 +03:00
Aliaksandr Valialkin
1a254ea20c
app/vmselect/netstorage: remove common contention points related to inter-CPU communcations
This should improve vmselect performance scalability on systems with many CPU cores.

The following tasks were done:

- Use separate temporary files for storing the data read from each vmstorage node.
  This may result in the following potential issues:
  - Up to N times higher memory usage for performing each query where N is the number
    of vmstorage nodes known to vmselect.
    This issue shouldn't increase chances of out of memory errors in most cases,
    since per-query memory overhead is quite low comparing to the overall vmselect memory usage.
  - Up to N times higher number of open temporary files where N is the number
    of vmstorage nodes known to vmselect.
    This issue should be fixed by increasing the limit on the number of open files.

- Use separate counters per each vmstorage node for various stats calculation
  when reading the data from vmstorage nodes.
2022-08-11 23:22:56 +03:00
Aliaksandr Valialkin
ec3df0b913
app/vmselect/netstorage: improve scalability of blocks processing on systems with multiple CPU cores
Previously a single syncwg.WaitGroup was used for tracking the lifetime of processBlock callbacks
across all the per-vmstorage goroutines. This could be slow on systems with many CPU cores
because of inter-CPU synchronization overhead.

Use a separate per-vmstorage sync.WaitGroup instead in order to reduce inter-CPU synchronization overhead.
This should imrpove performance for heavy queries over big number of blocks on multi-CPU systems.
2022-08-11 21:37:24 +03:00
Roman Khavronenko
f90f654cf2
vmalert: sort groups at /alerts page (#2968)
Sorting will produce deterministic output
of grops on the page.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-09 12:08:54 +03:00
Aliaksandr Valialkin
ceb158c5bb
docs/vmagent.md: typo fix in __meta_kubernetes_annotation_prometheus_io_tenant label name
It must be __meta_kubernetes_pod_annotation_prometheus_io_tenant
2022-08-08 14:53:28 +03:00
Aliaksandr Valialkin
310779d8b5
lib/promscrape: follow-up after 2c553d5a2f
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
2022-08-08 14:49:16 +03:00
Fury
59fdb4cb72
add support to scrape multi tenant metrics (#2950)
* add support to scrape multi tenant metrics

* add support to scrape multi tenant metrics

Co-authored-by: 赵福玉 <zhaofuyu@zhaofuyudeMac-mini.local>
2022-08-08 14:49:15 +03:00
Aliaksandr Valialkin
69d62d5736
docs: sync -help output with the latest changes 2022-08-08 14:05:36 +03:00
Aliaksandr Valialkin
221dd3a224
all: bump the minimum supported version of Go from 1.17 to 1.18
This is needed because some dependencies uses generics, which have been appeared in Go1.18

This is a follow-up for caf3dd4fa2
2022-08-08 13:45:39 +03:00
Roman Khavronenko
6af40f6275
vmalert: remove notions of vmalert being compatible with VM only (#2954)
vmalert can be successfully used with datasources
compatible with Prometheus HTTP API. So we remove comments or
notes in Readme which are saying opposite.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-08 13:44:15 +03:00
Aliaksandr Valialkin
1996e36cf0
app/vmselect/netstorage: prevent from calling processBlocks callback after the exit from ProcessBlocks function
This should prevent from panic at multi-level vmselect
when the top-level vmselect is configured with -replicationFactor > 1
2022-08-08 13:32:44 +03:00
laixintao
9f5fc040a7
bugfix: fix vmalert navbar url. (#2949)
the doc url should not be joined by `prefix` because it's an abs url.
2022-08-08 00:29:48 +03:00
Aliaksandr Valialkin
2635211bf4
app/vmselect/netstorage: properly detect and log timeout errors when querying vmstorage from vmselect
This change is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2937

Thanks to @isodude for the initial pull request.
2022-08-08 00:21:05 +03:00
Yury Molodov
a2ecf311b8
vmui: graph action on moush hold and move (#2915)
* fix: change event for graph panning

* fix: change detect key

* feat: add zoom in with mouse selection

* - document the change
- run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-07 23:56:08 +03:00
Aliaksandr Valialkin
f28f4ad551
deployment/docker: update Go builder from Go1.18.5 to Go1.19.0
See https://tip.golang.org/doc/go1.19

Notable changes:

* GOMEMLIMIT environment variable - see https://tip.golang.org/doc/gc-guide
* Faster CPU profiler
* Faster sort algorithm
2022-08-07 21:31:20 +03:00
Aliaksandr Valialkin
21f588e9cf
app/vmselect/promql: fix TestVmrangeBucketsToLE test after 8516670582
Remove 'AccountID=0, ProjectID=0, ' prefix for cluster version
2022-08-07 00:15:10 +03:00
Aliaksandr Valialkin
5e43145031
docs: add docs for scrape_configs section 2022-08-07 00:04:55 +03:00
Aliaksandr Valialkin
ecbe1ddf1b
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs
This option was ignored since d289ecded1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin
8ddad31eef
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
2022-08-04 22:28:21 +03:00
Igor Tiunov
0ba86fe87e
YC service discovery (#2923)
* YC service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386

* Fixed linter suggestions

* fixed golint errors
2022-08-04 22:28:20 +03:00
Aliaksandr Valialkin
dc58cb3fd8
app/vminsert/netstorage: pre-initialize the remaining throttled loggers
This is a follow-up after 6c66804fd3
2022-08-04 18:34:42 +03:00
Aliaksandr Valialkin
43185353bc
app/vmselect/netstorage: cleanup after 92630c1ab4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896
2022-08-04 18:34:38 +03:00
Aliaksandr Valialkin
98caffcfc4
docs: fix the recommended url for -vmalert.proxyURL accroding to 8667307d73 2022-08-04 18:03:24 +03:00
Aliaksandr Valialkin
a1e49606ed
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin
c81d2b4c18
app/vmselect/netstorage: initializes tsw.rowsProcessed before calling tsw.f, since tsw.f can modify r.Timestamps and r.Values lengths 2022-07-30 00:39:14 +03:00
Aliaksandr Valialkin
5ddae2e293
app/vmselect/netstorage: re-use random generator used for series shuffle in Result.RunParallel
This should reduce CPU usage needed for rand.Rand initialization
2022-07-30 00:31:00 +03:00
Dmytro Kozlov
ab645b5fab
vmselect/promql: add tests for vmrangeBucketsToLE (#2907)
* vmselect/promql: add tests for vmrangeBucketsToLE

* vmselect/promql: cleanup

* vmselect/promql: cleanup

* vmselect/promql: fix panic tests want result

* vmselect/promql: cleanup

* vmselect/promql: update test name

* vmselect/promql: fix linter error

* vmselect/promql: refactor testcases

* vmselect/promql: cleanup

* vmselect/promql: remove unused reassign to workers, fix typo

* wip

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-26 20:43:43 +03:00
Aliaksandr Valialkin
9f1e558c58
all: rename -pushmetrics.extraLabels to -pushmetrics.extraLabel for the sake of consistency 2022-07-26 19:25:26 +03:00
Aliaksandr Valialkin
8a83c59956
app/vmselect/promql: reduce the diff for f148cffc8a
This is a follow-up for c826f06366
2022-07-26 19:21:16 +03:00
Alan Liang
f97de473f4
vmselect: fix vmrangeBucketsToLE func may panic when ts value equal zero (#2902)
Co-authored-by: alanwzliang <alanwzliang@tencent.com>
2022-07-25 10:49:48 +03:00
Aliaksandr Valialkin
764fc04756
app/vmalert: fix after da10962d4c 2022-07-25 09:48:41 +03:00
Roman Khavronenko
da10962d4c
vmselect: cover special cases for vmalert's routing in single-node version (#2845)
* vmselect: cover special cases for vmalert's routing in single-node version

* remove trailing `/` from requests
* redirect to vmalert's home page when `/vmalert` is requested.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmselect/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-25 09:43:44 +03:00
Aliaksandr Valialkin
5d62f5a324
app/vmalert/config: add missing docs for ValidateTplFn 2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin
3d4c312ba2
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
2022-07-25 09:22:28 +03:00
Roman Khavronenko
970f36de17
vmalert: remove notifier dependency from config (#2906)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin
c0c9f30870
lib/pushmetrics: properly handle errors when initializing pushmetrics 2022-07-22 13:38:25 +03:00
Roman Khavronenko
01755fac38
vmalert: remove dependency on datasource pkg from config (#2905)
* vmalert: remove dependency on datasource pkg from config

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin
b1129fda0c
docs/vmgateway: typo fix in regexp filter: ~= is substituted with =~ 2022-07-21 22:37:42 +03:00
Aliaksandr Valialkin
59bb42be06
app/vmselect/vmui: make vmui-update after edecd2493c 2022-07-21 21:00:36 +03:00
Aliaksandr Valialkin
b49fc2f9f3
app/vmalert/utils: add missing docs to WithHeaders func added at 70a822f3a0 2022-07-21 20:49:38 +03:00
Roman Khavronenko
d0abdc2b5b
vmalert: allow configuring custom headers per group (#2901)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 20:48:05 +03:00
Roman Khavronenko
356d8b99e0
vmalert: allow configuring custom headers for URLs (#2897)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 20:45:12 +03:00
dependabot[bot]
272f207afc
build(deps): bump terser in /app/vmui/packages/vmui (#2895)
Bumps [terser](https://github.com/terser/terser) from 5.13.1 to 5.14.2.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)

---
updated-dependencies:
- dependency-name: terser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-21 20:42:01 +03:00
Yury Molodov
d1b88f3b5d
fix: change the z-index of the datepicker (#2891) 2022-07-21 20:41:16 +03:00
Roman Khavronenko
04c4f8bafd
vmselect: return correct error for second part of expression (#2893)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 20:40:47 +03:00
Aliaksandr Valialkin
fe68bb3ba7
all: follow-up after 46f803fa7a
Add -pushmetrics.* command-line flags to all the VictoriaMetrics apps
2022-07-21 20:18:25 +03:00
Aliaksandr Valialkin
f00a6bf837
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url 2022-07-21 20:15:29 +03:00
Aliaksandr Valialkin
536ee9fc18
deployment/docker: update alpine base image from 3.16.0 to 3.16.1
See https://alpinelinux.org/posts/Alpine-3.16.1-released.html
2022-07-19 19:51:36 +03:00
Aliaksandr Valialkin
7a06ee11fd
docs/vmauth.md: mention that multiple recrods for the same username are supported
This is a follow-up for 88029c521c
2022-07-19 19:43:20 +03:00
Nikolay
0102eb3869
app/vmauth: allow duplicate usernames (#2888)
Usernames could be duplicate if it has uniq password.
vmauth makes routing based on auth token and username + password combination must be unique for this case.
2022-07-19 19:35:28 +03:00
Aliaksandr Valialkin
f992f96a88
app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied
This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886
2022-07-19 14:29:41 +03:00
Roman Khavronenko
23e85e0fc5
vmagent: expose metric vmagent_remotewrite_queues (#2871)
The new metric `vmagent_remotewrite_queues` exports a static value of
number of configured remote write queus. This metric is useful to
calculate total saturation per each configured URL with given number
of queues. See corresponding changes to vmagent alerts and dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-18 14:41:04 +03:00
Aliaksandr Valialkin
d920b0afec
app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing 2022-07-18 14:24:23 +03:00
Aliaksandr Valialkin
f933d5f142
app/vmselect/promql/transform.go: reuse evalNumber() function for constructing timezone_offset() results 2022-07-18 14:24:22 +03:00
cui fliter
e0d30f6ec5
fix some typos (#2882)
Signed-off-by: cui fliter <imcusg@gmail.com>
2022-07-18 12:10:40 +03:00
Roman Khavronenko
80de6face8
vmalert: drop support of deprecated extra_filter_labels param (#2870)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-14 16:10:46 +03:00
Aliaksandr Valialkin
70b9925bf7
app: fix make publish-* after ed93330e66
Add missing `-linux` substring to built binary names for copying into Docker images
2022-07-14 11:01:34 +03:00
Aliaksandr Valialkin
e6ee940daf
docs: update descriptions for command-line flags according to the latest changes 2022-07-14 00:57:44 +03:00
Yury Molodov
1ca20caa4b
vmui: optimize table view (#2867)
* feat: optimize table view

* fix: add column display setting

* app/vmselect: `make vmui-update`

Also document the change at docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-14 00:16:57 +03:00
Dmytro Kozlov
99402541fb
vmui: update time picker behavior (#2847)
* vmui: update time picker behavior

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-14 00:06:22 +03:00
Yury Molodov
23c3b5f8c7
vmui: change selection from autocomplete (#2862)
* fix: change selection from autocomplete

* update docs/CHANELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:56:04 +03:00
Nikolay
c007b129cb
lib/promscrape: adds azure service discovery (#2743)
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:45:43 +03:00
Boris Petersen
cbf01ee4ae
fix typo introduced in pr #2604 (#2866)
Signed-off-by: Boris Petersen <boris.petersen@idealo.de>
2022-07-13 23:45:42 +03:00
Aliaksandr Valialkin
83221bc480
deployment/docker: update Go builder from go1.18.3 to go1.18.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.18.4+label%3ACherryPickApproved
2022-07-13 20:31:54 +03:00
Aliaksandr Valialkin
da6c85a2f6
all: follow-up for d99ba3481b 2022-07-13 17:17:08 +03:00
Dmytro Kozlov
4e4def9df8
Rename release packages (#2810)
* makefile: add os to each release file

* makefile: update vmutils arm64

* makefile: update victoria-metrics release process

* makefile: update publish with os

* makefile: update publish with os

* makefile: change tar library

* update release logic

* copy all releases

* sort command by GOOS

* rollback commands

* rollback OSARCH

* fix commands

* cleanup

* fix windows build

* sort build by GOOS, update README.md
2022-07-13 17:11:01 +03:00
guidao
f2d24a660b
add next retention metric (#2863)
Co-authored-by: wangfeng <wangfeng@zhihu.com>
2022-07-13 12:41:22 +03:00
Aliaksandr Valialkin
919939ee9f
app/vmselect/promql: consistency update after 93fbd0c54b 2022-07-13 12:34:28 +03:00
Roman Khavronenko
214e01499c
promql: return step as scrapeInterval when it can't be calculated (#2865)
The change allows to specify default value for `getScrapeInterval`
function when actual interval can't be calculated.

Before the change, function were returning `maxSilenceInterval` (5m)
in such cases, which may be not correct for instant queries processing.
The specific scenario where using `maxSilenceInterval` caused issues
is the following:
1. Series becomes stale;
2. Client (in this case vmalert) continues to request series every 15s;
3. Database returns empty results as expected;
4. But at some specific moment of time database returns datapoints from `now()-5m`,
because lookback window was extended to `maxSilenceInterval`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-13 12:34:28 +03:00
Aliaksandr Valialkin
5867708b3d
app/vmselect/promql: validate function name before evaluating its arguments
This avoids unneeded evaluation of args for unknown functions
2022-07-12 19:49:54 +03:00
Aliaksandr Valialkin
fbb403b5c0
app/vmselect/netstorage: optimize mergeSortBlocks() for the worst case when blocks contain interleaved samples 2022-07-12 12:30:24 +03:00
Aliaksandr Valialkin
aee08117e9
app/vmselect/netstorage: add mergeSortBlocks benchmark for the worstcase 2022-07-12 12:26:27 +03:00
Dmytro Kozlov
6734d77d97
vmselect/prometeus: Add limit param to api/v1/series api endpoint (#2851)
* issue-2841: Add limit param to api/v1/series api endpoint

* issue-2841: add change log

* issue-2841: update logic

* issue-2841: simplify logic

* issue-2841: simplify logic, add information to documentation
2022-07-11 20:37:20 +03:00
Aliaksandr Valialkin
ed2126aec3
app/vmselect: follow-up after 6dda254e01 2022-07-11 19:48:18 +03:00
Roman Khavronenko
6dda254e01
vmselect: cover special cases for vmalert's routing in cluster version (#2844)
* vmselect: cover special cases for vmalert's routing

* remove trailing `/` from requests
* redirect to vmalert's home page when `/vmalert` is requested.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmselect: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmselect/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-11 19:33:33 +03:00
Aliaksandr Valialkin
eab8ebbe11
all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
Aliaksandr Valialkin
3662ce8323
docs: make more clear the relation between replication and deduplication
This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2852
2022-07-11 19:23:24 +03:00
Aliaksandr Valialkin
c0af52228a
app/vmselect/netstorage: add benchmarks for mergeSortBlocks
This is a follow-up for 743ff84863
2022-07-11 12:53:46 +03:00
Aliaksandr Valialkin
d442ee4610
app/vmselect/netstorage: optimize mergeSortBlocks function
- Use binary search instead of linear scan when locating the run of smallest timestamps
  in blocks with intersected time ranges. This should improve performance
  when merging blocks with big number of samples

- Skip samples with duplicate timestamps. This should increase query performance
  in cluster version of VictoriaMetrics with the enabled replication.
2022-07-09 00:35:38 +03:00
Roman Khavronenko
78da074ad3
metricsql: properly evaluate timezone_offset over time interval (#2842)
* metricsql: properly evaluate `timezone_offset` over time interval

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-08 14:04:40 +03:00
Aliaksandr Valialkin
64419aa97c
app/vmalert/utils/links.go: document Prefix function, which has been added in b29fafa86b 2022-07-08 13:25:23 +03:00
Aliaksandr Valialkin
decbf84e55
app/vmselect/vmui: follow-up after 0bf6841140
* Document the bugfix in docs/CHANGELOG.md
* Run `make vmui-update` for updating static js files for vmui, which are included into vmselect
2022-07-08 13:22:57 +03:00
Dmytro Kozlov
5afe15f53a
vmui: fix query for json and table tabs (#2846)
* vmui: fix query for json and table

* vmui: add step param
2022-07-08 13:22:55 +03:00
Roman Khavronenko
b29fafa86b
vmalert: deprecate alert's status link (#2840)
* vmalert: deprecate alert's status link

Deprecate alert's status link `/api/v1/<groupID>/<alertID>/status` in favour of
`api/v1/alerts?group_id=<group_id>&alert_id=<alert_id>"`.

The change was needed for simplifying logic in vmselect for proxying vmalert's requests.

The old alert's status link will be still supported for a few versions but will be removed in the future.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-08 13:05:11 +03:00
Aliaksandr Valialkin
69718a6c16
docs/vmagent.md: clarify that vmagent supports Prometheus-compatible service discovery 2022-07-07 21:22:02 +03:00
Aliaksandr Valialkin
2a53b8961e
docs/vmagent.md: typo fixes after ef2eeeb642 2022-07-07 21:17:13 +03:00
Aliaksandr Valialkin
372ecc5c0e
docs/vmagent.md: actualize docs 2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin
95add1e8e4
app/{vmagent,vminsert}: follow-up after d19e46de55
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839
2022-07-07 01:32:11 +03:00
Pedro Gonçalves
af5f967307
app/vminsert: allow to ingest datadog metrics with simpler tags - not enforcing key:value (#2839) 2022-07-07 01:32:10 +03:00
Roman Khavronenko
218dfe7956
docs: mention deduplication issue for HA vmalert topology (#2838)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-07 01:13:56 +03:00
Aliaksandr Valialkin
6bb4bcd30c
app/vmui: tune visual presentation of trace view 2022-07-06 14:16:24 +03:00
Aliaksandr Valialkin
eee384c9c9
app/vmselect: add -clusternative.tls* options for mTLS setup in multi-level clusters 2022-07-06 13:48:21 +03:00
Aliaksandr Valialkin
195dccf678
app/vmselect: add ability to query vmselect from another vmselect 2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin
714958deab
all: follow-up after ed89106274 2022-07-06 12:48:52 +03:00
Roman Khavronenko
56f4058fe3
vmalert: make UI and assets links relative (#2831)
* make all links in vmalert relative, so links continue to work even if vmalert sits behind the proxy;
* update vmalert's routing to always have component-unique path prefix, e.g. /vmalert;

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-06 12:47:53 +03:00
Aliaksandr Valialkin
cdd89d9cc2
app/vmselect: properly generate response for /api/v1/series
The response has been broken in 7d5d33fd71
2022-07-06 12:46:23 +03:00
Aliaksandr Valialkin
1ec4dfd678
lib/vmselectapi: pass storage.SearchQuery to API calls instead of []*storage.TagFilters + storage.TimeRange + maxMetrics
This reduces the number of args to vmselectapi calls
2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin
2e721f7d16
lib/vmselectapi: rename Server.MustClose to more clear Server.MustStop 2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin
270e555f47
lib/vmselectapi: pass maxSuffixes arg to tagValueSuffixes RPC call 2022-07-06 12:46:22 +03:00
Aliaksandr Valialkin
f4df43f7cc
app/vmselect/netstorage: remove unused auth.Token arg 2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin
78eeca6f0d
lib/vmselectapi: rename deleteMetrics to more correct deleteSeries 2022-07-06 12:46:21 +03:00
Aliaksandr Valialkin
5afa54e845
lib/vmselectapi: use string type for tagKey and tagValuePrefix args at TagValueSuffixes()
This improves the API consistency
2022-07-06 12:46:21 +03:00
Roman Khavronenko
f51bc07d97
vmselect: allow proxying requests to vmalert (#2833)
The change allows to proxy requests with prefix `/vmalert`
to the vmalert component if `-vmalert.proxyURL` is set.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825
and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2831

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-06 10:47:51 +02:00
Aliaksandr Valialkin
10c63f5853
app/vmselect: make fmt after f3ece83e67 2022-07-05 14:56:53 +03:00
Aliaksandr Valialkin
8516670582
app/vmselect/promql: properly calculate histogram_quantile over unexpected le buckets
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819
2022-07-05 13:20:41 +03:00
Aliaksandr Valialkin
ecc11dc32d
lib/promauth: refactor NewConfig in order to improve maintainability
1. Split NewConfig into smaller functions
2. Introduce Options struct for simplifying construction of the Config with various options

This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2684
2022-07-04 14:31:43 +03:00
Dmytro Kozlov
7389fc9adf
vmui: disable ripple on nested component, enable copy text on button (#2808) 2022-07-04 11:47:39 +03:00
Aliaksandr Valialkin
fa08220d27
app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787
2022-06-30 22:40:26 +03:00
Aliaksandr Valialkin
1e6b0a1f54
app/vmagent/remotewrite: do not shadow headers global variable in getAuthConfig 2022-06-30 20:18:43 +03:00
Aliaksandr Valialkin
0f6c17ed06
app/vmagent/remotewrite: clarify descriptions for -remoteWrite.* options, which must be set per each -remoteWrite.url 2022-06-30 20:18:42 +03:00
Aliaksandr Valialkin
7fc03a1deb
app/vmagent/remotewrite: add -remoteWrite.header command-line flag for setting additional http headers to send to -remoteWrite.url
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805
2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin
4fb0f15322
all: readability improvements for query traces
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
2022-06-30 18:19:43 +03:00
Dmytro Kozlov
67753cc6f0
vmui: update render logic for nested component (#2795)
* vmui: update render logic for nested component, avoid rerender, remove local storage usage for tracing flag

* docs/url-examples.md: fix various documentation issues there

* docs: add Troubleshooting doc

This doc contains troubleshooting guides for typical problems with VictoriaMetrics.

* docs/Troubleshooting.md: add troubleshooting guide for cluster instability

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-30 15:48:22 +03:00
Aliaksandr Valialkin
92e6c55b75
docs: add Troubleshooting doc
This doc contains troubleshooting guides for typical problems with VictoriaMetrics.
2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin
daefb64f38
app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792
2022-06-28 20:18:31 +03:00
Aliaksandr Valialkin
9adff0b686
app/vmselect/promql: show the number of scanned samples in the query trace 2022-06-28 19:32:03 +03:00
Aliaksandr Valialkin
c21f5a508f
app/vmselect/prometheus: reduce the default value for -search.maxSeries from 100k to 30k
Production experience shows that 100k is too big for /api/v1/series .
It leads to increased CPU usage when Grafana queries /api/v1/series over VictoriaMetrics
with big number of time series during auto-completion and when modifying template variables.
2022-06-28 18:22:24 +03:00
Aliaksandr Valialkin
7d5d33fd71
lib/storage: return marshaled metric names from SearchMetricNames
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.

While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.

Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin
36edb1912b
app/vmstorage: rename "transport" package to "servers" package for better clarity 2022-06-28 14:04:14 +03:00
Aliaksandr Valialkin
399d4c36ae
app/vmselect: optimize /api/v1/series a bit for time ranges smaller than one day 2022-06-28 12:55:20 +03:00
Aliaksandr Valialkin
64505e924d
app/vmstorage: extract vmselect api server into a separate package - lib/vmselectapi
This opens doors for implementing vmselect api server at vmselect level,
so top-level vmselect could query lower-level vmselect nodes in the same way
as it queries vmstorage nodes.

This will create the ability to create highly available querying architecture
when multiple independent VictoriaMetrics clusters with the same data
are located in distinct availability zones. In this case we can use top-level
vmselect instead of Promxy for simultaneous querying of all the clusters
in all the AZs.
2022-06-27 14:20:41 +03:00
Aliaksandr Valialkin
a667d339be
app/vmselect/netstorage/netstorage.go: group metrics in order to improve readability a bit 2022-06-27 14:00:24 +03:00
Roman Khavronenko
50c0eb4c4e
vmalert: make __name__ available for templating in alerts (#2783)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-27 13:53:55 +03:00