Commit graph

1295 commits

Author SHA1 Message Date
Yury Molodov
daaea87931
vmui: timezone select (#3414)
* feat: add timezone selection

* vmui: provide feature timezone select

* fix: correct timezone with relative time

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:57:16 -08:00
Yury Molodov
08a68a2829
vmui: improve table view (#3377)
* vmui: add compact table view (#3365)

* feat: add compact table view

* fix: add overflow table

* fix: change table styles

* vmui: compact table view

* Update docs/CHANGELOG.md

Co-authored-by: Michal Kralik <michal.kralik@percona.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-05 22:56:53 -08:00
Aliaksandr Valialkin
98110a06d1
docs/CHANGELOG.md: document fd43b5bad0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3444
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3445
2022-12-05 22:02:16 -08:00
Aliaksandr Valialkin
c902500efc
app/vmselect/promql: add range_trim_spikes(phi, q) function for trimming phi percent of largest spikes per each time series returned by q 2022-12-05 21:55:24 -08:00
Zakhar Bessarab
c3276e57b4
app/vmctl: add option to migrate between clusters with automatic tenants discovery (#3450) 2022-12-05 17:20:12 -08:00
Aliaksandr Valialkin
d2e34b8052
{dashboards,alerts}: subtitute {type="indexdb"} with {type=~"indexdb.*"} inside queries after 8189770c50
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 16:00:42 -08:00
Aliaksandr Valialkin
7d5c64eb7a
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin
0b8e7deabd
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin
df4eca2807
docs/CHANGELOG.md: consistently add - prefix in front of command-line flags
This is a follow-up for bcba5d2a78
2022-12-02 19:14:02 -08:00
Aliaksandr Valialkin
86632658f0
docs: follow-up after 30fea30685
- Run `make docs-sync`, so app/vmalert/README.md is copied to docs/vmalert.md
- Clarify the feature description in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3408
2022-12-02 19:14:02 -08:00
Roman Khavronenko
31ca22109e
vmalert: fix replay step param (#3428)
The recent change in modifying default value
of `datasource.queryStep` flag resulted in situation
where replay mode was always running queries with
step=`datasource.queryStep`. When it should always
use rule's evaluation interval.

The fix is related not to replay mode only, but
for all Range requests. Now step param is set
individually for each mode.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-02 19:09:30 -08:00
Zakhar Bessarab
59f889cd3f
app/vmalert: add remoteWrite.sendTimeout command-line flag to configure timeout for sending data to remoteWrite.url (#3423)
* app/vmalert: add `remoteWrite.sendTimeout` command-line flag to configure timeout for sending data to `remoteWrite.url`

* vmalert: remove WriteTimeout from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the WriteTimeout.

* vmalert: remove DisablePathAppend from clients Cfg
No need to have it as a part of configuration struct:
* the client isn't used by other packages;
* there are no internal tests to check the DisablePathAppend.

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2022-12-02 19:03:34 -08:00
Roman Khavronenko
435f6f3add
vmalert: properly pass headers during the restore procedure (#3420)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3418

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-02 18:53:44 -08:00
Aliaksandr Valialkin
be6da5053f
lib/promscrape: optimize service discovery speed
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin
8440c822fa
all: follow-up after 05cf8a6ecc 2022-11-29 21:11:42 -08:00
Aliaksandr Valialkin
a547226b53
docs/CHANGELOG.md: document 027ab74efb
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402
2022-11-28 18:55:17 -08:00
Aliaksandr Valialkin
8ce5b095b7
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
2022-11-28 18:37:34 -08:00
Aliaksandr Valialkin
d0461bec25
docs/CHANGELOG.md: cut v1.84.0 2022-11-25 19:59:30 -08:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:42:38 -08:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants (#3348)
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* {app/vmstorage,app/vmselect}: add API to get list of existing tenants

* app/vmselect: fix error message

* {app/vmstorage,app/vmselect}: fix error messages

* app/vmselect: change log level for error handling

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 10:32:45 -08:00
Aliaksandr Valialkin
2f7152deb3
app/vmselect/vmui: make vmui-update after eb772aa50e 2022-11-24 17:37:33 -08:00
Yury Molodov
a59c7a0dd7
vmui: add set up series custom limits (#3368)
* feat: add set up series custom limits

* feat: add button for show series without limits

* fix: resolve merge conflicts
2022-11-22 15:36:17 +02:00
Aliaksandr Valialkin
afc35485c1
app/vminsert: add missing vm_relabel_config_* metrics after 03d88bc066
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
2022-11-22 00:48:13 +02:00
Roman Khavronenko
d1169c1559
vmagent: expose metrics for tracking config state (#3375)
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:48:12 +02:00
Yury Molodov
5924aaf7e5
vmui: add copy button to row on Table view (#3363)
* feat: add copy button to row on Table view

* vmui: add copy button to row on Table view

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-22 00:35:05 +02:00
Yury Molodov
05712cfc8d
vmui: sticky tooltip (#3376)
* feat: add ability to make tooltip "sticky"

* vmui: add ability to make tooltip "sticky"
2022-11-22 00:35:05 +02:00
Aliaksandr Valialkin
5c65b3c7dc
docs/CHANGELOG.md: link to the related issue for range_normalize() function 2022-11-21 23:27:11 +02:00
Aliaksandr Valialkin
ad548abd6a
app/vmselect/promql: add range_normalize(q1, ..., qN) function for normalizing query results into [0..1] value range
This may be useful for analyzing correlation between time series with different value ranges
2022-11-21 23:25:35 +02:00
Aliaksandr Valialkin
c33bcae457
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin
9cb76c9cd3
app/vmselect/promql: allow using SI and IEC suffixes in numeric values inside queries
For example, 10Ki is equivalent to 10*1024, while 5.3M is equivalent to 5.3*1000*1000
2022-11-21 21:28:34 +02:00
Aliaksandr Valialkin
2d361a834d
app/vmselect/promql: properly return an empty result from limit_offset() if offset exceeds the number of inner time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3312
2022-11-21 16:48:10 +02:00
Aliaksandr Valialkin
b0fefe562a
app/vmselect/promql: optimize e1 op e2 when e1 returns an empty result
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3349
2022-11-21 16:09:49 +02:00
Roman Khavronenko
0475f8a38e
vmalert: add default list of alerting rules (#3373)
The default list of alerting rules contains the basic
rules for checking vmalert's health state and is recommended
to use for monitoring vmalert deployments.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-21 16:09:47 +02:00
Aliaksandr Valialkin
0e51967970
app/{vminsert,vmselect}: add -storageNode.filter command-line flag for filtering the discovered storage nodes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3353
2022-11-21 15:20:20 +02:00
Yury Molodov
60aaf4b2c2
vmui: add trace analyzer (#3310)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add page with trace analyzer

* fix: change detect trace data

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* fix: change descriptions on trace page

* Update app/vmui/packages/vmui/src/pages/TracePage/index.tsx

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* feat: add base components

* feat: add reset styles

* docs: add description about trace analyzer

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

* feat: add paste json for trace analyzer

* vmui: update trace analyzer docs

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 22:23:05 +02:00
Aliaksandr Valialkin
f96bf852ea
docs/CHANGELOG.md: document the fix for CPU usage spikes
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-17 22:03:03 +02:00
Aliaksandr Valialkin
7969ca6c80
app/vmselect/vmui: make vmui-update after 51bfd1ab80 2022-11-17 18:57:44 +02:00
Yury Molodov
734172d2d0
vmui: add ability hide query (#3359)
* feat: add ability hide query

* fix: change logic hide query

* fix: remove console.log
2022-11-17 18:57:42 +02:00
Yury Molodov
423eb73051
vmui/refactor (#3298)
* refactor: change structure project

* refactor: change structure project

* fix: add hooks for set query params

* refactor: add index for pages

* docs: add TESTCASES.md

* refactor: restructure components

* feat: add base components

* feat: add reset styles

* feat: add styles for custom panel page

* feat: add styles for predefined panels

* feat: add style for TracingsView.tsx

* feat: add Alerts

* feat: add Tooltip.tsx

* fix: correct styles

* feat: add DatePicker.tsx

* feat: add tables

* feat: add theme provider

* fix: replace using callbacks as props to handlers

* fix: correct update time

* fix: change TimePicker.tsx

* fix: correct styles

* fix: update packages

* vmui: refactor code, remove material-ui

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-17 01:21:54 +02:00
Aliaksandr Valialkin
0cc59b9f95
app/vmselect/promql: add range_stdvar() and range_stddev() functions for calculating variance and deviation over time series on the selected time range 2022-11-17 01:03:02 +02:00
Aliaksandr Valialkin
facf7efa3a
app/vmselect/promql: add range_linear_regression(q) function for calculating simple linear regression for the selected time series on the selected time range 2022-11-17 00:39:17 +02:00
Aliaksandr Valialkin
6e4c9d5f6b
docs/CHANGELOG.md: document changes in v1.79.5 release 2022-11-11 01:28:12 +02:00
Aliaksandr Valialkin
6d984cdeff
docs/CHANGELOG.md: cut v1.83.1 2022-11-10 14:07:20 +02:00
Aliaksandr Valialkin
3bdd1727b7
docs/CHANGELOG.md: document the fix at 71335e6024 2022-11-10 13:47:32 +02:00
Aliaksandr Valialkin
2091693f16
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:36:15 +02:00
Aliaksandr Valialkin
28e4364fd2
docs/CHANGELOG.md: document 7130af7fd2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2814
2022-11-09 12:16:23 +02:00
Aliaksandr Valialkin
8540dd669b
app/vminsert/netstorage: move nodesHash from global state to storageNodesBucket
This should prevent from panics when the list of discovered vmstorage nodes changes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3329
2022-11-09 11:45:24 +02:00
Aliaksandr Valialkin
abf7e4e72f
lib/promscrape/discovery/consul: add __meta_consul_partition label in the same way as Prometheus does
See https://github.com/prometheus/prometheus/pull/11482
2022-11-07 15:26:45 +02:00
Aliaksandr Valialkin
d3035b1ca1
lib/storage: follow-up for 790768f20b
- Document the bugfix at docs/CHANGELOG.md
- Simplify the bugfix a bit
2022-11-07 14:18:06 +02:00
Aliaksandr Valialkin
ac7c43c246
docs: typo fixes 2022-10-29 04:53:03 +03:00
Aliaksandr Valialkin
1357107a11
docs/CHANGELOG.md: cut v1.83.0 2022-10-29 02:55:08 +03:00
Aliaksandr Valialkin
cf5efb93d9
app/vmbackupmanager: add functionality for automated restore from backup 2022-10-29 02:31:06 +03:00
Aliaksandr Valialkin
ac5528cb46
lib/promscrape: properly add exported_ prefix to labels, which clash with target labels if honor_labels: true option isn't set.
The issue was in the `labels := dst[offset:]` line in the beginning of appendExtraLabels() function.
The `dst` may be re-allocated when adding extra labels to it. In this case the addition of `exported_`
prefix to labels inside `labels` slice become invisible in the returned `dst` labels.

While at it, properly handle some corner cases:

- Add additional `exported_` prefix to clashing metric labels with already existing `exported_` prefix.
- Store scraped metric names in `exported___name__` label if scrape target contains `__name__` label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3278

Thanks to @jplanckeel for the initial attempt to fix this issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3281
2022-10-28 22:15:31 +03:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
7ae038766c
app/vmalert/templates: properly escape all the special chars in quotesEscape function
Previously the `quotesEscape` function was escaping only double quotes.
This wasn't enough, since the input string could contain other special chars,
which must be escaped when put inside JSON string. For example, carriage return and line feed chars (\n\r),
backslash char, etc. This led to the following issues, which were improperly fixed:

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890 - this issue
  was "fixed" by introducing the `crlfEscape` function, which led to unnecessary
  complications in user templates, while not fixing various corner cases
  such as backslash chars in the input string.
  See 1de15ad490

- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 - this issue
  was "fixed" by urlencoding the whole string passed to -external.alert.source
  command-line flag. This led to invalid urls, which couldn't be parsed by Grafana.
  See 00c838353d
  and 4bd0244599

This commit properly encodes the input string passed to `quotesEscape`, so it can be safely embedded inside JSON strings.

This commit deprecates crlfEscape template function and adds the following new template functions:

- strvalue and stripDomain - these functions are supported by Prometheus, so they were added
  for compatibility purposes.
- jsonEscape and htmlEscape for converting the input string to valid quoted JSON string
  and for html-escaping the input string, so it could be safely embedded as a plaintext
  into html.

This commit also documents all supported template functions at https://docs.victoriametrics.com/vmalert.html#template-functions
The deprecated crlfEscape function isn't documented on purpose, since its usefulness is negative in general case.
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
8a6898b625
Revert "vmalert: escape query params if external alert source defined (#3267)"
This reverts commit 00c838353d.

Reason for revert: it incorrectly fixes the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3139 .
Now `-external.alert.source=explore?orgId=1&left=...` is converted to the following invalid url, which cannot be handled by Grafana:

https://grafana.example.com/explore%3ForgId%3D1%26left%3D...

The next commit will contain the correct fix of the issue - the `quotesEscape` function must
properly escape the string, so it could be embedded into JSON string. This function must
properly escape \n\r chars too. In this case the `crlfEscape` function becomes unnecessary.
Actually, the next commit makes the `crlfEscape` function deprecated.
2022-10-28 00:08:50 +03:00
Dmytro Kozlov
3123059407
vmalert: escape query params if external alert source defined (#3267)
vmalert: escape query args if external alert source defined
2022-10-28 00:08:50 +03:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Yury Molodov
ab880afa50
vmui: add responsive styles for small screens (#3256)
* fix: add responsive styles for small screens

* fix: correct additional settings margins

* docs/CHANGELOG.md: add responsive styles
2022-10-26 02:40:41 +03:00
Aliaksandr Valialkin
36b92f07f7
lib/envflag: allow referring environment variables in command-line flags 2022-10-26 01:55:23 +03:00
Aliaksandr Valialkin
46a37fbeac
docs/CHANGELOG.md: document 274e235bf7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3240
2022-10-24 21:03:03 +03:00
Aliaksandr Valialkin
8ea84432ef
docs/enterprise.md: describe all the enteprise features in a short doc at https://docs.victoriametrics.com/enterprise.html 2022-10-24 18:03:22 +03:00
Aliaksandr Valialkin
754208b124
docs/CHANGELOG.md: typo fixes 2022-10-24 17:00:18 +03:00
Aliaksandr Valialkin
a6d4711ac6
lib/storage: add support for retention filters (aka multiple retentions for distinct sets of time series)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/143
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/289
2022-10-24 16:41:59 +03:00
Aliaksandr Valialkin
22a5405cfa
docs/CHANGELOG.md: document 69b27275d2b2bf1bdae0d8b887b44bde2a787649 2022-10-24 16:08:35 +03:00
Aliaksandr Valialkin
2dd93449d8
lib/storage: subsitute searchTSIDs functions with more lightweight searchMetricIDs function
The searchTSIDs function was searching for metricIDs matching the the given tag filters
and then was locating the corresponding TSID entries for the found metricIDs.

The TSID entries aren't needed when searching for time series names (aka MetricName),
so this commit removes the uneeded TSID search from the implementation of /api/v1/series API.
This improves perfromance of /api/v1/series calls.

This commit also improves performance a bit for /api/v1/query and /api/v1/query_range calls,
since now these calls cache small metricIDs instead of big TSID entries
in the indexdb/tagFilters cache (now this cache is named indexdb/tagFiltersToMetricIDs)
without the need to compress the saved entries in order to save cache space.

This commit also removes concurrency limiter during searching for matching time series,
which was introduced in 8f16388428, since the concurrency
for all the read queries is already limited with -search.maxConcurrentRequests command-line flag.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2022-10-23 12:43:44 +03:00
Aliaksandr Valialkin
c92aef39b5
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"} 2022-10-23 12:14:07 +03:00
Aliaksandr Valialkin
324e119172
lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal
Previously VictoriaMetrics apps could stop responding to SIGINT and SIGTERM signals
if they hang for some reason in graceful shutdown procedure.
2022-10-20 21:58:49 +03:00
Aliaksandr Valialkin
0126ea31e7
docs/CHANGELOG.md: move the BUGFIX line for 1059c4d84a into correct place 2022-10-18 20:40:37 +03:00
Nikolay
ea0596d9d8
lib/promscrape/discovery/kubernetes: correctly wrap error (#3250)
* lib/promscrape/discovery/kubernetes: correctly wrap error
follow-up after 1304824201

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-18 20:40:37 +03:00
Roman Khavronenko
f7d69c1735
vmalert: lower severity level for RW retries (#3237)
The message about dropped data still remains at `error` level.
The change supposed to make log message more clear about how
serious it is.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-18 20:40:37 +03:00
Aliaksandr Valialkin
d0288ea417
all: log error when environment variables referred from -promscrape.config are missing
This should prevent from using incorrect config files
2022-10-18 10:29:59 +03:00
Aliaksandr Valialkin
e3b9815fd9
docs/CHANGELOG.md: add a note that it is recommended to use v1.82.1 instead of v1.82.0 2022-10-14 13:41:29 +03:00
Aliaksandr Valialkin
7e8c9358d1
docs/CHANGELOG.md: release v1.82.1 2022-10-14 11:34:01 +03:00
Aliaksandr Valialkin
ed324aad66
lib/bytesutil: make sure that the string passed to FastStringMather.Match() is copied before using it as a key in the internal cache map
This prevents from possible corruption of the internal cache map
when the underlying byte slice used by the string key is modified.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3227
2022-10-14 09:52:18 +03:00
Yury Molodov
a50f2e141c
vmui: limit number of plotted series (#3229)
* feat: add maximum display series by tabs

* feat: add warning on PredefinedPanels.tsx

* docs/CHANGELOG.md: vmui limit number of plotted series

* docs/CHANGELOG.md: vmui limit number of plotted series

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:14:53 +03:00
Aliaksandr Valialkin
9bf8d64605
app/vmselect/promql: follow-up for 930f1ee153
Document the change at docs/CHANGELOG.md
Apply it to histogram_quantile() in the same way as to histogram_share()

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3225
2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
8f0e4901fe
docs/CHANGELOG.md: document b856581ad3 2022-10-13 10:36:13 +03:00
Aliaksandr Valialkin
3a14d00989
docs/CHANGELOG.md: document a4975ace86
The original commit, which led to the issue - 877940a131
2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin
4b7ca0597d
docs/CHANGELOG.md: document 9544b5cacf7446203fac19eb7c575779dc9b280e 2022-10-12 09:33:16 +03:00
Aliaksandr Valialkin
d6a65b1f44
docs/CHANGELOG.md: document e384d88abf 2022-10-10 21:54:18 +03:00
Aliaksandr Valialkin
087393bcef
lib/promrelabel: remove unconditional sorting of the labels in ParsedConfigs.Apply(), since the sorting isnt needed in many places
Sort labels explicitly after calling the ParsedConfigs.Apply() when needed.

This reduces CPU usage when performing metric-level relabeling, where labels' sorting isn't needed.
2022-10-09 14:53:35 +03:00
Aliaksandr Valialkin
938ff7bba6
app/vmselect: allow limiting per-query memory usage via -search.maxMemoryPerQuery command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3203
2022-10-08 01:14:01 +03:00
Aliaksandr Valialkin
708d45c8fe
docs/CHANGELOG.md: add a link to a feature request for the feature, which allows specifying full scrape urls in targets list and in the __address__ label
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3208
2022-10-07 23:46:34 +03:00
Aliaksandr Valialkin
3b828535f0
lib/promscrape: allow controlling staleness tracking on a per-scrape_config basis
Add support for no_stale_markers option at scrape_config section.
See https://docs.victoriametrics.com/sd_configs.html#scrape_configs and
https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers
2022-10-07 23:37:31 +03:00
Aliaksandr Valialkin
b710354067
docs/CHANGELOG.md: document the 27ed4b853e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3169
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3196#issuecomment-1269765205
2022-10-07 23:08:23 +03:00
Aliaksandr Valialkin
3987b0abd1
lib/promscrape: allow specifying full target url in __address__ label
Previously the `__address__` label could contain only `host:port` part of the target url,
while the scheme and metrics path were obtained from `__scheme__` and `__metrics_path__`
labels. Now it is possible to set the full url in `__address__` label.

This makes valid the following scrape config, which is frequently used by novice users:

scrape_configs:
- job_name: foo
  static_configs:
  - targets:
    - http://host1/metrics1
    - https://host2/metrics2
2022-10-07 22:46:29 +03:00
Aliaksandr Valialkin
8f28402cb3
docs/CHANGELOG.md: cut v1.82.0 2022-10-07 03:15:42 +03:00
Aliaksandr Valialkin
b736f60742
docs/CHANGELOG.md: typo fix 2022-10-07 03:11:03 +03:00
Aliaksandr Valialkin
1b2aeb621a
docs/CHANGELOG.md: add a changelog for v1.79.4 LTS release (copied from lts-1.79 branch) 2022-10-07 02:51:42 +03:00
Aliaksandr Valialkin
b05ff477c8
app/vmselect/promql: properly calculate vm_rows_scanned_per_query histogram for rollup functions, which take into account only a few samples on the provided lookbehind window 2022-10-06 23:23:17 +03:00
Aliaksandr Valialkin
baa11a778d
app/vmselect/promql: properly calculate quantiles_over_time() over a single raw sample 2022-10-06 22:37:38 +03:00
Aliaksandr Valialkin
8f1e218949
docs/CHANGELOG.md: remove duplicate description of the bugfix, which has been included in v1.81.2 2022-10-06 15:53:30 +03:00
Aliaksandr Valialkin
371f10ec78
docs/CHANGELOG.md: add missing link to /api/v1/export docs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 15:02:39 +03:00
Aliaksandr Valialkin
958c1f291c
app: follow-up after ec04fcac93
* Optimize fast path for /api/v1/import when importing numeric values
* Move the docs about the change from features to bugfixes at docs/CHANGELOG.md
* Update tests at lib/protoparser/vmimport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3161
2022-10-06 14:54:15 +03:00
Dmytro Kozlov
4064db27a8
Properly parse json when export import metric (#3180)
* app/vmselect: properly work when export import json from `api/v1/{export, import}` API

* app/vmselect: update convert function

* app/vmselect: export null if `math.IsNaN(v)`

* app/vmselect: get float from json

* lib/protoparser: add test

* docs: add change log

* lib/protoparser: make export import api compatible
2022-10-06 14:54:14 +03:00
Yury Molodov
112ae2978d
vmui: maximum queries (#3196)
vmui: allow using up 4 queries at the same time

The change also introduces UI updates to make using 
multiple queries more conveniently.
2022-10-06 13:30:47 +03:00
Aliaksandr Valialkin
e6d866b37c
docs: follow-up after 262ce77e2d
* Document the addition of Azure blob storage support in vmbackup / vmrestore
* List the supported storage system types at docs/vmrestore.md
* Mention about azblob storage system support at -src and -dst command-line flags
  for vmbackup / vmrestore tools.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1029
2022-10-06 00:36:20 +03:00
Aliaksandr Valialkin
3fc0da2c82
docs/CHANGELOG.md: document f8ac55d70ada9ef8490b322abefb05f28f75e2e9 2022-10-06 00:06:07 +03:00
Aliaksandr Valialkin
1459201de5
app/vmbackupmanager: expose vm_backup_in_flight metrics (follow-up after c6bca4a0b47b4f5626e1913d3480e62d657ed4cf) 2022-10-05 23:30:39 +03:00
Roman Khavronenko
6f6f6afae0
vmalert: allow using {{$labels}} for templating in -external.alert.source (#3194)
The change is supposed to provide additional flexibility for generating alert's
source link based on label values.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-10-05 22:53:02 +03:00
Aliaksandr Valialkin
43bdd96a6e
app/vmselect: improve performance scalability on multi-CPU systems for /api/v1/export/... endpoints 2022-10-01 22:16:07 +03:00
Aliaksandr Valialkin
ae6c7edf04
app/vmselect: do not export NaN values for stale metrics at /federate endpoint
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3185
2022-10-01 19:48:21 +03:00
Aliaksandr Valialkin
b857365b84
app/vmagent/remotewrite: allow specifying per--remoteWrite.url disk limits for persistent queue with pending data
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970
2022-10-01 18:41:21 +03:00
Aliaksandr Valialkin
e03a924236
app/vmauth: do not remove trailing slash from the proxied path
This should fix the issue with opening VMUI at /vmui/ page.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1752
2022-10-01 16:56:42 +03:00
Aliaksandr Valialkin
735de9ee54
lib/promscrape: add external_labels from global section of -promscrape.config after the relabeling is applied to the scraped metrics
This aligns with Prometheus behaviour.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3137
2022-10-01 16:15:00 +03:00
Aliaksandr Valialkin
d81285c814
app/vmagent: accept requests with /prometheus and /influx path prefixes in the same way as VictoriaMetrics does
This allows using vmagent as a drop-in replacement for VictoriaMetrics
for push protocols.
2022-10-01 14:11:09 +03:00
Aliaksandr Valialkin
b96fe2e265
lib/storage: optimize matching speed for non-trivial regexp filters
Wrap re.Match into bytesutil.FastStringMatcher.

This increases performance for `{foo=~"complex_regex_here"}` filters
by up to 4x.
2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
72d8653233
docs/CHANGELOG.md: clarify the description of the improvement in relabeling performance 2022-10-01 12:07:18 +03:00
Aliaksandr Valialkin
c628f5b6eb
lib/promscrape/discovery/ec2: expose __meta_ec2_region label in the same way as Prometheus 2.39 does
See https://github.com/prometheus/prometheus/pull/11326
2022-09-30 20:49:08 +03:00
Roman Khavronenko
408d7043a1
vmalert: support auth configs per static_target (#3188)
Allow configuring authorization params per list of targets
in vmalert's notifier config for `static_configs`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2690

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 18:38:11 +03:00
Aliaksandr Valialkin
c8000c029a
app/vminsert: remove support for undocumented VictoriaMetrics_AccountID and VictoriaMetrics_ProjectID labels in tcp-based data ingestion endpoints
These labels are substituted by documented vm_account_id and vm_project_id labels.

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy-via-labels

This is a follow up for 505d359b39
2022-09-30 18:06:00 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009)
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 17:28:35 +03:00
Aliaksandr Valialkin
9fc2817f41
lib/promrelabel: optimize action: replace for non-trivial regex values
Cache `action: replace` results for non-trivial regexs and return them next time
instead of performing CPU-intensive regex replacement.

Optimize also `action: labelmap_all` and `action: replace_all` in the same way.
2022-09-30 12:28:25 +03:00
Dmytro Kozlov
d0a9fff70c
docs, app/vmgateway: add description about new auth.httpHeader flag (#3134)
* docs, app/vmgateway: add description about new `auth.httpHeader` flag

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 08:56:43 +03:00
Aliaksandr Valialkin
98175493d7
app/vmselect/promql: remove empty series before applying aggregate function
Previously empty series (e.g. series with all NaN samples) were passed to aggregate functions.
Such series must be ingored by all the aggregate functions.
So it is better from consistency PoV filtering out empty series before applying aggregate functions.
2022-09-30 08:44:06 +03:00
Roman Khavronenko
273472dc20
app/vmselect: ignore empty series for limit_offset (#3178)
* app/vmselect: ignore empty series for `limit_offset`

VictoriaMetrics doesn't return empty series (with all NaN values) to
the user. But such series are filtered after transform functions.
It means `limit_offset` will account for empty series as well.

For example, let's consider following data set:
```
time series:
foo{label="1"} NaN, NaN, NaN, NaN // empty series
foo{label="2"} 1, 2, 3, 4
foo{label="3"} 4, 3, 2, 1
```

When user requests all series for metric `foo` the empty series
will be filtered out:
```
/query=foo:
foo{label="v2"} 1, 2, 3, 4
foo{label="v3"} 4, 3, 2, 1
```

But `limit_offset(1, 1, foo)` is applied to original series, not filtered yet.
So it will return `foo{label="v2"}` (skips the first in list)
```
/query=limit_offset(1, 1, foo):
foo{label="v2"} 1, 2, 3, 4
```

Expected result would be to apply `limit_offset` to already filtered list,
so in result we receive `foo{label="v3"}`:
```
/query=limit_offset(1, 1, foo):
foo{label="v3"} 4, 3, 2, 1
```

The change does exactly that - filters empty series before applying `limit_offset`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: ignore empty series for `limit_offset`

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-30 08:44:05 +03:00
Aliaksandr Valialkin
f1eebc0a99
lib/promrelabel: properly parse regex with escaped $ at the end
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3131

Thanks to @dmitryk-dk for the initial fix at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3179
2022-09-30 08:20:57 +03:00
Aliaksandr Valialkin
33590c5fb9
docs/CHANGELOG.md: document 39f559d22b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3013
2022-09-30 07:49:54 +03:00
Nikolay
909709346e
lib/awsapi: fixes sign encoding (#3183)
* lib/awsapi: fixes sign encoding

previously white spaces at filter were incorrectly encoded
encoding tip was copied from aws signing lib
For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do)
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3171

* Update lib/awsapi/sign.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-30 07:49:18 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Roman Khavronenko
fe71c73fe1
lib/mergeset: follow-up after a0e7432e42 (#3145)
* lib/mergeset: follow-up after a0e7432e42

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 16:43:17 +03:00
Aliaksandr Valialkin
d91bf89651
docs/CHANGELOG.md: document f022296d96
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3147
2022-09-26 16:32:54 +03:00
Aliaksandr Valialkin
89e1060101
docs/CHANGELOG.md: document 166d444159 2022-09-26 15:39:51 +03:00
Aliaksandr Valialkin
2b98f2bc1a
lib/protoparser/graphite: accept whitespace in metric names and tags according to the specification
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102

See the specification https://graphite.readthedocs.io/en/latest/tags.html
2022-09-26 15:20:11 +03:00
Aliaksandr Valialkin
dbc20091b1
lib/protoparser/datadog: sanitize metric names by default in the same way as DataDog does
This commit is based on the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105

Thanks to @PerGon for the idea and initial implementation.
2022-09-26 13:58:36 +03:00
Aliaksandr Valialkin
6a6cf9c590
app/{vmagent,vminsert}: add -usePromCompatibleNaming command-line flag for normalizing metric names and label names in the ingested samples
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113

Thanks to @erkexzcx for the idea and the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3146
2022-09-26 13:13:31 +03:00
Aliaksandr Valialkin
594a4ab345
docs/vmalert.md: follow-up for 0c95f928ae
- Clarify the description for -datasource.queryStep command-line flag
- Consistently use a single dash in front of -datasource.queryStep command-line flag
- Update -help output at docs/vmalert.md
2022-09-26 08:49:47 +03:00
Roman Khavronenko
cffceba0f5
vmalert: set default value for datasource.queryStep to 5m (#3149)
Change default value for command-line flag `datasource.queryStep` from `0s` to `5m`.
Param `step` is added by vmalert to every rule evaluation request sent to datasource.
Before this change, `step` was equal to group's evaluation interval by default.
Param `step` for instant queries defines how far VM can look back for the last written data point.
The change supposed to improve reliability of the rules evaluation when evaluation interval
is lower than scraping interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-26 08:40:17 +03:00
Aliaksandr Valialkin
19dc29abd7
docs/CHANGELOG.md: clarify the change at 622bbedbe1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3119
2022-09-19 15:04:37 +03:00
Roman Khavronenko
360b022603
vmalert: always re-evaluate Annotations (#3119)
* vmalert: always re-evaluate Annotations

Previously, Annotations were evaluated only:
1. On alert creating.
2. On alert's value change.

This is premature optimization. It was assumed that since annotations
could contain only text with alert's labels or value - there is no need
in spending resources to re-compile Annotations.

Later, template function `query` was added, which can execute
arbitrary queries and return different results on every evaluation.
So if it was used in annotations, it would be executed only on init
or value change.

Another case when optimization caused an issue - annotations hot reload.
In this case, annotations of the active alert won't change even if Rule's
annotations were changed.

This fix enables Annotations re-evaluation on each iteration to resolve
issues above. It would have some impact on performance, but it is unlikely
it will be noticeable.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add tp Changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-19 15:04:37 +03:00
Dmytro Kozlov
36ea8537d9
vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric) (#2982)
* vmselect/promql: add alphanumeric sort by label (sort_by_label_numeric)

* vmselect/promql: fix tests, add documentation

* vmselect/promql: update test

* vmselect/promql: update for alphanumeric sorting, fix tests

* vmselect/promql: remove comments

* vmselect/promql: cleanup

* vmselect/promql: avoid memory allocations, update functions descriptions

* vmselect/promql: make linter happy (remove ineffectual assigment)

* vmselect/promql: add test case, fix behavior when strings are equal

* vendor: update github.com/VictoriaMetrics/metricsql from v0.44.1 to v0.45.0

this adds support for sort_by_label_numeric and sort_by_label_numeric_desc functions

* wip

* lib/promscrape: read response body into memory in stream parsing mode before parsing it

This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-14 17:42:07 +03:00
Roman Khavronenko
6ae4f3526b
vmalert: add experimental feature of storing Rule's evaluation state (#3106)
vmalert: add experimental feature of storing Rule's evaluation state

The new feature keeps last 20 state changes of each Rule
in memory. The state are available for view on the Rule's
view page. The page can be opened by clicking on `Details`
link next to Rule's name on the `/groups` page.

States change suppose to help in investigating cases when Rule
doesn't generate alerts or records.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-09-14 15:08:40 +03:00
Aliaksandr Valialkin
2351468bc4
lib/promscrape: read response body into memory in stream parsing mode before parsing it
This reduces scrape duration for targets returning big responses.

The response body was already read into memory in stream parsing mode before this change,
so this commit shouldn't increase memory usage.
2022-09-14 13:29:39 +03:00
Aliaksandr Valialkin
20834c1757
app/vmalert: follow-up after 8441375da2
- Rename logDebug() to logDebugf() and pass format string together
  with format args directly to logDebugf(). This eliminates fmt.Sprintf()
  overhead at logDebug() call site when debugging is disabled.

- Format labels in debug message in Prometheus format, e.g. {label1="value1",...labelN="valueN"}

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025
2022-09-13 16:36:31 +03:00
Roman Khavronenko
a887c1bc07
vmalert: add debug mode for alerting rules (#3055)
* vmalert: add `debug` mode for alerting rules

Debug information includes alerts state changes and requests
sent to the datasource. Debug can be enabled only on rule's
level. It might be useful for debugging unexpected
behaviour of alerting rule.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3025

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update app/vmalert/alerting.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

* vmalert: go fmt

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-13 16:36:30 +03:00
Aliaksandr Valialkin
6a44e1bc06
docs/CHANGELOG.md: document atomic directory deletion
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
2022-09-13 16:16:12 +03:00
Yury Molodov
465894d720
vmui: fix data processing (#3092)
* fix: change data processing

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-12 10:56:39 +03:00
Yury Molodov
c8de98e03f
vmui: add lists of top queries (#3065)
* feat: add lists of top queries

* fix: change the field label

* refactor: add handlers for readability

* app/vmselect: `make vmui-update`

* docs: document `top queries` tab

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-08 21:44:43 +03:00
Aliaksandr Valialkin
052b527b39
docs/CHANGELOG.md: document ec273eafef
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3076
2022-09-08 21:21:26 +03:00
Aliaksandr Valialkin
f0eea5b02d
app/vmselect/netstorage: fix a typo, which leads to incorrect query results in VictoriaMetrics cluster
The typo has been introduced in the commit 1a254ea20c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067
2022-09-08 13:46:40 +03:00
Aliaksandr Valialkin
f6990871a3
docs/vmctl.md: make docs-sync after c5261d5f56
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:20:16 +03:00
Zakhar Bessarab
ef01f01b6c
vmctl: implement support of chunking data for vm-native export (#3044)
vmctl: implement support of chunking data for vm-native export process

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2733
2022-09-06 13:14:48 +03:00
Aliaksandr Valialkin
051e722112
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
This should reduce chances of unnoticed on-disk data corruption.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011

This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.

This is OK, since the native export format is undocumented.
2022-09-06 13:07:49 +03:00
Aliaksandr Valialkin
b7f3569522
app/vmselect/prometheus: follow-up after 50e2524bc2
- Add getCommonParamsWithDefaultDuration function and use it at /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the default behaviour for setting 5 minutes time range if start arg isn't passed to /api/v1/series, /api/v1/labels and /api/v1/label/.../values
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052
2022-09-05 11:57:08 +03:00
Aliaksandr Valialkin
c6b74148cf
app/vmselect/promql: consistently calculate rate_over_sum(m[d]) as sum_over_time(m[d])/d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045
2022-09-02 23:19:05 +03:00
Aliaksandr Valialkin
569843c0dd
docs/CHANGELOG.md: cut v1.81.1 2022-09-02 22:00:34 +03:00
Aliaksandr Valialkin
9cca3a0a1b
app/vmselect/netstorage: fix potential panic under high load
The panic may trigger during data blocks' processing received
from vmstorage nodes when some of vmstorage nodes return an error
or when `-replicationFactor` is set to values higher than 2 at `vmselect`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3058
2022-09-02 21:36:15 +03:00
Aliaksandr Valialkin
024e2f18da
app/vmselect/promql: evaluate union() args in parallel in order to increase query performance
Note that the parallel execution of `union()` args may take more memory and CPU time
than the sequential execution if args contain heavy queries, which may load all the available CPU,
disk and memory resources and vmselect and vmstorage levels.
2022-09-02 21:01:04 +03:00
Aliaksandr Valialkin
a11eae3a76
docs/CHANGELOG.md: cut v1.81.0 2022-08-31 02:33:36 +03:00
Aliaksandr Valialkin
30a1d0c622
docs/CHANGELOG.md: add missing link to vmagent 2022-08-31 02:22:52 +03:00
Aliaksandr Valialkin
cdb22c5ca3
docs/CHANGELOG.md: document v1.79.3 LTS release 2022-08-31 00:15:26 +03:00
Aliaksandr Valialkin
529bcc0761
docs/CHANGELOG.md: document the 044d51b668 2022-08-30 09:42:51 +03:00
Aliaksandr Valialkin
8aaaf221cc
app/vmselect/promql: follow-up after 2d71b4859c
- Use getScalar() function for obtaining the expected scalar from phi arg
- Reduce the error message returned to the user when incorrect phi is passed to histogram_quantiles
- Improve the description of this bugfix in the docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3026
2022-08-27 01:38:17 +03:00
Dmytro Kozlov
2cf29cc4d1
vmselect/promql: fix panic in histogram_quantiles function (#3029)
* vmselect/promql: fix panic in histogram_quantiles function

* Update docs/MetricsQL.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2022-08-27 01:34:54 +03:00
Aliaksandr Valialkin
86394b4179
lib/promscrape: optimize discoveryutils.SanitizeLabelName()
Cache sanitized label names and return them next time.
This reduces the number of allocations and speeds up the SanitizeLabelName()
function for common case when the number of unique label names is smaller than 100k
2022-08-27 00:18:19 +03:00
Aliaksandr Valialkin
e1bd38fa97
lib/promrelabel: optimize matching for commonly used regex patterns in if option
The following regex patterns are optimized:

- literal string match, e.g. "foo"
- prefix match, e.g. "foo.*" and "foo.+"
- substring match, e.g. ".*foo.*" and ".+foo.+"
- alternate values match, e.g. "foo|bar|baz"
2022-08-26 14:55:13 +03:00
Aliaksandr Valialkin
d60654eb0a
lib/promrelabel: optimize action: {labeldrop,labelkeep,keep,drop} with regex containing alternate values
For example, the following relabeling rule must work much faster now:

- action: labeldrop
  regex: "foo|bar|baz"
2022-08-24 17:55:54 +03:00
Aliaksandr Valialkin
d4cf52f092
app/vmagent: follow-up after 2b22aa1537
- Document the change at docs/CHANGELOG.md
- Move auth token parsing from app/vmagent/opentsdbhttp/ to app/vmagent/main.go,
  since it must be parsed only when multitenancy support is enabled at vmagent side.
  See https://docs.victoriametrics.com/vmagent.html#multitenancy
2022-08-24 16:21:23 +03:00
Dmytro Kozlov
d32a6359b0
vmselect/promql: enable search.maxPointsSubqueryPerTimeseries for sub-queries (#2963)
* vmselect/promql: enable search.maxPointsPerTimeSeriesSubquery for sub-queries

* vmselect/promql: cleanup

* vmselect/promql: rename config flag

* vmselect/promql: add tests

* vmselect/promql: use test object instead of log

* vmselect/promql: fix posible panic is subquery has more points. add description

* vmselect/promql: update tests descriptions

* vmselect/promql: update doInternal validation

* vmselect/promql: fix linter

* vmselect/promql: fix linter

* vmselect/promql: update documentation and release notes

* wip

- Properly apply -search.maxPointsSubqueryPerTimeseries limit to subqueries.
  Previously the -search.maxPointsPerTimeseries limit was unexpectedly applied to subqueries
  if it was smaller than the -search.maxPointsSubqueryPerTimeseries .
- Clarify docs for -search.maxPointsSubqueryPerTimeseries command-line flag .
- Document -search.maxPointsPerTimeseries and -search.maxPointsSubqueryPerTimeseries flags at https://docs.victoriametrics.com/#resource-usage-limits .
- Update docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2922

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-24 15:27:41 +03:00
Aliaksandr Valialkin
7b9ba456ff
app/vmstorage: expose vm_{hourly,daily}_series_limit_{max,current}_series metrics if -storage.max{Hourly,Daily}Series limits are set
These metrics allow alerting when the number of unique series approach the limit.
For example, the following query alerts when the number of series reaches 90% of the configured limit:

    vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9
2022-08-24 13:41:57 +03:00
Roman Khavronenko
555a202d80
docs: follow-up after 88425bb285 (#3007)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-24 01:23:37 +03:00
Aliaksandr Valialkin
c5325f2db5
docs/CHANGELOG.md: document d59d829cdb
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2673
2022-08-21 23:37:15 +03:00
Aliaksandr Valialkin
1c7f402598
app/vmagent: add ability to construct a label from multiple existing labels by referring them in the replacement field during relabeling
For example:

- target_label: composite-label
  replacement: {{source_label1}}-{{source_label2}}
2022-08-21 22:49:24 +03:00
Aliaksandr Valialkin
ecd8a03b28
docs: change links to Prometheus docs about instant and range queries to links to VictoriaMetrics docs 2022-08-21 19:02:46 +03:00
Aliaksandr Valialkin
76844e519b
docs/CHANGELOG.md: document 10402459d8 2022-08-19 11:44:29 +03:00
Roman Khavronenko
2256d51418
docs: fix docs formatting related to vmalert (#2994)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-19 11:05:08 +03:00
Aliaksandr Valialkin
bbfa52bd75
docs: follow-up after 68e56b6fc5 2022-08-17 21:27:24 +03:00
Roman Khavronenko
cfcb5ab15b
vmalert: set alert's source link to UI instead of JSON source (#2986)
We switch default alert's source link to redirect user
to vmalert's UI instead of previous JSON object. While it breaks
compatibility, it also supposed to improve user's experience.
The old behavior can be achieved by updating `-external.alert.source`
command-line flag.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-17 21:27:24 +03:00
Aliaksandr Valialkin
1812d33a2d
lib/promscrape: automatically generate additional per-target labels for targets with non-zero series limit
The following metrics are generated:

- scrape_series_limit
- scrape_series_current
- scrape_series_limit_samples_dropped

These metrics simplify alerting on targets, which expose too many time series

See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics
and https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for more details
2022-08-17 13:22:02 +03:00
Aliaksandr Valialkin
e5c61cac22
docs/CHANGELOG.md: group vmalert features closer to each other 2022-08-17 11:49:53 +03:00
Aliaksandr Valialkin
aa37e6b438
lib/promscrape: retry http requests if the server returns 429 status code
The 429 status code means that the server is overwhelmed with requests.
The client can retry the request after some wait time.
Implement this strategy for service discovery and scrape requests.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-16 14:57:26 +03:00
Aliaksandr Valialkin
dc929e0d16
lib/storage: improve performance for /api/v1/labels and /api/v1/label/.../values endpoints when match[] filter matches small number of time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2978
2022-08-16 13:34:23 +03:00
Roman Khavronenko
436b4d90af
docs: update vmalert docs (#2987)
* mention recently added `$alertID` and `$groupID` variables in the changelog
* properly escape template examples in the vmalert's README

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-16 12:10:08 +03:00
Aliaksandr Valialkin
7d7cf2b6fd
app/vmselect: follow-up after 63e0f16062
* Explicitly store a pointer to UserReadableError in the error interface.
  Previously Go automatically converted the value to a pointer before storing in the error interface.

* Add Unwrap() method to UserReadableError, so it can be used transparently with the other code,
  which calls errors.Is() and errors.As().

* Document the change in docs/CHANGELOG.md
2022-08-15 13:53:19 +03:00
Roman Khavronenko
79e396dac9
docs: mention default vmalert's behavior change in Update notes (#2981)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-15 13:31:59 +03:00
Yury Molodov
dc52b283a3
vmui: shortcut keys legend (#2971)
* feat: add shortcut modal

* feat: add shortcut descriptions

* app/vmselect/vmui: `make vmui-update`

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-15 12:04:30 +03:00
Aliaksandr Valialkin
c27bd63f6c
lib/promscrape: update links to sd_configs from Prometheus site to https://docs.victoriametrics.com/sd_configs.html 2022-08-15 01:40:48 +03:00
Aliaksandr Valialkin
1a00c9ef03
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_pod_container_image label in the same way as Prometheus 2.38 does
See https://github.com/prometheus/prometheus/pull/11034
2022-08-15 01:18:57 +03:00
Aliaksandr Valialkin
2fb63dda83
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_service_port_number label to role: service in the same way as Prometheus 2.38 does
See https://github.com/prometheus/prometheus/pull/11002
2022-08-15 01:07:19 +03:00
Aliaksandr Valialkin
198d8eeeaa
app/vmalert/templates: add toTime() template function in the same way as Prometheus 2.38 does
See https://github.com/prometheus/prometheus/pull/10993
2022-08-15 00:49:52 +03:00
Aliaksandr Valialkin
2b58bd9876
lib/promscrape/discovery/dns: add support for resolving MX records
See https://github.com/prometheus/prometheus/pull/10099
2022-08-15 00:33:06 +03:00
Aliaksandr Valialkin
78f584fb0b
docs/CHANGELOG.md: clarify the change at 28441711e6 2022-08-11 23:53:32 +03:00
Roman Khavronenko
36660bcfe2
vmalert: follow-up after 28441711e6 (#2972)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-08-11 23:52:18 +03:00
Aliaksandr Valialkin
ec3df0b913
app/vmselect/netstorage: improve scalability of blocks processing on systems with multiple CPU cores
Previously a single syncwg.WaitGroup was used for tracking the lifetime of processBlock callbacks
across all the per-vmstorage goroutines. This could be slow on systems with many CPU cores
because of inter-CPU synchronization overhead.

Use a separate per-vmstorage sync.WaitGroup instead in order to reduce inter-CPU synchronization overhead.
This should imrpove performance for heavy queries over big number of blocks on multi-CPU systems.
2022-08-11 21:37:24 +03:00
Roman Khavronenko
f42853275f
lib/storage: prevent excessive loops when storage is in RO (#2962)
* lib/storage: prevent excessive loops when storage is in RO

Returning nil error when storage is in RO mode results
into excessive loops and function calls which could
result into CPU exhaustion. Returning an err instead
will trigger delays in the for loop and save some resources.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-09 12:17:47 +03:00
Aliaksandr Valialkin
91d0b45537
docs/CHANGELOG.md: cut v1.80.0 2022-08-08 19:59:02 +03:00
Aliaksandr Valialkin
7e892d3322
docs/CHANGELOG.md: add changes for v1.79.2 2022-08-08 18:00:38 +03:00
Aliaksandr Valialkin
b8877da35b
docs/CHANGELOG.md: link to the issue related to vmselect panic in multi-level cluster setup
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2961
2022-08-08 15:09:39 +03:00
Aliaksandr Valialkin
310779d8b5
lib/promscrape: follow-up after 2c553d5a2f
- fix broken tests
- cosmetic code cleanup
- document the change at https://docs.victoriametrics.com/vmagent.html#multitenancy
- document the change at https://docs.victoriametrics.com/CHANGELOG.html
2022-08-08 14:49:16 +03:00
Aliaksandr Valialkin
1996e36cf0
app/vmselect/netstorage: prevent from calling processBlocks callback after the exit from ProcessBlocks function
This should prevent from panic at multi-level vmselect
when the top-level vmselect is configured with -replicationFactor > 1
2022-08-08 13:32:44 +03:00
Aliaksandr Valialkin
12f4c89ff7
docs/CHANGELOG.md: link to the issue with improper handling of enpoint option at ec2_sd_configs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2917
2022-08-08 03:33:17 +03:00
Aliaksandr Valialkin
9039f23bd1
lib/promrelabel: do not split regex into multiple lines if it contains groups
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2928
2022-08-08 03:16:15 +03:00
Aliaksandr Valialkin
23b7fa897a
docs/CHANGELOG.md: link to the issue regarding the increased load on Consul
This is a follow-up for 68de1f4e4a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2940
2022-08-08 02:23:01 +03:00
Yury Molodov
a2ecf311b8
vmui: graph action on moush hold and move (#2915)
* fix: change event for graph panning

* fix: change detect key

* feat: add zoom in with mouse selection

* - document the change
- run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-08-07 23:56:08 +03:00
Aliaksandr Valialkin
573f6c8d6c
Makefile: remove redundant -mod=vendor option when running Go tools
The `-mod=vendor` is automatically set when there is a `vendor` directory
starting from Go1.14 - see https://go.dev/doc/go1.14#go-command

Since the minimum supported Go version for VictoriaMetrics is Go1.17,
then the `-mod=vendor` option is no longer needed.
2022-08-07 20:40:41 +03:00
Aliaksandr Valialkin
5e43145031
docs: add docs for scrape_configs section 2022-08-07 00:04:55 +03:00
Aliaksandr Valialkin
77bd4e37cc
lib/promscrape/discovery/kubernetes: add missing __meta_kubernetes_ingress_class_name label for role: ingress
See 7e65ad3e43
and 7e1111ff14
2022-08-06 22:39:14 +03:00
Aliaksandr Valialkin
ecbe1ddf1b
lib/promscrape/discovery/ec2: properly handle custom endpoint option in ec2_sd_configs
This option was ignored since d289ecded1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1287
2022-08-05 18:52:37 +03:00
Aliaksandr Valialkin
80ecfcf759
lib/promscrape/discovery/dockerswarm: properly set __meta_dockerswarm_container_label_* labels instead of __meta_dockerswarm_task_label_* labels
See https://github.com/prometheus/prometheus/issues/9187
2022-08-05 16:20:29 +03:00
Aliaksandr Valialkin
85b04732ed
lib/promscrape/discovery/consul: allow stale responses from Consul service discovery by default
This aligns with Prometheus behaviour.

See `allow_stale` option description at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
2022-08-05 15:04:05 +03:00
Aliaksandr Valialkin
df34f8c5b8
docs: fixes after 83a4abda3f 2022-08-05 10:15:28 +03:00
Aliaksandr Valialkin
8ddad31eef
lib/promscrape/discovery/yandexcloud: follow-up after 6e5ac32fba
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1386
2022-08-04 22:28:21 +03:00
Aliaksandr Valialkin
64183d4416
docs/CHANGELOG.md: document v1.79.1 security fix 2022-08-02 13:31:14 +03:00
Aliaksandr Valialkin
a1e49606ed
app/{vmselect,vmalert}: properly generate http redirects if -http.pathPrefix command-line flag is set
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2918
2022-08-02 13:01:13 +03:00
Aliaksandr Valialkin
962ed46583
lib/mergeset: optimize mergeInmemoryBlocks() function
Do not spend CPU time on converting inmemoryBlock structs to inmemoryPart structs.
Just merge inmemoryBlock structs directly.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2249
2022-07-28 00:05:45 +03:00
Aliaksandr Valialkin
8722342819
docs/CHANGELOG.md: clarify docs about the ability to configure multiple sections for vmauth with identical username and different password values 2022-07-26 19:39:00 +03:00
Aliaksandr Valialkin
3d4c312ba2
app/vmselect/netstorage: improve the speed of queries over big number of time series on multi-CPU system
Reduce inter-CPU communications when processing the query over big number of time series.
This should improve performance for queries over big number of time series
on systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2896

Based on b596ac3745
Thanks to @zqyzyq for the idea.
2022-07-25 09:22:28 +03:00
Aliaksandr Valialkin
1b5799f894
lib/promscrape: set up=0 for partially failed scrape in stream parsing mode
This behaviour aligns with Prometheus behavior
2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin
aae522e501
docs/CHANGELOG.md: document bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2874
This is a follow-up for edecd2493c
2022-07-21 21:00:34 +03:00
Aliaksandr Valialkin
3cbe232186
docs/CHANGELOG.md: follow-up after 88edb3f6cf 2022-07-21 20:53:24 +03:00
Roman Khavronenko
d0abdc2b5b
vmalert: allow configuring custom headers per group (#2901)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 20:48:05 +03:00
Roman Khavronenko
356d8b99e0
vmalert: allow configuring custom headers for URLs (#2897)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2860

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-21 20:45:12 +03:00
Aliaksandr Valialkin
f00a6bf837
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url 2022-07-21 20:15:29 +03:00
Nikolay
0102eb3869
app/vmauth: allow duplicate usernames (#2888)
Usernames could be duplicate if it has uniq password.
vmauth makes routing based on auth token and username + password combination must be unique for this case.
2022-07-19 19:35:28 +03:00
Aliaksandr Valialkin
f992f96a88
app/vmselect/promql: execute q1 and q2 from q1 op q2 in parallel if labels pushdown cannot be applied
This should improve query performance if VictoriaMetrics has enough resources for processing `q1` and `q2` in parallel.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2886
2022-07-19 14:29:41 +03:00
Aliaksandr Valialkin
2d1366353c
lib/promscrape: reload all the scrape configs when the global section is changed inside -promscrape.config
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2884
2022-07-18 17:15:42 +03:00
Aliaksandr Valialkin
d920b0afec
app/vmselect/promql: properly return q1 series from q1 ifnot q2 when q2 returns nothing 2022-07-18 14:24:23 +03:00
Aliaksandr Valialkin
83fef7daf5
docs/CHANGELOG.md: document 2f9668eba5 2022-07-18 12:37:34 +03:00
Aliaksandr Valialkin
b2d9379bd0
docs/CHANGELOG.md: mention that v1.79.0 changes binary names to $(APP_NAME)-$(GOOS)-$(GOARCH)... naming 2022-07-18 12:00:45 +03:00
Aliaksandr Valialkin
a87f143e92
docs/CHANGELOG.md: cut v1.79.0 2022-07-14 16:15:35 +03:00
Roman Khavronenko
80de6face8
vmalert: drop support of deprecated extra_filter_labels param (#2870)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-14 16:10:46 +03:00
Aliaksandr Valialkin
36062d27c5
docs/CHANGELOG.md: formatting fixes for update notes 2022-07-14 12:48:17 +03:00
Yury Molodov
1ca20caa4b
vmui: optimize table view (#2867)
* feat: optimize table view

* fix: add column display setting

* app/vmselect: `make vmui-update`

Also document the change at docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-14 00:16:57 +03:00
Dmytro Kozlov
99402541fb
vmui: update time picker behavior (#2847)
* vmui: update time picker behavior

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-14 00:06:22 +03:00
Yury Molodov
23c3b5f8c7
vmui: change selection from autocomplete (#2862)
* fix: change selection from autocomplete

* update docs/CHANELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:56:04 +03:00
Nikolay
c007b129cb
lib/promscrape: adds azure service discovery (#2743)
* lib/promscrape: adds azure service discovery
Adds azure service discovery mechanism
implements authorization with oauth and msi
lists virtual machines and virtual machines managed by scaleSet

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1364

* makes linter happy

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-13 23:45:43 +03:00
Aliaksandr Valialkin
da6c85a2f6
all: follow-up for d99ba3481b 2022-07-13 17:17:08 +03:00
Aliaksandr Valialkin
be5fb139d6
docs/CHANGELOG.md: document 91faa152a5 2022-07-13 12:41:25 +03:00
Aliaksandr Valialkin
205c2a468a
docs/CHANGELOG.md: mention that the communication protocol between vmselect and vmstorage nodes is updated in the new release 2022-07-13 12:16:59 +03:00
Aliaksandr Valialkin
dd2b1f9ec4
app/vmselect/prometheus: follow-up after 3efe33b917 2022-07-11 20:37:21 +03:00
Dmytro Kozlov
6734d77d97
vmselect/prometeus: Add limit param to api/v1/series api endpoint (#2851)
* issue-2841: Add limit param to api/v1/series api endpoint

* issue-2841: add change log

* issue-2841: update logic

* issue-2841: simplify logic

* issue-2841: simplify logic, add information to documentation
2022-07-11 20:37:20 +03:00
Aliaksandr Valialkin
d442ee4610
app/vmselect/netstorage: optimize mergeSortBlocks function
- Use binary search instead of linear scan when locating the run of smallest timestamps
  in blocks with intersected time ranges. This should improve performance
  when merging blocks with big number of samples

- Skip samples with duplicate timestamps. This should increase query performance
  in cluster version of VictoriaMetrics with the enabled replication.
2022-07-09 00:35:38 +03:00
Roman Khavronenko
78da074ad3
metricsql: properly evaluate timezone_offset over time interval (#2842)
* metricsql: properly evaluate `timezone_offset` over time interval

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2771
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-07-08 14:04:40 +03:00
Aliaksandr Valialkin
decbf84e55
app/vmselect/vmui: follow-up after 0bf6841140
* Document the bugfix in docs/CHANGELOG.md
* Run `make vmui-update` for updating static js files for vmui, which are included into vmselect
2022-07-08 13:22:57 +03:00
Aliaksandr Valialkin
d0a943c487
docs/CHANGELOG.md: clarifications after e9b977859b 2022-07-08 13:05:11 +03:00
Aliaksandr Valialkin
a5588bc6f4
docs/CHANGELOG.md: recommend clearing caches after the upgrade from v1.78.0 to v1.78.1 2022-07-08 13:05:11 +03:00
Roman Khavronenko
b29fafa86b
vmalert: deprecate alert's status link (#2840)
* vmalert: deprecate alert's status link

Deprecate alert's status link `/api/v1/<groupID>/<alertID>/status` in favour of
`api/v1/alerts?group_id=<group_id>&alert_id=<alert_id>"`.

The change was needed for simplifying logic in vmselect for proxying vmalert's requests.

The old alert's status link will be still supported for a few versions but will be removed in the future.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-08 13:05:11 +03:00
Aliaksandr Valialkin
9ccae66337
docs/CHANGELOG.md: link to the issue related to multi-level vmselect 2022-07-08 05:23:51 +03:00
Aliaksandr Valialkin
e855744871
docs/CHANGELOG.md: recommend upgrading from v1.78.0 to v1.78.1 because of the bug in v1.78.0 2022-07-08 03:45:06 +03:00
Aliaksandr Valialkin
5a5c8534d1
docs/CHANGELOG.md: cut v1.78.1 2022-07-08 01:03:04 +03:00
Aliaksandr Valialkin
d8483f599e
docs/CHANGELOG.md: typo fixes 2022-07-07 20:37:20 +03:00
Aliaksandr Valialkin
f03d4b2aa2
docs/CHANGELOG.md: link another bugreport related to the bug with per-day inverted index in v1.78.0 2022-07-07 02:35:35 +03:00
Aliaksandr Valialkin
5794886662
lib/promscrape: properly set Host header when sending requests via http proxy
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2794
2022-07-07 02:28:47 +03:00
Aliaksandr Valialkin
9dd5d3a431
docs/CHANGELOG.md: link to the issue about incorrect per-day index handling in v1.78.0 2022-07-07 02:00:58 +03:00
Aliaksandr Valialkin
95add1e8e4
app/{vmagent,vminsert}: follow-up after d19e46de55
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2839
2022-07-07 01:32:11 +03:00
Aliaksandr Valialkin
4d03ac90fc
lib/promscrape/discovery/kubernetes: properly populate service-level labels for role: endpointslice targets
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823
2022-07-07 00:36:25 +03:00
Aliaksandr Valialkin
c4cc45d7f8
lib/promscrape/discovery/kubernetes: allow attaching node-level labels to role: endpoints and role: endpointlice targets in the same way as Prometheus does
See https://github.com/prometheus/prometheus/pull/10759
2022-07-07 00:36:24 +03:00
Aliaksandr Valialkin
195dccf678
app/vmselect: add ability to query vmselect from another vmselect 2022-07-06 13:19:45 +03:00
Aliaksandr Valialkin
714958deab
all: follow-up after ed89106274 2022-07-06 12:48:52 +03:00
Aliaksandr Valialkin
498c6d6e72
lib/promscrape: push scrape_samples_limit metric to remote storage if sample_limit option is set in scrape_config for this target
See https://github.com/VictoriaMetrics/operator/issues/497
2022-07-06 12:46:23 +03:00
Roman Khavronenko
f51bc07d97
vmselect: allow proxying requests to vmalert (#2833)
The change allows to proxy requests with prefix `/vmalert`
to the vmalert component if `-vmalert.proxyURL` is set.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825
and https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2831

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-06 10:47:51 +02:00
Aliaksandr Valialkin
78f9a8aafd
lib/storage: put the (date, metricID) entry in dateMetricIDCache just after the corresponding series is registered in the per-day inverted index
Previously the time series could be put into dateMetricIDCache without
registering in the per-day inverted index if GetOrCreateTSIDByName
finds TSID entry in the global index. This could lead to missing
series in query results.

The issue has been introduced in the commit 55e7afae3a,
which has been included in VictoriaMetrics v1.78.0
2022-07-05 14:56:55 +03:00
Aliaksandr Valialkin
8516670582
app/vmselect/promql: properly calculate histogram_quantile over unexpected le buckets
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819
2022-07-05 13:20:41 +03:00
Roman Khavronenko
78da4b5f5e
docs: warn about potential issue with read queries for 1.78.0 (#2818)
* docs: warn about potential issue with read queries for 1.78.0

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs: warn about potential issue with read queries for 1.78.0

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-07-04 11:47:41 +03:00
Aliaksandr Valialkin
fa08220d27
app/vmselect/promql: properly handle partial counter resets in rate(), irate(), increase() and remove_resets() functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787
2022-06-30 22:40:26 +03:00
Aliaksandr Valialkin
7fc03a1deb
app/vmagent/remotewrite: add -remoteWrite.header command-line flag for setting additional http headers to send to -remoteWrite.url
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805
2022-06-30 20:00:59 +03:00
Aliaksandr Valialkin
da3179667e
docs/CHANGELOG.md: document bdf9f4669a 2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin
92e6c55b75
docs: add Troubleshooting doc
This doc contains troubleshooting guides for typical problems with VictoriaMetrics.
2022-06-30 13:52:44 +03:00
Aliaksandr Valialkin
daefb64f38
app/vmselect: expose additional histograms at /metrics page, which may help get more insights for the query workload
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2792
2022-06-28 20:18:31 +03:00
Aliaksandr Valialkin
7d5d33fd71
lib/storage: return marshaled metric names from SearchMetricNames
Previously SearchMetricNames was returning unmarshaled metric names.
This wasn't great for vmstorage, which should spend additional CPU time
for marshaling the metric names before sending them to vmselect.

While at it, remove possible duplicate metric names, which could occur when
multiple samples for new time series are ingested via concurrent requests.

Also sort the metric names before returning them to the client.
This simplifies debugging of the returned metric names across repeated requests to /api/v1/series
2022-06-28 18:16:32 +03:00
Aliaksandr Valialkin
a38568ddfe
docs/CHANGELOG.md: document 45f20ad1aa 2022-06-27 13:53:26 +03:00
Aliaksandr Valialkin
6386f117c8
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds 2022-06-27 13:42:57 +03:00
Aliaksandr Valialkin
ea2aa617e5
docs/CHANGELOG.md: update after e40d015e9a 2022-06-24 18:03:56 +03:00
Aliaksandr Valialkin
94b81165d0
all: limit the maximum memory usage for regexp cache, which stores parsed regular expressions in MetricsQL queries
Previously the cache could store 10K unique regexps. When every regexp is huge (e.g. hundreds of kilobytes),
then the total cache size could grow to multiples of gigabytes. Now the cache size is limited by the total length
of all cached regexps. So huge regexps won't result in high memory usage for the cache.
2022-06-24 17:58:47 +03:00
hagen1778
e40d015e9a vmselect: make vm_partial_results_total consistent
Metrics `vm_partial_results_total` and `vm_requests_total` serving
the similar purpose, but contain inconsistent set of labels.
This change updates `vm_partial_results_total` labels to be consistent
with `vm_requests_total`.

The change breaks backward compatibility with assumption that
`vm_partial_results_total` wasn't widely used, since it is
not documented and absent in the alerts and dashboards.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-24 13:50:26 +02:00
Dmytro Kozlov
f28cbcc7b5
vmui: added query tracing (#2748)
* vmui: added query tracing

* vmui: updated ui

* vmui: update tracing logic, fix bugs, disable tracing by default

* vmui: use empty message as props

* vmui: fixed ui, added delete for each tacing data, show query in header

* vmui: added timelines

* vmui: speedup render

* vmui: use memo for sorting

* vmui: use Trace model, remove unused functions, simplify part of code

* vmui: update recursive logic

* vmui: fix set query to header

* vmui: code cleanup, remove unused code

* vmui: remove unused type, rename component

* wip

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-23 23:00:45 +03:00
Nikolay
ee5c502446
app/vmselect: fixes partial response with replicationFactor (#2777)
* app/vmselect: fixes partial response with replicationFactor
Allow partial response if it meets replicationFactor configured at vmselect
https://t.me/VictoriaMetrics_ru1/38490

* docs/CHANGELOG.md: document this change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-23 20:17:24 +03:00
Yurii Kravets
f0c1edb175
docs: Update CHANGELOG Update notes (#2776)
* docs: Update CHANGELOG Update notes

Specified the reason why `vmselect` and `vmstorage` nodes may log communication errors.
2022-06-23 19:40:20 +03:00
Aliaksandr Valialkin
71b0dfdefa
lib/promscrape: always send stale markers with the real scrape timestamp
This guarantees that query won't return data just after the series is disappeared.
2022-06-23 11:49:13 +03:00
Aliaksandr Valialkin
3ae6300497
lib/promauth: add ability to send additional http headers in requests to scrape targets
This solves https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header
2022-06-22 20:40:50 +03:00
Aliaksandr Valialkin
7429dfbe9e
app/vmselect: add -search.setLookbackToStep command-line flag for making the gap filling algorithm similar to InfluxDB data model
This option should override `-search.maxStalenessInterval` for most cases when users migrate from InfluxDB to VictoriaMetrics
2022-06-22 14:20:02 +03:00
Aliaksandr Valialkin
24097f2417
docs/CHANGELOG.md: cut v1.78.0 2022-06-20 18:11:04 +03:00
Aliaksandr Valialkin
a1629bd3be
lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful
If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped
until the background goroutine determines that the given address can be successfully dialed.

This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711

This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756

The main differences are:

- The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool.
  This makes possible re-using this feature everywhere lib/netutil.ConnPool is used.
- The check doesn't take into account handshake errors for already established connections.
  Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.
2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin
45e9732764
docs: follow-up after e4d6b750f6 2022-06-20 17:15:52 +03:00
Nikolay
15662c0f29
lib/httpserver: adds flagsAuthKey command-line flag (#2758)
* lib/httpserver: adds flagsAuthKey command-line flag
It protects /flags endpoint with authKey.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2753O

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-20 17:15:51 +03:00
Artem Navoiev
42c69ae74e
docs: replace bash code block type with console (#2746) 2022-06-19 23:02:00 +03:00
Aliaksandr Valialkin
4ae1c5655f
docs/CHANGELOG.md: document ef7f52e0e6 2022-06-19 22:49:11 +03:00
Aliaksandr Valialkin
88e1221b35
lib/storage: do not register new series if -storage.maxHourlySeries or -storage.maxDailySeries limits are exceeded
Previously samples for new series weren't added as expected when series limits were reached,
but new series were still registered in indexdb.
2022-06-19 22:03:02 +03:00
Aliaksandr Valialkin
c5ac176153
lib/storage: reset metric id caches for the previous and the current hour
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698
2022-06-19 22:02:51 +03:00
Dmytro Kozlov
5bc13e2fe8
vmui: added focusLabel, enable cardinality app configuratior (#2736)
* vmui: added focusLabel, enable app configuratior

* vmui: set focusLabel if {labelName!=""}

* wip

* docs/CHANGELOG.md: mention about focusLabel feature in cardinality explorer

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2730

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-17 13:04:09 +03:00
Roman Khavronenko
cbe39bbb8d
vmselect: limit end param max value by 2d in future (#2729)
* vmselect: limit `end` param max value by 2d in future

The change is applied only to service handlers like `/labels` or `/series`
and limits the `end` param by max value <= now() + 2 days. The same limit
is applied for the ingested data, so no reason to allow to request data
in future far than that.

The change is also needed for corner cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2669
where too high `end` value triggers inefficient global index search.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs/CHANGELOG.md: document the bugfix

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-16 20:51:40 +03:00
Aliaksandr Valialkin
450aa0ae5a
lib/promrelabel: support action: graphite relabeling
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737
2022-06-16 20:25:49 +03:00
Aliaksandr Valialkin
b4e75a0b89
docs/CHANGELOG.md: document dd327bfa9e2c69fe21ab1d92c14636733d7c5620 2022-06-15 18:41:26 +03:00
Aliaksandr Valialkin
14ce7b0e25
docs/CHANGELOG.md: document 00719e5779a3e4eeedb74cb3d25a9ecfe0e16063 2022-06-15 18:09:23 +03:00
Aliaksandr Valialkin
c846d0db01
docs/CHANGELOG.md: document 99dbe7f9d4 2022-06-13 10:07:19 +03:00
Yury Molodov
65107a4585
vmui: enhancements (#2638) (#2717)
* feat: make datepicker to be set to last 30 min by default

* fix: correct spinner while loading data

* feat: change legend style

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-13 09:49:58 +03:00
Aliaksandr Valialkin
9d8c37d280
docs/CHANGELOG.md: refer to the issue, which should be solved after the optimization to /api/v1/labels and /api/v1/label/.../values is added
The optimization has been added in 374beb350e

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1533
2022-06-12 14:32:57 +03:00
Aliaksandr Valialkin
61e03f172b
app/vmselect: optimize /api/v1/labels and /api/v1/label/.../values handlers when match[] query arg is passed to them 2022-06-12 14:06:24 +03:00
Aliaksandr Valialkin
4a94cd81ce
app/vmselect: add optional limit query arg to /api/v1/labels and /api/v1/label_values endpoints
This arg allows limiting the number of sample values returned from these APIs
2022-06-10 10:24:07 +03:00
Roman Khavronenko
3c09d25039
vmalert: followup for 76f05f8670 (#2706)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-06-09 13:15:35 +03:00
Aliaksandr Valialkin
2520b8efb8
docs: add a link to cardinality explorer playground 2022-06-08 19:40:12 +03:00
Dmytro Kozlov
f2754c3e90
Cardinality explorer (#2625)
* Cardinality explorer

* vmui, vmselect: updated field name, added description to spinner

* make vmui-update

* updated const name, make vmui-update

* lib/storage: changes calculation for totalSeries values

* added static files

* wip

* wip

* wip

* wip

* docs/CHANGELOG.md: document cardinality explorer feature

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2233

Co-authored-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-06-08 18:54:27 +03:00
Aliaksandr Valialkin
ba406ff28c
docs/CHANGELOG.md: document 63b538ecd1 2022-06-07 15:53:08 +03:00
Aliaksandr Valialkin
c07f99cd4f
docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2685 2022-06-07 15:39:53 +03:00
Aliaksandr Valialkin
c7ff4505d2
docs/CHANGELOG.md: document backwards-incompatible changes in communication protocol between vmselect and vmstorage
The changes are related to the added query tracing in afced37c0b

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
2022-06-07 15:25:58 +03:00
Aliaksandr Valialkin
2b72d21840
docs/CHANGELOG.md: document e755d0ec3f 2022-06-07 15:21:10 +03:00
Aliaksandr Valialkin
0b4f767c53
docs/CHANGELOG.md: document 1ee1e986da 2022-06-07 15:02:57 +03:00
Aliaksandr Valialkin
68b6ddfb14
all: follow-up after 8edb390e21
- Remove unused js bloatware from /targets page. This strips down binary size by more than 100Kb
- Add /service-discovery page for API compatibility with Prometheus
- Properly load bootstrap.min.css from /prometheus/targets
- Serve static contents for /targets page from app/vminsert instead of app/vmselect, because /targets page is served from there
2022-06-07 01:05:53 +03:00
Aliaksandr Valialkin
3dbb19d624
lib/promscrape/discovery/kubernetes: follow-up after 006b8c7534
- make more clear error logs
- simplify testing for newKubeConfig by passing only the path to kube_config file instead of SDConfig struct
2022-06-06 14:41:28 +03:00