Commit graph

3493 commits

Author SHA1 Message Date
Github Actions
3a96830ed6
Automatic update operator docs from VictoriaMetrics/operator@fec3f9d (#5381) 2023-11-23 21:06:27 +02:00
Aliaksandr Valialkin
a906a7d85c
app/vmagent/remotewrite: do not drop persistent queues when -remoteWrite.multitenantURL is set
It is unsafe to drop persistent queues when -remoteWrite.multitenantURL command-line flag is set,
since these queues are created on demand when a new sample for the given tenant is pushed
to the remote storage.

This addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5357
The issue has been appeared in the commit f3a51e8b1d
when implementing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014
2023-11-23 20:43:21 +02:00
Github Actions
28df725a37
Automatic update operator docs from VictoriaMetrics/operator@45bfa36 (#5373) 2023-11-22 20:25:46 +02:00
Aliaksandr Valialkin
10b4dfbbf9
app/vmalert/notifier: remove backticks from the description for -notifier.blackhole command-line flag
Backticks in flag description are automatically converted to flag type. See https://pkg.go.dev/flag#PrintDefaults

This is a follow-up for 20025d4fd6 and 25317b4e70
2023-11-22 20:17:45 +02:00
Aliaksandr Valialkin
db6dadf1f7
docs: convert png images to webp in all the docs except of docs/operator/*
This reduces the size of docs/* folder from 33MB to 18MB

Images inside docs/operator/* must be converted at the https://github.com/VictoriaMetrics/operator/tree/master/docs
and then the updated images must be automatically propagated to the docs/operator/*

This is a follow-up for d3f919df3e

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5206
2023-11-22 19:29:47 +02:00
Github Actions
bf934482c5
Automatic update operator docs from VictoriaMetrics/operator@b96131b (#5371) 2023-11-22 19:23:15 +02:00
Nikolay
c06044ef52
app/vmagent: adds google pubsub as remoteWrite dst and ingest consumer (#713)
it allows to push and receive metrics from google pubsub queue
Adds needed documentation and examples for it
2023-11-20 22:43:26 +02:00
hagen1778
34b7783461
docs: calrify version when vminsertConnsShutdownDuration was added
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-20 17:14:57 +01:00
hagen1778
0dbbffbdd5
docs: typo after 3f5a41e35e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 20025d4fd6)
2023-11-20 17:06:21 +01:00
hagen1778
6575d646c0
docs: follow-up after d3f919df3e
d3f919df3e
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 3ffa8975d4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-20 11:54:28 +01:00
Dmytro Kozlov
2362af3b0c
docs/managed-victoriametrics: use webp format to reduce image size (#5206)
(cherry picked from commit d3f919df3e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-20 11:53:06 +01:00
Hui Wang
91379331eb
lib/protoparser/promremotewrite: fall back to zstd decoding if Snappy-decoding fails (#5344)
This case is possible after the following steps:
1. vmagent successfully performed handshake with the -remoteWrite.url and the remote storage supports zstd-compressed data.
2. remote storage became unavailable or slow to ingest data, vmagent compressed the collected data into blocks with zstd and puts these blocks to persistent queue on disk.
3. vmagent restarts and the remote storage is unavailable during the handshake, then vmagent falls back to Snappy compression.
4. vmagent starts sending zstd-compressed data from persistent queue to the remote storage, while falsely advertizing it sends Snappy-compressed data.
5. The remote storage receives zstd-compressed data and fails unpacking it with Snappy.

The solution is the same as 12cd32fd75, just fall back to zstd decompression if Snappy decompression fails.
2023-11-17 15:53:18 +01:00
Aliaksandr Valialkin
39c56e8f65
docs/enterprise.md: update VictoriaMetrics version in examples from v1.95.0 to v1.95.1
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1
2023-11-17 15:47:24 +01:00
Github Actions
2e6c404cdd
Automatic update operator docs from VictoriaMetrics/operator@388745c (#5340) 2023-11-17 15:47:22 +01:00
luckyxiaoqiang
3419c14b35
docs/metricsql: remove duplicate sentence (#5349) 2023-11-17 15:41:21 +01:00
Aliaksandr Valialkin
1a15b0f57b
docs/CHANGELOG.md: cut v1.95.1 2023-11-16 20:32:27 +01:00
Aliaksandr Valialkin
5ad1c2b46a
docs/FAQ.md: add a link to https://docs.victoriametrics.com/#monitoring in questions where this is needed 2023-11-16 17:45:38 +01:00
Aliaksandr Valialkin
ef80a89a24
lib/handshake: add SetReadDeadline and SetWriteDeadline implementations additionally to SetDeadline
This is a follow-up for 27a5461785

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5327
2023-11-16 16:43:36 +01:00
Roman Khavronenko
27a5461785
lib/handshake: check for deadline in Read and Write methods (#5327)
The buffered connection could have exceeded the underlying connection
deadline during reading or writing to an internal buffer.
With this change, buffered connection struct additionally checks
for a deadline in Read/Write methods.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-16 16:33:40 +01:00
Github Actions
f430d506f2
Automatic update operator docs from VictoriaMetrics/operator@bc8b02f (#5331) 2023-11-16 16:29:56 +01:00
Roman Khavronenko
c0039ce7a3
docs/vmalert: clarify deduplication recommendations for HA setup (#5336)
Please see discussion here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5279

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-16 16:27:47 +01:00
Aliaksandr Valialkin
147fe45828
docs/CHANGELOG.md: remove duplicate word query after 2cbdb1db22 2023-11-16 16:24:15 +01:00
Aliaksandr Valialkin
7ca8ebef20
app/vmselect/promql: properly handle duplicate series when merging cached results with the results obtained from the database
evalRollupFuncNoCache() may return time series with identical labels (aka duplicate series)
when performing queries satisfying all the following conditions:

- It must select time series with multiple metric names. For example, {__name__=~"foo|bar"}
- The series selector must be wrapped into rollup function, which drops metric names. For example, rate({__name__=~"foo|bar"})
- The rollup function must be wrapped into aggregate function, which has no streaming optimization.
  For example, quantile(0.9, rate({__name__=~"foo|bar"})

In this case VictoriaMetrics shouldn't return `cannot merge series: duplicate series found` error.
Instead, it should fall back to query execution with disabled cache.

Also properly store the merged results. Previously they were incorrectly stored because of a typo
introduced in the commit 41a0fdaf39

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5332
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5337
2023-11-16 16:16:17 +01:00
Github Actions
16a41593e6
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@95d0711 (#5329) 2023-11-16 15:44:27 +01:00
hagen1778
7d72474a38
dashboards: use version instead of short_version in annotations
`version` label won't show the difference if various flavors of the same
version were deployed. But `short_version` will.

For example, on the sandbox env we test VM builds before new version release.
Without this change, the version update won't be visible on dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d389a4fcf3)
2023-11-16 09:27:42 +01:00
Aliaksandr Valialkin
844b0db67a
docs/Articles.md: add https://hackernoon.com/unleashing-vm-histograms-for-ruby-migrating-from-prometheus-to-victoriametrics-with-vm-client 2023-11-16 00:53:53 +01:00
Aliaksandr Valialkin
68c0038a5d
docs/vmbackup.md: fix links to https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages
This is a follow-up for 2fc7e9f47e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
2023-11-15 23:27:00 +01:00
Aliaksandr Valialkin
9a1354e8a9
docs/vmagent.md: refer to proper command-line flag: -remoteWrite.shardByURL.labels instead of -remoteWrite.shardByURLLabels
This is a follow-up for ed70a40669

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942
2023-11-15 23:03:30 +01:00
Aliaksandr Valialkin
f9355d34be
docs: mention that VictoriaMetrics and vmagent support data ingestion via New Relic protocol now
This is a follow-up for f60c08a7bd
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520
2023-11-15 22:56:34 +01:00
Aliaksandr Valialkin
772d2081ca
docs/Release-Guide.md: point to the proper location for CHANGELOG.md at github.com/VictoriaMetrics/operator repository 2023-11-15 21:56:11 +01:00
Aliaksandr Valialkin
656a6862d7
deployment: update reference to VictoriaLogs from v0.4.1-victorialogs to v0.4.2-victorialogs 2023-11-15 20:47:14 +01:00
Aliaksandr Valialkin
bc3a52cf50
docs/VictoriaLogs/README.md: cut v0.4.2-victorialogs 2023-11-15 20:47:14 +01:00
Aliaksandr Valialkin
bafe62b487
deployment: update references to VictoriaMetrics components from v1.94.0 to v1.95.0 2023-11-15 20:38:51 +01:00
Aliaksandr Valialkin
9ad4a8fffe
docs/CHANGELOG.md: cut v1.95.0 release 2023-11-15 17:46:02 +01:00
Aliaksandr Valialkin
bd5bbdf00c
docs/CHANGELOG.md: document v1.93.8 LTS release 2023-11-15 17:12:56 +01:00
Aliaksandr Valialkin
de3d5943eb
docs/stream-aggregation.md: clarify that stream aggregation is applied after all the configured relabeling
This is a follow-up after 68d2cb203d
2023-11-15 15:54:57 +01:00
Aliaksandr Valialkin
6a8911ad38
docs/CHANGELOG.md: document v1.87.11 LTS release 2023-11-15 15:54:57 +01:00
hagen1778
14df5af660
docs/stream-aggr: specify the relabeling order during aggregation
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68d2cb203d)
2023-11-15 14:29:29 +01:00
John Belmonte
7cd9757a2d
relabeling.md: fix link (#5325) 2023-11-15 12:57:49 +01:00
Aliaksandr Valialkin
d7a63529b5
docs/CHANGELOG.md: consistently prepend command-line flags with a single dash 2023-11-14 21:44:46 +01:00
hagen1778
cfc58dd932
docs: clarify vmalert flag changes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-14 21:44:46 +01:00
Aliaksandr Valialkin
d8220066dc
docs/Cluster-VictoriaMetrics.md: clarify how -storage.vminsertConnsShutdownDuration command-line flag works 2023-11-14 21:41:19 +01:00
Nikolay
0730c2586d
lib/querytracer: makes package concurrent safe to use (#5322)
* lib/querytracer: makes package concurrent safe to use
it must fix various issues with concurrent code usage.
Especially, when it's not reasonable to wait for all goroutines to be finished

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-11-14 20:58:28 +01:00
Artem Navoiev
83275af54b
docs: update Grafana setup section, use more direct link and add noti… (#5287)
(cherry picked from commit 5d61a7327d)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-14 14:36:35 +01:00
hagen1778
72a40539b0
dashboards: update description for RSS and anonymous memory panels to be consistent for single-node, cluster and vmagent dashboards.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d3ae2b2f62)
2023-11-14 10:00:11 +01:00
hagen1778
777424082b
deployment/dashboards: respect job and instance filters for alerts annotation in cluster and single-node dashboards
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d6ae082598)
2023-11-14 10:00:11 +01:00
Aliaksandr Valialkin
d6a2264709
docs/CHANGELOG.md: document 0e056ddb2d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5203
2023-11-14 01:24:29 +01:00
Aliaksandr Valialkin
9760221214
lib/logstorage: always check the previous indexBlockHeader for blocks with matching tenantID and/or streamID
The previous indexBlockHeader may contain blocks for the matching tenantID and/or streamID,
so it must be scanned unconditionally during the search.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856

This is a follow-up for 89dcbc2fe7
2023-11-14 01:02:02 +01:00
Zakhar Bessarab
f7834767c1
vmcluster: re-routing enhancement (#5293)
* app/vmstorage: close vminsert connections gradually before stopping storage

Implements graceful shutdown approach suggested here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1768146878

Test results for this can be found here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4922#issuecomment-1790640274

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update graceful shutdown logic

- close connections from vminsert in determenistic order
- update flag description
- lower default timeout to 25 seconds. 25 seconds value was chosen because the lowest default value used in default configuration deployments is 30s(default value in Kubernetes and ansible-playbooks).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/cluster: add information about re-routing enhancement during restart

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: add entry for new command-line flag

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* {app/vmstorage,lib/ingestserver}: address review feedback

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/cluster: add note to update workload scheduler timeout

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-11-14 01:00:42 +01:00
Aliaksandr Valialkin
c1f651a9f9
app/vmauth: add ability to drop the specified number of /-delimited prefix parts from request path
This can be done via `drop_src_path_prefix_parts` option at `url_map` and `user` levels.

See https://docs.victoriametrics.com/vmauth.html#dropping-request-path-prefix
2023-11-13 22:34:40 +01:00
Aliaksandr Valialkin
12cd32fd75
lib/protoparser/promremotewrite: fall back to Snappy decoding if zstd decoding fails
This case is possible after the following steps:

1. vmagent tries to perform handshake with the -remoteWrite.url in order to determine whether
   the remote storage supports zstd-compressed data.
2. The remote storage is unavailable during the handshake. In this case vmagent falls back to Snappy compression
   for the data sent to the remote storage.
3. vmagent compresses the collected data into blocks with Snappy and puts these blocks to persistent queue on disk.
4. The remote storage becomes available.
5. vmagent restarts, performs the handshake with the remote storage and detects that it supports zstd-compressed data.
6. vmagent starts sending Snappy-compressed data from persistent queue to the remote storage,
   while falsely advertizing it sends zstd-compressed data.
7. The remote storage receives Snappy-compressed data and fails unpacking it with zstd.

The solution is to just fall back to Snappy decompression if zstd decompression fails.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301
2023-11-13 21:25:39 +01:00
Aliaksandr Valialkin
356deada8c
lib/htmlcomponents: use relative links for the top page and for favicon.ico
This allows hiding VictoriaMetrics components behind proxies with arbitrary path prefixes.
For example, vmagent HTTP handlers can be served via /vmagent/ path prefix:

- http://proxy/vmagent/targets
- http://proxy/vmagent/service-discovery

The path prefix can be arbitrary. For example, below are vmagent urls
for /tenantID/vmagent/ path prefix:

- http://proxy/tenantID/vmagent/targets
- http://proxy/tenantID/vmagent/service-discovery

While at it, consistently serve favicon.ico from any path directory.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5306
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5307
2023-11-13 20:28:17 +01:00
Aliaksandr Valialkin
fb2071a01e
lib/regexutil: properly handle alternate regexps surrounded by .+ or .*
Previously the following regexps were improperly handled:

  .+foo|bar.+
  .*foo|bar.*

This could lead to unexpected regexp match results.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5297

Thanks to @Haleygo for the initial attempt to fix the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5308
2023-11-13 18:25:57 +01:00
Aliaksandr Valialkin
8256937d5e
docs/VictoriaLogs/CHANGELOG.md: follow-up for 66527c5981
Document the change

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5312
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300
2023-11-13 10:40:19 +01:00
Aliaksandr Valialkin
2a3352c70e
docs/CHANGELOG.md: remove trailing whitespace after bffd30b57a 2023-11-13 09:47:36 +01:00
Aliaksandr Valialkin
54c494ae8e
docs/vmauth.md: add missing dashes in front of command-line flags at the Backend TLS setup section
Dashes must be consistently used in front of command-line flags across the documentation.

This is a follow up for 61594d2bd8
2023-11-13 09:45:52 +01:00
Aliaksandr Valialkin
b9aba7edfb
app/vmauth: properly pass Host header to backends
Previously the `Host` header was remained unchanged when passing it in requests to backends.
This may improperly work if the backend uses host-based routing.

While at it, allows http/2.0 requests to backends. While VictoriaMetrics components
do not accept http/2.0 requests, other backends can require such requests.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240
2023-11-13 09:45:34 +01:00
Aliaksandr Valialkin
78bc816220
app/vmauth: follow-up for 323f3720ed
- Re-use identically configured http.Transport across multiple users.
  This fixes handling of the limit on the number of connection, which can be established per each backend
  via -maxIdleConnsPerBackend command-line flag. This limit stopped working after 323f3720ed

- Add docs about backend TLS setup at https://docs.victoriametrics.com/vmauth.html#backend-tls-setup

- Add ability to disable backend TLS verification for all the users via -backend.tlsInsecureSkipVerify command-line flag.
  This flag may be useful when -auth.config contains big number of users, and every user must disable backend TLS verification.

- Add ability to specify TLS Root CA via tls_ca_file option at per-user basis and via -backend.tlsCAFile command-line flag
  across all the users.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240
2023-11-13 09:45:16 +01:00
Aliaksandr Valialkin
4e621aaa0b
docs/Articles.md: typo fix 2023-11-13 09:44:53 +01:00
Aliaksandr Valialkin
76384b6d28
app/vmauth: improve docs a bit after 323f3720ed
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240
2023-11-13 09:44:25 +01:00
Aliaksandr Valialkin
bf12a49087
app/vmagent/README.md: sync with docs/vmagent.md after 930d26b2ff 2023-11-13 09:44:07 +01:00
Aliaksandr Valialkin
d9ecc3f6d7
lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args 2023-11-13 09:43:49 +01:00
Aliaksandr Valialkin
c916294b61
app/vmselect/promql: optimize instant queries with min_over_time() and max_over_time() rollup functions
This is a follow-up for 41a0fdaf39
2023-11-13 09:43:18 +01:00
Aliaksandr Valialkin
7bbdecb79a
deployment: update Go builder from Go1.21.3 to Go1.21.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved
2023-11-13 09:40:08 +01:00
Aliaksandr Valialkin
ed79f9806a
lib/blockcache: do not cache entries, which were attempted to be accessed 1 or 2 times
Previously entries which were accessed only 1 time weren't cached.
It has been appeared that some rarely executed heavy queries may read indexdb block twice
in a row instead of once. There is no need in caching such a block then.
This change should eliminate cache size spikes for indexdb/dataBlocks when such heavy queries are executed.

Expose -blockcache.missesBeforeCaching command-line flag, which can be used for fine-tuning
the number of cache misses needed before storing the block in the caching.
2023-11-13 09:38:57 +01:00
Aliaksandr Valialkin
f859b33808
docs/Articles.md: sort third-party articles by importance 2023-11-13 09:36:59 +01:00
Aliaksandr Valialkin
bc0b339507
docs/Articles.md: add a link to https://blog.cloudflare.com/introducing-http-traffic-anomalies-notifications/ 2023-11-13 09:36:41 +01:00
Aliaksandr Valialkin
18c5f20e50
docs/Single-server-VictoriaMetrics.md: make High availability section more clear 2023-11-13 09:36:16 +01:00
Zakhar Bessarab
7c7e0a5caa
docs/changelog: document vmbackupmanager bugfix (#5303)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-11-13 09:31:58 +01:00
Artem Navoiev
10435a44be
docs: vmagent change the codeblock languages
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-11-13 09:31:21 +01:00
Github Actions
bd29f2f1e2
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@cc18249 (#5305) 2023-11-13 09:30:45 +01:00
Github Actions
0cb932e11d
Automatic update Grafana datasource docs from VictoriaMetrics/grafana-datasource@52bdb4a (#5304) 2023-11-13 09:30:21 +01:00
Roman Khavronenko
becf7bf8df
app/vmalert: update remote-write process (#5284)
* app/vmalert: update remote-write process

* automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers.
* increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded.
* increment `vmalert_remotewrite_dropped_rows_total` amd `vmalert_remotewrite_dropped_bytes_total` metrics if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/CHANGELOG.md

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
2023-11-13 09:25:29 +01:00
Artem Navoiev
4bba3e39bd
docs: url example change the title h2->h3 h3->h4 for better indexing in search
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-11-13 09:20:55 +01:00
Artem Navoiev
b72dc10bb3
docs: fix formatting in stream aggregation more
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-11-13 09:20:23 +01:00
Artem Navoiev
55df212a76
docs: fix formatting in stream aggregation
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-11-13 09:19:49 +01:00
Github Actions
a01cc78f28
Automatic update operator docs from VictoriaMetrics/operator@b4b79da (#5291) 2023-11-13 09:18:53 +01:00
hagen1778
10da9e6e01
app/vmalert: fix typo in remoteWrite.concurrency description
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c07dc45786)
2023-11-03 22:05:00 +01:00
Yury Molodov
d7c6153f68
vmui: display query error on Explore metrics page (#5272)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5202

(cherry picked from commit f90d2ec843)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-11-03 16:25:21 +01:00
hagen1778
cf3812bd0b
docs: make docs-sync after 323f3720ed
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 054367c421)
2023-11-03 16:24:37 +01:00
Zakhar Bessarab
dea4695df5
app/vmauth: add option to skip TLS verification (#5256)
Add `tls_insecure_skip_verify` option on per-user basis which allows to disable TLS verification for all requests to backend on behalf of this user.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 323f3720ed)
2023-11-03 12:05:26 +01:00
Aliaksandr Valialkin
3d6f4da3b3
docs: update -help output after recent changes to VictoriaMetrics components 2023-11-02 20:27:16 +01:00
Aliaksandr Valialkin
bf01a97f17
docs/CHANGELOG.md: update the description of the optimization for SLO/SLI-like queries according to latest changes
See commits 4497a08e3d and 92826b0b4a
2023-11-02 20:09:22 +01:00
Roman Khavronenko
4e8c762fd9
app/vmalert: add label file pointing to the group's filename to metrics (#5281)
The filename should help identifying alerting rules belonging to specific groups
with identical names but different filenames.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b5254199c6)
2023-11-02 16:02:29 +01:00
Hui Wang
44fcdf0cf0
vmalert: reduce restore query request for each alerting rule (#5265)
reduce the number of queries for restoring alerts state on start-up.
The change should speed up the restore process and reduce pressure on `remoteRead.url`.

(cherry picked from commit 90d45574bf)
2023-11-02 16:02:28 +01:00
Aliaksandr Valialkin
44227c0287
docs/CHANGELOG.md: typo fix: tis -> this 2023-11-02 08:33:48 +01:00
Aliaksandr Valialkin
95f5984aae
docs/Single-server-VictoriaMetrics.md: document why data inside <-storageDataPath>/snapshots directory should be manipulated only via snapshot API 2023-11-02 08:31:08 +01:00
Aliaksandr Valialkin
c04e667f9d
docs/CHANGELOG.md: document v1.93.7 LTS release 2023-11-02 08:21:10 +01:00
Aliaksandr Valialkin
369d37749d
app/vmagent/remotewrite: add -remoteWrite.shardByURL.labels command-line flag
This command-line flag can be used for specifying a list of labels used for sharding
among -remoteWrite.url entries when -remoteWrite.shardByURL command-line flag is set.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942
2023-11-01 23:09:08 +01:00
Alexander Marshalov
ffeec24811
vmauth: add browser authorization request for http requests without… (#5234)
* vmauth: add browser authorization request for http requests without credentials to a route that is not in the `unauthorized_user` section (when `unauthorized_user` is specified).

* add link to issue in CHANGELOG

* Extend vmauth docs

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-11-01 21:00:52 +01:00
Github Actions
86aae00a60
Automatic update operator docs from VictoriaMetrics/operator@49826be (#5270)
Co-authored-by: Alexander Marshalov <_@marshalov.org>
2023-11-01 17:50:50 +01:00
Artem Navoiev
23d09684a6
add Try new Docs button in the current docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-11-01 17:50:27 +01:00
Aliaksandr Valialkin
6a98f9df54
app/vmui: show query execution duration in the header of query input field
This should simplify the process of query optimization
2023-11-01 16:46:42 +01:00
Hui Wang
4fafdda13e
vmalert: support specifying full http url in notifier static_configs target (#5261)
* vmalert: support specifying full http or https urls in notifier static_configs target address
* show right label results in ui
2023-11-01 16:44:54 +01:00
Github Actions
51afab3c7f
Automatic update operator docs from VictoriaMetrics/operator@57c1bf6 (#5266) 2023-11-01 09:58:50 +01:00
Aliaksandr Valialkin
28f0610e14
app/vmui: fix non-working Disable cache checkbox at JSON and Table views 2023-10-31 22:58:15 +01:00
Aliaksandr Valialkin
744f8c3fe7
app/vmselect/promql: add outliers_iqr(q) and outlier_iqr_over_time(m[d]) functions
These functions allow detecting anomalies in series and samples using Interquartile range method.
See Outliers section at https://en.wikipedia.org/wiki/Interquartile_range for more details.
2023-10-31 22:14:14 +01:00
Aliaksandr Valialkin
9661918bb4
app/vmselect/promql: optimize repeated SLI-like instant queries with lookbehind windows >= 1d
Repeated instant queries with long lookbehind windows, which contain one of the following rollup functions,
are optimized via partial result caching:

- sum_over_time()
- count_over_time()
- avg_over_time()
- increase()
- rate()

The basic idea of optimization is to calculate

  rf(m[d] @ t)

as

  rf(m[offset] @ t) + rf(m[d] @ (t-offset)) - rf(m[offset] @ (t-d))

where rf(m[d] @ (t-offset)) is cached query result, which was calculated previously

The offset may be in the range of up to 1 hour.
2023-10-31 20:08:38 +01:00
Aliaksandr Valialkin
5e7d495eb1
lib/httpserver: follow-up for 0638bbe69c
- Replace spaces with underscores in the `reason` label value for the vm_http_request_errors_total metric
  in order be consistent with Prometheus-like naming

- Clarify the description for the change at docs/CHANGELOG.md

Updates https://github.com/victoriaMetrics/victoriaMetrics/issues/4590
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5166
2023-10-31 19:10:48 +01:00
Aliaksandr Valialkin
a7a73b9845
docs/Cluster-VictoriaMetrics.md: clarify the description on why -dosnwampling.period must be set at both vmstorage and vmselect
This is a follow-up for ca7457d906
2023-10-31 19:08:21 +01:00
Aliaksandr Valialkin
ffcd757533
docs/Single-server-VictoriaMetrics.md: cosmetic fixes after 23369321f1 2023-10-31 19:05:45 +01:00
Aliaksandr Valialkin
40a53b516d
docs/CHANGELOG.md: move the description for -http.header.* command-line flags from SECURITY to FEATURE
The SECURITY label should be applied only to changes, which fix security issues.
The change at ad839aa492 adds new command-line flags, which can be used
for improving security in some cases. They do not fix any security issues.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5111
2023-10-31 19:05:01 +01:00
hagen1778
8c3bac8f40
dashboards/cluster: fix description about max threshold for Concurrent selects panel.
Before, it was mistakenly implying that `max` is equal to the double of available CPUs.

Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5214

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-31 19:03:21 +01:00
Roman Khavronenko
9d8f93050c
app/vmselect: expose vm_memory_intensive_queries_total counter metric (#5208)
The new metric gets increased each time `-search.logQueryMemoryUsage` memory limit
is exceeded by a query. This metric should help to identify expensive and heavy queries
without inspecting the logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-31 19:02:22 +01:00
hagen1778
f9c7822588
docs: follow-up for 0638bbe69c
0638bbe69c
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a8051d48c4)
2023-10-31 12:54:57 +01:00
hagen1778
9debdb497c
dashboards/vmalert: add new panel Missed evaluations
The new panel supposed to indicate alerting groups that miss their evaluations.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit aaf9e3d526)
2023-10-31 10:35:57 +01:00
hagen1778
659171686c
deployment/alerts: add TooManyMissedIterations alerting rule
The new rule for vmalert supposed to detect groups that miss their
evaulations due to slow queries.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9866974a53)
2023-10-31 10:35:57 +01:00
hagen1778
497c708aaa
dashboards: fix Errors rate to Alertmanager filter
The panel `Errors rate to Alertmanager` had `group` label filter
applied to the expression, while the metric `vmalert_alerts_send_errors_total`
doesn't have that label. This resulted into always empty results.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8874b525b7)
2023-10-31 10:35:57 +01:00
Roman Khavronenko
afd160f1dc
docs: mention information loss when downsampling gauges (#5204)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 23369321f1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 19:05:39 +01:00
Roman Khavronenko
c6d74108fb
docs: explain motivation behind having -downsampling.period on vmselect (#5205)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 15:29:35 +01:00
Hui Wang
8a786e5df4
vmalert: fix alert firing state in replay mode (#5192)
fix possible missing firing states for alerting rules in replay mode
Before if one firing stage is bigger than single query request range, like rule with a big `for`, alerting rule won't able to be detected as firing.

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit abcb21aa5e)
2023-10-30 13:55:48 +01:00
hagen1778
f0d10e2004
docs/troubleshooting: mention issue with un-ordered labels
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5219#issuecomment-1773441711

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e964df8039)
2023-10-30 13:55:45 +01:00
hagen1778
14b1997659
docs: rm mention of default values for security HTTP headers
The headers, their corresponding flags are mentioned at
https://docs.victoriametrics.com/#security

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a64b37cf24)
2023-10-30 11:46:49 +01:00
Dima Lazerka
ed8fc04898
lib/httpserver: add flags to specify HSTS / Frame-Options / CSP headers for httpserver (#5111)
support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options`
HTTP headers in all VictoriaMetrics components.
The values for headers can be specified by users via the following flags:
`-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`.

Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit ad839aa492)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 11:41:38 +01:00
Roman Khavronenko
733b73ffed
lib/storage: log warning about RO mode only on state change (#5191)
Before, vmstorage would log the same message each second producing excessive
amount of logs.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 29cebd82fb)
2023-10-30 11:29:49 +01:00
Aliaksandr Valialkin
20aeb8b65d
lib/promscrape/discovery/kubernetes: stop all the url watchers, which belong to a particular groupWatcher, at once
Previously url watchers for pod, service and node objects could be mistakenly closed
when service discovery was set up only for endpoints and endpointslice roles,
since watchers for these roles may start start pod, service and node url watchers
with nil apiWatcher passed to groupWatcher.startWatchersForRole().

Now all the url watchers, which belong to a particular groupWatcher, are stopped at once
when this groupWatcher has no apiWatcher subscribers.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5216

The issue has been introduced in v1.93.5 when addressing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4850
2023-10-27 14:34:25 +02:00
Hui Wang
69f4a58f76
do not print redundant error logs when failed to scrape consul or no… (#5239)
* do not print redundant error logs when failed to scrape consul or nomad target
prometheus performs the same because it uses consul lib which just drops the error(1806bcb38c/api/api.go (L1134))
2023-10-27 14:18:47 +02:00
Daria Karavaieva
076a796061
model list - isolation forest (#5235)
* model list - isolation forest

* curse of dimensionality

* isol forest definition change, minor fixes

* blank line fix
2023-10-27 14:08:30 +02:00
Aliaksandr Valialkin
f03e81c693
lib/promauth: follow-up for e16d3f5639
- Make sure that invalid/missing TLS CA file or TLS client certificate files at vmagent startup
  don't prevent from processing the corresponding scrape targets after the file becomes correct,
  without the need to restart vmagent.
  Previously scrape targets with invalid TLS CA file or TLS client certificate files
  were permanently dropped after the first attempt to initialize them, and they didn't
  appear until the next vmagent reload or the next change in other places of the loaded scrape configs.

- Make sure that TLS CA is properly re-loaded from file after it changes without the need to restart vmagent.
  Previously the old TLS CA was used until vmagent restart.

- Properly handle errors during http request creation for the second attempt to send data to remote system
  at vmagent and vmalert. Previously failed request creation could result in nil pointer dereferencing,
  since the returned request is nil on error.

- Add more context to the logged error during AWS sigv4 request signing before sending the data to -remoteWrite.url at vmagent.
  Previously it could miss details on the source of the request.

- Do not create a new HTTP client per second when generating OAuth2 token needed to put in Authorization header
  of every http request issued by vmagent during service discovery or target scraping.
  Re-use the HTTP client instead until the corresponding scrape config changes.

- Cache error at lib/promauth.Config.GetAuthHeader() in the same way as the auth header is cached,
  e.g. the error is cached for a second now. This should reduce load on CPU and OAuth2 server
  when auth header cannot be obtained because of temporary error.

- Share tls.Config.GetClientCertificate function among multiple scrape targets with the same tls_config.
  Cache the loaded certificate and the error for one second. This should significantly reduce CPU load
  when scraping big number of targets with the same tls_config.

- Allow loading TLS certificates from HTTP and HTTPs urls by specifying these urls at `tls_config->cert_file` and `tls_config->key_file`.

- Improve test coverage at lib/promauth

- Skip unreachable or invalid files specified at `scrape_config_files` during vmagent startup, since these files may become valid later.
  Previously vmagent was exitting in this case.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959
2023-10-26 09:55:47 +02:00
Aliaksandr Valialkin
02684a0b29
lib/promauth: properly parse string contents for ca, cert and key fields at tls_config
Previously yaml parser wasn't accepting string values for these fields,
because it was mistakenly expecting a list of uint8 values instead.
2023-10-26 09:54:18 +02:00
Alexander Marshalov
cf42a080af
lib/streamaggr: respect streamAgg.dropInput with empty stream aggr config (#5213)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5207
2023-10-26 09:30:12 +02:00
krakazyabra
c85ff74ac3
docs/case-studies: update Wedos info (#5211) 2023-10-26 09:26:31 +02:00
Aliaksandr Valialkin
94e061087f
docs: use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest instead of https://github.com/VictoriaMetrics/VictoriaMetrics/releases link where needed
The https://github.com/VictoriaMetrics/VictoriaMetrics/releases link may show non-latest
releases at the top, such as LTS releases or VictoriaLogs releases.
So it is better to use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest link,
which always redirect to the latest available release of VictoriaMetrics.
2023-10-26 09:23:17 +02:00
Roman Khavronenko
cd2247b24a
app/vmselect: limit the number of parallel workers by 32 (#5195)
* app/vmselect: limit the number of parallel workers by 32

The change should improve performance and memory usage during query processing
on machines with big number of CPU cores. The number of parallel workers for
query processing is controlled via `-search.maxWorkersPerQuery` command-line flag.
By default, the number of workers is limited by the number of available CPU cores,
but not more than 32. The limit can be increased via `-search.maxWorkersPerQuery`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

- The `-search.maxWorkersPerQuery` command-line flag doesn't limit resource usage,
  so move it from the `resource usage limits` to `troubleshooting` chapter at docs/Single-server-VictoriaMetrics.md

- Make more clear the description for the `-search.maxWorkersPerQuery` command-line flag

- Add the description of `-search.maxWorkersPerQuery` to docs/Cluster-VictoriaMetrics.md

- Limit the maximum value, which can be passed to `-search.maxWorkersPerQuery`, to GOMAXPROCS,
  because bigger values may worsen query performance and increase CPU usage

- Improve the the description of the change at docs/CHANGELOG.md. Mark it as FEATURE instead of BUGFIX,
  since it is closer to a feature than to a bugfix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-26 09:15:27 +02:00
Aliaksandr Valialkin
76606eed22
docs/Articles.md: add an article https://rtfm.co.ua/en/victoriametrics-vmauth-proxy-authentication-and-authorization/ 2023-10-26 09:06:54 +02:00
Aliaksandr Valialkin
52d7464095
docs/FAQ.md: add questions on why VictoriaMetrics doesnt rebalance data and doesnt restore replication factor between vmstorage nodes 2023-10-26 09:06:25 +02:00
Aliaksandr Valialkin
d3cab41426
docs/FAQ.md: refresh the answer to the question about how does VictoriaMetrics compare to competing solutions
- Mention Grafana Mimir
- Fix broken links
2023-10-26 09:06:01 +02:00
Github Actions
a35ae0716e
Automatic update operator docs from VictoriaMetrics/operator@2c826bb (#5188) 2023-10-26 09:05:33 +02:00
hagen1778
afab547821
lib/storage: follow-up after 188cfe3a85
188cfe3a85

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-26 09:04:49 +02:00
Hui Wang
d7dd7614eb
fix inconsistent behaviors with prometheus when scraping (#5153)
* fix inconsistent behaviors with prometheus when scraping

1. address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959. skip job with wrong syntax in `scrape_configs` with error logs instead of exiting;
2. show error messages on vmagent /targets ui if there are wrong auth configs in `scrape_configs`, previously will print error logs and do scrape without auth header;
3. don't send requests if there are wrong auth configs in:
    1. vmagent remoteWrite;
    2. vmalert datasource/remoteRead/remoteWrite/notifier.

* add changelogs

* address review comments

* fix ut
2023-10-26 08:56:54 +02:00
hagen1778
6c63ca18f5
app/vmalert: follow-up after c9375cac5e
c9375cac5e

Descriptions were updated in attempt to make it more clear for readers,
re-phrasing and linking missing docs.

`eval_delay` was added to tests to verify it can be unmarshalled.

`eval_delay` is now applied before timestamp alignment to make it more predictable.
Before, if delay < interval the timestamp won't be aligned.

`eval_delay` and `eval_offset` was added to API output.

`PreviouslySentSeriesToRW` converted to private `previouslySentSeriesToRW`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a216fe6728)
2023-10-25 14:39:49 +02:00
Hui Wang
86d861ec55
vmalert: add -rule.evalDelay flag and eval_delay as group attribute (#5185)
Also mark `-datasource.lookback` as will be deprecated, see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5155.

(cherry picked from commit c9375cac5e)
2023-10-25 14:39:49 +02:00
hagen1778
d349d6a9ce
deployment/alerts: make TooHighMemoryUsage more tolerable to spikes
Using `min_over_time` should reduce the amount of false positives when
component is running in near-the-threshold state. Now it should trigger
only if all collected samples were above the threshold on 10m interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 003ef3a518)
2023-10-25 14:39:48 +02:00
hagen1778
eb00a575c2
docs: mention key concepts as querying example
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5169

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-18 20:12:06 +02:00
hagen1778
46770409d9
dashboards/vmalert: respect job and instance filters in No data errors
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c2d252c045)
2023-10-17 10:26:32 +02:00
hagen1778
d7bae2b78f
dashboards/vmalert: use desc sorting for tooltips on panels
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit edba9f6266)
2023-10-17 10:26:32 +02:00
Aliaksandr Valialkin
767691b1a0
docs/Articles.md: add newly appeared articles about VictoriaMetrics at medium.com
- https://sarthak-acoustic.medium.com/solving-metrics-at-scale-with-victoriametrics-ac9c306826c3
- https://medium.com/@seifeddinerajhi/victoriametrics-a-comprehensive-guide-comparing-it-to-prometheus-and-implementing-kubernetes-03eb8feb0cc2
2023-10-17 00:55:57 +02:00
Aliaksandr Valialkin
ace11318da
docs/CHANGELOG.md: document v1.93.6 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.6
2023-10-17 00:53:32 +02:00
Aliaksandr Valialkin
a6ad9208ef
docs/CHANGELOG.md: document v1.87.10 release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.87.10
2023-10-16 23:25:55 +02:00
Aliaksandr Valialkin
627a4e9330
app/vmselect/promql: add labels_equal(q, "label1", "label2", ...) function
This function returns q series, which have identical values for the listed labels
"label1", "label2", ...

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148
2023-10-16 21:51:13 +02:00
Aliaksandr Valialkin
cdf62fb06b
docs/MetricsQL.md: typo fix after bdb743c88d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071
2023-10-16 21:09:50 +02:00
Aliaksandr Valialkin
b2f9b9d634
app/vmselect/promql: add drop_empty_series() function for dropping empty series before performing additional calculations
This can be useful in the following queries:

   drop_empty_series(temperature <= 30) default 40

This query drops temperature series with all the values bigger than 30 on the selected time range,
while replacing gaps in the remaining series with 40.

The query without drop_empty_series:

  (temperature <= 30) default 40

would leave all the temperature series with all the values bigger than 30 on the selected time range,
and replace all their values with 40. This is not what could be epxected in some cases
like here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071
2023-10-16 20:59:21 +02:00
Aliaksandr Valialkin
b28f904dfa
app/vmagent/remotewrite: move sas var initialization closer to the place where it is used
This makes the code sligthtly easier to understand.

This is a follow-up for 1d3d989be5

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5170
2023-10-16 20:54:35 +02:00
hagen1778
1152c30430
app/vmagent/remotewrite: follow-up after 4f102ff945
4f102ff945
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-16 20:54:35 +02:00
Aliaksandr Valialkin
2a0f77aaf7
lib/promscrape: add a link to https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets in descriptions for -promscrape.cluster.* command-line flags
This should help users figuring out the purpose of -promscrape.cluster.* command-line flags
2023-10-16 14:47:38 +02:00
Artem Navoiev
c8dfead6b1
docs/alert-tools add yaml output type for the correct displaying
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 14:12:34 +02:00
Haleygo
130e0ea5f0
vmalert-tool: implement unittest (#4789)
1. split package rule under /app/vmalert, expose needed objects
2. add vmalert-tool with unittest subcmd

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
2023-10-16 14:12:06 +02:00
Aliaksandr Valialkin
3b79f98459
Revert "docs victorialogs use relative links"
This reverts commit 3d7a77bf82.

Reason for revert: relative links do not work properly at GitHub code
and at GitHub wiki. For example, the following page contains broken links
before reverting this commit:

https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/VictoriaLogs/CHANGELOG.md

It is always better to use absolute links thank relative links, since the page contents
can be copy-n-pasted to other pages, which are located in vastly different directories,
and all the links will remain working.
2023-10-16 14:03:21 +02:00
Aliaksandr Valialkin
64b503ba1c
docs/CaseStudies.md: typ fix: vmgent -> vmagent
This is a follow-up for f5c46b8176
2023-10-16 14:02:58 +02:00
Aliaksandr Valialkin
d6f585ec0f
docs/vmbackup.md: clarify documentation about -deleteAllObjectVersions command-line flag
Updates 2fc7e9f47e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
2023-10-16 14:02:37 +02:00
Alexander Marshalov
1b4e7fcdb3
fixed error when creating a full backup using the -origin flag (#5180)
* fixed error when creating a full backup using the `-origin` flag (#5144)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-16 14:01:16 +02:00
Github Actions
700d83db93
Automatic update operator docs from VictoriaMetrics/operator@79298bf (#5177) 2023-10-16 13:56:49 +02:00
Aliaksandr Valialkin
815e9bf892
app/{vmagent,vminsert}: follow-up for NewRelic data ingestion protocol support
This is a follow-up for f60c08a7bd

Changes:

- Make sure all the urls related to NewRelic protocol start from /newrelic . Previously some urls were started from /api/v1/newrelic

- Remove /api/v1 part from NewRelic urls, since it has no sense

- Remove automatic transformation from CamelCase to snake_case for NewRelic labels and metric names,
  since it may complicate the transition from NewRelic to VictoriaMetrics. Preserve all the metric names and label names,
  so users could query metrics and labels by the same names which are used in NewRelic.
  The automatic transformation from CamelCase to snake_case can be added later as a special action for relabeling rules if needed.

- Properly update per-tenant data ingestion stats at app/vmagent/newrelic/request_handler.go . Previously it was always zero.

- Fix NewRelic urls in vmagent when multitenant data ingestion is enabled. Previously they were mistakenly started from `/`.

- Document NewRelic data ingestion url at docs/Cluster-VictoriaMetrics.md

- Remove superflouos memory allocations at lib/protoparser/newrelic

- Improve tests at lib/protoparser/newrelic/*

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712
2023-10-16 13:55:04 +02:00
Aliaksandr Valialkin
8cdf57bfb1
docs/Single-server-VictoriaMetrics.md: add a link to the original issue, which describes how to run VictoriaMetrics as Windows service
This is a follow-up for cc7d5b7bab

The original issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781 contains
up-to-date information on how to run VictoriaMetrics components as Windows service,
plus it may contain additional information about this case such as https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781#issuecomment-1708092680 ,
so it is better to refer this issue from the docs.
2023-10-16 13:50:47 +02:00
Artem Navoiev
d2bf73bd07
docs fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:44:39 +02:00
Artem Navoiev
146c4fc494
docs fix broken links in operator
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:43:56 +02:00
Artem Navoiev
1cab088282
docs vlogs changelog fix broken link
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:43:34 +02:00
Artem Navoiev
998a4f72e0
docs fix 404 external links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:43:14 +02:00
Artem Navoiev
bb18168b83
docs fix bad links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:42:48 +02:00
Artem Navoiev
13a3daf700
docs victorialogs use relative links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:42:26 +02:00
Artem Navoiev
798e349bc8
docs/vlogs/dataingetion remoev unknown language from codeblock
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:27:18 +02:00
Artem Navoiev
c9a1fc73a6
docs/vmlogs fix aliases
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:26:11 +02:00
hagen1778
5f621b2fa1
docs: fix typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-16 13:25:05 +02:00
hagen1778
709a2bad66
docs: remove extra / in the end of the link
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-16 13:24:30 +02:00
Artem Navoiev
860a8136f1
docs remove (( from the link to fix them
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:20:50 +02:00
Aliaksandr Valialkin
4fd99f50af
docs/Cluster-VictoriaMetrics.md: follow-up after f42c12e69a
Return back accidentally deleted text regarding KISS principles used in VictoriaMetrics development
2023-10-16 12:52:25 +02:00
Aliaksandr Valialkin
dfc67aedcb
lib/license: cleanups and prettifications for log messages and docs related to licensing
- Make more clear the docs at docs/enterprise.md, so readers could figure out faster
  on how to obtain enterprise key and how to pass it to VictoriaMetrics Enterprise components.

- Fix examples at docs/enterprise.md, which were referring to non-existing `-license-file` command-line flag.
  The `-licenseFile` command-line flag must be used instead.

- Improve the description of `-license*` command-line flags, so users could understand
  faster how to use them.

- Improve the warning message, which is emitted when the deprecated -eula command-line flag is passed,
  so the user could figure out how to switch faster to -license* command-line flags.

- Disallow running VictoriaMetrics components with both -license and -licenseFile command-line flags.

- Disallow running VictoriaMetrics components when -licensFile points to an empty file.

- Consistently use the phrase "This flag is available only in Enterprise binaries" across
  all the enterprise-specific command-line flags.

- Remove unneeded level of indirection for `noLicenseMessage` and `expiredMessage` string contants
  in order to improve code readability and maintainability.

- Remove unneded `return` statements after `logger.Fatalf()` calls, since these calls exit the app and never return.

- Make sure that the info log message about successful license verification is emitted
  when the license is verified successfully. Previously the error message could be logged
  when the license payload is invalid or if it misses some required features.
2023-10-16 12:51:37 +02:00
Artem Navoiev
0a0cd285db
docs: mention the free trial for enterprise
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 636d4ea196)
2023-10-16 12:47:19 +02:00
Artem Navoiev
365cbf2047
fix title for managed qs guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 12:43:32 +02:00
Aliaksandr Valialkin
10420603b2
docs/Single-server-VictoriaMetrics.md: remove question mark added by accident in the commit 3d5d62e38a 2023-10-16 12:41:53 +02:00
Aliaksandr Valialkin
b86bec8109
app/vmui: small UX enhancements
- Reduce vertical space usage, so more information is available on the screen without the need to scroll.
- Show information for lines with higher values at the top of the legend under the graph.
  This should simplify graph analysis when it contains many lines.
2023-10-16 12:39:32 +02:00
Aliaksandr Valialkin
ca35e0d66a
deployment/docker: update Go builder from Go1.21.1 to Go1.21.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved
2023-10-16 12:38:14 +02:00
Aliaksandr Valialkin
348c1bcec0
app/{vmselect,vlselect}: enable caching of static contents from /vmui/static/ folder at client side
This should improve repated VMUI page load times on slow networks

See https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/
2023-10-16 12:36:34 +02:00
Zakhar Bessarab
a353fb7e00
docs/vmbackupmaanger: reference permanent objects deletion (#5157)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-16 12:14:43 +02:00
hagen1778
b57e8b1bb9
dasbhoards: fix vminsert/vmstorage/vmselect metrics filtering
Fix vminsert/vmstorage/vmselect metrics filtering when dashboard is used
to display data from many sub-clusters with unique job names.
Before, only one specific job could have been accounted for component-specific panels,
instead of all available jobs for the component.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-16 12:13:01 +02:00
Dmytro Kozlov
d0df4f7c7f
docs/managed-victoriametrics: update documentation (#5135)
Co-authored-by: Ivan Yatskevich <ivan@yatskevich.com>
2023-10-16 02:02:42 +02:00
Roman Khavronenko
8b2c30c51b
lib/vmselect: bump maxSearchQuerySize to 5MB (#5158)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154#issuecomment-1757216612

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5154

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-11 12:25:54 +02:00
Zakhar Bessarab
8b42a1733c
lib/backup: add -deleteAllObjectVersions command-line flag (#5147)
New flag enforces removal of all versions of the object in remote object storage.

See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
- https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages

(cherry picked from commit 2fc7e9f47e)
2023-10-10 14:14:21 +02:00
Yury Molodov
65a9f3da03
vmui: transfer Top Queries time interval #5097 (#5145)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5097
(cherry picked from commit 6dc5306c9b)
2023-10-10 14:14:20 +02:00
Nikolay
4a50e9400c
app/vmselect: reduce lock contention for heavy aggregation requests (#5119)
reduce lock contention for heavy aggregation requests
previously lock contetion may happen on machine with big number of CPU due to enabled string interning. sync.Map was a choke point for all aggregation requests.
Now instead of interning, new string is created. It may increase CPU and memory usage for some cases.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087
2023-10-10 13:44:02 +02:00
Haleygo
b52f1d1f0a
vmalert: add evalAlignment for rule group and fix evalutaion timstamp (#5066)
* vmalert: add `query_time_alignment` for rule group

1. add `eval_alignment` attribute for group which by default is true. So group rule query stamp will be aligned with interval and propagated to ALERT metrics and the messages for alertmanager;
2. deprecate `datasource.queryTimeAlignment` flag.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5049
(cherry picked from commit 2aa0f5fc41)
2023-10-10 12:45:37 +02:00
Dmytro Kozlov
1cc6cd3d4f
app/vmalert: hide sensetive info in the vmalert (#5059)
Strip sensitive information such as auth headers or passwords from datasource, remote-read,
remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via
`-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044
(cherry picked from commit 244c887825)
2023-10-10 12:45:36 +02:00
hagen1778
a906837403
docs: follow-up after 636d4ea196
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit b57756734e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-10 12:45:36 +02:00
Yury Molodov
29487700d9
vmui: enhancement of autocomplete feature (#5051)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4993
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006
(cherry picked from commit c5044cdba9)
2023-10-10 12:45:08 +02:00
Artem Navoiev
d8c8a66c79
docs: mention the free trial for enterprise
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 636d4ea196)
2023-10-10 12:45:08 +02:00
Zakhar Bessarab
a92da26108
docs/enterprise: fix rendering of example alert (#5137)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 7183621d84)
2023-10-10 12:45:07 +02:00
Artem Navoiev
eea2e476cb
docs: fix markdown for k8s managed guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit dc79b25771)
2023-10-10 12:45:07 +02:00
hagen1778
9decf32799
docs: add YT video to "How to reduce expenses on monitoring"
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cb13ce9147)
2023-10-10 12:45:07 +02:00
Artem Navoiev
65b2a0ce60
docs: update the license flags description
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

(cherry picked from commit b3cc22b159)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-10 12:45:06 +02:00
Dmytro Kozlov
a89cf4ce12
app/(vminsert|vmagent): add support for new relic infrastructure agent (#4712)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit f60c08a7bd)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:06:05 +02:00
Zakhar Bessarab
99582f19d2
docs: add example service configuration for windows (#5129)
Use service configuration example from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781#issuecomment-1424671624

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit cc7d5b7bab)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:00:50 +02:00
Denys Holius
f42c12e69a
docs: update social links
(cherry picked from commit 3d5d62e38a)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:00:12 +02:00
Github Actions
c488f225ec
Automatic update operator docs from VictoriaMetrics/operator@26d1b3b (#5130)
(cherry picked from commit b576acff5e)
2023-10-05 14:59:51 +02:00
Aliaksandr Valialkin
c3908350ea
docs/VictoriaLogs/Roadmap.md: add integration with Grafana 2023-10-05 11:41:01 +02:00
Zakhar Bessarab
07586c1d8f
docs/enterprise: add info about license flags (#5085)
* docs/enterprise: add info about license flags

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: add more examples on how to run enterprise components

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: address review feedback

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: specify release version

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: fix image tag in example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: add docker-compose example, fix secret name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-05 11:39:22 +02:00
Artem Navoiev
5c87066f8c
docs: victorialogs change the doc ordering.2
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-05 11:37:01 +02:00
Artem Navoiev
8f96709219
docs: victorialogs change the doc ordering
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-05 11:36:27 +02:00
Github Actions
380a5d2c9f
Automatic update operator docs from VictoriaMetrics/operator@5da4bf6 (#5127) 2023-10-05 11:35:16 +02:00
Aliaksandr Valialkin
7ccefffe27
docs/VictoriaLogs/CHANGELOG.md: add release date for v0.4.1-victorialogs 2023-10-04 17:57:25 +02:00
Github Actions
e20f969ed3
Automatic update operator docs from VictoriaMetrics/operator@9fb79f4 (#5124) 2023-10-04 17:57:25 +02:00
Aliaksandr Valialkin
e275be05b0
docs/vmbackup.md: mention that -filestream.disableFadvise command-line flag can be used for reducing CPU usage on systems with big number of CPU cores
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120

This is a follow-up for 75dd7b30ba
2023-10-04 16:38:53 +02:00
Zakhar Bessarab
dbf0aa4b87
docs/vmbackupmanager: add missing step for restore flow in k8s (#5116)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-04 16:38:53 +02:00
Aliaksandr Valialkin
4e5d3d38f8
lib/filestream: add -filestream.disableFadvise syscall for unconditional disabling of fadvise syscall
This may be needed in rare cases when performing backups on systems with big number of CPU cores
and big value passed to -concurrency command-line flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120
2023-10-04 16:30:19 +02:00
Artem Navoiev
7a5a40e031
docs: fix wieghts for logs folders
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:29:31 +02:00
Artem Navoiev
1a2af79047
docs: remove id clashes for pages, properly cover victorialogs changelog page (#5123)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:28:52 +02:00
Artem Navoiev
12b0e3e5cb
add hugo front matter for operator docs (#5122)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:28:23 +02:00
Aliaksandr Valialkin
65e7a7cabb
deployment/docker: update VictoriaLogs from v0.4.0-victorialogs to v0.4.1-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v041
2023-10-04 16:27:39 +02:00
f41gh7
10569b9d80
cut v0.4.1-victorialogs release 2023-10-04 14:30:22 +02:00
Aliaksandr Valialkin
776174f1cd
docs/Single-server-VictoriaMetrics.md: follow-up for 612ec2df6b
Move the description of JSON line format into a separate chapter

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083
2023-10-03 17:53:15 +02:00
hagen1778
297f63a01e
alerting: account for vmauth component for alerts ServiceDown and TooManyRestarts
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-03 17:52:43 +02:00
Alexander Marshalov
44b51808ec
hide deprecated docs from menu (#5095) 2023-10-03 17:52:15 +02:00
Zakhar Bessarab
9f5e6fe0eb
docs/victorialogs: changelog followup for 650d72f3 (#5114)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-03 17:50:49 +02:00
hagen1778
c0b478d47f
docs: add more details explaining JSON line format
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-03 17:48:33 +02:00
Aliaksandr Valialkin
dc71db9470
deployment: update VictoriaLogs from v0.3.0-victorialogs to v0.4.0-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v040
2023-10-03 02:34:28 +02:00
Aliaksandr Valialkin
ff471d9f2c
docs/VictoriaLogs/CHANGELOG.md: cut v0.4.0-victorialogs 2023-10-03 01:46:42 +02:00
Aliaksandr Valialkin
cf93d27bca
docs/CHANGELOG.md: cut v1.94.0 2023-10-02 22:33:40 +02:00
Aliaksandr Valialkin
f0ff9c6f8c
docs/Articles.md: add a video from GopherCon 2023 for Writing a TSDB from scratch: performance optimization talk 2023-10-02 22:24:48 +02:00
Yury Molodov
f5521ce4d4
vmui: add storage for query history (#5022)
* vmui: add storage for query history

* docs/vmui: add storage for query history
2023-10-02 21:41:32 +02:00
Roman Khavronenko
1f2cb594d9
lib/promscrape: make concurrency control optional (#5073)
* lib/promscrape: make concurrency control optional

Before, `-maxConcurrentInserts` was limiting all calls to `promscrape.Parse`
function: during ingestion and scraping. This behavior is incorrect.
Cmd-line flag `-maxConcurrentInserts` should have effect onl on ingestion.

Since both pipelines use the same `promscrape.Parse` function, we extend it
to make concurrency limiter optional. So caller can decide whether concurrency
should be limited or not.

This commit makes c53b5788b4
obsolete.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "dashboards: move `Concurrent inserts` panel to Troubleshooting section"

This reverts commit c53b5788b4.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 21:34:41 +02:00
Dmytro Kozlov
d8d003af24
app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes (#5106)
* app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes

* app/vmagent: fix CHANGELOG.md description

* wip

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-02 21:18:47 +02:00
Aliaksandr Valialkin
d80ccf52a0
Revert "lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)"
This reverts commit 74301cdbf5.

Reason for revert:

vmagent already provides better approach for detecting slow scrape targets via the following query:

    scrape_duration_seconds / scrape_timeout_seconds > 1

This query depends on automatically generated per-target metrics.
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5074
2023-10-02 21:08:13 +02:00
Aliaksandr Valialkin
5fd79f47f1
app/vmselect/promql: follow-up for 896c85a4a4
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that bitmap_*(X, NaN) returns NaN

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5021
2023-10-02 21:07:46 +02:00
Aliaksandr Valialkin
7bb5f75a2a
lib/logstorage: follow-up for 94627113db
- Move uniqueFields from rows to blockStreamMerger struct.
  This allows localizing all the references to uniqueFields inside blockStreamMerger.mustWriteBlock(),
  which should improve readability and maintainability of the code.

- Remove logging of the event when blocks cannot be merged because they contain more than maxColumnsPerBlock,
  since the provided logging didn't provide the solution for the issue with too many columns.
  I couldn't figure out the proper solution, which could be helpful for end user,
  so decided to remove the logging until we find the solution.

This commit also contains the following additional changes:

- It truncates field names longer than 128 chars during logs ingestion.
  This should prevent from ingesting bogus field names.
  This also should prevent from too big columnsHeader blocks,
  which could negatively affect search query performance,
  since columnsHeader is read on every scan of the corresponding data block.

- It limits the maximum length of const column value to 256.
  Longer values are stored in an ordinary columns.
  This helps limiting the size of columnsHeader blocks
  and improving search query performance by avoiding
  reading too long const columns on every scan of the corresponding data block.

- It deduplicates columns with identical names during data ingestion
  and background merging. Previously it was possible to pass columns with duplicate names
  to block.mustInitFromRows(), and they were stored as is in the block.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4969
2023-10-02 21:06:49 +02:00
Dmytro Kozlov
e7b35455bd
docs: add clarification of the retention filter usage (#5103)
docs: add clarification of the retention filter usage

Updated documentation regarding retention filter usage if duration is set lower than
`-retentionPeriod` flag value.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 21:05:54 +02:00
Roman Khavronenko
0df0b0f29e
lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)
* lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total`

add metric `vm_promscrape_scrapes_skipped_total`to show whether vmagent skips the scrapes.
This could happen if vmagent is overloaded or target is responding too slow for configured `scrape_interval`.

The follow-up commit should add a corresponding alerting rule and panel to vmagent dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/docker: add `TooManyScrapeSkips` alerting rule for vmagent

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: add panels `Scrape duration 0.99 quantile` and `Skipped scrapes` to vmagent dashboard

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 20:38:23 +02:00
Github Actions
025a53dcb6
Automatic update operator docs from VictoriaMetrics/operator@44bdc27 (#5104) 2023-10-02 20:36:21 +02:00
Github Actions
835393a59c
Automatic update operator docs from VictoriaMetrics/operator@c7125bd (#5102) 2023-10-02 20:36:21 +02:00
Aliaksandr Valialkin
c3ece6d38e
docs/VictoriaLogs/CHANGELOG.md: remove duplicate lines about vl_http_request_duration_seconds metric
This is a follow-up after 8a23d08c21

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4945
2023-10-02 20:34:19 +02:00
Aliaksandr Valialkin
4c0402f118
docs/Single-server-VictoriaMetrics.md: refer to active queries and top queries pages at VMUI instead of refering to the corresponding HTTP endpoints
"Active queries" and "Top queries" pages at VMUI are user-friendly than the corresponding HTTP endpoints
2023-10-02 20:33:10 +02:00
Aliaksandr Valialkin
5f1492d978
docs/vmalert.md: refer to -evaluationInterval command-line flag instead of evaluation_interval option, which isnt supported by vmalert
This is follow-up for 5c42c1218a
2023-10-02 20:32:02 +02:00
Aliaksandr Valialkin
ac418281da
docs/Troubleshooting.md: describe how to optimize SLI/SLO queries with long lookbehind windows 2023-10-02 20:29:37 +02:00
Aliaksandr Valialkin
b5f9a6a5c6
docs/CHANGELOG.md: follow-up for f0e33700fc
Mention that the statistic inaccuracy is related to cardinality explorer
2023-10-02 20:29:07 +02:00
Dmytro Kozlov
90b189dab8
app/vmselect: fix bitmap_*() functions behavior (#5021)
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-02 20:13:27 +02:00
Aliaksandr Valialkin
6e613cb8e8
docs/Cluster-VictoriaMetrics.md: increase the minimum supported version of Go builder from 1.18 to 1.20
See the related commit 3da493ff62
2023-10-02 19:33:41 +02:00
Zakhar Bessarab
876bce5a57
lib/logstorage: prevent from panic during background merge (#4969)
* lib/logstorage: prevent from panic during background merge

Fixes panic during background merge when resulting block would contain more columns than maxColumnsPerBlock.
Buffered data will be flushed and replaced by the next block.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/logstorage: clarify field description and comment

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-02 19:29:31 +02:00
Zakhar Bessarab
dfdada055c
lib/logstorage: switch to read-only mode when running out of disk space (#4945)
* lib/logstorage: switch to read-only mode when running out of disk space

Added support of `--storage.minFreeDiskSpaceBytes` command-line flag to allow graceful handling of running out of disk space at `--storageDataPath`.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/logstorage: fix error handling logic during merge

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/logstorage: fix log level

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-02 17:09:57 +02:00
Dmytro Kozlov
10371eac60
vmui: update information about tsdb usage in cluster version (#5004)
* vmui: update information about tsdb usage in cluster version

* vmui: cleanup

* vmui: add CHANGELOG.md

* vmui: cleanup

* vmui: update logic, move information to the visible place

* app/vmui: remove values fetch, update documentation for cardinality explorer

* app/vmui: update CHANGELOG.md
2023-10-01 21:30:44 +02:00
Zakhar Bessarab
ff88e53e01
doc: address review feedback
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-01 21:27:49 +02:00
Zakhar Bessarab
9f6704d6cc
doc: mention InfluxDB v2 HTTP API support
Address: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5076
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-01 21:26:44 +02:00
Github Actions
18cc977956
Automatic update operator docs from VictoriaMetrics/operator@958ce2b (#5070) 2023-10-01 21:25:54 +02:00
hagen1778
d0641d6ea2
dashboards: move Concurrent inserts panel to Troubleshooting section
Moved because this panel is related to both: scraped and ingested data.
Before, it could have give a misleading impression that it is related to ingested metrics only.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-01 21:25:25 +02:00
Roman Khavronenko
d39c8525e2
Docker add vmauth (#5057)
* docker-compose: add vmauth to cluster env

vmauth acts as a balancer and used as an example of how to interconnect
VM components via vmauth.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docker-compose: add vmauth to cluster env

vmauth acts as a balancer and used as an example of how to interconnect
VM components via vmauth.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-01 21:24:01 +02:00
Aliaksandr Valialkin
15645c8a94
app/vmagent: follow-up for cfef814750
- Properly handle /insert/multitenant/api/put url for opentsdb handler at vmagent
- Document that the bug has been introduced in v1.93.2 at docs/CHANGELOG.md
- Add a link to multitenant url docs in bugfix description

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5061
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910
2023-10-01 20:52:29 +02:00
Alexander Marshalov
cfef814750
fixed ingestion via multitenant url for opentsdbhttp (#5061) (#5063) 2023-09-26 10:47:49 +02:00
Aliaksandr Valialkin
e341128096
docs/vmagent.md: make VictoriaMetrics remove_write protocol more visible by mentioning it at the top of the page 2023-09-25 17:42:25 +02:00
Aliaksandr Valialkin
308134970f
docs: run make docs-sync after 8e722e10ee 2023-09-25 17:35:34 +02:00
Aliaksandr Valialkin
f6b35a715d
docs/CaseStudies.md: add Criteo case study
This is a follow-up for bdbe616408

See https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61
2023-09-25 17:34:47 +02:00
Aliaksandr Valialkin
a421db5977
lib/storage: stop exposing vm_merge_need_free_disk_space metric
This metric confuses users and has no any useful information.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686#issuecomment-1733844128
2023-09-25 17:00:14 +02:00
Aliaksandr Valialkin
b43ff80d21
app/vmselect/promql: do not sort q1 or q2 results
This makes sure that `q2` series are returned after `q1` series in the same way as Prometheus does

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763
2023-09-25 16:15:02 +02:00
Aliaksandr Valialkin
c954019e43
app/vmselect/promql: completely substitute median_over_time() WITH template with regular median_over_time() rollup function
This is a follow-up for 34d7a670d0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034
2023-09-25 15:31:25 +02:00