Commit graph

7900 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8669584e9f
lib/{storage,mergeset}: convert beffered items into searchable in-memory parts exactly once per the given flush interval
Previously the interval between item addition and its conversion to searchable in-memory part
could vary significantly because of too coarse per-second precision. Switch from fasttime.UnixTimestamp()
to time.Now().UnixMilli() for millisecond precision. It is OK to use time.Now() for tracking
the time when buffered items must be converted to searchable in-memory parts, since time.Now()
calls aren't located in hot paths.

Increase the flush interval for converting buffered samples to searchable in-memory parts
from one second to two seconds. This should reduce the number of blocks, which are needed
to be processed during high-frequency alerting queries. This, in turn, should reduce CPU usage.

While at it, hardcode the maximum size of rawRows shard to 8Mb, since this size gives the optimal
data ingestion pefromance according to load tests. This reduces memory usage and CPU usage on systems
with big amounts of RAM under high data ingestion rate.
2024-02-23 01:11:57 +02:00
Aliaksandr Valialkin
5f1fa8e7f7
lib/storage: avoid superflouos copy of block header data 2024-02-23 01:11:31 +02:00
Fred Navruzov
c9d8627676
- v1.11 doc updates (#5852)
- fix dead links
2024-02-23 01:11:05 +02:00
Dan Dascalescu
6b5e8e7089
docs: CSV RFC3339 format uses server timezone (#5839) 2024-02-23 01:07:55 +02:00
Aliaksandr Valialkin
a982ab6bfb
app/vmstorage: expose vm_snapshots metric, which shows the current number of snapshots
While at it, refresh docs about snapshots - https://docs.victoriametrics.com/#how-to-work-with-snapshots
2024-02-23 01:07:04 +02:00
Aliaksandr Valialkin
3f9022bc08
lib/storage: do not pool rawRowsBlock when flushing rawRows to in-memory blocks
The pooled rawRowsBlock objects occupies big amounts of memory between flushes,
and the flushes are relatively rare. So it is better to don't use the pool
and to allocate rawRow blocks on demand. This should reduce the average
memory usage between flushes.
2024-02-23 01:06:28 +02:00
Aliaksandr Valialkin
bf07e2ac87
lib/storage: do not keep rawRows buffer across flush() calls
The buffer can be quite big under high ingestion rate (e.g. more than 100MB).
This leads to increased memory usage between buffer flushes.
So it is better to re-create the buffer on every flush in order to reduce memory usage
between buffer flushes.
2024-02-23 01:06:09 +02:00
Aliaksandr Valialkin
843f3ec94e
docs/MetricsQL.md: improve text formatting for better readability 2024-02-23 01:05:49 +02:00
Aliaksandr Valialkin
477fdc21aa
app/vmselect/promql: add count_values_over_time() MetricsQL function
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847
2024-02-23 01:05:31 +02:00
Aliaksandr Valialkin
65fb54ab8f
app/vmselect/promql: move needSilenceIntervalForRollupFunc from eval.go to rollup.go
This should improve maintainability of the code related to rollup functions,
since it is located in rollup.go

While at it, properly return empty results from holt_winters(), rate_over_sum(),
sum2_over_time(), geomean_over_time() and distinct_over_time() when there are no real samples
on the selected lookbehind window. Previously the previous sample value was mistakenly
returned from these functions.
2024-02-23 01:05:11 +02:00
Alexander Marshalov
8322425364
[lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5814)
* [lib/promutils, lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801)

* fixed tests

* fixed test

* Revert "fixed test"

This reverts commit 8a29764806.

* Revert "fixed tests"

This reverts commit 9ce13d1042.

* Revert "[lib/promutils, lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801)"

This reverts commit a7a04bd4

* [lib/httputils] fixed floating-point error when parsing time in RFC3339 format (#5801)

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-02-23 00:58:26 +02:00
Artem Navoiev
b46014c8ab
docs: change header from h1 to h2 1.97.2. The markdown requires the proper structure in hierachy of title so h1 can not be a child of h1,h2... only be a separate item, in our structure title is the parent h1
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-02-23 00:42:15 +02:00
Github Actions
e2bbab072a
Automatic update operator docs from VictoriaMetrics/operator@d88157b (#5845) 2024-02-23 00:41:10 +02:00
Github Actions
f08f33cd5a
Automatic update operator docs from VictoriaMetrics/operator@c393852 (#5844) 2024-02-23 00:40:38 +02:00
Github Actions
786679135b
Automatic update operator docs from VictoriaMetrics/operator@4791fd1 (#5843) 2024-02-23 00:40:04 +02:00
hagen1778
10fbda60c8
deployment/docker: add comments to components in docker-compose manifests
This should help readers to understand interconnectivity between components.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-23 00:39:05 +02:00
Aliaksandr Valialkin
af366a1f97
README.md: sync with docs/Cluster-VictoriaMetrics.md after e0569a355b 2024-02-23 00:22:54 +02:00
Anton L
8b7ff0f66e
#5833 Fix Deadlock when using shardByURL of VMAgent (#5834) 2024-02-22 11:54:53 +02:00
Dan Dascalescu
0c7eda7c88
app/vmselect: simplify wording for too many samples error (#5827)
(cherry picked from commit 17cf031fa1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-20 16:29:11 +01:00
Roman Khavronenko
2e172b9361
vmctl : Provide TLS config options for Open TSDB datasource #5797 (#5832)
Originally implemented here https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5797

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: khushijain21 <khushij393@gmail.com>
(cherry picked from commit bb1279bfc4)
2024-02-20 16:27:52 +01:00
Daria Karavaieva
550b589790
Vmanomaly Quickstart Fix absolute links (#5831)
* links fix

* typo fix

(cherry picked from commit 4034d081f4)
2024-02-20 16:27:52 +01:00
Daria Karavaieva
a68518c0e0
Vmanomaly QuickStart (#5800)
* first edit

* typo 1

* typo 2

* fixes 3

* fixes 4

* fixes 5

* fixes, cross links

* v1.10 config

* models why, self-monitoring fix

* config next steps

* fixes

* minor fix

(cherry picked from commit b2baf7d472)
2024-02-20 16:27:52 +01:00
hagen1778
c065287757
docs: move recent changes to Tip
These changes were mistakenly put to existing release

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit dc25c30fdc)
2024-02-20 16:27:51 +01:00
igorbernstein
b485e40823
deployment/docker: clean up loading of victoriametrics-datasource (#5793)
Currently the docker-compose examples for loading `victoriametrics-datasource` uses 2 environment variables:
-  `GF_ALLOW_LOADING_UNSIGNED_PLUGINS`
- `GF_DEFAULT_APP_MODE`

I believe both of the env vars are trying to achieve the same thing. `GF_DEFAULT_APP_MODE` disables code signing for all plugins and `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` intends to disable code signing for just `victoriametrics-datasource`.
Keeping the scope narrowed to just `victoriametrics-datasource` would be preferable in this case.

Unfortunately `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` is misspelled. According to [grafana docs](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables), the format is supposed to be `GF_<SectionName>_<KeyName>`. In other words the current env var is missing the section name.

This PR proposes to:
1. fix the typo
2. remove the global disablement of code signing

Alternatively, if you prefer to keep codesigning disabled globally, please remove `GF_ALLOW_LOADING_UNSIGNED_PLUGINS` env var as it confuses things

(cherry picked from commit cc5a274e4d)
2024-02-20 16:27:51 +01:00
hagen1778
4474c23aed
app/vmalert: consistently sort groups by name and filename on /groups page
This should prevent non-deterministic sorting for groups with identical names.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit e2dad3a2ac)
2024-02-20 13:51:31 +01:00
hagen1778
6c63fd831d
app/vmalert: follow-up after b60dcbe11f
* support case-insensitive search
* reflect search condition in URL, so link can be sharable
* support filtering on /alerts page
* fix collapseAll/expandAll logic to respect only shown entries
* add changelog

b60dcbe11f
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 11b03d9fc8)
2024-02-20 13:35:02 +01:00
Victor Amorim dos Santos
f79abd54b0
vmalert: add filter by group or rule name to UI (#5791)
Co-authored-by: Yury Molodov <yurymolodov@gmail.com>
(cherry picked from commit b60dcbe11f)
2024-02-20 13:35:02 +01:00
Yury Molodov
7d15c5abeb
vmui: update package-lock.json (#5822)
This should address detected security vulnerabilities

(cherry picked from commit 524c0a2e07)
2024-02-20 13:35:02 +01:00
Artem Navoiev
e0569a355b
docs: mention slack inviter and slack channel (#5817)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 5b652bccad)
2024-02-20 13:35:01 +01:00
Aliaksandr Valialkin
1de59610c8
docs/VictoriaLogs/CHANGELOG.md: document cafd6f08b3
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5400
2024-02-18 23:18:12 +02:00
Aliaksandr Valialkin
c4b16e3749
docs/VictoriaLogs/CHANGELOG.md: document 333bda8702
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5447
2024-02-18 23:13:30 +02:00
Aliaksandr Valialkin
b58c429044
app/vlselect: follow-up for 451d2abf50
- Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb.
  See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request.
- Document the change at docs/VictoriaLogs/CHANGELOG.md
- Document the `limit` query arg at docs/VictoriaLogs/querying/README.md
- Make the change less intrusive.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778
2024-02-18 23:06:08 +02:00
Dmytro Kozlov
2d674f98d4
Enable the limit query param for the /select/logsql/query (#5778)
* app/vlselect: add limit for logs query

* app/vlselect: CHANGELOG.md

* app/vlselect: stop search process if limit is reached, update logic, remove default limit

* app/vlselect: fix tests

* app/vlselect: fix filter tests

* app/vlselect: fix tests
2024-02-18 22:59:16 +02:00
Aliaksandr Valialkin
82e38e1627
lib/promscrape: add support for enable_compression option in the same way as Prometheus does
Updates https://github.com/prometheus/prometheus/pull/13166
Updates https://github.com/prometheus/prometheus/issues/12319

Do not document enable_compression option at docs/sd_configs.md, since vmagent already supports
more clear disable_compression option - see https://docs.victoriametrics.com/vmagent/#scrape_config-enhancements
2024-02-18 19:42:09 +02:00
Aliaksandr Valialkin
f0db7d474f
lib/promscrape/discovery/kuma: add support for client_id option
See https://github.com/prometheus/prometheus/pull/13278
2024-02-18 19:19:55 +02:00
Aliaksandr Valialkin
690f6c833c
vendor: update github.com/VictoriaMetrics/metricsql from v0.72.1 to v0.73.0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5383
2024-02-18 18:43:41 +02:00
Aliaksandr Valialkin
c1a3876e95
docs/MetricsQL.md: properly document how MetricsQL selects the lookbehind window
- rate(m) isn't equivalent to rate(m[1i]) when step is smaller than the interval between samples.
- default_rollup(m) isn't equivalent to default_rollup(m[1i]) when step is smaller than the interval between samples.

These changes have been made in v1.85.3 as a part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483 .
See the corresponding commit - 9fa3f1dc57 .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5816
2024-02-18 13:48:45 +02:00
Aliaksandr Valialkin
689c0b9817
vendor: update github.com/VictoriaMetrics/metrics from v1.31.0 to v1.32.0 2024-02-18 12:42:29 +02:00
Aliaksandr Valialkin
8053ff9135
docs/LTS-releases.md: cosmetic fixes 2024-02-17 18:09:43 +02:00
Aliaksandr Valialkin
55bba932d4
docs/CHANGELOG.md: document f8207e33a2 2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin
cf96c3416d
docs/CHANGELOG.md: add missing for in the description of the TLS configuration features for vmctl
This is a follow-up for 6a07cb1bdb and f973711e56
2024-02-17 17:55:01 +02:00
Aliaksandr Valialkin
efbead081c
docs/Single-server-VictoriaMetrics.md: enumerate all the VictoriaMetrics components 2024-02-17 17:54:34 +02:00
Aliaksandr Valialkin
9f7c9e9033
docs/LTS-releases.md: add a dedicated page describing LTS lines of releases for VictoriaMetrics 2024-02-17 17:53:56 +02:00
Alexander Marshalov
89e9bfc276
lib/httputils: fixed error message for getting zero duration (#5795) (#5812)
(cherry picked from commit f8207e33a2)
2024-02-16 15:31:59 +01:00
hagen1778
e53f53aaf5
app/vmctl: follow-up after 0c293a66ec
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f973711e56)
2024-02-16 15:31:58 +01:00
Khushi Jain
9ce7f21a63
app/vmctl : support TLS config options for remote read mode (#5798)
(cherry picked from commit 0c293a66ec)
2024-02-16 15:31:58 +01:00
hagen1778
025e52adad
app/vmctl: follow-up after 7cd1b7d047
* cleanup code
* update docs

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6a07cb1bdb)
2024-02-16 15:31:58 +01:00
Khushi Jain
02c8b5015c
app/vmctl : support TLS config options for InfluxDB datasource (#5783)
* vmctl: TLS flags for influx DB

* added httputils function

* Add changelog and doc

---------

Co-authored-by: Khushi Jain <khushi.jain@nokia.com>
(cherry picked from commit 7cd1b7d047)
2024-02-16 15:31:57 +01:00
hagen1778
f4578826b3
dashboards: add legend details to network panels in cluster dash
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ecccd2a1cc)
2024-02-16 15:31:57 +01:00
Fred Navruzov
d8de87aeb0
docs: vmanomaly - updates of v1.10.0 and model type section (#5813)
* - apply v1.10 changes
- chapter on model types (uni/multivariate and rolling)

* - update self-monitoring labels description
- fix typos

* fix duplicated text and link rendering

(cherry picked from commit 172e196ac9)
2024-02-16 15:31:56 +01:00