Commit graph

9052 commits

Author SHA1 Message Date
Dmytro Kozlov
91b28d0527
deployment/docker: update grafana datasources to the latest version (#7083)
### Describe Your Changes

Updated grafana plugins to the latest releases 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-24 11:51:18 +02:00
Github Actions
524579d9bd
Automatic update operator docs from VictoriaMetrics/operator@75bc1b4 (#7080)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-24 11:50:50 +02:00
hagen1778
a5c002edef
deployment/alerts: fix copy&paste typo in TooHighGoroutineSchedulingLatency
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-24 11:48:19 +02:00
Roman Khavronenko
4d0b41e63b
deployment: add panel and alerts for displying go scheduler latency (#7078)
The panel and alerting rule should help to understand whether VM
component doesn't have enough CPU resources or gets throttled. The alert
is applicable for all VM components.
The panel was added to vmalert, vmagent, vmsingle, vm clusert and
victorialogs dashes.

-------------------

This alerting rule should have help us identify resource shortage for
sandbox vmagent - see [this
link](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.range_input=23d13h25m25s424ms&g0.end_input=2024-09-23T14%3A11%3A00&g0.relative_time=none&g0.tab=0&g0.expr=histogram_quantile%280.99%2C+sum%28rate%28go_sched_latencies_seconds_bucket%7Bjob%3D%22vmagent-monitoring-vmagent%22%7D%5B5m%5D%29%29+by+%28le%2C+job%2C+instance%29%29+%3E+0.1)
for example. We weren't aware of resource shortage, because VM metrics
assumed this vmagent had 1vCPU while in fact its limit was 0.2vCPU.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-23 16:54:42 +02:00
Aliaksandr Valialkin
109772bdc4
lib/cgroup: round GOMAXPROCS to the lower integer value of cpuQuota
Rounding GOMAXPROCS to the upper interger value of cpuQuota increases chances of CPU starvation,
non-optimimal goroutine scheduling and additional CPU overhead related to context switching.

So it is better to round GOMAXPROCS to the lower integer value of cpuQuota.
2024-09-23 16:09:12 +02:00
Aliaksandr Valialkin
3964889705
app/vmselect/promql: consistently replace NaN data points with non-NaN values for range_first and range_last functions
It is expected that range_first and range_last functions return non-nan const value across all the points
if the original series contains at least a single non-NaN value. Previously this rule was violated for NaN data points
in the original series. This could confuse users.

While at it, add tests for series with NaN values across all the range_* and running_* functions, in order to maintain
consistent handling of NaN values across these functions.
2024-09-23 14:59:29 +02:00
hagen1778
3ed172eeeb
docs: add note about testing new releases on testing env
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-23 11:58:43 +02:00
Zakhar Bessarab
78ecfede06
license: add ability to reload keys (#775)
* lib/license: add support of license key hot-reload

* docs: add info about license key hot reload

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-09-23 09:04:17 +02:00
Aliaksandr Valialkin
57183c9b61
docs/changelog/CHANGELOG.md: moved the description of the fix for proper usage of -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval from FEATURE to BUGFIX section
The previous behaviour was incorrect, since it is unexpected that the -streamAggr.dedupInterval
and -remoteWrite.streamAggr.dedupInterval is applied to processed samples only if -streamAggr.config isn't set.

This is a follow-up for d523015f27
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6711
2024-09-23 08:56:33 +02:00
Github Actions
4dc85613a2
Automatic update helm docs from VictoriaMetrics/helm-charts@428cb36 (#7072)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-23 13:17:34 +08:00
Aliaksandr Valialkin
0ada781cf2
docs/changelog/CHANGELOG.md: document bugfix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7009
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7064
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7009

This is a follow-up for 55febc0920
2024-09-22 21:57:48 +02:00
Github Actions
8ed9978591
Automatic update operator docs from VictoriaMetrics/operator@d661554 (#7069)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-22 18:39:52 +02:00
Artem Fetishev
55febc0920
lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064)
### Describe Your Changes

Currently it the metricID list is empty it won't be mashalled and as the
result won't be put into the tagFiltersToMetricIDsCache which causes the
cache misses for the corresponding tagFilters. In some setups this
causes severe search speed detradation (see #7009).

The empty metric IDs was covered before but then was accidentally
removed in 6c21439.

This PR restores the coverage of this case.

A new unit test can be used as a proof that empty metricID lists are not
added to the cache (just remove the fix in index_db.go and run the test
to see the result)

Also a benchmark has been added to see the implications of the
compression.

```
user@laptop:~/p/github.com/rtm0/VictoriaMetrics/01/src$ go test ./lib/storage/ -run=NONE -bench BenchmarkMarshalUnmarshalMetricIDs --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-0-12             3237240               363.5 ns/op               0 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1-12             2831049               451.8 ns/op               0.4706 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10-12            1152764              1009 ns/op                 1.667 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100-12            297055              3998 ns/op                 5.755 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000-12            31172             34566 ns/op                 8.484 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000-12            4900            289659 ns/op                 9.416 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-100000-12            447           2341173 ns/op                 9.456 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-1000000-12            42          24926928 ns/op                 9.468 compression-rate
BenchmarkMarshalUnmarshalMetricIDs/numMetricIDs-10000000-12            5         204098872 ns/op                 9.467 compression-rate
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  15.018s
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-09-20 17:21:53 +02:00
Artem Navoiev
a2ecba4154
docs: cloud siplify menu items names for notification, user managemenr and notifs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 16:16:25 +02:00
Github Actions
8645438a79
Automatic update helm docs from VictoriaMetrics/helm-charts@f0e007f (#7059)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-09-20 16:11:23 +02:00
Artem Navoiev
9e9583357d
docs add root menu to _index.md as homepage
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 16:06:15 +02:00
Github Actions
75a29655f5
Automatic update operator docs from VictoriaMetrics/operator@fdd3f9f (#7065)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: tenmozes <1381404+tenmozes@users.noreply.github.com>
2024-09-20 16:03:52 +02:00
Andrii Chubatiuk
5708a85499
docs: updated root menu items (#7061)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 06:14:29 -07:00
Aliaksandr Valialkin
787b9cd9a0
lib/storage: improve performance for indexSearch.containsTimeRange()
The indexSearch.containsTimeRange() function is called for the current indexDB and the previous indexDB
every time when searching for metricIDs by label filters. This function consumes a lot of additional CPU time
for cases when queries with lightweight label filters are sent to VictoriaMetrics at high rate (e.g. thousands of RPS),
like in the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7009 .

Optimize indexSearch.containsTimeRange() function in the following ways:

- Unconditionally return true if this function is called for the current indexDB, since there are very high
  chances that the current indexDB contains the data with timestamps in the requested time range.

- Cache the minimum timestamp, which is missing in the indexed data for the previous indexDB.
  This is safe to do, since the previous indexDB is readonly.
  This optimization eliminates potentially slow lookup in the previous indexDB for typical
  use cases when the requested time range is close to the current time.
2024-09-20 13:07:20 +02:00
Aliaksandr Valialkin
6f61e9d49d
lib/storage: simplify indexDB.doExtDB() usage by removing the returned value
Previously indexDB.doExtDB() was returning boolean value, which was indicating whether f callback was called.
There is no need in returning this boolean value, since the f callback can determine on itself whether it was called.

This simplifies the code a bit.

While at it, document indexDB.doExtDB().
2024-09-20 11:59:57 +02:00
Roman Khavronenko
218c533874
lib/storage: follow-up after d8f8822fa5 (#7036)
Make function name and comments more clear.

d8f8822fa5

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-09-20 11:50:47 +02:00
Artem Navoiev
7596e239eb
docs: set operator menu item weight to 30
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 11:22:03 +02:00
Artem Navoiev
36d58588a7
docs: clarify operator name.2
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 11:20:16 +02:00
Artem Navoiev
cd384c7547
docs: clarify operator name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-20 11:11:38 +02:00
Hui Wang
d6d02d7aeb
vmalert: fix variable $activeAt value when templating rule annotation in replay mode
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-09-20 11:07:40 +02:00
hagen1778
6167bccc5a
docs: fix more typos in the changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 10:54:43 +02:00
hagen1778
59281d5358
docs: rm update node about loggerMaxArgLen as it doesn't have incompatibility effect
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 10:42:07 +02:00
hagen1778
6726a5aaed
docs: fix typo in link in change line about NaN
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 10:39:13 +02:00
Thomas Danielsson
258201af04
docs: fix typo in the changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-20 08:46:29 +02:00
Aliaksandr Valialkin
a3d8077959
lib/logstorage: make sure that getCommonTokens returns common tokens in the original order of tokens inside tokenSets arg
This fixes flaky test TestGetCommonTokensForOrFilters:

    filter_or_test.go:143: unexpected tokens for field "_msg"; got ["foo" "bar"]; want ["bar" "foo"]
2024-09-19 15:59:48 +02:00
hagen1778
c00b64726c
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Executed after https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6972
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6900

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-19 15:39:40 +02:00
f41gh7
61721303fd
docs/changelog: mention vmagent kafka consumer bugfix
Changes were made to the enteprise repository
2024-09-19 15:35:48 +02:00
Yury Molodov
4e976f66f3
vmui: optimize public directory by cleaning up files (#6972)
### Describe Your Changes

### Pull Request Description:

1. **HTML File Structure Optimization**: Adjusted the location of HTML
files for different builds to prevent redundant files in the final
output. See issue #6900
2. **Metadata Fixes**: Corrected metadata in HTML files for each build
configuration.
3. **Favicon Update**: Replaced PNG favicon (`14 KB` and `1.58 KB`) with
SVG (`1.35 KB`).
4. **Social Media Optimization**: Optimized the social preview image,
reducing its size by `60.2 KB`.
5. **Git Ignore Update**: Added `public/index.html` to `.gitignore` as
it is dynamically generated during the build process.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-19 14:37:16 +02:00
Thomas Danielsson
69b976f08e
docs: typo filebeat -> fluent-bit (#7000)
### Describe Your Changes

Fix a typo; `filebeat.yml` -> `fluent-bit.conf`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Thomas Danielsson <thomas@elajt.se>
2024-09-19 14:33:12 +02:00
Yury Molodov
b0bdb92729
vmui: change the query_range request method from GET to POST (#7039)
### Describe Your Changes

change the `/query_range` and `/query` requests method from `GET` to
`POST`. See #6288.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-09-19 14:30:54 +02:00
Roman Khavronenko
e115b85770
lib/logger: increase default value of -loggerMaxArgLen cmd-line fla… (#7008)
…g from 1e3 to 5e3

This should improve visibility on errors produced by very long queries.

The change is classified as BUG in order to port it to LTS releases.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Mathias Palmersheim <mathias@victoriametrics.com>
2024-09-19 14:29:18 +02:00
Yury Molodov
7491f49e9e
vmui: update dependencies in package.json to latest versions (#7007)
Update dependencies in `package.json` to latest versions
2024-09-19 11:43:52 +02:00
Yury Molodov
bc9cb69170
vmui/logs: add auto refresh (#7038)
### Describe Your Changes

Add auto refresh 
#7017


![image](https://github.com/user-attachments/assets/20ed1102-d5e4-4d3f-9c24-7d298d93400a)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-09-19 11:11:16 +02:00
Thomas Danielsson
75cc7922c3
typo: added missing '{' (#7044)
### Describe Your Changes

Added missing `{` in vmalert rule.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-19 00:11:30 -07:00
Aliaksandr Valialkin
e86891b010
app/vlselect/logsql: call Query.Optimize() on the cloned query in order to replace * filter with filterNoop inside getLastNQueryResults()
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785
2024-09-18 18:24:17 +02:00
Aliaksandr Valialkin
b82e2cabc5
app/vmselect/promql: properly calculate c1 and c2 and c1 or c2 by upgrading github.com/VictoriaMetrics/metricsql to v0.79.0
The fix is in the https://github.com/VictoriaMetrics/metricsql/pull/34
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6637
2024-09-18 17:38:19 +02:00
Github Actions
8bc30b68cd
Automatic update operator docs from VictoriaMetrics/operator@fe90f57 (#7026)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-17 19:30:09 +02:00
Dmytro Kozlov
4c228f1e18
docs/victoriametrics-cloud: updated pictures and descriptions with new functionality (#7014)
Improved VictoriaMetrics documentation for cloud

Related issue: https://github.com/VictoriaMetrics/cloud/issues/2143

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-16 23:33:40 -07:00
Nikolay
d8f8822fa5
lib/storage: consistently check for missing metricID index records (#6967)
* Previously, only metricID->metricName missing index records were
tracked with deadline But it was possible a case for missing
metricID->TSID index records. IndexDB metrics fix exposed misleading
metric for such missing records.

* This commit adds check for metricID->TSID missing index records. And
delete missing metricID entry if it hit 60 second deadline.

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6931

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-16 10:05:08 +02:00
Nikolay
264c2ec6bd
lib/fs: properly call windows APIs (#6998)
Previously we manually imported system windows DDLs
and made direct syscall.

 But golang exposes syscall wrappers with sys/windows package.
It seems, that direct syscall was broken at 1.23 golang release. It was
`GetDiskFreeSpace` syscall in our case.

This commit replaces all manual syscalls with wrappers

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6973

Related golang issue:
https://github.com/golang/go/issues/69029

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-13 12:22:25 +02:00
Dima Lazerka
8207879fa3
docs: fixes misspelled typos
Also tried to make it catch "Authorisation" in the future, fixed a lot
of other misspells along the way, but didn't make it catch
"Authorisation" anyway.

- Fix misspelled "Authorization" header name
- Fix misspelled "organization"
- Fix more misspells
2024-09-13 12:14:24 +02:00
Aliaksandr Valialkin
e92f347336
docs/VictoriaLogs/querying/README.md: typo fixes: use field_values instead field_names where needed 2024-09-11 09:39:22 +02:00
Alexander Marshalov
c0272463d9
[vmcloud]: Cloud tiers assumptions docs (#6948)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-10 12:41:25 +02:00
Artem Navoiev
277fed9990
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 14:33:05 -07:00
Artem Navoiev
153cceb124
docs: adjust image size
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 21:30:54 +02:00