Commit graph

4937 commits

Author SHA1 Message Date
Zhu Jiekun
44d856922a
lib/promscrape/discovery: properly apply the resource_group filter for Azure service discovery
Previously, this filter did not apply to virtual
machine scale sets, causing all virtual machines to be discovered.

 This commit conditionally adds `resource_group` filter for Azure service discovery on virtual
machine scale sets. 

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7630.
2024-11-26 19:08:31 +01:00
Roman Khavronenko
1753c3850b
app/vmauth: dump requests that failed the route rules to stderr (#7649)
Additional info from the dump can be used to debug rotuing rules.

https://pkg.go.dev/net/http/httputil#DumpRequest

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-26 18:50:31 +01:00
Artem Navoiev
d8b4faf50c
docs: update formatting on cloud billing page
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-11-26 18:50:31 +01:00
matty
ecbdbb8e49
docs: Bump LTS versions (#7658)
### Describe Your Changes

LTS versions are out of date, linking to old releases. This updates them
to the latest.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-26 18:50:31 +01:00
Artem Navoiev
103bd5bd88
docs: add cloud billing page (#7655)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-11-26 18:50:30 +01:00
Github Actions
f122f35a37
Automatic update helm docs from VictoriaMetrics/helm-charts@ebc6d67 (#7653)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-25 18:08:32 +01:00
Andrei Baidarov
728ceefca1
vmagent: set up a timeout for tcp connection establishment during k8s discovery
Previously, default dial timeout was used for kubernetes API server connection.

 This commit changes it for custom dialer used by the all VictoriaMetrics components. It has lower connection timeout (30s by default). 


 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7127

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-25 18:08:32 +01:00
hagen1778
7a06f05221
docs: re-structure multitenancy docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-25 18:08:32 +01:00
hagen1778
a099f0a70e
docs: update otel docs
* cross-link related doc chapters and guides about otel
* mention different URL format for cluster version of VM

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-25 18:08:32 +01:00
hagen1778
9a0b263e53
docs: update link for tsdb stats word
The link was pointing to itself, which doesn't make sense.
Changed it to point to https://docs.victoriametrics.com/url-examples/#apiv1statustsdb

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-25 18:08:31 +01:00
Hui Wang
82e42b1d4a
vmalert-tool: exit immediately with error message if no test file found in -files (#7627) 2024-11-25 11:52:33 +01:00
Nikolay
93061dfa7b
app/vmselect: fixes multitenant cache init
Previously multitenant cache was inited before flag.Parse call. It
didn't allow to change cache expiration value and default value was
always used.

 This commit adds cache init at the first time cache was called.

 Also this commit adds small cache improvements:
* chore for cleanup cache, it now uses common pattern for in-place items
filtering
* fail cache request fast if item is already expired


---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-25 11:49:34 +01:00
Andrei Baidarov
439d1b932e
app/vmselect: fix panic/incorrect tenant in key
This is a follow-up after 3120dc2
  
- Consistently use key for rollupCache in multitenant mode cache keys use different authTokens. Previously it could lead to panic in rare cases when cache state was inconsistent. 
- Do not share `err` variable across goroutines for `processBlock` function. It could lead to data races. 

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7549
---------
Signed-off-by: Andrei Baidarov <abaidarov@yandex.ru>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-25 11:47:24 +01:00
Github Actions
ba5e0c8745
Automatic update helm docs from VictoriaMetrics/helm-charts@6a5b14b (#7642)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-25 11:42:16 +01:00
dmitry-shur
857144826f
docs: fixed typo in FAQ (#7643)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-25 11:42:16 +01:00
Github Actions
65ed9759ed
Automatic update helm docs from VictoriaMetrics/helm-charts@a582432 (#7641)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-25 11:42:16 +01:00
hagen1778
65a2752cd8
docs: add vmui section to cluster docs
It looks like users get confused with vmui availability in vmselect.
Adding a note about it to cluster docs should improve situation.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7631

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-25 11:42:16 +01:00
Artem Navoiev
a1b2cf43c4
docs: cloud mention silver level in support
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-11-25 11:42:16 +01:00
Hui Wang
af4c6f3a29
vmalert: fix alert states restoration (#7624)
Previously, when the alert got resolved shortly before the vmalert
process shuts down, this could result in false alerts.

This change switches vmalert to use MetricsQL function during alerts state restore, which makes it
incompatible for state restoration with PromQL.

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-22 17:38:14 +01:00
Github Actions
53c2214940
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@a5e2cb4 (#7622) 2024-11-22 17:38:13 +01:00
hagen1778
1dd56f022d
docs: update differences between vmagent and Prometheus in FAQ
* mention stream aggregation
* rm statement that Prometheus can only pull data, which is not true anymore
* mention absence of backfilling limitations

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-22 17:38:13 +01:00
Ivan Yurochko
60ac4e0c00
lib/streamaggr: add ignore_first_sample_interval param for streamaggr cfg (#7313)
### Describe Your Changes

As of right now by default aggregated output in streaming aggregation
takes a staleness interval and only starts sending first samples after
the staleness interval passes. We have a use case where we prefer to
start sending data as soon as we have any. This adds the option to
configure when we start sending first samples

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7116

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-11-22 17:38:13 +01:00
Github Actions
66cd3f0a9f
Automatic update helm docs from VictoriaMetrics/helm-charts@b0095d3 (#7618)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-22 17:38:13 +01:00
hagen1778
257b0dc10c
docs: mention Raw Query tab in vmui
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7024
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-22 17:38:12 +01:00
Yury Molodov
db917d522d
vmui: add Raw Query tab (#7575)
### Describe Your Changes

1. **Add new `Raw Query` tab**  
A new `Raw Query` tab has been added to the
[vmui](https://docs.victoriametrics.com/#vmui) interface for displaying
raw data. The tab uses the `/api/v1/export` API endpoint. Related issue:
[#7024](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7024)


2. **Fix rendering of isolated points on the graph**  
Previously, isolated points (not connected to other points on the left
or right) were not visible on the graph. Now, they are rendered
correctly.
 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-22 17:38:12 +01:00
Nikolay
8357c22cc8
app/vmselect: properly return binary pow function result (#7619)
Previously, for `^` aka pow function calls, VictoriaMetrics returned `1`
if left arg was Nan. For example, given query=`(hour()==2)^1` returns 1
for NaN produced by hour() == 2 function. It added additional non-exist
datapoints to the timeseries.

This commit port bugfix from `metricql` package and adds test for it.
Now, VictoriaMetrics
correctly returns `NaN` for such cases.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7359

Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit bb399518db)
2024-11-21 15:23:49 +01:00
Hui Wang
5bb1e3a063
vmalert: remove deprecated cmd-line flags -datasource.lookback, `da… (#6779)
…tasource.queryTimeAlignment` and `remoteRead.ignoreRestoreErrors`

Those flags were all deprecated before
[v1.101.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.101.0).

(cherry picked from commit 1bd927e3fe)
2024-11-21 15:23:49 +01:00
Github Actions
f29c2b5a75
Automatic update operator docs from VictoriaMetrics/operator@b3e19a2 (#7608)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit f07574a78e)
2024-11-21 15:23:48 +01:00
Github Actions
426466b2df
Automatic update helm docs from VictoriaMetrics/helm-charts@8a9669e (#7616)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 689196048f)
2024-11-21 15:23:48 +01:00
Alexander Frolov
61050e2003
app/vmagent: respect Pushgateway protocol in multi-tenant vmagent handler (#7571)
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636 for
multi-tenant vmagent handler

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-20 18:37:35 +01:00
Will Jordan
209a5024ce
lib/tenantmetrics: improves CounterMap performance with large numbers of tenants
Previously, map for storing tenant metrics was re-created to each newly ingested tenant. It has significant performance impact for systems with large number of tenants. 

 This commit addresses this issue by changing algorithm of creating tenant metric records at map. Instead of map re-creation, it uses `sync.Map` primitive. 


Benchmark results:

```
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/tenantmetrics
cpu: AMD Ryzen 9 5900X 12-Core Processor            
                                            │ lib/tenantmetrics/orig.bench │     lib/tenantmetrics/new.bench     │
                                            │            sec/op            │    sec/op     vs base               │
CounterMapGrowth/n=100,nProcs=GOMAXPROCS-24                  1943.2µ ±  5%   248.0µ ± 11%  -87.24% (p=0.001 n=7)
CounterMapGrowth/n=100-24                                    434.63µ ±  5%   98.82µ ± 16%  -77.26% (p=0.001 n=7)
CounterMapGrowth/n=1000-24                                   32.719m ± 20%   1.425m ±  5%  -95.65% (p=0.001 n=7)
CounterMapGrowth/n=10000-24                                 3653.60m ±  5%   18.00m ±  2%  -99.51% (p=0.001 n=7)
geomean                                                       17.83m         890.4µ        -95.00%
```

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7482

---
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
2024-11-20 18:36:06 +01:00
Hui Wang
92e2e6f951
doc: clarify rule group type (#7595)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7585
2024-11-20 16:32:47 +01:00
Renato Monteiro
5838e4ce23
docs: fix small grammatical error in CONTRIBUTING.md (#7596) 2024-11-20 16:32:47 +01:00
Github Actions
d39f6c4e53
Automatic update operator docs from VictoriaMetrics/operator@c2b7a5a (#7598)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-11-20 16:32:46 +01:00
Github Actions
d4a140705f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@c4d6669 (#7602) 2024-11-20 16:32:46 +01:00
Hui Wang
04c07c085a
vmalert: revert the default value of -remoteWrite.maxQueueSize from… (#7570)
… `1_000_000` to `100_000`

It was bumped in
[v1.104.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.104.0),
which increases memory usage and is not needed for most setups. See
[this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7471).
2024-11-20 16:32:46 +01:00
Github Actions
a4e0fff34a
Automatic update operator docs from VictoriaMetrics/operator@1e7c2c5 (#7582)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: Haleygo <39937150+Haleygo@users.noreply.github.com>
2024-11-20 16:32:46 +01:00
Github Actions
3109d5b5ff
Automatic update helm docs from VictoriaMetrics/helm-charts@131846f (#7581)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-20 16:32:45 +01:00
Mathias Palmersheim
8bf2b46545
Updated vmanomaly product link (#7580)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

fixes #7579 7579
2024-11-20 16:32:45 +01:00
Fred Navruzov
e53a69c27d
docs/vmanomaly: add self-monitoring section (#7558)
- Added self-monitoring guide for `vmanomaly`.
- Added cross-referencing on other pages.
- Slight improvements in wording on related pages
- Update references to v1.18.4
- [x] publish Grafana dashboard to
https://grafana.com/orgs/victoriametrics/dashboards:
https://grafana.com/grafana/dashboards/22337-victoriametrics-vmanomaly/

@AndrewChubatiuk , JFYI if it somehow impacts your work on supporting
`vmanomaly` in operator.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-20 16:32:45 +01:00
JAYICE
1694ddb305
docs: correct the wrong description of vector support in the docs (#7566)
### Describe Your Changes

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7557

Vector only supports OpenTelemetry as a source.
2024-11-20 16:31:56 +01:00
f41gh7
99bc72df8c
{deployment/,docs/}: point app versions to the lastest v1.106.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-11-20 16:31:56 +01:00
f41gh7
3b60aa4c84
docs/changelog: mention 1.106.1,1.102.7 and 1.97.12 releases
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-11-20 16:31:56 +01:00
f41gh7
0babe43391
CHANGELOG.md: cut v1.106.1 release
remove upcoming v1.107.0 release from changelog
2024-11-15 23:38:16 +01:00
f41gh7
bd34dbaf5e
CHANGELOG.md: cut v1.107.0 release 2024-11-15 18:06:57 +01:00
Zakhar Bessarab
3391c2dce1
lib/storage/downsampling: handle dedup separately from downsampling rules with filters
Previously, dedup was added as a downsampling rule with 0s offset to all downsmapling rules with filters. That enforced a metric name lookup even in cases it is not needed.
For example, the following configuration: `-dedup.minScrapeInterval=10s -downsampling.period={__name__=~"node.*"}:1h:1m` would be parsed as: `{__name__=~"node.*"}:1h:1m {}:0s:10s`

This commit changes this logic and treats dedup as a separate case. This allows to perform metric name lookups only in cases when timestamp of current partition can be eligible to use some of downsampling filters. Newer parts will not trigger metric name lookup and will apply deduplication directly.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7440
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-15 17:57:05 +01:00
Nikolay
59d2b4c7fc
lib/storage: properly check for minMissingTimestamps
After changes at commit 787b9cd. Minimal timestamps for extDB check was performed without context of the index search prefix.
It worked fine for Single node version, but for cluster version a different prefix was used for
metricID search requests. It may lead to incomplete results, if minimal missing timestamp was cached
for the tenant with different ingestion patterns.

 Minimal reproducible case is:
- metrics were ingested for tenants 0 and 1
- at some point in time metrics ingestion for tenant 1 stopped
- index records have the following timestamps layout:
 tenant 0:  1,2,3,4,5,6
 tenant 1:  1,2,3,4
- after indexDB rotation, containsTimeRange lookups may produce
  incorrect results:
 time range request for tenant 1 - 5:6 caches 5 as min timestamp
 request for the same or smaller time range for tenant 0 now returns
empty results.

Second case:
- requests for the tenant without metrics always updates atomic value with incorrect minimal time range for other tenants.

 This commit replaces single atomic with map of search prefix keys. It should have slight performance overhead,
but work consistently for cluster version. minMissingTimestamp is cached by prefix search key, which included tenantID.

 Since it will be only populated at runtime, it doesn't hold unused tenants for queries.

Related issue: 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7417
2024-11-15 16:18:32 +01:00
Andrei Baidarov
3120dc2054
app/vmselect: fixes possible panics for multitenant queries
This commit fixes panic for multitenant requests and empty storage node responses for tenants api.

 It also optimizes `populateSqTenantTokensIfNeeded` function calls, by making it only once for query request. Previously it was incorrectly called multiple times per each storage node request.

Related issue: 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7549
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-15 16:12:30 +01:00
Aliaksandr Valialkin
c73a9cbdaa
docs: refer to the https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f in the appropriate places 2024-11-15 16:08:18 +01:00
Aliaksandr Valialkin
27125c6d74
docs/Articles.md: fix broken link to Open-sourcing VictoriaMetrics article 2024-11-15 16:08:17 +01:00