Commit graph

9162 commits

Author SHA1 Message Date
hagen1778
bd84f8a35d
docs: bump VM latest LTS releases
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 16:34:06 +02:00
hagen1778
07902baa8e
deployment: bump VM to v1.104.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 16:33:00 +02:00
hagen1778
4c6b7ce6da
docs: mention to test releases before publishing them in release guide
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 16:31:11 +02:00
hagen1778
8592fc3162
docs: add link to docs for multitenant reads
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 16:27:20 +02:00
Fred Navruzov
150ee902fd
docs/vmanomaly: patch release 1.16.1 (#7169)
### Describe Your Changes

`vmanomaly` patch release 1.16.1 updates

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 18:10:08 +04:00
hagen1778
01bc28eda2
docs: mention version available for multitenant reads
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 14:58:06 +02:00
hagen1778
cd8a478a8d
docs: actualize multitenancy docs
Follow-up after https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6346

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 14:35:29 +02:00
hagen1778
36acde1d11
docs: add missing release notes
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 14:20:55 +02:00
Roman Khavronenko
0d4f4b8f7d
(app|lib)/vmstorage: do not increment vm_rows_ignored_total on NaNs (#7166)
`vm_rows_ignored_total` metric is a metric for users to signalize about
ingestion issues, such as bad timestamp or parsing error.
In commit
a5424e95b3
this metric started to increment each time vmstorage gets NaN. But NaN
is a valid value for Prometheus data model and for Prometheus metrics
exposition format. Exporters from Prometheus ecosystem could expose NaNs
as values for metrics and these values will be delivered to vmstorage
and increment the metric.
Since there is nothing user can do with this, in opposite to parsing
errors or bad timestamps, there is not much sense in incrementing this
metric. So this commit rolls-back `reason="nan_value"` increments.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 12:37:27 +02:00
Artem Navoiev
c04c377d09
docs: cloud API remove link to guide as we don't have it
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-10-02 11:10:38 +02:00
Artem Navoiev
504da7d02b
docs: add API doc for cloud (#7164)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-10-02 10:58:35 +02:00
Fred Navruzov
dde2a0cb25
docs/vmanomaly - update versions in examples (#7163)
### Describe Your Changes

update `vnanomaly` versions in examples

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 11:53:05 +03:00
Fred Navruzov
0e54cfe350
docs/vmanomaly - release 1.16.0 docs (#7159)
### Describe Your Changes

doc updates for vmanomaly v1.16.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-02 00:19:14 +02:00
Aliaksandr Valialkin
456aeda605
app/vlogscli: preserve less output
This simplifies logs' investigation, since it allows copying some text from the previous query output
2024-10-01 21:46:36 +02:00
Aliaksandr Valialkin
3634fefc64
docs/VictoriaLogs/querying/vlogscli.md: typo fixes 2024-10-01 21:10:05 +02:00
f41gh7
776c501cb2
CHANGELOG.md: cut v1.104.0 release 2024-10-01 16:55:04 +02:00
f41gh7
076a1f84e1
vmselect: add support of multi-tenant queries
Added  ability to query data across multiple tenants. See:
VictoriaMetrics/VictoriaMetrics#1434

Currently, the following endpoints work with multi-tenancy:
- /prometheus/api/v1/query
- /prometheus/api/v1/query_range
- /prometheus/api/v1/series
- /prometheus/api/v1/labels
- /prometheus/api/v1/label/<label_name>/values
- /prometheus/api/v1/status/active_queries
- /prometheus/api/v1/status/top_queries
- /prometheus/api/v1/status/tsdb
- /prometheus/api/v1/export
- /prometheus/api/v1/export/csv
- /vmui

A note regarding VMUI: endpoints such as `active_queries` and
`top_queries` have been updated to indicate whether query was a
single-tenant or multi-tenant, but UI needs to be updated to display
this info.
cc: @Loori-R

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-10-01 16:49:46 +02:00
Aliaksandr Valialkin
496015aa0e
docs/VictoriaLogs/querying/vlogscli.md: provide instructions on how to download run vlogscli 2024-10-01 16:13:12 +02:00
Aliaksandr Valialkin
b05fbee63d
deployment: update VictoriaLogs docker image from v0.32.1-victorialogs to v0.33.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.33.0-victorialogs
2024-10-01 13:57:56 +02:00
Aliaksandr Valialkin
234c81754e
docs/VictoriaLogs/CHANGELOG.md: cut v0.33.0-victorialogs release 2024-10-01 13:42:18 +02:00
Roman Khavronenko
d57d8b5e60
docs: add note that otel is not using streaming parsing (#7148)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-01 13:39:40 +02:00
Aliaksandr Valialkin
a350be48b6
lib/logstorage: do not count dictionary values which have no matching logs in count_uniq stats function
Create blockResultColumn.forEachDictValue* helper functions for visiting matching
dictionary values. These helper functions should prevent from counting dictionary values
without matching logs in the future.

This is a follow-up for 0c0f013a60
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7152
2024-10-01 13:34:45 +02:00
Aliaksandr Valialkin
630211cfed
app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:23:07 +02:00
Aliaksandr Valialkin
61ae077e43
docs/VictoriaLogs/FAQ.md: clarify answers a bit 2024-10-01 12:08:27 +02:00
Roman Khavronenko
ead3250b80
docs: old logo seems not referenced anymore (#7151)
6y old :)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-01 10:26:34 +02:00
Nikolay
fbaa026ae6
dashboards: updates operator dashboard (#7139)
* Replaces deprecated graphs with Timeseries panels
* Adds new latency dashboards for rest client and golang scheduler
* Adds new overview panels
* Adds VM Datasource version of dashboard

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-09-30 15:35:39 +02:00
Artem Navoiev
0b2d3d7752
docs: cloud add support page. remove title as far it's already there
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-30 14:45:33 +02:00
Zhu Jiekun
7bb8853a5c
feature: [vmagent] Add service discovery support for OVH Cloud VPS and dedicated server (#6160)
### Describe Your Changes
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6071

#### Added
- Added service discovery support for OVH Cloud:
    - VPS.
    - Dedicated server.

#### Docs
- `CHANGELOG.md`, `sd_configs.md`, `vmagent.md` are updated.

#### Note
- Useful links: 
    - OVH Cloud VPS API: https://eu.api.ovh.com/console/#/vps~GET
- OVH Cloud Dedicated server API:
https://eu.api.ovh.com/console/#/dedicated/server~GET
    - OVH Cloud SDK: https://github.com/ovh/go-ovh
- Prometheus SD:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ovhcloud_sd_config

Tested on OVH Cloud VPS and dedicated server.
<img width="1722" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/d3f0adc8-b0ef-423e-9379-8a9b9b0792ee">

<img width="1724" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/18b5b730-3512-4fc0-8b2c-f2450ac550fd">

---
Signed-off-by: Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-09-30 14:42:46 +02:00
Aliaksandr Valialkin
aafa9262c5
deployment/docker: update VictoriaLogs from v0.32.0-victorialogs to v0.32.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.1-victorialogs
2024-09-30 14:35:40 +02:00
Aliaksandr Valialkin
82482fca4b
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.1-victorialogs release 2024-09-30 14:31:17 +02:00
Artem Navoiev
f5f04c903b
docs: cloud add support page (#7143)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-30 14:29:26 +02:00
Hui Wang
664f337c70
stream aggregation: fix possible duplicated aggregation results (#7118)
When ingesting samples with the same labels(duplicated samples or
samples with the same labels after `by` or `without` options). They
could register different entries for the same labelset in
LabelsCompressor.
For example, both index 99 and 100 can be assigned to label `foo=1` in
two concurrent pushes. Then due to differing label indexes in encoded
keys, the samples will appear as distinct in aggrState, resulting in
duplicated results after decompressing the label indexes.

fbde238cdc/lib/streamaggr/streamaggr.go (L933)

In this pull request, since we need to store `idxToLabel` first to
ensure the idx can be searched after `lc.labelToIdxStore`,
the `lc.idxToLabel` still could contain a duplicated entries
[100]="foo=1". But given the low likelihood of this issue and the size
of idxToLabel, it should be fine.
2024-09-30 14:24:59 +02:00
Aliaksandr Valialkin
0c0f013a60
lib/logstorage: skip values with zero hits for 'uniq', 'top' and 'field_values' pipes
See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/72#issuecomment-2352078483
2024-09-30 14:15:07 +02:00
Artem Fetishev
85ea0f80fc
Change the default value of the maxDeleteSeries flag to 1 million (#7140)
Change the default value of the maxDeleteSeries flag to 1 million. This
is a follow up for ed5da38ede
---

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-09-30 12:40:49 +02:00
f41gh7
758f42fc12
docs: add Update Note for upcoming release changes
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-30 12:37:30 +02:00
Artem Fetishev
ed5da38ede
Introduce a flag for limiting the number of time series to delete (#7091)
### Describe Your Changes

Introduce the `-search.maxDeleteSeries` flag that limits the number of
time series that can be deleted with a single
`/api/v1/admin/tsdb/delete_series` call.

Currently, any number can be deleted and if the number is big (millions)
then the operation may result in unaccounted CPU and memory usage spikes
which in some cases may result in OOM kill (see #7027). The flag limits
the number to 30k by default and the users may override it if needed at
the vmstorage start time.


---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-09-30 10:02:21 +02:00
Alexander Frolov
80a3c410d4
vmselect: ensure default -search.maxConcurrentRequests is non-decreasing (#6996)
### Describe Your Changes

vmselect determines the default value of `-search.maxConcurrentRequests`
multiplying the number of available CPUs by 2 if and only if the number
is small (to be precise <= 4). That leads
`-search.maxConcurrentRequests` is decreasing at the edge of these two
cases as shown below:
| CPUs | MaxConcurrentRequests | MaxConcurrentRequests (original
proposal) | MaxConcurrentRequests (updated proposal) |
|--------|--------|--------|--------|
| 1 | 2 | 2 | 2 |
| 2 | 4 (prev+2) | 4 (prev+2) | 4 (prev+2) |
| 3 | 6 (prev+2) | 6 (prev+2) | 6 (prev+2) |
| 4 | 8 (prev+2) | 8 (prev+2) | 8 (prev+2) |
| 5 | 5 __(prev-3)__ | 9 __(prev+1)__ | 10 __(prev+2)__ |
| 6 | 6 (prev+1) | 10 (prev+1) | 12 (prev+2) |
| 7 | 7 (prev+1) | 11 (prev+1) | 14 (prev+2) |
| 8 | 8 (prev+1) | 12 (prev+1) | 16 (prev+2) |

I propose to make the default value non-decreasing.
2024-09-30 09:51:54 +02:00
hagen1778
2df2e9b92f
docs: use global dedup flag in vmagent's FlexibleDeduplication docs
Current doc is using per-url deduplication, and users might use this example
when they have more than 1 remoteWrite URL. Which would result into extra resource usage.
Changing the example to use global dedup, as it makes more sense.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-30 09:08:56 +02:00
Github Actions
3d01bc3fbe
Automatic update helm docs from VictoriaMetrics/helm-charts@de4356f (#7138)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-30 13:03:15 +08:00
Github Actions
5e67e611b8
Automatic update operator docs from VictoriaMetrics/operator@0e44654 (#7136)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-29 22:30:24 +02:00
Aliaksandr Valialkin
2047ad20ef
deployment: update VictoriaLogs docker image from v0.31.0-victorialogs to v0.32.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.0-victorialogs
2024-09-29 14:54:21 +02:00
Aliaksandr Valialkin
45cfb6b526
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.0-victorialogs 2024-09-29 14:47:31 +02:00
Aliaksandr Valialkin
1da4650143
lib/logstorage: allow using ! in unescaped phrase
Previously the phrase filter with `!` was treated unexpectedly.
For example, `foo!bar` filter was treated at `foo AND NOT bar`,
while most users expect that it matches "foo!bar" phrase.

This commit aligns with users' expectations.
2024-09-29 11:14:15 +02:00
Aliaksandr Valialkin
60183c7c79
lib/logstorage: allow using - instead of ! in front of (...) 2024-09-29 11:12:22 +02:00
Nikolay
3bbb2aed72
fscore: rollback trailing space trim (#7106)
Previous commit 201fd6de1e removed
trailing space trim from data read from file. But common practice is to
remove such trailing space. And it leaded to the authorization errors
for the major group of users.

In first place, this change must help to mitigate an issue with
kubernetes. When authorization information was read from Secret content.
Changes to the operator was made to mitigate such problem at commit
1cf64358c8

We could introduce later optional flag for VictoriaMetrics to disable
trim space behavior.

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6986
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7089 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6947

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
2024-09-29 10:59:25 +02:00
Aliaksandr Valialkin
b52862badf
lib/logstorage: return the expected hits results from uniq pipe when the number of unique values reaches the specified limit
Previously `uniq` pipe could return zero `hits` if the number of found unique values equals the specified limit.
This wasn't expected in most cases.
2024-09-29 10:51:09 +02:00
Aliaksandr Valialkin
55eb321f77
lib/logstorage: clear hits slice obtained from encoding.GetUint64s() before updating it with hits for valueTypeDict column
encoding.GetUint64s() returns uninitialized slice, which may contain arbitrary values.
So values in this slice must be reset to zero before using it for counting hits in `uniq` and `top` pipes.
2024-09-29 10:29:13 +02:00
Aliaksandr Valialkin
94afcbd9a9
lib/logstorage: postpone initialization of per-shard stateSizeBudget until the first call to pipeProcessor.writeBlock()
This simplifies pipeProcessor initialization logic a bit.
This also doesn't mangle the original maxStateSize value, which is used in error messages when the state size exceeds maxStateSize.
2024-09-29 10:29:13 +02:00
Andrii Chubatiuk
dfcab4a47f
deployment/victorialogs: rename not executable compose files (#7124)
### Describe Your Changes

Renamed base compose files to prevent envs to be created from them

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-29 09:54:45 +02:00
Aliaksandr Valialkin
0b91452ca4
lib/logstorage: add non-empty if (...) condition to automatically generated result names in stats pipe
This allows executing queries with `stats` pipe, which calculate multiple results with the same functions,
but with different `if (...)` conditions. For example:

  _time:5m | count(), count() if (error)

Previously such queries couldn't be executed becasue automatically generated name for the second result
didn't include `if (error)`, so names for both results were identical - `count(*)`.
2024-09-29 09:51:28 +02:00