Commit graph

9243 commits

Author SHA1 Message Date
Roman Khavronenko
dfb2ad4ab4
(app|lib)/vmstorage: do not increment vm_rows_ignored_total on NaNs (#7166)
`vm_rows_ignored_total` metric is a metric for users to signalize about
ingestion issues, such as bad timestamp or parsing error.
In commit
a5424e95b3
this metric started to increment each time vmstorage gets NaN. But NaN
is a valid value for Prometheus data model and for Prometheus metrics
exposition format. Exporters from Prometheus ecosystem could expose NaNs
as values for metrics and these values will be delivered to vmstorage
and increment the metric.
Since there is nothing user can do with this, in opposite to parsing
errors or bad timestamps, there is not much sense in incrementing this
metric. So this commit rolls-back `reason="nan_value"` increments.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 0d4f4b8f7d)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-02 12:43:13 +02:00
Artem Navoiev
608f364ff7
docs: cloud API remove link to guide as we don't have it
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit c04c377d09)
2024-10-02 12:41:15 +02:00
Artem Navoiev
38f7cae398
docs: add API doc for cloud (#7164)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 504da7d02b)
2024-10-02 12:41:15 +02:00
Fred Navruzov
d205998501
docs/vmanomaly - update versions in examples (#7163)
### Describe Your Changes

update `vnanomaly` versions in examples

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit dde2a0cb25)
2024-10-02 12:41:14 +02:00
Fred Navruzov
42ddf7cfcc
docs/vmanomaly - release 1.16.0 docs (#7159)
### Describe Your Changes

doc updates for vmanomaly v1.16.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 0e54cfe350)
2024-10-02 12:41:14 +02:00
Aliaksandr Valialkin
17c8beca18
app/vlogscli: preserve less output
This simplifies logs' investigation, since it allows copying some text from the previous query output
2024-10-01 21:48:13 +02:00
Aliaksandr Valialkin
ff151f2caa
docs/VictoriaLogs/querying/vlogscli.md: typo fixes 2024-10-01 21:11:51 +02:00
f41gh7
8105c92d14
app/vminsert: fixes multiarch Dockerfile
Follow-up after 395894688c

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-10-01 18:11:40 +02:00
f41gh7
ee3db9b7c7
CHANGELOG.md: cut v1.104.0 release 2024-10-01 16:55:18 +02:00
Zakhar Bessarab
44b071296d
vmselect: add support of multi-tenant queries (#6346)
### Describe Your Changes

Added an ability to query data across multiple tenants. See:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1434

Currently, the following endpoints work with multi-tenancy:
- /prometheus/api/v1/query
- /prometheus/api/v1/query_range
- /prometheus/api/v1/series
- /prometheus/api/v1/labels
- /prometheus/api/v1/label/<label_name>/values
- /prometheus/api/v1/status/active_queries
- /prometheus/api/v1/status/top_queries
- /prometheus/api/v1/status/tsdb
- /prometheus/api/v1/export
- /prometheus/api/v1/export/csv
- /vmui


A note regarding VMUI: endpoints such as `active_queries` and
`top_queries` have been updated to indicate whether query was a
single-tenant or multi-tenant, but UI needs to be updated to display
this info.
cc: @Loori-R 

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-10-01 16:37:18 +02:00
Aliaksandr Valialkin
856c189688
docs/VictoriaLogs/querying/vlogscli.md: provide instructions on how to download run vlogscli 2024-10-01 16:13:35 +02:00
Aliaksandr Valialkin
ee784f9334
deployment: update VictoriaLogs docker image from v0.32.1-victorialogs to v0.33.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.33.0-victorialogs
2024-10-01 13:58:36 +02:00
Aliaksandr Valialkin
b2c3dbef09
docs/VictoriaLogs/CHANGELOG.md: cut v0.33.0-victorialogs release 2024-10-01 13:42:27 +02:00
Roman Khavronenko
5702b354a7
docs: add note that otel is not using streaming parsing (#7148)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-01 13:40:03 +02:00
Aliaksandr Valialkin
81f3e07e1e
lib/logstorage: do not count dictionary values which have no matching logs in count_uniq stats function
Create blockResultColumn.forEachDictValue* helper functions for visiting matching
dictionary values. These helper functions should prevent from counting dictionary values
without matching logs in the future.

This is a follow-up for 0c0f013a60
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7152
2024-10-01 13:36:27 +02:00
Aliaksandr Valialkin
8c55b699f4
app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:24:53 +02:00
Aliaksandr Valialkin
285a4dde11
docs/VictoriaLogs/FAQ.md: clarify answers a bit 2024-10-01 12:24:03 +02:00
Roman Khavronenko
301fa4191f
docs: old logo seems not referenced anymore (#7151)
6y old :)

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-01 12:24:03 +02:00
Nikolay
88e7f1b837
dashboards: updates operator dashboard (#7139)
* Replaces deprecated graphs with Timeseries panels
* Adds new latency dashboards for rest client and golang scheduler
* Adds new overview panels
* Adds VM Datasource version of dashboard

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-10-01 12:24:02 +02:00
Artem Navoiev
584cc36b06
docs: cloud add support page. remove title as far it's already there
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-30 15:06:15 +02:00
Zhu Jiekun
d1d59d6348
feature: [vmagent] Add service discovery support for OVH Cloud VPS and dedicated server (#6160)
### Describe Your Changes
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6071

#### Added
- Added service discovery support for OVH Cloud:
    - VPS.
    - Dedicated server.

#### Docs
- `CHANGELOG.md`, `sd_configs.md`, `vmagent.md` are updated.

#### Note
- Useful links: 
    - OVH Cloud VPS API: https://eu.api.ovh.com/console/#/vps~GET
- OVH Cloud Dedicated server API:
https://eu.api.ovh.com/console/#/dedicated/server~GET
    - OVH Cloud SDK: https://github.com/ovh/go-ovh
- Prometheus SD:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ovhcloud_sd_config

Tested on OVH Cloud VPS and dedicated server.
<img width="1722" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/d3f0adc8-b0ef-423e-9379-8a9b9b0792ee">

<img width="1724" alt="image"
src="https://github.com/VictoriaMetrics/VictoriaMetrics/assets/30280396/18b5b730-3512-4fc0-8b2c-f2450ac550fd">

---
Signed-off-by: Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-09-30 15:06:14 +02:00
Aliaksandr Valialkin
975f0f3610
deployment/docker: update VictoriaLogs from v0.32.0-victorialogs to v0.32.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.1-victorialogs
2024-09-30 14:38:27 +02:00
Artem Navoiev
c9b20d4a1b
docs: cloud add support page (#7143)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-30 14:31:31 +02:00
Aliaksandr Valialkin
c07746c1af
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.1-victorialogs release 2024-09-30 14:30:34 +02:00
Hui Wang
bf3d9ba57b
stream aggregation: fix possible duplicated aggregation results (#7118)
When ingesting samples with the same labels(duplicated samples or
samples with the same labels after `by` or `without` options). They
could register different entries for the same labelset in
LabelsCompressor.
For example, both index 99 and 100 can be assigned to label `foo=1` in
two concurrent pushes. Then due to differing label indexes in encoded
keys, the samples will appear as distinct in aggrState, resulting in
duplicated results after decompressing the label indexes.

fbde238cdc/lib/streamaggr/streamaggr.go (L933)

In this pull request, since we need to store `idxToLabel` first to
ensure the idx can be searched after `lc.labelToIdxStore`,
the `lc.idxToLabel` still could contain a duplicated entries
[100]="foo=1". But given the low likelihood of this issue and the size
of idxToLabel, it should be fine.
2024-09-30 14:30:34 +02:00
Aliaksandr Valialkin
dbcf06cd85
lib/logstorage: skip values with zero hits for 'uniq', 'top' and 'field_values' pipes
See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/72#issuecomment-2352078483
2024-09-30 14:16:21 +02:00
f41gh7
ba037a9777
docs: add Update Note for upcoming release changes
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-30 12:45:03 +02:00
Alexander Frolov
1eb7ab1f77
vmselect: ensure default -search.maxConcurrentRequests is non-decreasing (#6996)
### Describe Your Changes

vmselect determines the default value of `-search.maxConcurrentRequests`
multiplying the number of available CPUs by 2 if and only if the number
is small (to be precise <= 4). That leads
`-search.maxConcurrentRequests` is decreasing at the edge of these two
cases as shown below:
| CPUs | MaxConcurrentRequests | MaxConcurrentRequests (original
proposal) | MaxConcurrentRequests (updated proposal) |
|--------|--------|--------|--------|
| 1 | 2 | 2 | 2 |
| 2 | 4 (prev+2) | 4 (prev+2) | 4 (prev+2) |
| 3 | 6 (prev+2) | 6 (prev+2) | 6 (prev+2) |
| 4 | 8 (prev+2) | 8 (prev+2) | 8 (prev+2) |
| 5 | 5 __(prev-3)__ | 9 __(prev+1)__ | 10 __(prev+2)__ |
| 6 | 6 (prev+1) | 10 (prev+1) | 12 (prev+2) |
| 7 | 7 (prev+1) | 11 (prev+1) | 14 (prev+2) |
| 8 | 8 (prev+1) | 12 (prev+1) | 16 (prev+2) |

I propose to make the default value non-decreasing.
2024-09-30 12:44:54 +02:00
hagen1778
39b3e73f55
docs: use global dedup flag in vmagent's FlexibleDeduplication docs
Current doc is using per-url deduplication, and users might use this example
when they have more than 1 remoteWrite URL. Which would result into extra resource usage.
Changing the example to use global dedup, as it makes more sense.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-30 12:44:53 +02:00
Github Actions
b4bd1cf643
Automatic update helm docs from VictoriaMetrics/helm-charts@de4356f (#7138)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-30 12:44:53 +02:00
Github Actions
68e0130779
Automatic update operator docs from VictoriaMetrics/operator@0e44654 (#7136)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-30 12:44:53 +02:00
Artem Fetishev
91c2b5b24d
Introduce a flag for limiting the number of time series to delete (cluster version) (#7112)
### Describe Your Changes

Introduce the `-search.maxDeleteSeries` flag that limits the number of
time series that can be deleted with a single
`/api/v1/admin/tsdb/delete_series` call.

Currently, any number can be deleted and if the number is big (millions)
then the operation may result in unaccounted CPU and memory usage spikes
which in some cases may result in OOM kill (see #7027). The flag limits
the number to 30k by default and the users may override it if needed at
the vmstorage start time.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7027
---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-09-30 12:43:11 +02:00
Aliaksandr Valialkin
bf746f9975
deployment: update VictoriaLogs docker image from v0.31.0-victorialogs to v0.32.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.32.0-victorialogs
2024-09-29 14:54:42 +02:00
Aliaksandr Valialkin
3babcb0bbd
docs/VictoriaLogs/CHANGELOG.md: cut v0.32.0-victorialogs 2024-09-29 14:48:36 +02:00
Nikolay
cb50408dc6
fscore: rollback trailing space trim (#7106)
Previous commit 201fd6de1e removed
trailing space trim from data read from file. But common practice is to
remove such trailing space. And it leaded to the authorization errors
for the major group of users.

In first place, this change must help to mitigate an issue with
kubernetes. When authorization information was read from Secret content.
Changes to the operator was made to mitigate such problem at commit
1cf64358c8

We could introduce later optional flag for VictoriaMetrics to disable
trim space behavior.

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6986
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7089 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6947

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
2024-09-29 14:48:36 +02:00
Aliaksandr Valialkin
7456cbc653
lib/logstorage: allow using ! in unescaped phrase
Previously the phrase filter with `!` was treated unexpectedly.
For example, `foo!bar` filter was treated at `foo AND NOT bar`,
while most users expect that it matches "foo!bar" phrase.

This commit aligns with users' expectations.
2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
b7a3d575da
lib/logstorage: allow using - instead of ! in front of (...) 2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
fce6fc745a
lib/logstorage: return the expected hits results from uniq pipe when the number of unique values reaches the specified limit
Previously `uniq` pipe could return zero `hits` if the number of found unique values equals the specified limit.
This wasn't expected in most cases.
2024-09-29 10:53:44 +02:00
Aliaksandr Valialkin
58d1e517de
lib/logstorage: clear hits slice obtained from encoding.GetUint64s() before updating it with hits for valueTypeDict column
encoding.GetUint64s() returns uninitialized slice, which may contain arbitrary values.
So values in this slice must be reset to zero before using it for counting hits in `uniq` and `top` pipes.
2024-09-29 10:29:50 +02:00
Aliaksandr Valialkin
b5d94f06f5
lib/logstorage: postpone initialization of per-shard stateSizeBudget until the first call to pipeProcessor.writeBlock()
This simplifies pipeProcessor initialization logic a bit.
This also doesn't mangle the original maxStateSize value, which is used in error messages when the state size exceeds maxStateSize.
2024-09-29 10:29:49 +02:00
Andrii Chubatiuk
a18d4f79d4
deployment/victorialogs: rename not executable compose files (#7124)
### Describe Your Changes

Renamed base compose files to prevent envs to be created from them

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-29 10:29:49 +02:00
Aliaksandr Valialkin
7f8b1300a9
lib/logstorage: add non-empty if (...) condition to automatically generated result names in stats pipe
This allows executing queries with `stats` pipe, which calculate multiple results with the same functions,
but with different `if (...)` conditions. For example:

  _time:5m | count(), count() if (error)

Previously such queries couldn't be executed becasue automatically generated name for the second result
didn't include `if (error)`, so names for both results were identical - `count(*)`.
2024-09-29 09:52:19 +02:00
Aliaksandr Valialkin
04c73d54d4
lib/logstorage: support order alias for sort pipe
Now the following queries are equivalents:

    _time:5s | sort by (_time)

    _time:5s | order by (_time)

This is needed for convenience, since `order by` is commonly used in other query languages such as SQL.
2024-09-29 09:52:18 +02:00
Artem Navoiev
55eae927bd
docs: changelog fix typo in url
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-29 09:52:18 +02:00
Artem Navoiev
0dff44be8f
docs: mention new create backup api in docs and changelog (#7104)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-29 09:52:17 +02:00
Aliaksandr Valialkin
0f1b3852dd
app/vlinsert: support unix timestamps in seconds and milliseconds in JSON stream data ingestion API 2024-09-28 21:57:19 +02:00
Aliaksandr Valialkin
b8fa213310
app/vlinsert: accept unix timestamp in seconds additionally to milliseconds at ElasticSearch bulk API
Timestamps in seconds are sometimes used for data ingestion via ElasticSearch bulk API
2024-09-28 21:21:19 +02:00
Roman Khavronenko
3e5cba3e78
deployment: separate datasource between single and cluster versions (#7119)
Before, single and cluster deployments were provisioned with both
Grafana datasources: single and cluster. But this resulted into a
problem: single DS didn't work for cluster and vice versa. See
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7113

This PR splits datasource file into 2 files: single and cluster. Now,
these files are separately provisioned to single and cluster deployments
correspondingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-28 21:21:18 +02:00
Github Actions
8044c8f2d5
Automatic update operator docs from VictoriaMetrics/operator@d39fc94 (#7129)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-09-28 21:21:18 +02:00
Andrii Chubatiuk
3df920c60d
docs/victorialogs: change fluentd input type in examples (#7123)
### Describe Your Changes

Use fluentd logging driver in examples to have enriched data in
VictoriaLogs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-28 21:21:17 +02:00