Commit graph

10253 commits

Author SHA1 Message Date
f41gh7
e0b2c1c4f5
docs/changelog: removes duplicate record for maxIngestionRate feature
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-12-24 19:40:44 +01:00
Github Actions
5afbee5f6f
Automatic update operator docs from VictoriaMetrics/operator@43f5554 (#7923)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-12-24 19:31:27 +01:00
Github Actions
51459196f9
Automatic update helm docs from VictoriaMetrics/helm-charts@1f33c21 (#7929)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-12-24 19:30:43 +01:00
Dima Shur
7941877233
docs: changed typo in label (enterpriSe instead of enterpriZe) (#7925)
### Describe Your Changes

Fixed typo in contributing.md (enterpriZe -> enterpriSe in the label
name)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-24 15:51:35 +04:00
Phuong Le
f303081304
vminsert: sort the storage nodes during initialization (#7899)
Fixes #7898
2024-12-23 19:41:17 +01:00
Ted Possible
a84628f701
app/vminsert: support for rate limiting number of samples/sec with -maxIngestionRate
This commit adds feature to limit sample ingestion rate globally for ingestion protocols. 

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7377
2024-12-23 17:37:30 +01:00
Github Actions
f823a225ac
Automatic update operator docs from VictoriaMetrics/operator@471f183 (#7916)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-12-23 16:48:49 +01:00
Andrii Chubatiuk
79f1a37ee6
vlinsert: take into account order of msgfields to have predictable _msg field selection in case of multiple matches (#7784)
### Describe Your Changes

Currently if multiple msgFields are present in a log row it's not
obvious which field is selected as a _msg field. With this PR and order
of msgfield values defined either via headers or query arg params
defines a priority of these values

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-23 10:10:02 +01:00
Andrii Chubatiuk
f9cd408ca9
datadog-serverless: fixed metrics and logs ingestion from Datadog serverless extensions for AWS and GCP (#7769)
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7761

### Describe Your Changes

- datadog /api/v2/logs api supports message field in json format, which
is not documented and is used by serverless extension. This PR allows
message field to be both string and object type. Also added support of
not documented timestamp field
- added `-datadog.streamFields` and `-datadog.ignoreFields` flags to
configure default stream fields for datadog logs, where there's no
alternative option to pass extra headers and query args
- added ingest `max` and `min` values of data, which are ingested using
`datadogsketches` API, which is also actively used by serverless
extensions
- use default `.` separator instead of `_` for sketches metric names
until metrics are not sanitized
2024-12-23 09:57:48 +01:00
Aliaksandr Valialkin
c2811d8d11
docs/VictoriaLogs/LogsQL.md: fix a link to count_uniq_hash stats function docs
It must be consistent with the other stats functions

This is a follow-up for de0ae735aa
2024-12-22 14:39:27 +01:00
Aliaksandr Valialkin
8d981b15c9
deployment: update VictoriaLogs Docker image from v1.3.2-victorialogs to v1.4.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.4.0-victorialogs
2024-12-22 14:36:49 +01:00
Aliaksandr Valialkin
58f09fe3f8
docs/VictoriaLogs/CHANGELOG.md: cut v1.4.0-victorialogs release 2024-12-22 14:31:31 +01:00
Aliaksandr Valialkin
afd926a0b0
lib/logstorage: limit the maximum number of logs and/or log streams, which can be passed to stream_context pipe
This should prevent from excess usage of CPU, RAM and other resources when too many logs
are passed to 'stream_context' pipe.

It is expected that 'stream_context' pipe results are investigated by humans, who cannot inspect
surrounding logs for millions of initial logs. That's why it is OK to limit the number of logs
and/or log streams, which can be passed to 'stream_context' pipe.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7766
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7903
2024-12-22 14:28:50 +01:00
Aliaksandr Valialkin
204c102342
app/vlselect/vmui: run make vmui-logs-update after the commit 1fbc2c0db1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7288
2024-12-22 13:53:45 +01:00
Aliaksandr Valialkin
c5949af9e8
lib/logstorage: reduce memory allocations when splitting in(...) values into tokens and calculating hashes for these tokens
While at it, reduce memory allocations at Storage.getFieldValuesNoHits and make it more scalable on multi-CPU systems.

This improves performance of in(<query>) filter when the <query> returns big number of values.
2024-12-22 13:13:44 +01:00
Aliaksandr Valialkin
5dc0413bc0
lib/logstorage: allow specifying hits column name in the top pipe via top ... hits as <column_name> syntax 2024-12-22 11:23:19 +01:00
Aliaksandr Valialkin
f919783de9
lib/logstorage: uncommend accidentally commented tests at 60f9f44150 2024-12-22 02:20:57 +01:00
Aliaksandr Valialkin
60f9f44150
lib/logstorage: reduce memory allocations at stats and top pipes
Use chunked allocator in order to reduce memory allocations. It allocates objects from slices of up to 64Kb size.
This improves performance for `stats` and `top` pipes by up to 2x when they are applied to big number of `by (...)` groups.

Also parallelize execution of `count_uniq`, `count_uniq_hash` and `uniq_values` stats functions,
so they are executed faster on hosts with many CPU cores when applied to fields with big number
of unique values.
2024-12-22 02:13:02 +01:00
Github Actions
0fcbe8fdae
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@b18583c (#7910) 2024-12-21 09:42:48 -08:00
Github Actions
458b602938
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@cbff3fa (#7909) 2024-12-21 09:31:07 -08:00
Aliaksandr Valialkin
471f1d0a09
lib/logstorage: fixed a typo in blockResult.reset()
The commit 4599429f51 improperly set br.cs to nil,
while it should set br.bs to nil instead. This resulted in excess memory allocations
at br.csInit() and br.csInitFast().
2024-12-21 13:39:25 +01:00
hagen1778
7f80c1633f
docs: mention filebeat version requirement for vlogs integration
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-20 16:21:13 +01:00
Yury Molodov
186b00df6b
vmui: add export button for raw query data (#7828)
### Describe Your Changes

1. Added the ability to export data from the `Raw Query` page and import
exported data to the `Query Analyzer` page (related issue #7628).
2. Added a `Title` input field; the `Title` is displayed when importing
data on the `Query Analyzer` page.
3. Implemented `Markdown` support for comments in exported data.  
4. Updated the styling of the `Query Analyzer` page.  
5. Fixed an issue where the `Upload JSON` button on the `Query Analyzer`
page was only clickable on the button text (now clickable on the entire
button area).
6. Added a tooltip with `Deduplication` information on the `Raw Query`
page (related to issue #7763).

<details>
  <summary>Screenshots</summary>
  
#### Data export and `Markdown` preview

<img width="400"
src="https://github.com/user-attachments/assets/bbab31bb-81d3-4335-98c3-d01c8786bde4"/>
<img width="400"
src="https://github.com/user-attachments/assets/3cfd9938-b518-45d6-8ded-e3e7e6ab9299"/>

#### `Query Analyzer` page displaying data from `Raw Query`

<img width="900"
src="https://github.com/user-attachments/assets/008e0e93-92f2-4c25-a20e-3cee90a03397"/>

#### Viewing stats and comments on the `Query Analyzer` page  
    
<img width="600"
src="https://github.com/user-attachments/assets/18bfbba1-a11c-420e-84f2-78229ac7bd25"/>

#### Viewing stats data from the `Query` page

<img width="900"
src="https://github.com/user-attachments/assets/0f7a3009-9fb5-4727-b0c4-257aa196a9c1"/>

#### Tooltip on the `Raw Query` page  

<img width="900"
src="https://github.com/user-attachments/assets/400f86e7-f362-4307-8b1d-24af3c67020e"/>
  
</details>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-12-20 15:51:06 +01:00
Github Actions
4205ae3011
Automatic update helm docs from VictoriaMetrics/helm-charts@feb0675 (#7897)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-12-20 15:27:51 +01:00
Daria Karavaieva
491028774a
docs/vmanomaly: popup deprecated_from and available_from for all docs (#7905)
### Describe Your Changes
added deprecated form and available from popups in vmanomaly docs

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-20 15:24:16 +01:00
Mathias Palmersheim
565b79c9ca
docs: update vmalert+victorialogs doc with multitenant recording (#7779)
### Describe Your Changes
 
- Adds Headers to FAQ questions in vmalert for Victorialogs
- Adds FAQ for multitenant recording rules described in #7656

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Haley Wang <haley@victoriametrics.com>
2024-12-20 15:02:00 +01:00
Aliaksandr Valialkin
5478cc61c2
lib/cgroup: add missing initialization of gogc variable inside SetGOGC
This is a follow-up for 79c08ecac4

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7902
2024-12-20 14:56:59 +01:00
Aliaksandr Valialkin
79c08ecac4
lib/cgroup: use the default GOGC=100 for the most of VictoriaMetrics components
Historically some of VictoriaMetrics components were optimized for the low rate of memory allocations.
These are: vmagent, single-node VictoriaMetrics and vmstorage. These components benefit from the low
GOGC value, since this allow reducing their memory usage in steady state on typical workloads.

Other VictoriaMetrics components aren't optimized for the reduced rate of memory allocations.
This results in the increased CPU usage spent on garbage collection (GC) in these components,
since it must be triggered at higher rate. See https://tip.golang.org/doc/gc-guide#GOGC for details.

These components do not use too much memory, so it is OK increasing the GOGC for these components
from 30 to 100 - this won't affect the most users.

Keep GOGC to 30 only for vmagent, single-node VictoriaMetrics and vmstorage components.
See 077193d87c and 54b9e1d3cb .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7902
2024-12-20 14:48:28 +01:00
hagen1778
f47fd83e54
docs: add example with dots in label name to vlogs rules
This change adds an example of how to use labels with `.` dots
in rule annotations.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-20 14:07:48 +01:00
Aliaksandr Valialkin
9c39bac565
lib/logstorage: fix imroper sorting of numeric fields when they are stored as const values at sort pipe
Numeric fields can be stored as const values in the block of logs. In this case the `sort` pipe
was incorrectly comparing such values as strings instead of numbers. This results in incorrect
sort results. For example, 123 was smaller than 2. Fix this by removing the incorrect case
for comparing const fields.

While at it, replace lessString() with strings.LessNatural() in the sortBlockLess.
This improves sorting performance a bit, since the sortBlockLess function already tried
comparing numeric values, and it doesn't need to spend CPU time on such a comparison again inside lessString() call.
The commit 42c9183281 wasn't correct by replacing strings.LessNatural() with lessString()
inside the sortBlockLess() function.
2024-12-20 13:26:20 +01:00
Roman Khavronenko
1042f07498
docs: update OTEL guide (#7887)
* simplify wording
* update styles
* remove extra info about go application details. The details are likely
not needed and we didn't have details for rolling-dice app anyway. So
keep it simple for consstency and brevity.
* update navigation for simplicity sake
* fix typos

follow-up after
40b47601d1

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-19 15:13:03 +01:00
Nikolay
79a595c6d0
app/vmauth: properly log host at debugInfo function (#7886)
vmauth started to use request.Host after commit
f4776fec1b for`src_hosts` routing rules.

This commit adds http.Request.Host to the debugInfo output in order to
be consistent with routing logic.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-12-19 15:04:37 +01:00
Andrii Chubatiuk
40b47601d1
docs/guides/otel: added logs integration, updated old otel dependencies
### Describe Your Changes

- added VictoriaLogs to OpenTelemetry guide
- updated deprecated dependencies
- added deltatocumulative processor to example and deltatemporality
selector to one of examples to use for counters by default
- added exponential histograms to example

---
Signed-off-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
2024-12-19 12:32:41 +01:00
Zakhar Bessarab
6bfcbe66f7
docs/release-guide: add a note about versioning in helm charts and ansible
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-12-19 12:28:32 +01:00
f41gh7
94118c63f6
docs: update VM apps version to v1.108.1
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-12-19 12:25:37 +01:00
f41gh7
9605d73809
CHANGELOG.md: cut v1.108.1 release 2024-12-18 23:34:58 +01:00
f41gh7
3237c64ef3
make vmui-update 2024-12-18 23:08:22 +01:00
Yury Molodov
1fbc2c0db1
vmui: fix cursor reset in query input
Fix cursor reset in query input field. 

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7288.
2024-12-18 22:30:08 +01:00
Nikolay
71bb9fc0d0
app/vminsert: properly apply relabeling at ingestion
Regression was introduced at 564e6ea024
after implementing:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6928

ctx.Labels array could be incorrectly updated and changes to it after
relabeling rules can be lost.
E.g. ctx.Labels passed to WriteDataPoint function as slice copy, but
results of relabeling only changed an actual slice at ctx.Labels.

This commit replaces implicit relabeling call with explicit
`TryPrepareLabels` function.
It also reduces code diffs with cluster version and adds integration tests

 related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7865

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-12-18 22:27:51 +01:00
Github Actions
0210f4ebd2
Automatic update operator docs from VictoriaMetrics/operator@9b337c1 (#7879)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-12-18 16:27:37 +01:00
Andrii Chubatiuk
891ad8f202
app/vlinsert: loki healthcheck endpoint (#7864)
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7824

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-12-18 14:59:44 +01:00
Github Actions
e501640f44
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@d830b2a (#7855) 2024-12-18 12:27:15 +01:00
Daria Karavaieva
21082405ec
docs/vmanomaly: add version popup (#7860)
### Describe Your Changes

Added `available_from` popup into documentation of vmanomaly

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-18 12:26:42 +01:00
Github Actions
094a5ab58f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@b8fd925 (#7862) 2024-12-18 12:26:11 +01:00
hagen1778
bbc84fa119
docs: add requirements to commit message to contributing
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-18 12:22:20 +01:00
Github Actions
9d1a72aca8
Automatic update operator docs from VictoriaMetrics/operator@5c3657d (#7871)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-12-18 12:13:58 +01:00
Dmytro Kozlov
05d3db248b
deployment/docker: rename victorialogs-datasource to victoriametrics-logs-datasource (#7874)
### Describe Your Changes

Renamed victorialogs-datasource to victoriametrics-logs-datasource.

We prepared the victorialogs Grafana plugin for sign and updated the
plugin ID. This action require to update configs in our ops repository

Please check this
[release](https://github.com/VictoriaMetrics/victorialogs-datasource/releases/tag/v0.13.0)
and https://github.com/VictoriaMetrics/victorialogs-datasource/pull/161
with changes

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-18 11:41:16 +01:00
Github Actions
59d739ff0b
Automatic update helm docs from VictoriaMetrics/helm-charts@3b7bfbd (#7876)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-12-18 11:40:39 +01:00
hagen1778
b54d10be63
docs: port LTS changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-17 20:20:19 +01:00
Aliaksandr Valialkin
524f0e8d8b
lib/logstorage: eliminate memory allocations when finalizing per-group values calculated by stats pipe
This improves query performance a bit when `stats by (...)` returns millions of individual `by (...)` groups
2024-12-17 15:17:01 +01:00