Commit graph

684 commits

Author SHA1 Message Date
Naveen
63ba8fc149
docs: update vlogs README.md (#8460)
### Describe Your Changes

Fixed the typo in the documentation. Updated `Ir provides` to `It
provides`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 179c530095)
2025-03-10 13:54:23 +01:00
Andrii Chubatiuk
5a05788719
app/vlinsert: support floats for elasticseach timestamps (#8472)
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8470

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 67f8fa66ed)
2025-03-10 13:54:23 +01:00
Zakhar Bessarab
dc7a953dd3
docs/victoria-logs/data-ingestion/promtail: fix typo (#8451)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit fd7b016c5b)
2025-03-07 11:03:52 +04:00
hagen1778
fbec45af0c
docs/changelog: fix metric name in changelog for vlogs request duration
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit bcbe5e80b3)
2025-03-06 15:18:44 +01:00
Andrii Chubatiuk
c72d5690cc
lib/protoparser/opentelemetry: properly marshal nested attributes into JSON
Previously, opentelemetry attribute parsed added extra field names according to 
golang JSON parser spec for structs:

```
struct AnyValue{
 StringValue string
}
```
 Was serialized into:
```
{"StringValue": "some-string"}
```
 While opentelemetry-collector serializes it as
```
"some-string"
```

 This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
2025-03-05 18:38:25 +01:00
Aliaksandr Valialkin
be8ed94b42
docs/VictoriaLogs/README.md: mention about JSONBench benchmark results in the benchmarks section
See https://docs.victoriametrics.com/victorialogs/#benchmarks
2025-03-04 18:35:58 +01:00
hagen1778
ff91443d06
docs/victorialogs/vmalert: apply consistent formatting and fix typos
* use unified numeric list;
* apply line width limits;
* remove time filter from quantile examples, as we suggest to
not use time filters.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 065a3d068c)
2025-03-03 14:14:57 +01:00
Zakhar Bessarab
c9b0de5728
docs/all: add refs to Quay mirror for docker images (#8406)
### Describe Your Changes

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4116

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 6ff61a1c09)
2025-02-28 15:23:01 +01:00
Aliaksandr Valialkin
8beb52eb31
docs/VictoriaLogs/CHANGELOG.md: typo fix: filter -> filters 2025-02-28 00:09:07 +01:00
Aliaksandr Valialkin
08d31bc6c2
deployment/docker: update VictoriaLogs Docker image tag from v1.14.0-victorialogs to v1.15.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.15.0-victorialogs
2025-02-28 00:04:00 +01:00
Aliaksandr Valialkin
f9082185c0
docs/VictoriaLogs/CHANGELOG.md: cut v1.15.0-victorialogs release 2025-02-27 22:58:26 +01:00
Aliaksandr Valialkin
c8a12435ec
lib/logstorage: add ability to specify field name prefixes inside fields (...) lists passed to pack_json and pack_logfmt pipes 2025-02-27 22:56:14 +01:00
Aliaksandr Valialkin
a1aa4b7aa9
lib/logstorage: allow passing * at in(*), contains_any(*) and contains_all(*)
Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for Grafana.

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673
2025-02-27 11:41:39 +01:00
Roman Khavronenko
727b8f06c9
dashboards: update victorialogs dashboard (#8398)
* stats panels:
* * add ingested logs/bytes over 24h
* * fix correctness compression ratio and disk space usage panels
* overview panels:
* * add requests errors rate, logs rate, query duration panels
* add troubleshooting section:
* * add non-default falgs, dropped logs panels
* update descriptions of the panels and version requirements

Depends on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8396

--------------

<img width="1489" alt="image"
src="https://github.com/user-attachments/assets/5257b8e1-a2e7-4143-94c7-bb42b19e4206"
/>

<img width="1499" alt="image"
src="https://github.com/user-attachments/assets/1b521c58-6e12-4d38-b052-93abdac3b4fc"
/>

<img width="1468" alt="image"
src="https://github.com/user-attachments/assets/3010cf0a-8ca7-48d4-9967-328b37fdbc28"
/>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f39ce2aeef)
2025-02-27 11:17:48 +01:00
Roman Khavronenko
1471deee71
vlogs: add metrics to track select queries duration (#8396)
This change would help to track slow queries via alerts or dashboards.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ba2bf9e73a)
2025-02-27 11:17:48 +01:00
Aliaksandr Valialkin
a3ff49def0
lib/logstorage: do not treat a string with leading zeros as a number at tryParseUint64
The "00123" string shouldn't be treated as 123 number.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-02-26 16:07:47 +01:00
Aliaksandr Valialkin
c103a1a7d6
docs/VictoriaLogs/CHANGELOG.md: fix link to elasticsearch data ingestion docs 2025-02-25 21:32:14 +01:00
Aliaksandr Valialkin
7e02baeeea
deployment/docker: update VictoriaLogs Docker image tag from v1.13.0-victorialogs to v1.14.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.14.0-victorialogs
2025-02-25 21:32:14 +01:00
Aliaksandr Valialkin
e9611d18a3
docs/VictoriaLogs/CHANGELOG.md: cut v1.14.0-victorialogs release 2025-02-25 21:32:13 +01:00
Zhu Jiekun
acf0aef973
fix: [stats_query_range] fix inconsistent result caused by shared variable (#8385)
### Describe Your Changes

Previously, `timestamp` variable was shared by goroutines and modified
during `writeBlock`, verified in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8312.

This pull request create new variable within each goroutine to avoid
race condition.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ]  Test on sandbox.

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-25 21:32:12 +01:00
Aliaksandr Valialkin
5941b03b34
docs/VictoriaLogs/FAQ.md: add a question on how to estimate the needed compute resources for the particular workload
(cherry picked from commit 128f6d78ff)
2025-02-25 19:13:31 +01:00
Aliaksandr Valialkin
9e0581533c
lib/logstorage: add le_field and lt_field filters
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.

(cherry picked from commit 30974e7f3f)
2025-02-25 19:13:31 +01:00
Andrii Chubatiuk
6d62620c9d
vlinsert: accept ES ping requests to endpoint without trailing slash (#8354)
### Describe Your Changes

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8353

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 7a1c84b6ec)
2025-02-25 19:13:30 +01:00
Aliaksandr Valialkin
37112ab003
docs/VictoriaLogs/LogsQL.md: typo fixes 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
e6fb17aad6
deployment/docker: update VictoriaLogs Docker image tag from v1.12.0-victorialogs to v1.13.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.13.0-victorialogs
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
283b3f1bfb
docs/VictoriaLogs/CHANGELOG.md: cut v1.13.0-victorialogs 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
3ee4b3ef24
lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
a2d0846e86
lib/logstorage: add an ability to drop duplicate words at unpack_words pipe 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
518ed87a3a
lib/logstorage: rename unpack_tokens to unpack_words pipe
The LogsQL defines a word at https://docs.victoriametrics.com/victorialogs/logsql/#word ,
so it is more natural to use unpack_words instead of unpack_tokens name for the pipe.
2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
1f11bc948e
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
959282090a
lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
aef939dc20
lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-24 15:34:55 +01:00
Aliaksandr Valialkin
afd74d82db
lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-24 15:34:55 +01:00
Aliaksandr Valialkin
bfc65952df
deployment/docker: update VictoriaLogs Docker image tag from v1.11.0-victorialogs to v1.12.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.12.0-victorialogs
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
2dfd6bb689
lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
31a5bb8ce8
docs/VictoriaLogs/CHANGELOG.md: cut v1.12.0-victorialogs 2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
80d173471f
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
a6bf74443a
docs/VictoriaLogs/keyConcepts.md: small clarifications 2025-02-21 12:43:25 +01:00
Aliaksandr Valialkin
28b7684adb
docs/VictoriaLogs/CHANGELOG.md: move the description of the change at ca1d1bc12b to the appropriate release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8315
2025-02-21 12:43:25 +01:00
Andrii Chubatiuk
94bf90842a
app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:12:40 +01:00
Aliaksandr Valialkin
23a1274e4a
deployment: update VictoriaLogs Docker image tag from v1.10.1-victorialogs to v1.11.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.11.0-victorialogs
2025-02-19 17:40:30 +01:00
Aliaksandr Valialkin
d781b5045a
docs/VictoriaLogs/CHANGELOG.md: cut v1.11.0-victorialogs
(cherry picked from commit 77aec3fe77)
2025-02-19 13:30:05 +01:00
Yury Molodov
99cec5cdd3
vmui: optimize group view (#8334)
### Describe Your Changes

This PR introduces several enhancements and optimizations for the Group
view:

1. **Disable hover effect:**
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.

2. **Limit entries per Group:**
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.

3. **Display group info:**
Include the group number in the title along with the total count of
groups to improve clarity and navigation.

4. **Performance improvement:**
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.

Related issue: #8135

<details>
  <summary>Demo UI</summary>

<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>

</details>

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
(cherry picked from commit b2a99d7c6f)
2025-02-19 13:30:04 +01:00
Aliaksandr Valialkin
0a8d52376e
lib/bytesutil: drop ByteBuffer.B when its capacity is bigger than 64KB at Reset
There is little sense in keeping too big buffers - they just waste RAM and do not reduce
the load on GC too much. So it is better dropping such buffers at Reset instead of keeping them around.

(cherry picked from commit b58e2ab214)
2025-02-19 13:30:03 +01:00
Aliaksandr Valialkin
a842114070
lib/logstorage: make sure that the data for every log field is stored in a separate file until the number of files is smaller than 256
This should improve query performance for logs with hundreds of fields (aka wide events).
Previously there was a high chance that the data for multiple log fields is stored in the same file.
This could result in query performance slowdown and/or increased disk read IO,
since the operating system could read unnecessary data for the fields, which aren't used in the query.

Now log fields are guaranteed to be stored in separate files until the number of fields exceeds 256.
After that multiple log fields start sharing files.

(cherry picked from commit 9bb5ba5d2f)
2025-02-19 13:30:02 +01:00
Andrii Chubatiuk
bd269dfde3
docs: disable operator, helm docs sync (#8279)
### Describe Your Changes

exclude operator and helm charts docs sync as these repos will be synced
with vmdocs directly

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 5bdbafb9b6)
2025-02-19 13:29:58 +01:00
Zhu Jiekun
ea30d1d014
app/vlinsert: properly ingest journald logs with single-character name entity
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to  `^[A-Z_][A-Z0-9_]*`.

 It's needed to properly support entities with single-character
names.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
2025-02-17 16:08:49 +01:00
Aliaksandr Valialkin
b306b81966
docs/VictoriaLogs/CHANGELOG.md: typo fix 2025-02-17 15:36:37 +01:00
Aliaksandr Valialkin
a2cac11735
deployment: update VictoriaLogs Docker image tag from v1.10.0-victorialogs to v1.10.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.10.1-victorialogs
2025-02-17 15:36:37 +01:00
Aliaksandr Valialkin
15294ed030
docs/VictoriaLogs/CHANGELOG.md: cut v1.10.1-victorialogs release 2025-02-17 15:36:37 +01:00