Commit graph

678 commits

Author SHA1 Message Date
hagen1778
065a3d068c
docs/victorialogs/vmalert: apply consistent formatting and fix typos
* use unified numeric list;
* apply line width limits;
* remove time filter from quantile examples, as we suggest to
not use time filters.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-03 13:57:30 +01:00
Zakhar Bessarab
6ff61a1c09
docs/all: add refs to Quay mirror for docker images (#8406)
### Describe Your Changes

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4116

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-02-28 11:14:10 +01:00
Aliaksandr Valialkin
bfa9fee424
docs/VictoriaLogs/CHANGELOG.md: typo fix: filter -> filters 2025-02-28 00:09:00 +01:00
Aliaksandr Valialkin
d7431e69fd
deployment/docker: update VictoriaLogs Docker image tag from v1.14.0-victorialogs to v1.15.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.15.0-victorialogs
2025-02-28 00:02:55 +01:00
Aliaksandr Valialkin
0e5e90fe74
docs/VictoriaLogs/CHANGELOG.md: cut v1.15.0-victorialogs release 2025-02-27 22:58:18 +01:00
Aliaksandr Valialkin
744ac496bd
lib/logstorage: add ability to specify field name prefixes inside fields (...) lists passed to pack_json and pack_logfmt pipes 2025-02-27 22:54:18 +01:00
Aliaksandr Valialkin
84d5771b41
lib/logstorage: allow passing * at in(*), contains_any(*) and contains_all(*)
Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for Grafana.

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673
2025-02-27 11:37:43 +01:00
Roman Khavronenko
f39ce2aeef
dashboards: update victorialogs dashboard (#8398)
* stats panels:
* * add ingested logs/bytes over 24h
* * fix correctness compression ratio and disk space usage panels
* overview panels:
* * add requests errors rate, logs rate, query duration panels
* add troubleshooting section:
* * add non-default falgs, dropped logs panels
* update descriptions of the panels and version requirements

Depends on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8396

--------------

<img width="1489" alt="image"
src="https://github.com/user-attachments/assets/5257b8e1-a2e7-4143-94c7-bb42b19e4206"
/>

<img width="1499" alt="image"
src="https://github.com/user-attachments/assets/1b521c58-6e12-4d38-b052-93abdac3b4fc"
/>

<img width="1468" alt="image"
src="https://github.com/user-attachments/assets/3010cf0a-8ca7-48d4-9967-328b37fdbc28"
/>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 11:15:52 +01:00
Roman Khavronenko
ba2bf9e73a
vlogs: add metrics to track select queries duration (#8396)
This change would help to track slow queries via alerts or dashboards.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 11:11:26 +01:00
Aliaksandr Valialkin
ae5e28524e
lib/logstorage: do not treat a string with leading zeros as a number at tryParseUint64
The "00123" string shouldn't be treated as 123 number.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-02-25 22:10:36 +01:00
Aliaksandr Valialkin
b484112f22
docs/VictoriaLogs/CHANGELOG.md: fix link to elasticsearch data ingestion docs 2025-02-25 21:31:16 +01:00
Aliaksandr Valialkin
1da04db33a
deployment/docker: update VictoriaLogs Docker image tag from v1.13.0-victorialogs to v1.14.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.14.0-victorialogs
2025-02-25 21:24:42 +01:00
Aliaksandr Valialkin
ef2415e0c5
docs/VictoriaLogs/CHANGELOG.md: cut v1.14.0-victorialogs release 2025-02-25 21:19:58 +01:00
Zhu Jiekun
521af4d7a3
fix: [stats_query_range] fix inconsistent result caused by shared variable (#8385)
### Describe Your Changes

Previously, `timestamp` variable was shared by goroutines and modified
during `writeBlock`, verified in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8312.

This pull request create new variable within each goroutine to avoid
race condition.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
- [ ]  Test on sandbox.

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-25 20:56:03 +01:00
Aliaksandr Valialkin
128f6d78ff
docs/VictoriaLogs/FAQ.md: add a question on how to estimate the needed compute resources for the particular workload 2025-02-25 18:24:51 +01:00
Aliaksandr Valialkin
30974e7f3f
lib/logstorage: add le_field and lt_field filters
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.
2025-02-25 18:24:50 +01:00
Andrii Chubatiuk
7a1c84b6ec
vlinsert: accept ES ping requests to endpoint without trailing slash (#8354)
### Describe Your Changes

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8353

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-25 15:48:40 +01:00
Aliaksandr Valialkin
b8daa8afb4
docs/VictoriaLogs/LogsQL.md: typo fixes 2025-02-22 22:35:54 +01:00
Aliaksandr Valialkin
e977d6f63a
deployment/docker: update VictoriaLogs Docker image tag from v1.12.0-victorialogs to v1.13.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.13.0-victorialogs
2025-02-22 22:24:48 +01:00
Aliaksandr Valialkin
ce7e408c14
docs/VictoriaLogs/CHANGELOG.md: cut v1.13.0-victorialogs 2025-02-22 21:55:58 +01:00
Aliaksandr Valialkin
f1eac36a80
lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-22 21:55:58 +01:00
Aliaksandr Valialkin
c372e10937
lib/logstorage: add an ability to drop duplicate words at unpack_words pipe 2025-02-22 21:55:57 +01:00
Aliaksandr Valialkin
7da98b540b
lib/logstorage: rename unpack_tokens to unpack_words pipe
The LogsQL defines a word at https://docs.victoriametrics.com/victorialogs/logsql/#word ,
so it is more natural to use unpack_words instead of unpack_tokens name for the pipe.
2025-02-22 21:55:56 +01:00
Aliaksandr Valialkin
dfcfaba374
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-22 21:55:54 +01:00
Aliaksandr Valialkin
d33e24ab9b
lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
cd73c1bafb
lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
d32c697361
lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-22 21:55:52 +01:00
Aliaksandr Valialkin
2318e57bbe
deployment/docker: update VictoriaLogs Docker image tag from v1.11.0-victorialogs to v1.12.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.12.0-victorialogs
2025-02-20 22:41:07 +01:00
Aliaksandr Valialkin
1ea3f72d50
lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-20 22:36:09 +01:00
Aliaksandr Valialkin
384a757c3d
docs/VictoriaLogs/CHANGELOG.md: cut v1.12.0-victorialogs 2025-02-20 20:49:54 +01:00
Aliaksandr Valialkin
ffbd0ebbae
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-20 19:04:51 +01:00
Aliaksandr Valialkin
dabe9ff389
docs/VictoriaLogs/keyConcepts.md: small clarifications 2025-02-20 19:04:50 +01:00
Aliaksandr Valialkin
531ecdfa0d
docs/VictoriaLogs/CHANGELOG.md: move the description of the change at ca1d1bc12b to the appropriate release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8315
2025-02-20 16:14:19 +01:00
Andrii Chubatiuk
58a80a9718
app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:01:54 +01:00
Aliaksandr Valialkin
8d79b28b37
deployment: update VictoriaLogs Docker image tag from v1.10.1-victorialogs to v1.11.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.11.0-victorialogs
2025-02-19 13:34:11 +01:00
Aliaksandr Valialkin
77aec3fe77
docs/VictoriaLogs/CHANGELOG.md: cut v1.11.0-victorialogs 2025-02-19 12:46:25 +01:00
Yury Molodov
b2a99d7c6f
vmui: optimize group view (#8334)
### Describe Your Changes

This PR introduces several enhancements and optimizations for the Group
view:

1. **Disable hover effect:**  
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.

2. **Limit entries per Group:**  
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.

3. **Display group info:**  
Include the group number in the title along with the total count of
groups to improve clarity and navigation.

4. **Performance improvement:**  
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.
  
Related issue: #8135  


<details>
  <summary>Demo UI</summary>
  
<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>

</details>
  
### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-19 12:42:54 +01:00
Aliaksandr Valialkin
b58e2ab214
lib/bytesutil: drop ByteBuffer.B when its capacity is bigger than 64KB at Reset
There is little sense in keeping too big buffers - they just waste RAM and do not reduce
the load on GC too much. So it is better dropping such buffers at Reset instead of keeping them around.
2025-02-19 12:35:06 +01:00
Aliaksandr Valialkin
9bb5ba5d2f
lib/logstorage: make sure that the data for every log field is stored in a separate file until the number of files is smaller than 256
This should improve query performance for logs with hundreds of fields (aka wide events).
Previously there was a high chance that the data for multiple log fields is stored in the same file.
This could result in query performance slowdown and/or increased disk read IO,
since the operating system could read unnecessary data for the fields, which aren't used in the query.

Now log fields are guaranteed to be stored in separate files until the number of fields exceeds 256.
After that multiple log fields start sharing files.
2025-02-19 01:48:14 +01:00
Andrii Chubatiuk
5bdbafb9b6
docs: disable operator, helm docs sync (#8279)
### Describe Your Changes

exclude operator and helm charts docs sync as these repos will be synced
with vmdocs directly

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-18 14:24:14 +02:00
Zhu Jiekun
ca1d1bc12b
app/vlinsert: properly ingest journald logs with single-character name entity
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to  `^[A-Z_][A-Z0-9_]*`.

 It's needed to properly support entities with single-character
names.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
2025-02-17 15:49:20 +01:00
Aliaksandr Valialkin
9619a84af6
docs/VictoriaLogs/CHANGELOG.md: typo fix 2025-02-14 16:20:06 +01:00
Aliaksandr Valialkin
35f3570abf
deployment: update VictoriaLogs Docker image tag from v1.10.0-victorialogs to v1.10.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.10.1-victorialogs
2025-02-14 15:39:20 +01:00
Aliaksandr Valialkin
a9e8661a67
docs/VictoriaLogs/CHANGELOG.md: cut v1.10.1-victorialogs release 2025-02-14 15:35:37 +01:00
Aliaksandr Valialkin
f768c15bb2
lib/logstorage: make sure that chunkedAllocator is isn't used from concurrently running goroutines
This is needed in order to avoid data races
2025-02-14 15:34:08 +01:00
Aliaksandr Valialkin
95422e8f7a
lib/logstorage: ensure that statsProcessor.updateStatsForAllRows() is called on non-empty blockResult
This eliminates a class of potential bugs with incorrect stats calculations when an additional filter
is applied to the blockResult before passing it to the stats function, and this filter removes
all the rows from blockResult.
2025-02-14 15:34:07 +01:00
Aliaksandr Valialkin
40bde6bd28
lib/logstorage: properly initialize minValue and maxValue at pipeLenProcessorShard and pipeHashProcessorShard
Previously this could result in incorrect 0 result of min() stats function applied to the len() results.

This is a follow-up for eddeccfcfb
2025-02-14 15:34:07 +01:00
Aliaksandr Valialkin
6d7f82ddb6
docs/VictoriaLogs/data-ingestion: remove configuration examples for ingesting logs into VictoriaLogs via Loki JSON protocol
There are many subtle issues while ingesting logs from popular log shippers into VictoriaLogs via Loki JSON protocol.
For example, the common issue is that structured logs are ingested as a JSON string at _msg field.
This is not what most users expect - they expect that fields in structured logs are ingested as separate log fields.
It is better removing examples with configs for Loki JSON protocol, since other supported protocols work much better
without any issues. This should reduce the confusion level for new users, who try Loki protocol and hit its issues.

- Elasticsearch ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api )
- JSON stream ( https://docs.victoriametrics.com/victorialogs/data-ingestion/#json-stream-api )
2025-02-13 18:16:51 +01:00
Aliaksandr Valialkin
98241cfa78
docs/VictoriaLogs/README.md: mention high performance and resource efficiency points to the top of VictoriaLogs features 2025-02-13 15:55:17 +01:00
Aliaksandr Valialkin
815ff805e6
docs/VictoriaLogs/LogsQL.md: add a chapter about subquery filters 2025-02-13 09:44:25 +01:00