Commit graph

235 commits

Author SHA1 Message Date
Yury Molodov
10c42668a1
vmui/logs: optimize memory consumption (#7524)
### Describe Your Changes

- **Memory Optimization**: Reduced memory consumption on the "Group" and
"JSON" tabs by approximately 30%.

- **Table Pagination**: Added pagination to the "Table" view with an
option to select the number of rows displayed (from 10 to 1000 items per
page, with a default of 1000). This change significantly reduced memory
usage by approximately 75%.

Related to #7185

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-27 13:49:06 +01:00
Aliaksandr Valialkin
99607e2f3b
docs/VictoriaLogs/CHANGELOG.md: mention that VictoriaLogs gained all the planned features 2024-11-12 19:07:09 +01:00
Aliaksandr Valialkin
1bf58b2f13
docs/VictoriaLogs: cut v1.0.0-victorialogs 2024-11-12 18:31:59 +01:00
Aliaksandr Valialkin
0eb3a0a902
docs/VictoriaLogs/CHANGELOG.md: cut v0.42.0-victorialogs 2024-11-08 23:29:18 +01:00
Aliaksandr Valialkin
546bf7d579
lib/logstorage: properly skip filtered out dict values when calculating uniq_values, min, max, row_min and row_max stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-08 23:21:21 +01:00
Aliaksandr Valialkin
2f1ce74d97
lib/logstorage: properly clone field values at values stats function
Previously field values weren't properly cloned, which could lead to garbage output for `values` stats function

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-08 23:16:42 +01:00
Aliaksandr Valialkin
342f84c569
app/vlinsert/loki: show the original request body on parse errors
This should simplify debugging.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7490
2024-11-08 22:00:58 +01:00
Aliaksandr Valialkin
b7c290ea5e
docs/VictoriaLogs/CHANGELOG.md: refer to the issue related to adding fields to Syslog logs
This is a follow-up for cd60a4c589
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354
2024-11-08 21:50:14 +01:00
Aliaksandr Valialkin
4f0bec6f03
app/vlinsert/syslog: allow changing the default set of log fields to use as stream fields during syslog data ingestion
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7488
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7480

See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#stream-fields
2024-11-08 21:21:08 +01:00
Aliaksandr Valialkin
cd60a4c589
app/vlinsert/syslog: add an ability to drop and add fields during data ingestion via Syslog protocol
See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#dropping-fields
and https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#adding-extra-fields
2024-11-08 20:57:59 +01:00
Aliaksandr Valialkin
63c76b9b27
lib/logstorage: support for [label1=value1 ... labelN=valueN] syntax inside syslog messages for adding arbitrary labels (fields) to log entries 2024-11-08 19:57:22 +01:00
Aliaksandr Valialkin
f55791f20b
app/vlogscli: allow toggling wrapping long lines with \wrap_long_lines command 2024-11-08 17:08:50 +01:00
Aliaksandr Valialkin
a44787372f
app/vlselect/logsql: add an ability to delay returning matching logs from live tailing via offset query arg
By default the delay equals to 1 second.

While at it, document refresh_interval query arg at /select/logsql/tail endpoint.

Thanks to @Fusl for the idea and the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428
2024-11-08 16:43:55 +01:00
Aliaksandr Valialkin
e5537bc64d
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters 2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
a98fb495c6
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.
2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin
5a6531b329
lib/logstorage: add an ability to add prefix to resulting query field names in join pipe
See https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe
2024-11-08 16:43:53 +01:00
Andrii Chubatiuk
b399f1c656
victorialogs: added missing changelog entry about DataDog support (#7464)
### Describe Your Changes

Added missing changelog entry for PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5536

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-07 14:26:02 +01:00
Aliaksandr Valialkin
a4ce80b4ea
docs/VictoriaLogs/CHANGELOG.md: cut v0.41.0-victorialogs 2024-11-06 20:10:21 +01:00
Yury Molodov
1e1952acf5
vmui/logs: add log sorting (#7344)
### Describe Your Changes

add sorting of logs by groups and within each group by time in desc
order. See #7184 and #7045

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-11-06 19:33:35 +01:00
Aliaksandr Valialkin
3d75c39ff4
app/vlinsert/loki: follow-up for 3aeb1b96a2
- Disallow more than 3 items in Loki line entry, since it must contain two mandatory entries: timestamp and message,
  plus one optional entry - structured metadata. See https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs

- Update references to structured metadata docs in Loki, in order to simplify further maintenance of the code

- Move the change from bugfix to feature at docs/VictoriaLogs/CHANGELOG.md, since VictoriaLogs never supported
  structured metadata over JSON Loki protocol. The support for structured metadata in protobuf Loki protocol
  has been added in ac06569c49 , which has been included in v0.28.0-victorialogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7432
2024-11-06 19:23:38 +01:00
Aliaksandr Valialkin
42c9183281
docs/VictoriaLogs: properly sort log fields with floating-point numbers 2024-11-06 18:56:13 +01:00
Aliaksandr Valialkin
5ed54ebadf
lib/logstorage: add block_stats pipe for analyzing per-block storage stats 2024-11-06 18:55:45 +01:00
Aliaksandr Valialkin
f9e23bf8e3
lib/logstorage: add join pipe for joining multiple query results 2024-11-06 18:53:29 +01:00
Aliaksandr Valialkin
3a5f1019ba
app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing 2024-11-06 18:53:28 +01:00
Zhu Jiekun
3aeb1b96a2
app/vlinisert/loki: properly parse json logs with structured metadata
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously,  loki request json parsers rejected log line if tuple size != 2.

This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the  log message stream.


related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-06 17:25:05 +01:00
Arie Heinrich
2e8f420d84
docs: spelling fixes (#7420)
### Describe Your Changes

Christmas is early and you get the first present in the shape of
spelling fixes.
Sorry for the big amount :)

### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-05 16:44:23 +01:00
Aliaksandr Valialkin
4478e48eb6
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445
2024-11-01 20:06:17 +01:00
Aliaksandr Valialkin
1a7b55009b
docs/VictoriaLogs/CHANGELOG.md: cut v0.40.0-victorialogs release 2024-10-30 23:38:06 +01:00
Aliaksandr Valialkin
c5d08d317c
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Aliaksandr Valialkin
2e635a42d8
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162
2024-10-30 22:29:52 +01:00
Aliaksandr Valialkin
7603446850
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365
2024-10-30 17:59:49 +01:00
hagen1778
6494606924
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-30 16:06:57 +01:00
Aliaksandr Valialkin
13822f6d11
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release 2024-10-30 15:06:00 +01:00
Aliaksandr Valialkin
16ee470da6
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.
2024-10-30 14:59:38 +01:00
Aliaksandr Valialkin
ed73f8350b
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.
2024-10-30 14:17:33 +01:00
Aliaksandr Valialkin
102e9d4f4e
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.
2024-10-30 13:39:28 +01:00
Aliaksandr Valialkin
c963d7d10d
docs/VictoriaLogs/CHANGELOG.md: remove unnededed with prefix in front of rank at top pipe example 2024-10-29 18:27:40 +01:00
Aliaksandr Valialkin
12223cf5d0
docs/VictoriaLogs/CHANGELOG.md: cut v0.38.0 release 2024-10-29 18:08:36 +01:00
Aliaksandr Valialkin
3c06d083ea
lib/logstorage: add an ability to return rank from top pipe results 2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
7a62eefa34
lib/logstorage: dynamically adjust the number of (bloom, values) shards in a part depending on the number of non-const columns
This allows reducing the amounts of data, which must be read during queries over logs with big number of fields (aka "wide events").
This, in turn, improves query performance when the data, which needs to be scanned during the query, doesn't fit OS page cache.
2024-10-29 16:44:45 +01:00
Aliaksandr Valialkin
8d968acd0a
lib/logstorage: avoid reading columnsHeader data when field_values pipe is applied directly to log filters
This improves performance of `field_values` pipe when it is applied to large number of data blocks.
This also improves performance of /select/logsql/field_values HTTP API.
2024-10-29 16:44:44 +01:00
Andrii Chubatiuk
7e60afb6fc
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:36:33 +01:00
Yury Molodov
dd89745a34
vmui/logs: fix query and limit update issue (#7294)
### Describe Your Changes

Fixes issues with incorrect updating of query and limit fields, and
resolves the problem where the display tab resets.

Related issue: #7279 and #7290

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-25 09:32:20 +02:00
Aliaksandr Valialkin
51cd3ba02b
docs/VictoriaLogs/CHANGELOG.md: cut v0.37.0-victorialogs release 2024-10-18 02:32:37 +02:00
Yury Molodov
423df09d7d
vmui/logs: add ability to hide hits chart (#7206)
### Describe Your Changes

**Added ability to hide the hits chart**

- Users can now hide or show the hits chart by clicking the "eye" icon
located in the upper-right corner of the chart.
- When the chart is hidden, it will stop sending requests to
`/select/logsql/hits`.
- Upon displaying the chart again, it will automatically refresh. If a
relative time range is set, the chart will update according to the time
period of the logs currently being displayed.

**Hits chart visible:**

![image](https://github.com/user-attachments/assets/577e877b-6417-4b83-8d84-c55e3d39864a)

**Hits chart hidden:**

![image](https://github.com/user-attachments/assets/068b1143-d140-4d72-8d65-663900124f32)

Related issue: #7117

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-18 02:30:56 +02:00
Yury Molodov
36a86c3aaf
vmui/logs: fix display of hits chart (#7167)
### Describe Your Changes

Fixed the display of hits chart in VictoriaLogs.
See #7133

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-18 02:28:23 +02:00
Aliaksandr Valialkin
064b9a6314
docs/VictoriaLogs/CHANGELOG.md: remove "index.html" trailer from the link to docs for the sake of consistency with other links to docs
This is a follow-up for 3538869942
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7252
2024-10-18 02:26:01 +02:00
Aliaksandr Valialkin
2023f017b1
lib/logstorage: optimize performance for queries, which select all the log fields for logs containing hundreds of log fields (aka "wide events")
Unpack the full columnsHeader block instead of unpacking meta-information per each individual column
when the query, which selects all the columns, is executed. This improves performance when scanning
logs with big number of fields.
2024-10-18 02:22:42 +02:00
Aliaksandr Valialkin
78c6fb0883
lib/logstorage: improve performance of top and field_values pipes on systems with many CPU cores
- Parallelize mering of per-CPU results.
- Parallelize writing the results to the next pipe.
2024-10-18 02:22:42 +02:00
Aliaksandr Valialkin
c4b2fdff70
lib/logstorage: optimize 'stats by(...)' calculations for by(...) fields with millions of unique values on multi-CPU systems
- Parallelize merging of per-CPU `stats by(...)` result shards.
- Parallelize writing `stats by(...)` results to the next pipe.
2024-10-18 02:22:41 +02:00