Commit graph

129 commits

Author SHA1 Message Date
Aliaksandr Valialkin
2dd845fa53
lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields
Previously (f1:foo OR f2:bar) was incorrectly returning `foo` token for `f1` and `bar` token for `f2`.
These tokens were used for checking against bloom filter for every data block, so the data block,
which didn't contain simultaneously `foo` token for `f1` field and `bar` token for `f2` field, was skipped.
This was incorrect, since such a block may contain logs matching the original OR filter.

The fix is to return common tokens from `OR`-delimted filters only if these tokens exist at EVERY such filter
for the given field name. If some `OR`-delimited filter misses the given field name, then `OR`-delimited filters
do not contain common tokens, which could be used for checking against bloom filter.

While at it, add more tests covering various edge cases for filters delimited by AND and OR.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6554
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6556
2024-09-05 14:29:50 +02:00
f41gh7
b8bbea8896
docs/changelog: moves victorialogs changes to proper file
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-04 15:36:33 +02:00
f41gh7
8b36529b32
follow-up after 1731c0eabf
* updates change log
* adds VL-Debug http header
* updates doc
* extracts only the first value of http headers for VL-Stream-Fields and VL-Ignore-Fields.
  It makes behaviour the same as Query string args. And allows to easily configure client applications.
  Since most of the client collectors don't support multi value headers.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 19:16:10 +02:00
jackyin
975ed27a76
lib/logstorage: and filter results in unexpected response (#6556)
fix #6554
andfilter shouldn't return orfilter field which result in bloomfilter
return false.

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-09-03 10:17:44 +02:00
hagen1778
00d09d67dc
docs/victorialogs/CHANGELOG.md: mention issue #4750 in the change
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-29 15:09:52 +02:00
Yury Molodov
e35237920a
vmui: add column search in table settings (#6804)
### Describe Your Changes

Add search functionality to the column display settings in the table
 #6668
 

![image](https://github.com/user-attachments/assets/e9bd52c3-6428-4d4f-8b7f-d83dd80b6912)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 16:57:26 +02:00
hagen1778
2dba4165e8
docs: move change from VM changelog to VL changelog
The change for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6764
was added to VM changelog by mistake

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-20 17:16:04 +02:00
jackyin
5f5bc46b3e
vlogs: add select/deselect all button to table settings in UI (#6680)
fix #6668, just add **select all** and "unselect all" func.


https://github.com/user-attachments/assets/0c31385b-def0-4618-aa9c-5ba4bb6f56c3

---------

Co-authored-by: Yury Molodov <yurymolodov@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-08-13 10:20:07 -04:00
Andrii Chubatiuk
35d77a3bed
docs: updated guides structure, removed deprecated sort option (#6767)
### Describe Your Changes

* `sort` param is unused by the current website engine, and was present only for compatibility
with previous website engine. It is time to remove it as it makes no effect
* re-structure guides content into folders to simplify assets management

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-07 16:48:08 +02:00
Yury Molodov
04c2232e45
vmui/logs: add display top streams in the hits graph (#6647)
### Describe Your Changes

- Adds support for displaying the top 5 log streams in the hits graph,
grouping the remaining streams into an "other" label.
   #6545

- Adds options to customize the graph display with bar, line, stepped
line, and points views.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-08-06 16:28:44 +02:00
Zakhar Bessarab
58b6c54da2
app/vlinsert/elasticsearch: add fake response for logstash requests (#6742)
### Describe Your Changes

This is needed in order to support standard Elasticsearch output in
Logstash pipelines.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6660

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-08-06 15:43:33 +02:00
Yury Molodov
e06a19d85f
vmui/logs: improve UI functionality (#6688)
* add a toggle button to the "Group" tab that allows users to expand or collapse all groups at once
* introduce the ability to select a key for grouping logs within the "Group" tab
* display the number of entries within each log group.
* move the Markdown toggle to the general settings panel in the upper left corner.
2024-08-02 15:48:36 +02:00
Yury Molodov
a05317f61f
vmui/logs: add fields for tenant configuration (#6661)
Added fields for configuring AccountID and ProjectID
#6631
2024-08-02 09:57:39 +02:00
Andrii Chubatiuk
88ea9c2fb3
docs: removed redundant 'VictoriaLogs' from title (#6715)
### Describe Your Changes

After breadcrumb was added to docs.victoriametrics.com there's no need
to specify parent page name in a title

<img width="1437" alt="Screenshot 2024-07-27 at 10 20 09"
src="https://github.com/user-attachments/assets/733f41f4-a727-4f52-a7c0-6019edf1b803">

Also added vmdocs to gitignore to avoid committing it

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-29 14:29:15 +02:00
Aliaksandr Valialkin
8cb1822b94
Revert "removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)"
This reverts commit 2e9b1efeb9.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 12:48:13 +02:00
Aliaksandr Valialkin
f357ee57ef
Revert "fixed victorialogs relative links (#6693)"
This reverts commit 683a69d53d.

Reason for revert: relative links in docs are much harder to maintain in consistent state
comparing to absolute links:

- It is non-trivial to figure out the proper relative link path when creating and editing docs.
- Relative links break after moving the doc files to another paths, and it is non-trivial
  to figure which links are broken after that.
- The updated relative links do not work properly right now in the docs.
  For example, the https://docs.victoriametrics.com/victorialogs/quickstart.md#building-from-source-code
  link at https://docs.victoriametrics.com/victorialogs/changelog/ leads to 404 page.

This is documented at https://docs.victoriametrics.com/#images-in-documentation .
2024-07-25 12:48:12 +02:00
Andrii Chubatiuk
683a69d53d
fixed victorialogs relative links (#6693)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 11:26:02 -07:00
Andrii Chubatiuk
2e9b1efeb9
removed unneeded ref shortcodes, updated VM changelog to use relative markdown links (#6691)
### Describe Your Changes

Use relative markdown references, removed `{{< ref >}}` shortcodes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 13:33:37 +03:00
Andrii Chubatiuk
ce4cc4cbb2
view documentation locally (#6677)
### Describe Your Changes

- moved files from root to VictoriaMetrics folder to be able to mount
operator docs and VictoriaMetrics docs independently
- added ability to run website locally

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-24 01:00:31 -07:00
Arkadii Yakovets
fabf0b928e
docs: add spellcheck command (#6562)
### Describe Your Changes

Implement spellcheck command:
  - add cspell configuration files
  - dockerize spellchecking process
  - add Makefile targets
 
This PR adds a standalone `make spellcheck` target to check `docs/*.md` files for spelling
errors. The target process is dockerized to be run in a separate npm environment.

Some `docs/` typo fixes also included.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-11 12:39:42 +02:00
Aliaksandr Valialkin
b173f24041
docs/VictoriaLogs/CHANGELOG.md: cut v0.28.0-victorialogs release 2024-07-10 03:08:16 +02:00
Aliaksandr Valialkin
00c666a6c3
app/vlselect/vmui: run make vmui-logs-update after 662e026279 2024-07-10 00:50:10 +02:00
Aliaksandr Valialkin
aa9bb99527
lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:28 +02:00
Yury Molodov
662e026279
vmui/logs: add spinner to bar chart (#6577)
Add a spinner to the bar chart 

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-09 14:58:48 +02:00
Aliaksandr Valialkin
bbf1f33818
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.1-victorialogs 2024-07-05 01:22:24 +02:00
Aliaksandr Valialkin
c0caa69939
lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings
The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-05 01:22:23 +02:00
Aliaksandr Valialkin
d506e47c5a
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.0-victorialogs 2024-07-02 01:43:46 +02:00
Aliaksandr Valialkin
e11f0aa9ec
app/vlinsert/insertutils: flush the ingested logs from in-memory buffer to storage every second
Previously the in-memory buffer could remain unflushed for long periods of time under low ingestion rate.
The ingested logs weren't visible for search during this time.
2024-07-02 01:38:19 +02:00
Aliaksandr Valialkin
ba6f82069f
app/vlinsert/syslog: add an ability to use log ingestion time as the _time field 2024-07-02 01:38:19 +02:00
Aliaksandr Valialkin
387b3b7fb7
docs/VictoriaLogs/CHANGELOG.md: use new url https://docs.victoriametrics.com/victorialogs/querying/ instead of old one https://docs.victoriametrics.com/VictoriaLogs/querying/ 2024-07-01 16:31:42 +02:00
Aliaksandr Valialkin
839596c00d
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.1-victorialogs 2024-07-01 02:32:51 +02:00
Aliaksandr Valialkin
6bb66cb3e9
lib/logstorage: properly search for the surrounding logs in stream_context pipe
The set of log fields in the found logs may differ from the set of log fields present in the log stream.
So compare only the log fields in the found logs when searching for the matching log entry in the log stream.

While at it, return _stream field in the delimiter log entry, since this field is used by VictoriaLogs Web UI
for grouping logs by log streams.
2024-07-01 02:29:50 +02:00
Aliaksandr Valialkin
353e26cdfc
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.0-victorialogs 2024-07-01 01:49:49 +02:00
Aliaksandr Valialkin
bb0deb7ac4
lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
Aliaksandr Valialkin
dc291d8980
lib/logstorage: add delimiter between log chunks returned from | stream_context pipe 2024-07-01 01:30:37 +02:00
Aliaksandr Valialkin
c9568d45dc
docs/VictoriaLogs: typo fixes 2024-06-28 19:26:24 +02:00
Aliaksandr Valialkin
9877c086b9
docs/VictoriaLogs/CHANGELOG.md: cut v0.25.0-victorialogs 2024-06-28 19:17:49 +02:00
Aliaksandr Valialkin
d4ca651547
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
Aliaksandr Valialkin
d7185f1b77
app/vlinsert/syslog: properly skip empty lines in Syslog protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548
2024-06-28 14:09:28 +02:00
Aliaksandr Valialkin
e8322147e9
app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint
This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545
in order to return top N groups with the biggest number of hits.
2024-06-28 03:08:40 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
Yury Molodov
43342745ac
vmui/logs: fix the update of the relative time range (#6517)
### Describe Your Changes

- Fixed the update of the relative time range when `Execute Query` is
clicked
- Optimized server requests: now, if an error occurs in the `/query`
request, the `/hits` request will not be executed.

#6345 (duplicates: #6440, #6312)
2024-06-26 11:23:22 +02:00
Aliaksandr Valialkin
dff5008392
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:33 +02:00
Aliaksandr Valialkin
0c90eeb388
docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release 2024-06-25 14:59:15 +02:00
Aliaksandr Valialkin
3eacd43fff
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
Aliaksandr Valialkin
9e1c037249
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:53:38 +02:00
Aliaksandr Valialkin
de7450b7e0
lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
Aliaksandr Valialkin
7229dd8c33
lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
Yury Molodov
32fbffedd9
vmui/logs: add bar chart (#6461)
- Added a bar chart displaying the number of log entries over a time
range.
   #6404
- When `_msg` is empty, all fields are displayed in a single line.
- Added double quotes when copying pairs: `key: "value"`.
- Minor style adjustments.
2024-06-18 15:23:21 +02:00
Aliaksandr Valialkin
4e912c3c72
docs/VictoriaLogs/CHANGELOG.md: typo fix in the release date for v0.20.2-victorialogs 2024-06-18 00:28:40 +02:00