Commit graph

9436 commits

Author SHA1 Message Date
Aliaksandr Valialkin
871bfc6ed8
deployment: update Go builder from Go1.23.1 to Go1.23.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.23.3+label%3ACherryPickApproved
2024-11-14 17:21:28 +01:00
Zhu Jiekun
d043c31fe5
docs: [VictoriaLogs] FAQ add logs without msg field (#7487)
### Describe Your Changes

Add FAQs to VictoriaLogs:
- I want to ingest logs without message field, is that possible?
- What if my logs have multiple message fields candidates

Preview:
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/docs/VL-FAQ-empty-msg/docs/VictoriaLogs/FAQ.md#i-want-to-ingest-logs-without-message-field-is-that-possible

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
c15925912c
docs/VictoriaLogs/CHANGELOG.md: cut v0.42.0-victorialogs 2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
75e4a8e64b
lib/logstorage: properly skip filtered out dict values when calculating uniq_values, min, max, row_min and row_max stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
8b287e8da4
lib/logstorage: properly clone field values at values stats function
Previously field values weren't properly cloned, which could lead to garbage output for `values` stats function

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
a7e1575ebe
lib/logstorage: simplify the code for uniq_values stats function a bit
Move the repeated check for an empty value into statsUniqValuesProcessor.updateState() function.
This allow removing duplicate code for this check from statsUniqValuesProcessor.updateState() call sites.
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
8a86575128
app/vlinsert/loki: show the original request body on parse errors
This should simplify debugging.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7490
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
8e0fa84dc8
docs/VictoriaLogs/CHANGELOG.md: refer to the issue related to adding fields to Syslog logs
This is a follow-up for cd60a4c589
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
ab2ce3728b
app/vlinsert/syslog: allow changing the default set of log fields to use as stream fields during syslog data ingestion
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7488
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7480

See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#stream-fields
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
bab5348b8b
app/vlinsert/syslog: add an ability to drop and add fields during data ingestion via Syslog protocol
See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#dropping-fields
and https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#adding-extra-fields
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
5b0b7d509f
lib/logstorage: support for [label1=value1 ... labelN=valueN] syntax inside syslog messages for adding arbitrary labels (fields) to log entries 2024-11-14 17:21:26 +01:00
Aliaksandr Valialkin
29c3a13836
app/vlogscli: allow toggling wrapping long lines with \wrap_long_lines command 2024-11-14 17:21:25 +01:00
Aliaksandr Valialkin
0e9767cd77
docs/VictoriaLogs/README.md: add Security chapter
It is inspired by https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428#discussion_r1831555716

(cherry picked from commit 62e6c9bd6f)
2024-11-08 17:07:58 +01:00
Aliaksandr Valialkin
2818ce29f8
app/vlselect/logsql: add an ability to delay returning matching logs from live tailing via offset query arg
By default the delay equals to 1 second.

While at it, document refresh_interval query arg at /select/logsql/tail endpoint.

Thanks to @Fusl for the idea and the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428

(cherry picked from commit a44787372f)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
a02d26e853
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters
(cherry picked from commit e5537bc64d)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
f82cfa16bf
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.

(cherry picked from commit a98fb495c6)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
a4ea3b87d7
lib/logstorage: optimize query imeediately after its parsing
This eliminates possible bugs related to forgotten Query.Optimize() calls.

This also allows removing optimize() function from pipe interface.

While at it, drop filterNoop inside filterAnd.

(cherry picked from commit 66b2987f49)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
52929c060a
app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint.
This reduces the probability of forgotten call to Query.Optimize().

(cherry picked from commit 0550093802)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
7d078dd591
lib/logstorage: add an ability to add prefix to resulting query field names in join pipe
See https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe

(cherry picked from commit 5a6531b329)
2024-11-08 17:07:56 +01:00
Andrii Chubatiuk
c077a6e0f3
docs: add make command for docs version update (#7430)
### Describe Your Changes

added make target, which updates `{{% available_from "#" %}}` shortcode
to `{{% available_from "$(PKG_TAG)" %}}` if PKG_TAG matches expression
`v.*`. `{{% available_from %}}` shortcode was introduced in
https://github.com/VictoriaMetrics/vmdocs/pull/89 to show a reference to
a version in a changelog since which a feature was introduced

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7376

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 30dd4cdc0d)
2024-11-08 17:07:55 +01:00
nemobis
534e0c11ad
docs: Update RELEX Oy figures (#7491)
Update figures for the existing RELEX Oy case study.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 690328028d)
2024-11-08 17:07:55 +01:00
hagen1778
dfc70b1de3
docs: test available_from shortcode for versioning
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8e0fbb0ed2)
2024-11-08 17:07:55 +01:00
hagen1778
35706bc79d
docs: fix OpenTSDB port for cluster version in API examples
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit acb9f47e8d)
2024-11-08 17:07:55 +01:00
Roman Khavronenko
07ad4f82cd
app/vmauth: properly check for backend health before leastLoaded policy (#7489)
Previously, vmauth could have pick `buMin` as least loaded backend
without checking its status. In result, vmauth could have respond to the
user with an error even if there were healthy backends. That could
happen if healthy backends already had non-zero amount of concurrent
requests executing at the moment of least-loaded backend choosing logic.

Steps to reproduce:
1. Setup vmauth with two backends: healthy and non-healthy
2. Execute a bunch of concurrent requests against vmauth (i.e. Grafana
dash reload)
3. Observe that some requests will fail with message that all backends
are unavailable

Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3061
---
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a0a154511a)
2024-11-08 17:07:54 +01:00
Roman Khavronenko
e60cce54a8
tests: couple applications and test suit (#7476)
* make test suite responisble for stopping apps
* reuse test suite fields to simplify function signatures

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-08 14:49:00 +01:00
Andrii Chubatiuk
f240f12a67
victorialogs: added missing changelog entry about DataDog support (#7464)
### Describe Your Changes

Added missing changelog entry for PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5536

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-08 10:16:24 +01:00
Github Actions
f3d8a00112
Automatic update helm docs from VictoriaMetrics/helm-charts@1d4f6d6 (#7473)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
2024-11-08 10:16:24 +01:00
Github Actions
2e32b8aace
Automatic update operator docs from VictoriaMetrics/operator@ee35f90 (#7474)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-11-08 10:16:24 +01:00
Github Actions
f507331e37
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@0b86b62 (#7477) 2024-11-08 10:16:24 +01:00
Github Actions
bfc9ddd8da
Automatic update Grafana datasource docs from VictoriaMetrics/victoriametrics-datasource@aad9171 (#7478) 2024-11-08 10:16:24 +01:00
Andrii Chubatiuk
d682eb6ab6
dashboards/victorialogs: allow multiple values for instance variable (#7465)
### Describe Your Changes

Related issue https://github.com/VictoriaMetrics/helm-charts/issues/1699

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-08 10:16:23 +01:00
hagen1778
da1e89e190
tests: fix minor typos
* rename vmsingle test to actually match the mask
* swap got/want arguments, as they were misplaced

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f553628f46)
2024-11-07 13:27:38 +01:00
hagen1778
790eab3026
tests: integration tests for vmsingle (#7434)
### Describe Your Changes

This PR continues the implementation of integration tests (#7199). It
adds the support for vm-single:

-    A vmsingle app wrapper has been added
- Sample vmsingle tests that test the VM documentation related to
querying data (#7435)
- The tests use the go-cmp/{cmp,/cmpopts} packages, therefore they have
been added to ./vendor
-    Minor refactoring: data objects have been moved to model.go

Advice on porting things to cluster branch:

- The build rule must include tests that start with TestVmsingle
(similarly to how TestCluster tests are skipped in master branch)
- The build rule must depend on `vmstorage vminsert vmselect` instead of
`victoria-metrics`
- The query_test.go can actually be implemented for cluster as well. To
do this the tests need to be renamed to start with TestCluster and the
tests must instantiace vm{storage,insert,select} instead of vmsingle.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-11-07 13:25:01 +01:00
Phuong Le
9ad958212d
deployment/docker/base: fix the typo that causes InvalidDefaultArgInFrom
(cherry picked from commit 4d383fdb9a)
2024-11-07 13:00:22 +01:00
Aliaksandr Valialkin
4d5f364ad9
docs/VictoriaLogs/LogsQL.md: add missing reference to pipes docs from join pipe docs
This is needed for consistency with other pipe docs

(cherry picked from commit d1af84a49d)
2024-11-07 13:00:22 +01:00
Aliaksandr Valialkin
722873e18b
deployment/docker: update VictoriaLogs from v0.40.0-victorialogs to v0.41.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.41.0-victorialogs

(cherry picked from commit a549240852)
2024-11-07 13:00:21 +01:00
Aliaksandr Valialkin
bf81ef667b
docs/VictoriaLogs/CHANGELOG.md: cut v0.41.0-victorialogs
(cherry picked from commit a4ce80b4ea)
2024-11-07 13:00:21 +01:00
Aliaksandr Valialkin
bcf8a788eb
app/vlselect/vmui: run make vmui-logs-update after 1e1952acf5
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7344
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7184
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7045

(cherry picked from commit bd64c7e3ca)
2024-11-07 13:00:21 +01:00
Yury Molodov
200926c2bc
vmui/logs: add log sorting (#7344)
### Describe Your Changes

add sorting of logs by groups and within each group by time in desc
order. See #7184 and #7045

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
(cherry picked from commit 1e1952acf5)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
1c7d8adc65
app/vlinsert/loki: follow-up for 3aeb1b96a2
- Disallow more than 3 items in Loki line entry, since it must contain two mandatory entries: timestamp and message,
  plus one optional entry - structured metadata. See https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs

- Update references to structured metadata docs in Loki, in order to simplify further maintenance of the code

- Move the change from bugfix to feature at docs/VictoriaLogs/CHANGELOG.md, since VictoriaLogs never supported
  structured metadata over JSON Loki protocol. The support for structured metadata in protobuf Loki protocol
  has been added in ac06569c49 , which has been included in v0.28.0-victorialogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7432

(cherry picked from commit 3d75c39ff4)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
364a2e3e1f
docs/VictoriaLogs: properly sort log fields with floating-point numbers
(cherry picked from commit 42c9183281)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
7a39f526ec
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
(cherry picked from commit 5ed54ebadf)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
83c9d42263
lib/logstorage: add join pipe for joining multiple query results
(cherry picked from commit f9e23bf8e3)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
ab3b4020b4
app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing
(cherry picked from commit 3a5f1019ba)
2024-11-07 13:00:19 +01:00
Evgeniy Negriy
98fe1950a1
app/vmselect: fixes graphite function transformRemoveEmptySeries
Previously it incorrectly applied xFilesFactor, if it's value equal to 0.

 This commit properly handles this case and returns result according to
the graphite documentation:

`xFilesFactor follows the same semantics as in Whisper storage schemas. Setting it to 0 (the default) means that only a single value in the series needs to be non-null for it to be considered non-empty, setting it to 1 means that all values ​​in the series must be non-null. A setting of 0.5 means that at least half the values ​​in the series must be non-null.`

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Evgeniy Negriy <einegriy@avito.ru>
(cherry picked from commit d27dfac5c6)
2024-11-07 13:00:18 +01:00
Zhu Jiekun
e113a40142
app/vlinisert/loki: properly parse json logs with structured metadata
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously,  loki request json parsers rejected log line if tuple size != 2.

This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the  log message stream.

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>

(cherry picked from commit 3aeb1b96a2)
2024-11-07 13:00:18 +01:00
Andrii Chubatiuk
93bc205e05
promql: exclude limit_offset from default by metric name sorting (#7402)
### Describe Your Changes

I don't like this solution, but it works. Other possible solutions
described in an issue

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7068

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a88f896b43)
2024-11-06 15:27:29 +01:00
hagen1778
1cc066a1b2
docs/victorialogs: recommend using separate installations of vmalert for vm and vl
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 05f2e9548d)
2024-11-06 15:27:28 +01:00
hagen1778
a0779a86fa
docs/victorialogs: clarify usage of -rule.defaultRuleType=vlogs
User experience suggests that examples shouldn't have `-rule.defaultRuleType=vlogs` set,
as it may confuse users who run vmalert with their existing rules or only use
rules from examples for testing purposes.

This change is supposed to remove the confusion by removing `-rule.defaultRuleType=vlogs`
from default recommendations and explcitily specifying `type` on group level in examples.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a5f1764171)
2024-11-06 15:27:28 +01:00
Roman Khavronenko
d6eefb86af
docs: fix typos in vlogs rules examples (#7457)
* fix typos in rules definition. Otherwise, they can't pass validation
* add code types for rendered examples

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0390d58a34)
2024-11-06 13:58:20 +01:00