Commit graph

460 commits

Author SHA1 Message Date
Aliaksandr Valialkin
0e9767cd77
docs/VictoriaLogs/README.md: add Security chapter
It is inspired by https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428#discussion_r1831555716

(cherry picked from commit 62e6c9bd6f)
2024-11-08 17:07:58 +01:00
Aliaksandr Valialkin
2818ce29f8
app/vlselect/logsql: add an ability to delay returning matching logs from live tailing via offset query arg
By default the delay equals to 1 second.

While at it, document refresh_interval query arg at /select/logsql/tail endpoint.

Thanks to @Fusl for the idea and the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428

(cherry picked from commit a44787372f)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
a02d26e853
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters
(cherry picked from commit e5537bc64d)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
f82cfa16bf
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.

(cherry picked from commit a98fb495c6)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
7d078dd591
lib/logstorage: add an ability to add prefix to resulting query field names in join pipe
See https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe

(cherry picked from commit 5a6531b329)
2024-11-08 17:07:56 +01:00
Andrii Chubatiuk
c077a6e0f3
docs: add make command for docs version update (#7430)
### Describe Your Changes

added make target, which updates `{{% available_from "#" %}}` shortcode
to `{{% available_from "$(PKG_TAG)" %}}` if PKG_TAG matches expression
`v.*`. `{{% available_from %}}` shortcode was introduced in
https://github.com/VictoriaMetrics/vmdocs/pull/89 to show a reference to
a version in a changelog since which a feature was introduced

related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7376

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 30dd4cdc0d)
2024-11-08 17:07:55 +01:00
Andrii Chubatiuk
f240f12a67
victorialogs: added missing changelog entry about DataDog support (#7464)
### Describe Your Changes

Added missing changelog entry for PR
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5536

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-11-08 10:16:24 +01:00
Github Actions
f507331e37
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@0b86b62 (#7477) 2024-11-08 10:16:24 +01:00
Aliaksandr Valialkin
4d5f364ad9
docs/VictoriaLogs/LogsQL.md: add missing reference to pipes docs from join pipe docs
This is needed for consistency with other pipe docs

(cherry picked from commit d1af84a49d)
2024-11-07 13:00:22 +01:00
Aliaksandr Valialkin
722873e18b
deployment/docker: update VictoriaLogs from v0.40.0-victorialogs to v0.41.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.41.0-victorialogs

(cherry picked from commit a549240852)
2024-11-07 13:00:21 +01:00
Aliaksandr Valialkin
bf81ef667b
docs/VictoriaLogs/CHANGELOG.md: cut v0.41.0-victorialogs
(cherry picked from commit a4ce80b4ea)
2024-11-07 13:00:21 +01:00
Yury Molodov
200926c2bc
vmui/logs: add log sorting (#7344)
### Describe Your Changes

add sorting of logs by groups and within each group by time in desc
order. See #7184 and #7045

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
(cherry picked from commit 1e1952acf5)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
1c7d8adc65
app/vlinsert/loki: follow-up for 3aeb1b96a2
- Disallow more than 3 items in Loki line entry, since it must contain two mandatory entries: timestamp and message,
  plus one optional entry - structured metadata. See https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs

- Update references to structured metadata docs in Loki, in order to simplify further maintenance of the code

- Move the change from bugfix to feature at docs/VictoriaLogs/CHANGELOG.md, since VictoriaLogs never supported
  structured metadata over JSON Loki protocol. The support for structured metadata in protobuf Loki protocol
  has been added in ac06569c49 , which has been included in v0.28.0-victorialogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7432

(cherry picked from commit 3d75c39ff4)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
364a2e3e1f
docs/VictoriaLogs: properly sort log fields with floating-point numbers
(cherry picked from commit 42c9183281)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
7a39f526ec
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
(cherry picked from commit 5ed54ebadf)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
83c9d42263
lib/logstorage: add join pipe for joining multiple query results
(cherry picked from commit f9e23bf8e3)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
ab3b4020b4
app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing
(cherry picked from commit 3a5f1019ba)
2024-11-07 13:00:19 +01:00
Zhu Jiekun
e113a40142
app/vlinisert/loki: properly parse json logs with structured metadata
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously,  loki request json parsers rejected log line if tuple size != 2.

This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the  log message stream.

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>

(cherry picked from commit 3aeb1b96a2)
2024-11-07 13:00:18 +01:00
hagen1778
1cc066a1b2
docs/victorialogs: recommend using separate installations of vmalert for vm and vl
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 05f2e9548d)
2024-11-06 15:27:28 +01:00
hagen1778
a0779a86fa
docs/victorialogs: clarify usage of -rule.defaultRuleType=vlogs
User experience suggests that examples shouldn't have `-rule.defaultRuleType=vlogs` set,
as it may confuse users who run vmalert with their existing rules or only use
rules from examples for testing purposes.

This change is supposed to remove the confusion by removing `-rule.defaultRuleType=vlogs`
from default recommendations and explcitily specifying `type` on group level in examples.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a5f1764171)
2024-11-06 15:27:28 +01:00
Roman Khavronenko
d6eefb86af
docs: fix typos in vlogs rules examples (#7457)
* fix typos in rules definition. Otherwise, they can't pass validation
* add code types for rendered examples

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0390d58a34)
2024-11-06 13:58:20 +01:00
Andrii Chubatiuk
23dcec3911
vlinsert: support datadog logs
This commit adds the following changes:

- Added support to push datadog logs with examples of how to ingest data
using Vector and Fluentbit
- Updated VictoriaLogs examples directory structure to have single
container image for victorialogs, agent (fluentbit, vector, etc) but
multiple configurations for different protocols

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632

(cherry picked from commit e0930687f1)
2024-11-06 13:58:16 +01:00
Arie Heinrich
7a819ae5ce
docs: spelling fixes (#7420)
### Describe Your Changes

Christmas is early and you get the first present in the shape of
spelling fixes.
Sorry for the big amount :)

### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2e8f420d84)
2024-11-06 13:58:16 +01:00
Zakhar Bessarab
48d3052901
docs/CHANGELOG.md: cut v1.106.0
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 11:14:18 -03:00
Aliaksandr Valialkin
cf8a971421
docs/VictoriaLogs/querying: refer to vlogscli from docs for /select/logsql/query and /select/logsql/tail HTTP endpoints
These endpoints are easier to query via vlogscli

(cherry picked from commit 5733e56e40)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
c8213b30af
docs/VictoriaLogs/data-ingestion: mention Grafana Agent and Grafana Alloy additionally to Promtail as log collectors for Grafana Loki
(cherry picked from commit 37a58677fa)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
5e7f0eb976
docs/VictoriaLogs/LogsQL.md: remove misleading information that min and max stats functions work only for numeric values
These functions work for both numeric and string values after ad505a7a9a

(cherry picked from commit c7e242b8ce)
2024-11-04 10:23:18 -03:00
Aliaksandr Valialkin
43998ff82b
docs/VictoriaLogs/LogsQL.md: typo fix in the link to description for sum_len stats function: sum-len-stats -> sum_len-stats
This is a follow-up for 364f084b43

(cherry picked from commit a86df52adf)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
fced48d540
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445

(cherry picked from commit 4478e48eb6)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
21d2c417da
docs/VictoriaLogs/keyConcepts.md: remove duplicate sentence about missing timezone information in the ingested _time field values
(cherry picked from commit b3e2db5647)
2024-11-04 10:23:15 -03:00
Github Actions
5ad662dd0f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3ecaebd (#7395)
(cherry picked from commit 955f3660de)
2024-10-31 14:11:17 +01:00
Aliaksandr Valialkin
dd6781b179
deployment/docker: update VictoriaLogs from v0.39.0-victorialogs to v0.40.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.40.0-victorialogs

(cherry picked from commit 24b6e117dd)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
4895ec79c5
docs/VictoriaLogs/CHANGELOG.md: cut v0.40.0-victorialogs release
(cherry picked from commit 1a7b55009b)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
2d1bc39fe5
docs/VictoriaLogs/README.md: follow-up for 0f6b9e9490: consistently use canonical url for Grafana dashboards
See also 0a5ffb3bc1

(cherry picked from commit f88e2ae9fb)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
cce047c27a
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886

(cherry picked from commit 0f6b9e9490)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
1ea65d662f
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit c5d08d317c)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
b74bcb7886
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit 2e635a42d8)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
fb7f8dec60
docs/VictoriaLogs/querying/README.md: typo fix after f2cd284cf4a7429848fb4be1522483a9b93a43e7: Alls -> All
(cherry picked from commit f62502a943)
2024-10-31 14:11:07 +01:00
Aliaksandr Valialkin
0c657a95dc
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365

(cherry picked from commit 7603446850)
2024-10-31 14:11:07 +01:00
hagen1778
0e6ed4171b
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6494606924)
2024-10-30 16:44:51 +01:00
Aliaksandr Valialkin
50741d548e
deployment/docker: update VictoriaLogs from v0.38.0-victorialogs to v0.39.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.39.0-victorialogs

(cherry picked from commit 25bca94d04)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
310005f063
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release
(cherry picked from commit 13822f6d11)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
b4f5e5c1b8
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.

(cherry picked from commit 16ee470da6)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
44468e8d8b
docs/VictoriaLogs/data-ingestion/README.md: formatting fixes after 8b36529b32
- Remove leading whitespace from the first lines in 'HTTP parameters' chapter.
  This whitespace isn't needed for the markdown formatting.

- Add leading whitespace for the second sentence in the list bullet describing AccountID and ProjectID HTTP headers.
  This fixes markdown formatting for this list bullet.

(cherry picked from commit 96466562b6)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
8baa5177aa
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.

(cherry picked from commit ed73f8350b)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
bf243df9ce
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.

(cherry picked from commit 102e9d4f4e)
2024-10-30 15:19:51 +01:00
hagen1778
bdfef9700d
docs: add frequently asked questions to vmalert integration with vlogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2c93353173)
2024-10-30 13:13:04 +01:00
hagen1778
8d64847f5e
docs: mark alerting feature as done in VictoriaLogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b789a9dc83)
2024-10-30 13:13:04 +01:00
hagen1778
f923bcba5e
docs: update version placeholder to point to the next version
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 77b690ab26)
2024-10-30 13:13:03 +01:00
Artem Navoiev
19327643b4
docs: logs vmalert add identifier
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 3473440d26)
2024-10-30 09:52:54 +01:00