Commit graph

8 commits

Author SHA1 Message Date
Zhu Jiekun
ea30d1d014
app/vlinsert: properly ingest journald logs with single-character name entity
This commit changes journald ingestion validation regex:
from `^[A-Z_][A-Z0-9_]+` to  `^[A-Z_][A-Z0-9_]*`.

 It's needed to properly support entities with single-character
names.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
2025-02-17 16:08:49 +01:00
Aliaksandr Valialkin
66b5c619fa
app/vlinsert: add a link to the pull request at systemd repository, which enables compression support
This should simplify maintenance of this code in the future.
While at it, clarify the change at the docs/VictoriaLogs/CHANGELOG.md.

This is a follow-up commit for 3c9f9f49b0.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8264
Updates https://github.com/systemd/systemd/pull/34822
2025-02-12 22:10:05 +01:00
Andrii Chubatiuk
6254810e12
app/vlinsert: add journald content negotiation, which enables compression on a client
Adding Accept-Encoding response header to support content negotiation,
which was introduced in [this
PR](https://github.com/systemd/systemd/pull/34822) and enables
compression on journald

(cherry picked from commit 3c9f9f49b0)
2025-02-12 12:47:42 +01:00
Aliaksandr Valialkin
bd37715c7b
app/vlinsert: use default set of log stream fields for Loki and OpenTelemetry protocols if _stream_fields query arg is empty
Loki protocol supports a list of log stream labels - see https://grafana.com/docs/loki/latest/get-started/labels/

OpenTelemetry protocol also supports a list of log stream labels, which are named resource attributes there.
See https://opentelemetry.io/docs/concepts/resources/#semantic-attributes-with-sdk-provided-default-value

Simplify logs' ingestion into VictoriaLogs for these protocols by allowing the data ingestion without
the need to specify _stream_fields query arg or VL-Stream-Fields HTTP header. In this case the upstream log stream fields
are used during data ingestion. The set of log stream fields can be overriden via _stream_fields query arg
and via VL-Stream-Fields HTTP header if needed.

Thanks to @AndrewChubatiuk for the initial idea and implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7554
2024-12-05 15:16:53 +01:00
Aliaksandr Valialkin
955b83bb9e
app/vlinsert: track vl_rows_ingested_total metric in a single place
Previously vl_rows_ingested_total metric was tracked individually per each supported data ingestion protocols.
It is better from maintainability PoV tracking this metric consistently in a single place - at logMessageProcessor.AddRow() function
in the same way as vl_bytes_ingested_total metric is tracked.

This is a follow-up for 50bfa689c9
2024-12-05 15:16:52 +01:00
Aliaksandr Valialkin
b2555491bb
app/vlinsert: expose vl_bytes_ingested_total metric
This metric tracks an approximate amounts of bytes processed when parsing the ingested logs.
The metric is exposed individually per every supported data ingestion protocol. The protocol name
is exposed via "type" label in order to be consistent with vl_rows_ingested_total metric.

Thanks to @tenmozes for the initial idea and implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7682

While at it, remove the unneeded "format" label from vl_rows_ingested_total metric.
The "type" label must be enough for encoding the data ingestion format.
2024-11-30 17:26:17 +01:00
Aliaksandr Valialkin
8baa5177aa
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.

(cherry picked from commit ed73f8350b)
2024-10-30 15:19:52 +01:00
Andrii Chubatiuk
da14c7b26e
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:42:41 +01:00