Aliaksandr Valialkin
ac06569c49
app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages
2024-07-10 03:05:17 +02:00
Aliaksandr Valialkin
aa9bb99527
lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API
2024-07-10 00:39:28 +02:00
Aliaksandr Valialkin
3c02937a34
all: consistently use 'any' instead of 'interface{}'
...
'any' type is supported starting from Go1.18. Let's consistently use it
instead of 'interface{}' type across the code base, since `any` is easier to read than 'interface{}'.
2024-07-10 00:20:37 +02:00
Aliaksandr Valialkin
a9525da8a4
lib: consistently use f-tests instead of table-driven tests
...
This makes easier to read and debug these tests. This also reduces test lines count by 15% from 3K to 2.5K
See https://itnext.io/f-tests-as-a-replacement-for-table-driven-tests-in-go-8814a8b19e9e
While at it, consistently use t.Fatal* instead of t.Error*, since t.Error* usually leads
to more complicated and fragile tests, while it doesn't bring any practical benefits over t.Fatal*.
2024-07-09 22:40:50 +02:00
Aliaksandr Valialkin
c0caa69939
lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings
...
The strconv.AppendQuote improperly encodes special chars such as \x1b . They must be encoded as \u001b .
See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-05 01:22:23 +02:00
Aliaksandr Valialkin
3b6c78c26c
lib/logstorage: allow writing after N in front of before N at stream_context pipe
2024-07-02 01:38:20 +02:00
Aliaksandr Valialkin
6bb66cb3e9
lib/logstorage: properly search for the surrounding logs in stream_context pipe
...
The set of log fields in the found logs may differ from the set of log fields present in the log stream.
So compare only the log fields in the found logs when searching for the matching log entry in the log stream.
While at it, return _stream field in the delimiter log entry, since this field is used by VictoriaLogs Web UI
for grouping logs by log streams.
2024-07-01 02:29:50 +02:00
Aliaksandr Valialkin
bb0deb7ac4
lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax
2024-07-01 01:44:17 +02:00
Aliaksandr Valialkin
dc291d8980
lib/logstorage: add delimiter between log chunks returned from | stream_context pipe
2024-07-01 01:30:37 +02:00
Aliaksandr Valialkin
d4ca651547
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs
2024-06-28 19:14:29 +02:00
Aliaksandr Valialkin
0730f1324d
lib/logstorage: it is safe using | unroll pipe in live tailing
...
`| unroll` pipe can make multiple copies of rows from the input row.
This doesn't break live tailing, so allow `| unroll` pipe in live tailing.
2024-06-27 19:44:57 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress
2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin
dff5008392
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage
2024-06-25 17:30:33 +02:00
Aliaksandr Valialkin
3eacd43fff
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data
2024-06-25 14:53:39 +02:00
Aliaksandr Valialkin
9e1c037249
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
...
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.
While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:53:38 +02:00
Aliaksandr Valialkin
7252c5d258
lib/logstorage: make golangci-lint happy
2024-06-25 03:04:21 +02:00
Aliaksandr Valialkin
de7450b7e0
lib/logstorage: work-in-progress
2024-06-24 23:27:12 +02:00
Aliaksandr Valialkin
7229dd8c33
lib/logstorage: work-in-progress
2024-06-20 03:10:08 +02:00
Aliaksandr Valialkin
e498fa6960
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports
2024-06-17 23:16:34 +02:00
Aliaksandr Valialkin
2b6a634ec0
lib/logstorage: work-in-progress
2024-06-17 12:13:18 +02:00
Aliaksandr Valialkin
8f5dc966f6
lib/logstorage: work-in-progress
2024-06-11 17:50:32 +02:00
Aliaksandr Valialkin
0521e58a09
lib/logstorage: work-in-progress
2024-06-10 18:42:19 +02:00
Aliaksandr Valialkin
55d8379ae6
lib/logstorage: work-in-progress
2024-06-06 12:27:05 +02:00
Aliaksandr Valialkin
80a7c65ab7
lib/logstorage: allow using eval keyword instead of math keyword in math pipe
2024-06-05 10:07:49 +02:00
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress
2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin
96c29ab403
lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc
2024-06-04 02:29:10 +02:00
Aliaksandr Valialkin
539fce9227
lib/logstorage: work-in-progress
2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin
b30e80b071
lib/logstorage: work-in-progress
2024-05-30 16:19:23 +02:00
Aliaksandr Valialkin
1de187bcb7
lib/logstorage: work-in-progress
2024-05-29 01:52:13 +02:00
Aliaksandr Valialkin
0aafca29be
lib/logstorage: work-in-progress
2024-05-28 19:29:41 +02:00
Aliaksandr Valialkin
99138e15c0
lib/logstorage: fix golangci-lint warnings
2024-05-26 02:01:32 +02:00
Aliaksandr Valialkin
1e203f35f7
lib/logstorage: work-in-progress
2024-05-26 01:55:21 +02:00
Aliaksandr Valialkin
7ac529c235
lib/logstorage: work-in-progress
2024-05-25 22:59:13 +02:00
Aliaksandr Valialkin
0b629ce5a5
lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll
2024-05-25 22:13:32 +02:00
Aliaksandr Valialkin
dc55146752
lib/logstorage: work-in-progress
2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin
e2590f0485
lib/logstorage: work-in-progress
2024-05-25 00:30:58 +02:00
Aliaksandr Valialkin
4b458370c1
lib/logstorage: work-in-progress
2024-05-24 03:06:55 +02:00
Alexander Marshalov
7da541360e
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:46:50 +02:00
Aliaksandr Valialkin
22107421eb
lib/logstorage: work-in-progress
2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin
bc4a0b8f37
lib/logstorage: fix golangci-lint warnings
2024-05-20 11:04:12 +02:00
Aliaksandr Valialkin
ad505a7a9a
lib/logstorage: work-in-progress
2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin
0aa19a2837
lib/logstorage: work-in-progress
2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin
da3af090c6
lib/logstorage: work-in-progress
2024-05-14 03:05:03 +02:00
Aliaksandr Valialkin
cb35e62e04
lib/logstorage: work-in-progress
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258
2024-05-14 01:49:23 +02:00
Aliaksandr Valialkin
cc2647d212
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit
...
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.
This improves performance of these functions in hot loops of VictoriaLogs a bit.
2024-05-14 01:23:54 +02:00
hagen1778
17283fab6c
lib/logstorage: make linter happy
...
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-13 15:35:11 +02:00
Aliaksandr Valialkin
9dbd0f9085
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:33:31 +02:00
Aliaksandr Valialkin
590160ddbb
lib/slicesutil: add helper functions for setting slice length and extending its capacity
...
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
2024-05-12 11:32:17 +02:00
wanshuangcheng
83216e956c
chore: fix function names in comment ( #6076 )
...
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 01:11:12 -07:00
Aliaksandr Valialkin
918cccaddf
all: fix golangci-lint(revive) warnings after 0c0ed61ce7
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001
2024-04-02 23:16:29 +03:00
