Aliaksandr Valialkin
430bebb5f0
lib/logstorage: add top pipe, which returns top N field sets with the biggest number of matching logs
2024-06-13 18:49:15 +02:00
Aliaksandr Valialkin
c86edb59e6
wip
2024-06-11 15:13:55 +02:00
Aliaksandr Valialkin
c94c52e864
wip
2024-06-11 14:28:45 +02:00
Aliaksandr Valialkin
256a83ce99
wip
2024-06-10 18:39:48 +02:00
Aliaksandr Valialkin
e1689e1300
wip
2024-06-08 01:32:25 +02:00
Aliaksandr Valialkin
62fc9479b2
wip
2024-06-08 01:15:58 +02:00
Aliaksandr Valialkin
dc99281295
wip
2024-06-08 00:36:30 +02:00
Aliaksandr Valialkin
bcf3ded60a
wip
2024-06-07 23:59:35 +02:00
Aliaksandr Valialkin
dd5d494cdf
wip
2024-06-06 12:10:14 +02:00
Aliaksandr Valialkin
80a7c65ab7
lib/logstorage: allow using eval keyword instead of math keyword in math pipe
2024-06-05 10:07:49 +02:00
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress
2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin
96c29ab403
lib/logstorage: allow typing asc in sort pipe for the sake of consistency with desc
2024-06-04 02:29:10 +02:00
Aliaksandr Valialkin
539fce9227
lib/logstorage: work-in-progress
2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin
b30e80b071
lib/logstorage: work-in-progress
2024-05-30 16:19:23 +02:00
Aliaksandr Valialkin
1de187bcb7
lib/logstorage: work-in-progress
2024-05-29 01:52:13 +02:00
Aliaksandr Valialkin
0aafca29be
lib/logstorage: work-in-progress
2024-05-28 19:29:41 +02:00
Aliaksandr Valialkin
99138e15c0
lib/logstorage: fix golangci-lint warnings
2024-05-26 02:01:32 +02:00
Aliaksandr Valialkin
1e203f35f7
lib/logstorage: work-in-progress
2024-05-26 01:55:21 +02:00
Aliaksandr Valialkin
7ac529c235
lib/logstorage: work-in-progress
2024-05-25 22:59:13 +02:00
Aliaksandr Valialkin
0b629ce5a5
lib/logstorage: re-use per-shard fields across processed blocks in pipePackJSON and pipeUnroll
2024-05-25 22:13:32 +02:00
Aliaksandr Valialkin
dc55146752
lib/logstorage: work-in-progress
2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin
e2590f0485
lib/logstorage: work-in-progress
2024-05-25 00:30:58 +02:00
Aliaksandr Valialkin
4b458370c1
lib/logstorage: work-in-progress
2024-05-24 03:06:55 +02:00
Alexander Marshalov
7da541360e
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:46:50 +02:00
Aliaksandr Valialkin
22107421eb
lib/logstorage: work-in-progress
2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin
bc4a0b8f37
lib/logstorage: fix golangci-lint warnings
2024-05-20 11:04:12 +02:00
Aliaksandr Valialkin
ad505a7a9a
lib/logstorage: work-in-progress
2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin
0aa19a2837
lib/logstorage: work-in-progress
2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin
da3af090c6
lib/logstorage: work-in-progress
2024-05-14 03:05:03 +02:00
Aliaksandr Valialkin
cb35e62e04
lib/logstorage: work-in-progress
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6258
2024-05-14 01:49:23 +02:00
Aliaksandr Valialkin
cc2647d212
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit
...
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.
This improves performance of these functions in hot loops of VictoriaLogs a bit.
2024-05-14 01:23:54 +02:00
hagen1778
17283fab6c
lib/logstorage: make linter happy
...
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-05-13 15:35:11 +02:00
Aliaksandr Valialkin
9dbd0f9085
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:33:31 +02:00
Aliaksandr Valialkin
590160ddbb
lib/slicesutil: add helper functions for setting slice length and extending its capacity
...
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
2024-05-12 11:32:17 +02:00
wanshuangcheng
83216e956c
chore: fix function names in comment ( #6076 )
...
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 01:11:12 -07:00
Aliaksandr Valialkin
918cccaddf
all: fix golangci-lint(revive) warnings after 0c0ed61ce7
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6001
2024-04-02 23:16:29 +03:00
XLONG96
a5795f533d
lib/logstorage: avoid panic when parsing regex with stream filter ( #5897 )
2024-02-29 15:31:54 +02:00
Aliaksandr Valialkin
4617dc8bbe
lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types
...
See ea9e2b19a5
2024-02-23 23:46:13 +02:00
Aliaksandr Valialkin
f81b480905
lib/mergeset: consistently use atomic.* types instead of atomic.* function calls on ordinary types
...
See ea9e2b19a5
2024-02-23 23:29:35 +02:00
Aliaksandr Valialkin
275335c181
lib/logstorage: consistently use atomic.* type for refCount and mustDrop fields in datadb and storage structs in the same way as it is used in lib/storage
...
See ea9e2b19a5a204fd69f1
2024-02-23 23:04:42 +02:00
Aliaksandr Valialkin
0514091948
app/vlselect: follow-up for 451d2abf50
...
- Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb.
See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request.
- Document the change at docs/VictoriaLogs/CHANGELOG.md
- Document the `limit` query arg at docs/VictoriaLogs/querying/README.md
- Make the change less intrusive.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778
2024-02-18 23:05:51 +02:00
Dmytro Kozlov
451d2abf50
Enable the limit query param for the /select/logsql/query ( #5778 )
...
* app/vlselect: add limit for logs query
* app/vlselect: CHANGELOG.md
* app/vlselect: stop search process if limit is reached, update logic, remove default limit
* app/vlselect: fix tests
* app/vlselect: fix filter tests
* app/vlselect: fix tests
2024-02-18 22:58:47 +02:00
noodles2hg
cafd6f08b3
lib/logstorage: proper exit during block search ( #5400 )
2024-02-01 12:11:05 +00:00
Jiajing LU
333bda8702
count inmemoryParts that have not been taken for merge ( #5447 )
2024-02-01 12:06:28 +00:00
Aliaksandr Valialkin
2655c02d5e
lib/logstorage: make sure that WaitGroup.Add isnt called after stopCh is closed and WaitGroup.Wait is called
...
This protects from rare panic, which may occur during graceful shutdown of VictoriaLogs
2024-01-26 21:17:02 +01:00
Aliaksandr Valialkin
3449d563bd
all: add up to 10% random jitter to the interval between periodic tasks performed by various components
...
This should smooth CPU and RAM usage spikes related to these periodic tasks,
by reducing the probability that multiple concurrent periodic tasks are performed at the same time.
2024-01-22 18:40:32 +02:00
Aliaksandr Valialkin
cef7a39ba3
lib/logstorage: always check the previous indexBlockHeader for blocks with matching tenantID and/or streamID
...
The previous indexBlockHeader may contain blocks for the matching tenantID and/or streamID,
so it must be scanned unconditionally during the search.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856
This is a follow-up for 89dcbc2fe7
2023-11-13 23:13:53 +01:00
XLONG96
89dcbc2fe7
lib/logstorage: fix streamID and tenantID search ( #4856 ) ( #5295 )
2023-11-13 23:09:39 +01:00
Aliaksandr Valialkin
42dd71bb63
all: consistently use %w instead of %s in when error is passed to fmt.Errorf()
...
This allows consistently using errors.Is() for verifying whether the given error wraps some other known error.
2023-10-25 21:24:03 +02:00
Zakhar Bessarab
b296c8e95a
lib/logstorage: fix free space check ( #5113 )
...
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-03 12:39:41 +02:00
