17 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	1892e357c3	lib/logstorage: consistently use "pHits := m[..]" pattern ... Consistency improves maintainability of the code a bit.	2024-10-18 02:22:43 +02:00
Aliaksandr Valialkin	78c6fb0883	lib/logstorage: improve performance of top and field_values pipes on systems with many CPU cores ... - Parallelize mering of per-CPU results. - Parallelize writing the results to the next pipe.	2024-10-18 02:22:42 +02:00
Aliaksandr Valialkin	a350be48b6	lib/logstorage: do not count dictionary values which have no matching logs in count_uniq stats function ... Create blockResultColumn.forEachDictValue* helper functions for visiting matching dictionary values. These helper functions should prevent from counting dictionary values without matching logs in the future. This is a follow-up for 0c0f013a60 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7152	2024-10-01 13:34:45 +02:00
Aliaksandr Valialkin	0c0f013a60	lib/logstorage: skip values with zero hits for 'uniq', 'top' and 'field_values' pipes ... See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/72#issuecomment-2352078483	2024-09-30 14:15:07 +02:00
Aliaksandr Valialkin	b52862badf	lib/logstorage: return the expected hits results from uniq pipe when the number of unique values reaches the specified limit ... Previously `uniq` pipe could return zero `hits` if the number of found unique values equals the specified limit. This wasn't expected in most cases.	2024-09-29 10:51:09 +02:00
Aliaksandr Valialkin	55eb321f77	lib/logstorage: clear hits slice obtained from encoding.GetUint64s() before updating it with hits for valueTypeDict column ... encoding.GetUint64s() returns uninitialized slice, which may contain arbitrary values. So values in this slice must be reset to zero before using it for counting hits in `uniq` and `top` pipes.	2024-09-29 10:29:13 +02:00
Aliaksandr Valialkin	94afcbd9a9	lib/logstorage: postpone initialization of per-shard stateSizeBudget until the first call to pipeProcessor.writeBlock() ... This simplifies pipeProcessor initialization logic a bit. This also doesn't mangle the original maxStateSize value, which is used in error messages when the state size exceeds maxStateSize.	2024-09-29 10:29:13 +02:00
Aliaksandr Valialkin	4599429f51	lib/logstorage: read timestamps column when it is really needed during query execution ... Previously timestamps column was read unconditionally on every query. This could significantly slow down queries, which do not need reading this column like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070 .	2024-09-25 19:17:47 +02:00
Aliaksandr Valialkin	87f1c8bd6c	lib/logstorage: work-in-progress	2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin	99138e15c0	lib/logstorage: fix golangci-lint warnings	2024-05-26 02:01:32 +02:00
Aliaksandr Valialkin	dc55146752	lib/logstorage: work-in-progress	2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin	4b458370c1	lib/logstorage: work-in-progress	2024-05-24 03:06:55 +02:00
Aliaksandr Valialkin	22107421eb	lib/logstorage: work-in-progress	2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin	ad505a7a9a	lib/logstorage: work-in-progress	2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin	0aa19a2837	lib/logstorage: work-in-progress	2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin	cc2647d212	lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit ... Change the return values for these functions - now they return the unmarshaled result plus the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling. This improves performance of these functions in hot loops of VictoriaLogs a bit.	2024-05-14 01:23:54 +02:00
Aliaksandr Valialkin	9dbd0f9085	lib/logstorage: initial implementation of pipes in LogsQL ... See https://docs.victoriametrics.com/victorialogs/logsql/#pipes	2024-05-12 16:33:31 +02:00