Commit graph

9 commits

Author SHA1 Message Date
Aliaksandr Valialkin
f768c15bb2
lib/logstorage: make sure that chunkedAllocator is isn't used from concurrently running goroutines
This is needed in order to avoid data races
2025-02-14 15:34:08 +01:00
Aliaksandr Valialkin
95422e8f7a
lib/logstorage: ensure that statsProcessor.updateStatsForAllRows() is called on non-empty blockResult
This eliminates a class of potential bugs with incorrect stats calculations when an additional filter
is applied to the blockResult before passing it to the stats function, and this filter removes
all the rows from blockResult.
2025-02-14 15:34:07 +01:00
Aliaksandr Valialkin
186aa3bb0e
lib/logstorage: explicitly pass statsFunc to statsProcessor methods
This allows reducing the state of every statsProcessor by removing pointer to the corresponding statsFunc.
For example, this reduces statsCountProcessor size by 2x.
2025-01-13 04:49:39 +01:00
Aliaksandr Valialkin
e368f687a7
lib/logstorage: stats pipe: stop finalizeStats() as soon as the query is canceled
Previoysly finalizeStats() for some functions such as count_uniq() could run for long periods
of time after the query is canceled, since stopCh wan't propagated to finalizeStats().
2025-01-13 03:38:09 +01:00
Aliaksandr Valialkin
60f9f44150
lib/logstorage: reduce memory allocations at stats and top pipes
Use chunked allocator in order to reduce memory allocations. It allocates objects from slices of up to 64Kb size.
This improves performance for `stats` and `top` pipes by up to 2x when they are applied to big number of `by (...)` groups.

Also parallelize execution of `count_uniq`, `count_uniq_hash` and `uniq_values` stats functions,
so they are executed faster on hosts with many CPU cores when applied to fields with big number
of unique values.
2024-12-22 02:13:02 +01:00
Aliaksandr Valialkin
524f0e8d8b
lib/logstorage: eliminate memory allocations when finalizing per-group values calculated by stats pipe
This improves query performance a bit when `stats by (...)` returns millions of individual `by (...)` groups
2024-12-17 15:17:01 +01:00
Aliaksandr Valialkin
4599429f51
lib/logstorage: read timestamps column when it is really needed during query execution
Previously timestamps column was read unconditionally on every query.
This could significantly slow down queries, which do not need reading this column
like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070 .
2024-09-25 19:17:47 +02:00
Aliaksandr Valialkin
539fce9227
lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin
b30e80b071
lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00