VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin 45a3713bdb
lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters
Previously tokens from AND filters were extracted in random order. This could slow down
checking them agains bloom filters if the most specific tokens go at the beginning of the AND filters.
Preserve the original order of tokens when matching them against bloom filters,
so the user could control the performance of the query by putting the most specific AND filters
at the beginning of the query.

While at it, add tests for getCommonTokensForAndFilters() and getCommonTokensForOrFilters().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6554
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6556
2024-09-08 12:27:30 +02:00
..
arena.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
arena_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
bitmap.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_timing_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
block.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
block_header_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_result.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
block_search.go lib/logstorage: work-in-progress 2024-06-11 17:50:32 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
block_stream_reader.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_stream_writer.go chore: fix function names in comment (#6076) 2024-04-08 01:11:12 -07:00
block_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
bloomfilter_test.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
consts.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
fields_set.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
fields_set_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_and.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_and_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_any_case_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_any_case_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_in.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_in_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_ipv4_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_ipv4_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_len_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_len_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_noop.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
filter_not.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_not_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_or.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_or_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_regexp.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_regexp_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_sequence.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_sequence_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_stream.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
filter_stream_id.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_stream_id_test.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_string_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_string_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_test.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
filter_time.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
filter_time_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_timing_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
filter_week_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_week_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types 2024-02-23 23:46:13 +02:00
if_filter.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
indexdb_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
json_parser.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
json_parser_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
log_rows.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
logfmt_parser.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
logfmt_parser_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
parser.go lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions 2024-09-08 11:24:44 +02:00
parser_test.go lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions 2024-09-08 11:24:44 +02:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
partition_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
pattern.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_timing_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
pipe_copy.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_copy_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_delete.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_delete_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_drop_empty_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_drop_empty_fields_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_extract.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_extract_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_field_names.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_names_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pipe_field_values.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_values_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_fields_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_filter.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_filter_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
pipe_format.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_format_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_limit.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_limit_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_math.go lib/logstorage: substitute | operator with or operator at math pipe 2024-09-06 22:44:14 +02:00
pipe_math_test.go lib/logstorage: substitute | operator with or operator at math pipe 2024-09-06 22:44:14 +02:00
pipe_offset.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_offset_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_pack.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_pack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_json_test.go lib/logstorage: work-in-progress 2024-05-29 01:52:13 +02:00
pipe_pack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_rename.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_rename_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_replace.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_replace_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_sort.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_sort_test.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_sort_topk.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_stats.go app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert 2024-09-06 23:06:43 +02:00
pipe_stats_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_stream_context.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_stream_context_test.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_top.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_top_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_uniq.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_uniq_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_json_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_syslog.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_syslog_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_unroll.go lib/logstorage: it is safe using | unroll pipe in live tailing 2024-06-27 19:44:57 +02:00
pipe_unroll_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_update.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_utils_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
rows.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
rows_test.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_avg.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_avg_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_row_any.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_any_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_max.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_max_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_min.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_min_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_sum.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_uniq_values.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_uniq_values_test.go lib/logstorage: work-in-progress 2024-05-25 00:30:58 +02:00
stats_values.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_values_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
storage.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
storage_search.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
storage_search_test.go lib/logstorage: properly search for the surrounding logs in stream_context pipe 2024-07-01 02:29:50 +02:00
storage_test.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
stream_filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_filter_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_id.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_id_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_tags.go app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:17 +02:00
stringbucket.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
syslog_parser.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
syslog_parser_test.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
tenant_id.go app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:34 +02:00
tenant_id_test.go lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:40:50 +02:00
tokenizer.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_timing_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00