VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin 5763a957ef
lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields
Previously (f1:foo OR f2:bar) was incorrectly returning `foo` token for `f1` and `bar` token for `f2`.
These tokens were used for checking against bloom filter for every data block, so the data block,
which didn't contain simultaneously `foo` token for `f1` field and `bar` token for `f2` field, was skipped.
This was incorrect, since such a block may contain logs matching the original OR filter.

The fix is to return common tokens from `OR`-delimted filters only if these tokens exist at EVERY such filter
for the given field name. If some `OR`-delimited filter misses the given field name, then `OR`-delimited filters
do not contain common tokens, which could be used for checking against bloom filter.

While at it, add more tests covering various edge cases for filters delimited by AND and OR.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6554
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6556
2024-09-05 16:57:47 +02:00
..
arena.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
arena_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
bitmap.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
bitmap_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
bitmap_timing_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
block.go lib/logstorage: work-in-progress 2024-05-14 01:49:58 +02:00
block_data.go lib/logstorage: work-in-progress 2024-05-14 01:49:58 +02:00
block_data_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
block_header_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_result.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
block_search.go lib/logstorage: work-in-progress 2024-06-11 17:51:01 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
block_stream_reader.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_stream_writer.go chore: fix function names in comment (#6076) 2024-04-08 15:38:51 +02:00
block_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
block_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
bloomfilter.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
bloomfilter_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
consts.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
datadb.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
encoding.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-06 22:22:08 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
fields_set.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
fields_set_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filenames.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
filter.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_and.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 16:57:47 +02:00
filter_and_test.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 16:57:47 +02:00
filter_any_case_phrase.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_any_case_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_any_case_prefix.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_any_case_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_day_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_day_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_exact.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
filter_exact_prefix.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
filter_exact_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_exact_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_in.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
filter_in_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_ipv4_range.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_ipv4_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_len_range.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_len_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_noop.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
filter_not.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_not_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_or.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 16:57:47 +02:00
filter_or_test.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 16:57:47 +02:00
filter_phrase.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
filter_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_prefix.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
filter_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_regexp.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
filter_regexp_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_sequence.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
filter_sequence_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_stream.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
filter_stream_id.go lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
filter_stream_id_test.go lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
filter_string_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
filter_string_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_test.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
filter_time.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
filter_time_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_timing_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
filter_week_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
filter_week_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
hash128.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
hash128_timing_test.go lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types 2024-02-24 00:29:39 +02:00
if_filter.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
index_block_header.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
indexdb.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
indexdb_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
inmemory_part.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-06 22:24:18 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
json_parser.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
json_parser_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
log_rows.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
logfmt_parser.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
logfmt_parser_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
parser.go lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:16 +02:00
parser_test.go lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:16 +02:00
part.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
part_header.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
partition.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 20:38:00 +02:00
partition_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
pattern.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pattern_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pattern_timing_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
pipe_copy.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_copy_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_delete.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_delete_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_drop_empty_fields.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_drop_empty_fields_test.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
pipe_extract.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_extract_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_extract_regexp_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_extract_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_field_names.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_field_names_test.go lib/logstorage: work-in-progress 2024-05-24 03:07:07 +02:00
pipe_field_values.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_field_values_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_fields.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_fields_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_filter.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_filter_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:31 +02:00
pipe_format.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
pipe_format_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
pipe_limit.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_limit_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
pipe_math.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_math_test.go lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:08:08 +02:00
pipe_offset.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_offset_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_pack.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
pipe_pack_json.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_pack_json_test.go lib/logstorage: work-in-progress 2024-05-29 01:52:34 +02:00
pipe_pack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_pack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:25 +02:00
pipe_rename.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_rename_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
pipe_replace.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_replace_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_replace_regexp_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_replace_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_sort.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
pipe_sort_test.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:46:03 +02:00
pipe_sort_topk.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:46:03 +02:00
pipe_stats.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_stats_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
pipe_stream_context.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:39:45 +02:00
pipe_stream_context_test.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:39:45 +02:00
pipe_top.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_top_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
pipe_uniq.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_uniq_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_unpack.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
pipe_unpack_json.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack_json_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:24 +02:00
pipe_unpack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
pipe_unpack_syslog.go lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
pipe_unpack_syslog_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:25 +02:00
pipe_unroll.go lib/logstorage: it is safe using | unroll pipe in live tailing 2024-06-27 19:45:12 +02:00
pipe_unroll_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
pipe_update.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
pipe_utils_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:50 +02:00
rows.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
rows_test.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
stats_avg.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_avg_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_empty.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_empty_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_uniq.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_count_uniq_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_max.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_max_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_median.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_median_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_min.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_min_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_quantile.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_quantile_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_row_any.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
stats_row_any_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_row_max.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
stats_row_max_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_row_min.go lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
stats_row_min_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:36 +02:00
stats_sum.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum_len.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum_len_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_sum_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_uniq_values.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:49 +02:00
stats_uniq_values_test.go lib/logstorage: work-in-progress 2024-05-25 00:31:55 +02:00
stats_values.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
stats_values_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:28 +02:00
storage.go app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
storage_search.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
storage_search_test.go lib/logstorage: properly search for the surrounding logs in stream_context pipe 2024-07-01 02:33:00 +02:00
storage_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
stream_filter.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
stream_filter_test.go lib/logstorage: work-in-progress 2024-05-20 04:09:15 +02:00
stream_id.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
stream_id_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00
stream_tags.go app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:55 +02:00
stringbucket.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
syslog_parser.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
syslog_parser_test.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
tenant_id.go app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:58 +02:00
tenant_id_test.go lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:39:13 +02:00
tokenizer.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
tokenizer_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:36:01 +02:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
u128.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
u128_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
values_encoder.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
values_encoder_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00
values_encoder_timing_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:54:24 +02:00