github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib/logstorage
History
Aliaksandr Valialkin 258ccfb953
lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 
Previously per-token hashes for per-block bloom filters were re-calculated on every scanned block.
This could be slow when the number of tokens is big or when the number of blocks to scan is big.
Pre-calculate hashes for bloom filters and then use them for searching in bloom filters.
This improves performance by 2.5x for in(...) filters with many values to search inside `in()`.
2024-09-05 19:44:17 +02:00
..
arena.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
arena_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
bitmap.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_timing_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
block.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
block_header_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_result.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
block_search.go lib/logstorage: work-in-progress 2024-06-11 17:50:32 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
block_stream_reader.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_stream_writer.go chore: fix function names in comment (#6076) 2024-04-08 01:11:12 -07:00
block_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
bloomfilter_test.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
consts.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
fields_set.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
fields_set_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_and.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_and_test.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 14:29:50 +02:00
filter_any_case_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_any_case_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_in.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_in_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_ipv4_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_ipv4_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_len_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_len_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_noop.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
filter_not.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_not_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_or.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_or_test.go lib/logstorage: properly fix incorrect extraction of common tokens for OR filters at distinct log fields 2024-09-05 14:29:50 +02:00
filter_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_regexp.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_regexp_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_sequence.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_sequence_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_stream.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
filter_stream_id.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_stream_id_test.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_string_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_string_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_test.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
filter_time.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
filter_time_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_timing_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
filter_week_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_week_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types 2024-02-23 23:46:13 +02:00
if_filter.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
indexdb_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
json_parser.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
json_parser_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
log_rows.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
logfmt_parser.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
logfmt_parser_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
parser.go lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:28 +02:00
parser_test.go lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API 2024-07-10 00:39:28 +02:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
partition_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
pattern.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_timing_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
pipe_copy.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_copy_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_delete.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_delete_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_drop_empty_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_drop_empty_fields_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_extract.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_extract_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_field_names.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_names_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pipe_field_values.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_values_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_fields_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_filter.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_filter_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
pipe_format.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_format_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_limit.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_limit_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_math.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_math_test.go lib/logstorage: allow using eval keyword instead of math keyword in math pipe 2024-06-05 10:07:49 +02:00
pipe_offset.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_offset_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_pack.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_pack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_json_test.go lib/logstorage: work-in-progress 2024-05-29 01:52:13 +02:00
pipe_pack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_rename.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_rename_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_replace.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_replace_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_sort.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_sort_test.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_sort_topk.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_stats.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_stats_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_stream_context.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_stream_context_test.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_top.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_top_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_uniq.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_uniq_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_json_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_syslog.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_syslog_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_unroll.go lib/logstorage: it is safe using | unroll pipe in live tailing 2024-06-27 19:44:57 +02:00
pipe_unroll_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_update.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_utils_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
rows.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
rows_test.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_avg.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_avg_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_row_any.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_any_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_max.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_max_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_min.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_min_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_sum.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_uniq_values.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_uniq_values_test.go lib/logstorage: work-in-progress 2024-05-25 00:30:58 +02:00
stats_values.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_values_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
storage.go app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:33 +02:00
storage_search.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
storage_search_test.go lib/logstorage: properly search for the surrounding logs in stream_context pipe 2024-07-01 02:29:50 +02:00
storage_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_filter_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_id.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_id_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_tags.go app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:17 +02:00
stringbucket.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
syslog_parser.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
syslog_parser_test.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
tenant_id.go app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:34 +02:00
tenant_id_test.go lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:40:50 +02:00
tokenizer.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_timing_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00