VictoriaMetrics/lib/logstorage
Aliaksandr Valialkin 657988ac3a
app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg
Previously the query could return incorrect results, since the query timestamp was updated with every Query.Clone() call
during iterative search for the time range with up to limit=N rows.

While at it, optimize queries, which find low number of matching logs, while spend a lot of CPU time for searching
across big number of logs. The optimization reduces the upper bound of the time range to search if the current time range
contains zero matching rows.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785
2024-09-08 14:32:23 +02:00
..
arena.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
arena_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
bitmap.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
bitmap_timing_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
block.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data.go lib/logstorage: work-in-progress 2024-05-14 01:49:23 +02:00
block_data_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
block_header_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_result.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
block_search.go lib/logstorage: work-in-progress 2024-06-11 17:50:32 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
block_stream_reader.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_stream_writer.go chore: fix function names in comment (#6076) 2024-04-08 01:11:12 -07:00
block_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
block_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
bloomfilter.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
bloomfilter_test.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
consts.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
datadb_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
encoding.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-04 13:14:30 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
fields_set.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
fields_set_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filenames.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_and.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_and_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_any_case_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_any_case_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_any_case_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_day_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_exact_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_exact_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_in.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_in_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_ipv4_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_ipv4_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_len_range.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_len_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_noop.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
filter_not.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
filter_not_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_or.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_or_test.go lib/logstorage: preserve the order of tokens to check against bloom filters in AND filters 2024-09-08 12:27:30 +02:00
filter_phrase.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_phrase_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_prefix.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_prefix_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_regexp.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_regexp_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_sequence.go lib/logstorage: pre-calculate hashes from tokens used in bloom filter search 2024-09-05 19:44:17 +02:00
filter_sequence_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_stream.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
filter_stream_id.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_stream_id_test.go lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
filter_string_range.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
filter_string_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_test.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
filter_time.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
filter_time_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_timing_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
filter_week_range.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
filter_week_range_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
hash128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
hash128_timing_test.go lib/logstorage: consistently use atomic.* types instead of atomic.* functions on regular types 2024-02-23 23:46:13 +02:00
if_filter.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
index_block_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
indexdb.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
indexdb_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
inmemory_part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-04 14:13:14 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
json_parser.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
json_parser_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
log_rows.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
logfmt_parser.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
logfmt_parser_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
parser.go app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg 2024-09-08 14:32:23 +02:00
parser_test.go app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg 2024-09-08 14:32:23 +02:00
part.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
partition.go lib/logstorage: follow-up for 8a23d08c21 2023-10-02 16:52:23 +02:00
partition_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
pattern.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pattern_timing_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe.go lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:14:29 +02:00
pipe_copy.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_copy_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_delete.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_delete_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_drop_empty_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_drop_empty_fields_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_extract.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_extract_regexp_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_extract_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_field_names.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_names_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
pipe_field_values.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_field_values_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_fields.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_fields_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_filter.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_filter_test.go lib/logstorage: work-in-progress 2024-06-10 18:42:19 +02:00
pipe_format.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_format_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_limit.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_limit_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
pipe_math.go lib/logstorage: substitute | operator with or operator at math pipe 2024-09-06 22:44:14 +02:00
pipe_math_test.go lib/logstorage: substitute | operator with or operator at math pipe 2024-09-06 22:44:14 +02:00
pipe_offset.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_offset_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_pack.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_pack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_json_test.go lib/logstorage: work-in-progress 2024-05-29 01:52:13 +02:00
pipe_pack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_pack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
pipe_rename.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_rename_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
pipe_replace.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_replace_regexp_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_replace_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_sort.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
pipe_sort_test.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_sort_topk.go lib/logstorage: add ability to store sorted log position into a separate field with sort ... rank <fieldName> syntax 2024-07-01 01:44:17 +02:00
pipe_stats.go app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert 2024-09-06 23:06:43 +02:00
pipe_stats_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_stream_context.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_stream_context_test.go lib/logstorage: allow writing after N in front of before N at stream_context pipe 2024-07-02 01:38:20 +02:00
pipe_top.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_top_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_uniq.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_uniq_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_json.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_json_test.go lib/logstorage: work-in-progress 2024-05-25 21:36:16 +02:00
pipe_unpack_logfmt.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_logfmt_test.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
pipe_unpack_syslog.go lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
pipe_unpack_syslog_test.go lib/logstorage: work-in-progress 2024-06-17 12:13:18 +02:00
pipe_unroll.go lib/logstorage: it is safe using | unroll pipe in live tailing 2024-06-27 19:44:57 +02:00
pipe_unroll_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_update.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
pipe_utils_test.go lib/logstorage: work-in-progress 2024-05-28 19:29:41 +02:00
rows.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
rows_test.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_avg.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_avg_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_empty_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_count_uniq_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_max_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_median_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_min_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_quantile_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_row_any.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_any_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_max.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_max_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_row_min.go lib/logstorage: work-in-progress 2024-06-04 01:49:02 +02:00
stats_row_min_test.go lib/logstorage: work-in-progress 2024-05-30 16:19:23 +02:00
stats_sum.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_len_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_sum_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_uniq_values.go lib/logstorage: use quicktemplate.AppendJSONString instead of strconv.AppendQuote for encoding JSON strings 2024-07-05 01:22:23 +02:00
stats_uniq_values_test.go lib/logstorage: work-in-progress 2024-05-25 00:30:58 +02:00
stats_values.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
stats_values_test.go lib/logstorage: work-in-progress 2024-05-22 21:01:20 +02:00
storage.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
storage_search.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
storage_search_test.go lib/logstorage: properly search for the surrounding logs in stream_context pipe 2024-07-01 02:29:50 +02:00
storage_test.go lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant 2024-09-06 16:17:04 +02:00
stream_filter.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_filter_test.go lib/logstorage: work-in-progress 2024-05-20 04:08:30 +02:00
stream_id.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_id_test.go lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
stream_tags.go app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:17 +02:00
stringbucket.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
syslog_parser.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
syslog_parser_test.go lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:53:39 +02:00
tenant_id.go app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:34 +02:00
tenant_id_test.go lib: consistently use f-tests instead of table-driven tests 2024-07-09 22:40:50 +02:00
tokenizer.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_test.go lib/logstorage: initial implementation of pipes in LogsQL 2024-05-12 16:33:31 +02:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
u128_test.go app/victoria-logs: initial code release 2023-06-19 22:55:12 -07:00
values_encoder.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00
values_encoder_timing_test.go lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano() 2024-06-25 14:53:38 +02:00