37 commits

Author	SHA1	Message	Date
Aliaksandr Valialkin	e5537bc64d	lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters	2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin	66b2987f49	lib/logstorage: optimize query imeediately after its parsing ... This eliminates possible bugs related to forgotten Query.Optimize() calls. This also allows removing optimize() function from pipe interface. While at it, drop filterNoop inside filterAnd.	2024-11-08 16:43:54 +01:00
Aliaksandr Valialkin	0550093802	app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint. ... This reduces the probability of forgotten call to Query.Optimize().	2024-11-08 16:43:53 +01:00
Aliaksandr Valialkin	3a5f1019ba	app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing	2024-11-06 18:53:28 +01:00
Aliaksandr Valialkin	7603446850	app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs ... These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365	2024-10-30 17:59:49 +01:00
Aliaksandr Valialkin	bac193e50b	app/vlselect: do not show empty fields in query results ... Empty fields are treated as non-existing fields by VictoriaLogs data model. So there is no sense in returning empty fields in query results, since they may mislead and confuse users.	2024-10-14 23:43:58 +02:00
Aliaksandr Valialkin	fe022ed795	app/vlselect/logsql: add missing return from ProcessLiveTailRequest() when the query cannot be live tailed	2024-10-09 16:30:10 +02:00
Aliaksandr Valialkin	255d1d4e13	app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop ... Previously the original timestamp was used in the copied query, so _time:duration filters were applied to the original time range: (timestamp-duration ... timestamp]. This resulted in stopped live tailing, since new logs have timestamps bigger than the original time range. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7028	2024-09-26 08:57:23 +02:00
Aliaksandr Valialkin	e86891b010	app/vlselect/logsql: call Query.Optimize() on the cloned query in order to replace * filter with filterNoop inside getLastNQueryResults() ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785	2024-09-18 18:24:17 +02:00
Aliaksandr Valialkin	657988ac3a	app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg ... Previously the query could return incorrect results, since the query timestamp was updated with every Query.Clone() call during iterative search for the time range with up to limit=N rows. While at it, optimize queries, which find low number of matching logs, while spend a lot of CPU time for searching across big number of logs. The optimization reduces the upper bound of the time range to search if the current time range contains zero matching rows. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785	2024-09-08 14:32:23 +02:00
Aliaksandr Valialkin	eaee2d7db4	lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions	2024-09-08 11:24:44 +02:00
Aliaksandr Valialkin	0a40064a6f	app/vlselect: add /select/logsql/stats_query_range endpoint for building time series panels in VictoriaLogs plugin for Grafana ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6943 Updates https://github.com/VictoriaMetrics/victorialogs-datasource/issues/61	2024-09-07 00:41:47 +02:00
Aliaksandr Valialkin	c9bb4ddeed	app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706	2024-09-06 23:06:43 +02:00
Aliaksandr Valialkin	aa9bb99527	lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API	2024-07-10 00:39:28 +02:00
Aliaksandr Valialkin	d4ca651547	lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs	2024-06-28 19:14:29 +02:00
Aliaksandr Valialkin	e8322147e9	app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint ... This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545 in order to return top N groups with the biggest number of hits.	2024-06-28 03:08:40 +02:00
Aliaksandr Valialkin	7c8c040502	app/vlselect: properly return live tailing results	2024-06-27 15:05:57 +02:00
Aliaksandr Valialkin	87f1c8bd6c	lib/logstorage: work-in-progress	2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin	8f5dc966f6	lib/logstorage: work-in-progress	2024-06-11 17:50:32 +02:00
Aliaksandr Valialkin	43cf221681	lib/logstorage: work-in-progress	2024-06-05 03:18:12 +02:00
Aliaksandr Valialkin	539fce9227	lib/logstorage: work-in-progress	2024-06-04 01:49:02 +02:00
Aliaksandr Valialkin	dc55146752	lib/logstorage: work-in-progress	2024-05-25 21:36:16 +02:00
Aliaksandr Valialkin	4b458370c1	lib/logstorage: work-in-progress	2024-05-24 03:06:55 +02:00
Aliaksandr Valialkin	22107421eb	lib/logstorage: work-in-progress	2024-05-22 21:01:20 +02:00
Aliaksandr Valialkin	ad505a7a9a	lib/logstorage: work-in-progress	2024-05-20 04:08:30 +02:00
Aliaksandr Valialkin	0aa19a2837	lib/logstorage: work-in-progress	2024-05-15 04:55:44 +02:00
Aliaksandr Valialkin	da3af090c6	lib/logstorage: work-in-progress	2024-05-14 03:05:03 +02:00
Aliaksandr Valialkin	9dbd0f9085	lib/logstorage: initial implementation of pipes in LogsQL ... See https://docs.victoriametrics.com/victorialogs/logsql/#pipes	2024-05-12 16:33:31 +02:00
Nikolay	43b5d8bc7a	app/vlselect: follow-up for 0514091948 (#6004) ... removes println lines	2024-03-22 08:46:40 +01:00
Aliaksandr Valialkin	0514091948	app/vlselect: follow-up for 451d2abf50 ... - Consistently return the first `limit` log entries if the total size of found log entries doesn't exceed 1Mb. See app/vlselect/logsql/sort_writer.go . Previously random log entries could be returned with each request. - Document the change at docs/VictoriaLogs/CHANGELOG.md - Document the `limit` query arg at docs/VictoriaLogs/querying/README.md - Make the change less intrusive. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5674 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5778	2024-02-18 23:05:51 +02:00
Dmytro Kozlov	451d2abf50	Enable the limit query param for the /select/logsql/query (#5778) ... * app/vlselect: add limit for logs query * app/vlselect: CHANGELOG.md * app/vlselect: stop search process if limit is reached, update logic, remove default limit * app/vlselect: fix tests * app/vlselect: fix filter tests * app/vlselect: fix tests	2024-02-18 22:58:47 +02:00
Roman Khavronenko	c32a01c52e	docs: follow-up after aec4b5db81 (#4638) ... Signed-off-by: hagen1778 <roman@victoriametrics.com>	2023-07-19 10:10:51 +02:00
Aliaksandr Valialkin	57541d5cea	Revert "app/vlselect/logsql: use buffered writer in order to save syscalls when sending big amounts of data to clients" ... This reverts commit c19048dc13. Reason for revert: it has been appeared that the net/http.ResponseWriter is already buffered, so there in no need in double bufferring	2023-06-21 20:40:01 -07:00
Aliaksandr Valialkin	c19048dc13	app/vlselect/logsql: use buffered writer in order to save syscalls when sending big amounts of data to clients	2023-06-21 20:25:32 -07:00
Aliaksandr Valialkin	83a1249299	app/vlselect/logsql: properly handle the error from ParseLogMessage	2023-06-21 10:28:47 -07:00
Aliaksandr Valialkin	7346bb4f03	app/vlselect/logsql: sort query results by _time if their summary size doesnt exceed -select.maxSortBufferSize	2023-06-21 01:11:25 -07:00
Aliaksandr Valialkin	87b66db47d	app/victoria-logs: initial code release	2023-06-19 22:55:12 -07:00