Hui Wang
9616814728
vmalert: integrate with victorialogs ( #7255 )
...
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706 .
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md .
Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254 .
Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling )
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289 )
for details.
Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68bad22fd2
2024-10-29 16:32:00 +01:00
Aliaksandr Valialkin
54ccf09fdd
lib/logstorage: follow-up for 72941eac36
...
- Allow dropping metrics if the query result contains at least a single metric.
- Allow copying by(...) fields.
- Disallow overriding by(...) fields via `math` pipe.
- Allow using `format` pipe in stats query. This is useful for constructing some labels from the existing by(...) fields.
- Add more tests.
- Remove the check for time range in the query filter according to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254/files#r1803405826
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254
2024-10-17 11:09:16 -03:00
Hui Wang
21864de527
victorialogs: add more checks for stats query APIs ( #7254 )
...
1. Verify if field in [fields
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe )
exists. If not, it generates a metric with illegal float value "" for
prometheus metrics protocol.
2. check if multiple time range filters produce conflicted query time
range, for instance:
```
query: _time: 5m | stats count(),
start:2024-10-08T10:00:00.806Z,
end: 2024-10-08T12:00:00.806Z,
time: 2024-10-10T10:02:59.806Z
```
must give no result due to invalid final time range.
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-17 11:09:16 -03:00
Aliaksandr Valialkin
b3bbf94310
lib/logstorage: disallow using pipe names as the first unquoted words in filter pipe
...
Improperly written pipes could be silently parsed as filter pipe.
For example, the following query:
* | by (x)
was silently parsed to:
* | filter "by" x
It is better to return error, so the user could identify and fix invalid pipe
instead of silently executing invalid query with `filter` pipe.
(cherry picked from commit 7b475ed95d
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
834e2ad855
lib/logstorage: disallow using by as the first word in log filters, since it frequently clashes with stats by(...) pipe where stats word is omitted
...
(cherry picked from commit 6acf543b90
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
8c55b699f4
app/vlogscli: add interactive command-line tool for querying VictoriaLogs
2024-10-01 12:24:53 +02:00
Aliaksandr Valialkin
7456cbc653
lib/logstorage: allow using ! in unescaped phrase
...
Previously the phrase filter with `!` was treated unexpectedly.
For example, `foo!bar` filter was treated at `foo AND NOT bar`,
while most users expect that it matches "foo!bar" phrase.
This commit aligns with users' expectations.
2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
b7a3d575da
lib/logstorage: allow using - instead of ! in front of (...)
2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
1a6313ca68
lib/logstorage: allow using - instead of ! as a shorthand for NOT operator in LogsQL
2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
b60cb98377
lib/logstorage: support skipping _stream: prefix for stream filters
...
'_stream:{...}' can be written as '{...}'
This simplifies writing queries with stream filters, and makes them more familier to Loki users.
2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
3a556bd15a
app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop
...
Previously the original timestamp was used in the copied query, so _time:duration filters
were applied to the original time range: (timestamp-duration ... timestamp]. This resulted
in stopped live tailing, since new logs have timestamps bigger than the original time range.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7028
2024-09-26 08:57:48 +02:00
Aliaksandr Valialkin
55ecf4f766
lib/logstorage: add blocks_count pipe
...
This pipe is useful for debugging purposes when the number of processed blocks must be calculated for the given query:
<query> | blocks_count
This helps detecting the root cause of query performance slowdown in cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070
2024-09-25 19:18:38 +02:00
Aliaksandr Valialkin
cad236003b
app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg
...
Previously the query could return incorrect results, since the query timestamp was updated with every Query.Clone() call
during iterative search for the time range with up to limit=N rows.
While at it, optimize queries, which find low number of matching logs, while spend a lot of CPU time for searching
across big number of logs. The optimization reduces the upper bound of the time range to search if the current time range
contains zero matching rows.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785
2024-09-08 14:34:46 +02:00
Aliaksandr Valialkin
4b49b62a58
lib/logstorage: improve error logging for incorrect queries passed to /select/logsql/stats_query and /select/logsql/stats_query_range functions
2024-09-08 12:28:33 +02:00
Aliaksandr Valialkin
c448189f69
app/vlselect: add /select/logsql/stats_query_range endpoint for building time series panels in VictoriaLogs plugin for Grafana
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6943
Updates https://github.com/VictoriaMetrics/victorialogs-datasource/issues/61
2024-09-07 00:44:34 +02:00
Aliaksandr Valialkin
01c8e12370
app/vlselect: add /select/logsql/stats_query endpoint, which is going to be used by vmalert
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6942
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706
2024-09-06 23:00:58 +02:00
Aliaksandr Valialkin
08fe7949d1
lib/logstorage: consistently use nsecsPerDay constant and remove nsecPerDay constant
2024-09-06 16:18:15 +02:00
Aliaksandr Valialkin
b8a8d3d6f1
lib/logstorage: drop all the pipes from the query when calculating the number of matching logs at /select/logsql/hits API
2024-07-10 00:39:16 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs
2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress
2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress
2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress
2024-06-20 03:10:37 +02:00
Aliaksandr Valialkin
1750991119
lib/logstorage: work-in-progress
2024-06-17 12:13:25 +02:00
Aliaksandr Valialkin
9135b404d9
lib/logstorage: work-in-progress
2024-06-11 17:51:01 +02:00
Aliaksandr Valialkin
37a8cc0b12
lib/logstorage: work-in-progress
2024-06-10 18:42:31 +02:00
Aliaksandr Valialkin
53382ae837
lib/logstorage: work-in-progress
2024-06-06 12:27:11 +02:00
Aliaksandr Valialkin
b45e466a1b
lib/logstorage: work-in-progress
2024-06-05 03:18:25 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress
2024-06-04 01:50:55 +02:00
Aliaksandr Valialkin
79c03fc35f
lib/logstorage: work-in-progress
2024-05-28 19:29:50 +02:00
Aliaksandr Valialkin
afa597ce2a
lib/logstorage: work-in-progress
2024-05-26 01:56:12 +02:00
Aliaksandr Valialkin
03fe4c8963
lib/logstorage: work-in-progress
2024-05-25 21:36:24 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress
2024-05-25 00:31:55 +02:00
Aliaksandr Valialkin
7a2a2f173e
lib/logstorage: work-in-progress
2024-05-24 03:07:07 +02:00
Alexander Marshalov
0b70c4c1f1
[vmlogs] fixed time parsing with millisecond precision time ( #6293 ) ( #6295 )
...
fix for #6293
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-05-22 21:54:50 +02:00
Aliaksandr Valialkin
04d0dd2542
lib/logstorage: work-in-progress
2024-05-22 21:01:28 +02:00
Aliaksandr Valialkin
582e7d5439
lib/logstorage: work-in-progress
2024-05-20 04:09:15 +02:00
Aliaksandr Valialkin
28626db066
lib/logstorage: work-in-progress
...
(cherry picked from commit 0aa19a2837
2024-05-16 09:35:55 +02:00
Aliaksandr Valialkin
b1ee7bca1a
lib/logstorage: work-in-progress
2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL
...
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
2024-05-12 16:36:01 +02:00
XLONG96
88b9088499
lib/logstorage: avoid panic when parsing regex with stream filter ( #5897 )
2024-02-29 15:32:25 +02:00
Aliaksandr Valialkin
5819d4e6f7
lib/logstorage: properly encode "offset" search word just after _time filter
2023-07-18 16:03:57 -07:00
Aliaksandr Valialkin
da2ef397fa
lib/logstorage: add abilty to speficy offset for the selected _time filter
...
The following syntax is supported: _time:filter offset off
For example:
- _time:5m offset 1h - 5-minute duration one hour before the current time
- _time:2023 offset 2w - 2023 year with the 2 weeks offset in the past
2023-07-17 19:07:14 -07:00
Aliaksandr Valialkin
6751a08071
lib/logstorage: support for short form of _time:(now-duration, now] filter: _time:duration
2023-07-17 18:23:43 -07:00
Aliaksandr Valialkin
8fdfd13a29
lib/logstorage: LogsQL: replace exact_prefix("...") with exact("..."*)
...
This makes LogsQL queries more consistent with i("...") and i("..."*) syntax
2023-07-17 17:19:45 -07:00
Aliaksandr Valialkin
374890294e
app/victoria-logs: initial code release
2023-07-06 17:30:05 -07:00
