Commit graph

82 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a1aa4b7aa9
lib/logstorage: allow passing * at in(*), contains_any(*) and contains_all(*)
Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for Grafana.

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673
2025-02-27 11:41:39 +01:00
Aliaksandr Valialkin
a3ff49def0
lib/logstorage: do not treat a string with leading zeros as a number at tryParseUint64
The "00123" string shouldn't be treated as 123 number.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-02-26 16:07:47 +01:00
Aliaksandr Valialkin
dd1c0e3bb7
lib/logstorage: optimize common regex filters generated by Grafana
For example, `field:~".+"`, `field:~".*"` or `field:""`

Replace such filters to faster ones. For example, `field:~".*"` is replaced with `*`,
while `field:~".+"` is replaced with `field:*`.
2025-02-25 20:35:04 +01:00
Aliaksandr Valialkin
9e0581533c
lib/logstorage: add le_field and lt_field filters
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.

(cherry picked from commit 30974e7f3f)
2025-02-25 19:13:31 +01:00
Aliaksandr Valialkin
3ee4b3ef24
lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
bc3e557f02
lib/logstorage: improve error logging for improperly escaped backslashes inside quoted strings
This should simplify debugging LogsQL queries by users
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
1f11bc948e
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
061fd098b5
lib/logstorage: properly handle _time:<=max_time filter
_time:<=max_time filter must include logs with timestamps matching max_time.
For example, _time:<=2025-02-24Z must include logs with timestamps until the end of February 24, 2025.
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
80d173471f
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
00d8e7a373
lib/logstorage: allow calling visitSubqueries on nil Query
This makes the code, which calls Query.visitSubquery, less error prone

(cherry picked from commit 910f307ca2)
2025-02-19 13:30:01 +01:00
Aliaksandr Valialkin
3ba095a875
lib/logstorage: remove needExecuteQuery from filterIn and filterStreamID, since it isn't needed
(cherry picked from commit 6afd66dcc8)
2025-02-19 13:30:01 +01:00
Aliaksandr Valialkin
88363b46b5
lib/logstorage: consistently use Query.cloneShallow() for shallow cloning of the original query 2025-02-17 15:36:38 +01:00
Aliaksandr Valialkin
5e4b5f9969
lib/logstorage: move common code for parsing a query inside parens into a separate function 2025-02-17 15:36:37 +01:00
Roman Khavronenko
c1861bdf8b
bump golangci-lint to v1.64.4
See https://github.com/golangci/golangci-lint/releases/tag/v1.64.4

* address linting errors

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-13 11:18:09 +01:00
Aliaksandr Valialkin
fea934936b
lib/logstorage: properly propagate extra filters to all the subqueries
The purpose of extra filters ( https://docs.victoriametrics.com/victorialogs/querying/#extra-filters )
is to limit the subset of logs, which can be queried. For example, it is expected that all the queries
with `extra_filters={tenant=123}` can access only logs, which contain `123` value for the `tenant` field.

Previously this wasn't the case, since the provided extra filters weren't applied to subqueries.
For example, the following query could be used to select all the logs outside `tenant=123`, for any `extra_filters` arg:

    * | union({tenant!=123})

This commit fixes this by propagating extra filters to all the subqueries.

While at it, this commit also properly propagates [start, end] time range filter from HTTP querying APIs
into all the subqueries, since this is what most users expect. This behaviour can be overriden on per-subquery
basis with the `options(ignore_global_time_filter=true)` option - see https://docs.victoriametrics.com/victorialogs/logsql/#query-options

Also properly apply apply optimizations across all the subqueries. Previously the optimizations at Query.optimize()
function were applied only to the top-level query.
2025-01-26 22:05:05 +01:00
Aliaksandr Valialkin
7b62086609
lib: consistently use logger.Panicf("BUG: ...") for logging programming bugs
logger.Fatalf("BUG: ...") complicates investigating the bug, since it doesn't show the call stack,
which led to the bug. So it is better to consistently use logger.Panicf("BUG: ...") for logging programming bugs.
2025-01-24 16:40:50 +01:00
Phuong Le
3ada13dd48
lib/logstorage: remove redundant error check 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
45cc9974ab
lib/logstorage: inherit query options by nested queries
This is a follow-up for b620b5cff5
2025-01-24 07:52:51 +01:00
Aliaksandr Valialkin
0a586ecfd8
lib/logstorage: add an ability to set query concurrency on a per-query basis
This is done via 'options(concurrency=N)' prefix for the query.
For example, the following query is executed on at most 4 CPU cores:

    options(concurrency=4) _time:1d | count_uniq(user_id)

This allows reducing RAM and CPU usage at the cost of longer query execution times,
since by default every query is executed in parallel on all the available CPU cores.

See https://docs.victoriametrics.com/victorialogs/logsql/#query-options
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
fb311d3ad5
lib/logstorage: always pass the current timestamp to newLexer()
Also always initialize Query.timestamp with the timestamp from the lexer.

This should avoid potential problems with relative timestamps inside inner queries.
For example, the `_time:1h` filter in the following query is correctly executed
relative to the current timestamp:

   foo:in(_time:1h | keep foo)
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
cb6f69b3ee
lib/logstorage: merge top-level _stream:{...} filters in the query
This should improve performance of queries, which contain multiple top-level _stream:{...} filters.

This should help the case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8037#issuecomment-2595854592

(cherry picked from commit 2eb15cf30c)
2025-01-17 13:26:52 +04:00
Aliaksandr Valialkin
7cbfe32d7e
lib/logstorage: add union pipe, which allows uniting results from multiple queries
(cherry picked from commit f27e120aeb)
2025-01-16 17:07:34 +01:00
Aliaksandr Valialkin
9ff6128102
lib/logstorage: add value_type filter to LogsQL
This filter can be used when debugging and exploring logs in order to understand better
which value types are used for storing the particular log fields.

The `value_type` filter complements `block_stats` pipe.
2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
53d726eca0
app/vlselect: allow passing arbitrary LogsQL filters to extra_filters and extra_stream_filters query args
While at at, allow passing an array of string values per each JSON entry at extra_filters and extra_stream_filters.
For example, `extra_filters={"foo":["bar","baz"]}` is converted into `foo:in("bar", "baz")` extra filter,
while `extra_stream_fitlers={"foo":["bar","baz"]}` is converted into `{foo=~"bar|baz"}` extra filter.

This should simplify creating faceted search when multiple values per a single log field must be selected.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365#issuecomment-2447964259

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5542
2024-12-18 22:40:52 +01:00
Aliaksandr Valialkin
63b0f02878
lib/logstorage: do not return log fields with the same constant value across all the selected logs from facets pipe
Such log fields do not give any useful information during logs' exploration.
They just clutter the output of the `facets` pipe. So it is better to drop such fields by default.

If these fields are needed, then `keep_const_fields` option can be added to `facets` pipe.
2024-12-18 22:40:51 +01:00
Aliaksandr Valialkin
3e37e6c08e
app/vlselect: allow passing max_value_len query arg to /select/logsql/facets API
The max_value_len query arg allows controlling the maximum length of values
per every log field. If the length is exceeded, then the log field is dropped
from the results, since it contains incomplete (misleading) set of most frequently seen field values.

(cherry picked from commit 48540ac409)
2024-12-09 12:23:33 +01:00
Aliaksandr Valialkin
80d4c7b50a
app/vlselect: add /select/logsql/facets endpoint
This endpoint returns the most frequent values per each field seen in the selected logs.
This endpoint is going to be used by VictoriaLogs web UI for faceted search.

(cherry picked from commit 740548ccfc)
2024-12-09 12:23:27 +01:00
Aliaksandr Valialkin
e71a8e3a6c
lib/logstorage: add facets pipe for returning the most frequent values across all the log fields seen in the selected logs
(cherry picked from commit dbec34bafc)
2024-12-09 12:23:27 +01:00
Aliaksandr Valialkin
659782ff75
lib/logstorage: add rate and rate_sum stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7415
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7646

(cherry picked from commit c3b8da81cd)
2024-12-09 12:23:26 +01:00
Aliaksandr Valialkin
cef135f2e8
lib/logstorage: add first and last pipes
The `first N by (field)` pipe is a shorthand to `sort by (field) limit N`,
while the `last N by (field)` pipe is a shorthand to `sort by (field) desc limit N`.

While at it, add support for partitioning sort results by log groups and applying
individual limit per each group.

For example, the following query returns up to 3 logs per each host with the biggest value
for the `request_duration` field:

_time:5m | last 3 by (request_duration) partition by (host)

This query is equivalent to the following one:

_time:5m | sort by (request_duration) desc limit 3 partition by (host)

Automatically add the 'partition by (_time)` into `sort`, `first` and `last` pipes
used in the query to `/select/logsql/stats_query_range` API.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7699
2024-12-05 15:16:54 +01:00
Aliaksandr Valialkin
4892d4d805
lib/logstorage: allow special chars in unquoted _stream tag names and values
This simplifies writing _stream filters. For example,

{foo-bar=abc:de}

can be written instead of

{"foo-bar"="abc:de"}
2024-11-30 17:27:58 +01:00
Aliaksandr Valialkin
a02d26e853
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters
(cherry picked from commit e5537bc64d)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
f82cfa16bf
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.

(cherry picked from commit a98fb495c6)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
a4ea3b87d7
lib/logstorage: optimize query imeediately after its parsing
This eliminates possible bugs related to forgotten Query.Optimize() calls.

This also allows removing optimize() function from pipe interface.

While at it, drop filterNoop inside filterAnd.

(cherry picked from commit 66b2987f49)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
52929c060a
app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint.
This reduces the probability of forgotten call to Query.Optimize().

(cherry picked from commit 0550093802)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
7a39f526ec
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
(cherry picked from commit 5ed54ebadf)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
0c657a95dc
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365

(cherry picked from commit 7603446850)
2024-10-31 14:11:07 +01:00
Hui Wang
9616814728
vmalert: integrate with victorialogs (#7255)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.

Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.

Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.

Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68bad22fd2)
2024-10-29 16:32:00 +01:00
Aliaksandr Valialkin
54ccf09fdd
lib/logstorage: follow-up for 72941eac36
- Allow dropping metrics if the query result contains at least a single metric.
- Allow copying by(...) fields.
- Disallow overriding by(...) fields via `math` pipe.
- Allow using `format` pipe in stats query. This is useful for constructing some labels from the existing by(...) fields.
- Add more tests.
- Remove the check for time range in the query filter according to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254/files#r1803405826

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254
2024-10-17 11:09:16 -03:00
Hui Wang
21864de527
victorialogs: add more checks for stats query APIs (#7254)
1. Verify if field in [fields
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe)
exists. If not, it generates a metric with illegal float value "" for
prometheus metrics protocol.
2. check if multiple time range filters produce conflicted query time
range, for instance:
```
query: _time: 5m | stats count(), 
start:2024-10-08T10:00:00.806Z, 
end: 2024-10-08T12:00:00.806Z, 
time: 2024-10-10T10:02:59.806Z
```
must give no result due to invalid final time range.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-17 11:09:16 -03:00
Aliaksandr Valialkin
b3bbf94310
lib/logstorage: disallow using pipe names as the first unquoted words in filter pipe
Improperly written pipes could be silently parsed as filter pipe.
For example, the following query:

   * | by (x)

was silently parsed to:

   * | filter "by" x

It is better to return error, so the user could identify and fix invalid pipe
instead of silently executing invalid query with `filter` pipe.

(cherry picked from commit 7b475ed95d)
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
834e2ad855
lib/logstorage: disallow using by as the first word in log filters, since it frequently clashes with stats by(...) pipe where stats word is omitted
(cherry picked from commit 6acf543b90)
2024-10-11 14:27:46 +02:00
Aliaksandr Valialkin
8c55b699f4
app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:24:53 +02:00
Aliaksandr Valialkin
7456cbc653
lib/logstorage: allow using ! in unescaped phrase
Previously the phrase filter with `!` was treated unexpectedly.
For example, `foo!bar` filter was treated at `foo AND NOT bar`,
while most users expect that it matches "foo!bar" phrase.

This commit aligns with users' expectations.
2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
b7a3d575da
lib/logstorage: allow using - instead of ! in front of (...) 2024-09-29 11:18:04 +02:00
Aliaksandr Valialkin
1a6313ca68
lib/logstorage: allow using - instead of ! as a shorthand for NOT operator in LogsQL 2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
b60cb98377
lib/logstorage: support skipping _stream: prefix for stream filters
'_stream:{...}' can be written as '{...}'

This simplifies writing queries with stream filters, and makes them more familier to Loki users.
2024-09-27 13:15:55 +02:00
Aliaksandr Valialkin
3a556bd15a
app/vlselect/logsql: clone the query with the current timestamp when performing live tailing requests in the loop
Previously the original timestamp was used in the copied query, so _time:duration filters
were applied to the original time range: (timestamp-duration ... timestamp]. This resulted
in stopped live tailing, since new logs have timestamps bigger than the original time range.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7028
2024-09-26 08:57:48 +02:00
Aliaksandr Valialkin
55ecf4f766
lib/logstorage: add blocks_count pipe
This pipe is useful for debugging purposes when the number of processed blocks must be calculated for the given query:

    <query> | blocks_count

This helps detecting the root cause of query performance slowdown in cases like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7070
2024-09-25 19:18:38 +02:00
Aliaksandr Valialkin
cad236003b
app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg
Previously the query could return incorrect results, since the query timestamp was updated with every Query.Clone() call
during iterative search for the time range with up to limit=N rows.

While at it, optimize queries, which find low number of matching logs, while spend a lot of CPU time for searching
across big number of logs. The optimization reduces the upper bound of the time range to search if the current time range
contains zero matching rows.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785
2024-09-08 14:34:46 +02:00