github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/docs/VictoriaLogs/LogsQL.md
Aliaksandr Valialkin ed00b03ecb
lib/logstorage: add abilty to speficy offset for the selected _time filter 
The following syntax is supported: _time:filter offset off
For example:

- _time:5m offset 1h - 5-minute duration one hour before the current time
- _time:2023 offset 2w - 2023 year with the 2 weeks offset in the past
2023-07-17 19:07:42 -07:00

56 KiB
Raw Blame History

sort weight title menu aliases
2 2 LogsQL
docs
parent weight
victorialogs 2
/VictoriaLogs/LogsQL.html

LogsQL

LogsQL is a simple yet powerful query language for VictoriaLogs. It provides the following features:

LogsQL tutorial

If you aren't familiar with VictoriaLogs, then start with key concepts docs.

Then follow these docs:

The simplest LogsQL query is just a word, which must be found in the log message. For example, the following query finds all the logs with error word:

error

If the queried word clashes with LogsQL keywords, then just wrap it into quotes. For example, the following query finds all the log messages with and word:

"and"

It is OK to wrap any word into quotes. For example:

"error"

Moreover, it is possible to wrap phrases containing multiple words in quotes. For example, the following query finds log messages with the error: cannot find file phrase:

"error: cannot find file"

Queries above match logs with any timestamp, e.g. they may return logs from the previous year alongside recently ingested logs.

Usually logs from the previous year aren't so interesting comparing to the recently ingested logs. So it is recommended adding time filter to the query. For example, the following query returns logs with the error word, which were ingested into VictoriaLogs during the last 5 minutes:

error AND _time:5m

This query consists of two filters joined with AND operator:

The AND operator means that the log entry must match both filters in order to be selected.

Typical LogsQL query constists of multiple filters joined with AND operator. It may be tiresome typing and then reading all these AND words. So LogsQL allows omitting AND words. For example, the following query is equivalent to the query above:

error _time:5m

The query returns the following log fields by default:

Logs may contain arbitrary number of other fields. If you need obtaining some of these fields in query results, then just refer them in the query with field_name:* filter. See these docs for more details.

For example, the following query returns host.hostname field additionally to _msg, _stream and _time fields:

error _time:5m host.hostname:*

Suppose the query above selects too many rows because some buggy app pushes invalid error logs to VictoriaLogs. Suppose the app adds buggy_app word to every log line. Then the following query removes all the logs from the buggy app, allowing us paying attention to the real errors:

_time:5m error NOT buggy_app

This query uses NOT operator for removing log lines from the buggy app. The NOT operator is used frequently, so it can be substituted with ! char. So the following query is equivalent to the previous one:

_time:5m error !buggy_app

Suppose another buggy app starts pushing invalid error logs to VictoriaLogs - it adds foobar word to every emitted log line. No problems - just add !foobar to the query in order to remove these buggy logs:

_time:5m error !buggy_app !foobar

This query can be rewritten to more clear query with the OR operator inside parentheses:

_time:5m error !(buggy_app OR foobar)

Note that the parentheses are required here, since otherwise the query won't return the expected results. The query error !buggy_app OR foobar is interpreted as (error AND NOT buggy_app) OR foobar. This query may return error logs from the buggy app if they contain foobar word. This query also continues returning all the error logs from the second buggy app. This is because of different priorities for NOT, AND and OR operators. Read these docs for more details. There is no need in remembering all these priority rules - just wrap the needed query parts into explicit parentheses if you aren't sure in priority rules. As an additional bonus, explicit parentheses make queries easier to read and maintain.

Queries above assume that the error word is stored in the log message. This word can be stored in other field such as log.level. How to select error logs in this case? Just add the log.level: prefix in front of the error word:

_time:5m log.level:error !(buggy_app OR foobar)

The field name can be wrapped into quotes if it contains special chars or keywords, which may clash with LogsQL syntax. Any word also can be wrapped into quotes. So the following query is equivalent to the previous one:

"_time":"5m" "log.level":"error" !("buggy_app" OR "foobar")

What if the application identifier - such as buggy_app and foobar - is stored in the app field? Correct - just add app: prefix in front of buggy_app and foobar:

_time:5m log.level:error !(app:buggy_app OR app:foobar)

The query can be simplified by moving the app: prefix outside the parentheses:

_time:5m log.level:error !app:(buggy_app OR foobar)

The app field uniquely identifies the application instance if a single instance runs per each unique app. In this case it is recommended associating the app field with log stream fields during data ingestion. This usually improves both compression rate and query performance when querying the needed streams via _stream filter. If the app field is associated with the log stream, then the query above can be rewritten to more performant one:

_time:5m log.level:error _stream:{app!~"buggy_app|foobar"}

This query completely skips scanning for logs from buggy_app and foobar apps, thus significantly reducing disk read IO and CPU time needed for performing the query.

Finally, it is recommended reading performance tips.

Now you are familiar with LogsQL basics. Read query syntax if you want to continue learning LogsQL.

Key concepts

Word

LogsQL splits all the log fields into words delimited by non-word chars such as whitespace, parens, punctuation chars, etc. For example, the foo: (bar,"тест")! string is split into foo, bar and тест words. Words can contain arbitrary utf-8 chars. These words are taken into account by full-text search filters such as word filter, phrase filter and prefix filter.

Query syntax

LogsQL query consists of the following parts delimited by |:

  • Filters, which select log entries for further processing. This part is required in LogsQL. Other parts are optional.
  • Optional stream context, which allows selecting surrounding log lines for the matching log lines.
  • Optional transformations for the selected log fields. For example, an additional fields can be extracted or constructed from existing fields.
  • Optional post-filters for post-filtering of the selected results. For example, post-filtering can filter results based on the fields constructed by transformations.
  • Optional stats transformations, which can calculate various stats across selected results.
  • Optional sorting, which can sort the results by the sepcified fields.
  • Optional limiters, which can apply various limits on the selected results.

Filters

LogsQL supports various filters for searching for log messages (see below). They can be combined into arbitrary complex queries via logical filters.

Filters are applied to _msg field by default. If the filter must be applied to other log field, then its' name followed by the colon must be put in front of the filter. For example, if error word filter must be applied to the log.level field, then use log.level:error query.

Field names and filter args can be put into quotes if they contain special chars, which may clash with LogsQL syntax. LogsQL supports quoting via double quotes ", single quotes ' and backticks:

"some 'field':123":i('some("value")') AND `other"value'`

If doubt, it is recommended quoting field names and filter args.

The list of LogsQL filters:

Time filter

VictoriaLogs scans all the logs per each query if it doesn't contain the filter on _time field. It uses various optimizations in order to speed up full scan queries without the _time filter, but such queries can be slow if the storage contains large number of logs over long time range. The easiest way to optimize queries is to narrow down the search with the filter on _time field.

For example, the following query returns log messages ingested into VictoriaLogs during the last hour, which contain the error word:

_time:1h AND error

The following formats are supported for _time filter:

  • _time:duration matches logs with timestamps on the time range (now-duration, now]. Examples:
    • _time:5m - returns logs for the last 5 minutes
    • _time:2.5d15m42.345s - returns logs for the last 2.5 days, 15 minutes and 42.345 seconds
    • _time:1y - returns logs for the last year
  • _time:YYYY-MM-DD - matches all the logs for the particular day by UTC. For example, _time:2023-04-25 matches logs on April 25, 2023 by UTC.
  • _time:YYYY-MM - matches all the logs for the particular month by UTC. For example, _time:2023-02 matches logs on February, 2023 by UTC.
  • _time:YYYY - matches all the logs for the particular year by UTC. For example, _time:2023 matches logs on 2023 by UTC.
  • _time:YYYY-MM-DDTHH - matches all the logs for the particular hour by UTC. For example, _time:2023-04-25T22 matches logs on April 25, 2023 at 22 hour by UTC.
  • _time:YYYY-MM-DDTHH:MM - matches all the logs for the particular minute by UTC. For example, _time:2023-04-25T22:45 matches logs on April 25, 2023 at 22:45 by UTC.
  • _time:YYYY-MM-DDTHH:MM:SS - matches all the logs for the particular second by UTC. For example, _time:2023-04-25T22:45:59 matches logs on April 25, 2023 at 22:45:59 by UTC.
  • _time:[min_time, max_time] - matches logs on the time range [min_time, max_time], including both min_time and max_time. The min_time and max_time can contain any format specified here. For example, _time:[2023-04-01, 2023-04-30] matches logs for the whole April, 2023 by UTC, e.g. it is equivalent to _time:2023-04.
  • _time:[min_time, max_time) - matches logs on the time range [min_time, max_time), not including max_time. The min_time and max_time can contain any format specified here. For example, _time:[2023-02-01, 2023-03-01) matches logs for the whole February, 2023 by UTC, e.g. it is equivalent to _time:2023-02.

It is possible to specify time zone offset for all the absolute time formats by appending +hh:mm or -hh:mm suffix. For example, _time:2023-04-25+05:30 matches all the logs on April 25, 2023 by India time zone, while _time:2023-02-07:00 matches all the logs on February, 2023 by California time zone.

It is possible to specify generic offset for the selected time range by appending offset after the _time filter. Examples:

  • _time:5m offset 1h matches logs on the time range (now-1h5m, now-1h].
  • _time:2023-07 offset 5h30m matches logs on July, 2023 by UTC with offset 5h30m.
  • _time:[2023-02-01, 2023-03-01) offset 1w matches logs the week before the time range [2023-02-01, 2023-03-01) by UTC.

Performance tips:

  • It is recommended specifying the smallest possible time range during the search, since it reduces the amounts of log entries, which need to be scanned during the query. For example, _time:1h is usually faster than _time:5h.

  • While LogsQL supports arbitrary number of _time:... filters at any level of logical filters, it is recommended specifying a single _time filter at the top level of the query.

  • See other performance tips.

See also:

Stream filter

VictoriaLogs provides an optimized way to select log entries, which belong to particular log streams. This can be done via _stream:{...} filter. The {...} may contain arbitrary Prometheus-compatible label selector over fields associated with log streams. For example, the following query selects log entries with app field equal to nginx:

_stream:{app="nginx"}

This query is equivalent to the following exact() query, but the upper query usually works much faster:

app:exact("nginx")

Performance tips:

  • It is recommended using the most specific _stream:{...} filter matching the smallest number of log streams, which needs to be scanned by the rest of filters in the query.

  • While LogsQL supports arbitrary number of _stream:{...} filters at any level of logical filters, it is recommended specifying a single _stream:... filter at the top level of the query.

  • See other performance tips.

See also:

Word filter

The simplest LogsQL query consists of a single word to search in log messages. For example, the following query matches log messages with error word inside them:

error

This query matches the following log messages:

  • error
  • an error happened
  • error: cannot open file

This query doesn't match the following log messages:

  • ERROR, since the filter is case-sensitive by default. Use i(error) for this case. See these docs for details.
  • multiple errors occurred, since the errors word doesn't match error word. Use error* for this case. See these docs for details.

By default the given word is searched in the _msg field. Specify the field name in front of the word and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the error word in the log.level field:

log.level:error

Both the field name and the word in the query can contain arbitrary utf-8-encoded chars. For example:

поле:значение

Both the field name and the word in the query can be put inside quotes if they contain special chars, which may clash with the query syntax. For example, the following query searches for the ip 1.2.3.45 in the field ip:remote:

"ip:remote":"1.2.3.45"

See also:

Phrase filter

Is you need to search for log messages with the specific phrase inside them, then just wrap the phrase in quotes. The phrase can contain any chars, including whitespace, punctuation, parens, etc. They are taken into account during the search. For example, the following query matches log messages with ssh: login fail phrase inside them:

"ssh: login fail"

This query matches the following log messages:

  • ERROR: ssh: login fail for user "foobar"
  • ssh: login fail!

This query doesn't match the following log messages:

  • ssh login fail, since the message misses : char just after the ssh. Use seq("ssh", "login", "fail") query if log messages with the sequence of these words must be found. See these docs for details.
  • login fail: ssh error, since the message doesn't contain the full phrase requested in the query. If you need matching a message with all the words listed in the query, then use ssh AND login AND fail query. See these docs for details.
  • ssh: login failed, since the message ends with failed word instead of fail word. Use "ssh: login fail"* query for this case. See these docs for details.
  • SSH: login fail, since the SSH word is in capital letters. Use i("ssh: login fail") for case-insensitive search. See these docs for details.

By default the given phrase is searched in the _msg field. Specify the field name in front of the phrase and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the cannot open file phrase in the event.original field:

event.original:"cannot open file"

Both the field name and the phrase can contain arbitrary utf-8-encoded chars. For example:

сообщение:"невозможно открыть файл"

The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query searches for the cannot open file phrase in the field some:message:

"some:message":"cannot open file"

See also:

Prefix filter

If you need to search for log messages with words / phrases containing some prefix, then just add * char to the end of the word / phrase in the query. For example, the following query returns log messages, which contain words with err prefix:

err*

This query matches the following log messages:

  • err: foobar
  • cannot open file: error occurred

This query doesn't match the following log messages:

  • Error: foobar, since the Error word starts with capital letter. Use i(err*) for this case. See these docs for details.
  • fooerror, since the fooerror word doesn't start with err. Use re("err") for this case. See these docs for details.

Prefix filter can be applied to phrases. For example, the following query matches log messages containing phrases with unexpected fail prefix:

"unexpected fail"*

This query matches the following log messages:

  • unexpected fail: IO error
  • error:unexpected failure

This query doesn't match the following log messages:

  • unexpectedly failed, since the unexpectedly doesn't match unexpected word. Use unexpected* AND fail* for this case. See these docs for details.
  • failed to open file: unexpected EOF, since failed word occurs before the unexpected word. Use unexpected AND fail* for this case. See these docs for details.

By default the prefix filter is applied to the _msg field. Specify the needed field name in front of the prefix filter in order to apply it to the given field. For example, the following query matches log.level field containing any word with the err prefix:

log.level:err*

If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches log:level field containing any word with the err prefix.

"log:level":err*

Performance tips:

See also:

Empty value filter

Sometimes it is needed to find log entries without the given log field. This can be performed with log_field:"" syntax. For example, the following query matches log entries without host.hostname field:

host.hostname:""

See also:

Any value filter

Sometimes it is needed to find log entries containing any non-empty value for the given log field. This can be performed with log_field:* syntax. For example, the following query matches log entries with non-empty host.hostname field:

host.hostname:*

See also:

Exact filter

The word filter and phrase filter return log messages, which contain the given word or phrase inside them. The message may contain additional text other than the requested word or phrase. If you need searching for log messages or log fields with the exact value, then use the exact(...) filter. For example, the following query returns log messages wih the exact value fatal error: cannot find /foo/bar:

exact("fatal error: cannot find /foo/bar")

The query doesn't match the following log messages:

  • fatal error: cannot find /foo/bar/baz or some-text fatal error: cannot find /foo/bar, since they contain an additional text other than the specified in the exact() filter. Use "fatal error: cannot find /foo/bar" query in this case. See these docs for details.

  • FATAL ERROR: cannot find /foo/bar, since the exact() filter is case-sensitive. Use i("fatal error: cannot find /foo/bar") in this case. See these docs for details.

By default the exact() filter is applied to the _msg field. Specify the field name in front of the exact() filter and put a colon after it if it must be searched in the given field. For example, the following query returns log entries with the exact error value at log.level field:

log.level:exact("error")

Both the field name and the phrase can contain arbitrary utf-8-encoded chars. For example:

log.уровень:exact("ошибка")

The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query matches the error value in the field log:level:

"log:level":exact("error")

See also:

Exact prefix filter

Sometimes it is needed to find log messages starting with some prefix. This can be done with the exact("prefix"*) filter. For example, the following query matches log messages, which start from Processing request prefix:

exact("Processing request"*)

This filter matches the following log messages:

  • Processing request foobar
  • Processing requests from ...

It doesn't match the following log messages:

  • processing request foobar, since the log message starts with lowercase p. Use exact("processing request"*) OR exact("Processing request"*) query in this case. See these docs for details.
  • start: Processing request, since the log message doesn't start with Processing request. Use "Processing request" query in this case. See these docs for details.

By default the exact() filter is applied to the _msg field. Specify the field name in front of the exact() filter and put a colon after it if it must be searched in the given field. For example, the following query returns log entries with log.level field, which starts with err prefix:

log.level:exact("err"*)

Both the field name and the phrase can contain arbitrary utf-8-encoded chars. For example:

log.уровень:exact("ошиб"*)

The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query matches log:level values starting with err prefix:

"log:level":exact("err"*)

See also:

Multi-exact filter

Sometimes it is needed to locate log messages with a field containing one of the given values. This can be done with multiple exact filters combined into a single logical filter. For example, the following query matches log messages with log.level field containing either error or fatal exact values:

log.level:(exact("error") OR exact("fatal"))

While this solution works OK, LogsQL provides simpler and faster solution for this case - the in() filter.

log.level:in("error", "fatal")

It works very fast for long lists passed to in().

The future VictoriaLogs versions will allow passing arbitrary queries into in() filter. For example, the following query selects all the logs for the last hour for users, who visited pages with admin word in the path during the last day:

_time:1h AND user_id:in(_time:1d AND path:admin | fields user_id)

See the Roadmap for details.

See also:

Case-insensitive filter

Case-insensitive filter can be applied to any word, phrase or prefix by wrapping the corresponding word filter, phrase filter or prefix filter into i(). For example, the following query returns log messages with error word in any case:

i(error)

The query matches the following log messages:

  • unknown error happened
  • ERROR: cannot read file
  • Error: unknown arg
  • An ErRoR occured

The query doesn't match the following log messages:

  • FooError, since the FooError word has superflouos prefix Foo. Use re("(?i)error") for this case. See these docs for details.
  • too many Errors, since the Errors word has superflouos suffix s. Use i(error*) for this case.

By default the i() filter is applied to the _msg field. Specify the needed field name in front of the filter in order to apply it to the given field. For example, the following query matches log.level field containing error word in any case:

log.level:i(error)

If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches log:level field containing error word in any case.

"log:level":i("error")

Performance tips:

See also:

Sequence filter

Sometimes it is needed to find log messages with words or phrases in a particular order. For example, if log messages with error word followed by open file phrase must be found, then the following LogsQL query can be used:

seq("error", "open file")

This query matches some error: cannot open file /foo/bar message, since the open file phrase goes after the error word. The query doesn't match the cannot open file: error message, since the open file phrase is located in front of the error word. If you need matching log messages with both error word and open file phrase, then use error AND "open file" query. See these docs for details.

By default the seq() filter is applied to the _msg field. Specify the needed field name in front of the filter in order to apply it to the given field. For example, the following query matches event.original field containing (error, "open file") sequence:

event.original:seq(error, "open file")

If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches event:original field containing (error, "open file") sequence:

"event:original":seq(error, "open file")

See also:

Regexp filter

LogsQL supports regular expression filter with re2 syntax via re(...) expression. For example, the following query returns all the log messages containing err or warn susbstrings:

re("err|warn")

The query matches the following log messages, which contain either err or warn substrings:

  • error: cannot read data
  • 2 warnings have been raised
  • data trasferring finished

The query doesn't match the following log messages:

  • ERROR: cannot open file, since the ERROR word is in uppercase letters. Use re("(?i)(err|warn)") query for case-insensitive regexp search. See these docs for details. See also case-insenstive filter docs.
  • it is warmer than usual, since it doesn't contain neither err nor warn substrings.

By default the re() filter is applied to the _msg field. Specify the needed field name in front of the filter in order to apply it to the given field. For example, the following query matches event.original field containing either err or warn substrings:

event.original:re("err|warn")

If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches event:original field containing either err or warn substrings:

"event:original":re("err|warn")

Performance tips:

  • Prefer combining simple word filter with logical filter instead of using regexp filter. For example, the re("error|warning") query can be substituted with error OR warning query, which usually works much faster. Note that the re("error|warning") matches errors as well as warnings words, while error OR warning matches only the specified words. See also multi-exact filter.
  • Prefer moving the regexp filter to the end of the logical filter, so lightweighter filters are executed first.
  • Prefer using exact("some prefix"*) instead of re("^some prefix"), since the exact() works much faster than the re() filter.
  • See other performance tips.

See also:

Range filter

If you need to filter log message by some field containing only numeric values, then the range() filter can be used. For example, if the request.duration field contains the request duration in seconds, then the following LogsQL query can be used for searching for log entries with request durations exceeding 4.2 seconds:

request.duration:range(4.2, Inf)

The lower and the upper bounds of the range are excluded by default. If they must be included, then substitute the corresponding parentheses with square brackets. For example:

  • range[1, 10) includes 1 in the matching range
  • range(1, 10] includes 10 in the matching range
  • range[1, 10] includes 1 and 10 in the matching range

Note that the range() filter doesn't match log fields with non-numeric values alongside numeric values. For example, range(1, 10) doesn't match the request took 4.2 seconds log message, since the 4.2 number is surrounded by other text. Extract the numeric value from the message with parse(_msg, "the request took <request_duration> seconds") transformation and then apply the range() post-filter to the extracted request_duration field.

Performance tips:

See also:

IPv4 range filter

If you need to filter log message by some field containing only IPv4 addresses such as 1.2.3.4, then the ipv4_range() filter can be used. For example, the following query matches log entries with user.ip address in the range [127.0.0.0 - 127.255.255.255]:

user.ip:ipv4_range(127.0.0.0, 127.255.255.255)

The ipv4_range() accepts also IPv4 subnetworks in CIDR notation. For example, the following query is equivalent to the query above:

user.ip:ipv4_range("127.0.0.0/8")

If you need matching a single IPv4 address, then just put it inside ipv4_range(). For example, the following query matches 1.2.3.4 IP at user.ip field:

user.ip:ipv4_range("1.2.3.4")

Note that the ipv4_range() doesn't match a string with IPv4 address if this string contains other text. For example, ipv4_range("127.0.0.0/24") doesn't match request from 127.0.0.1: done log message, since the 127.0.0.1 ip is surrounded by other text. Extract the IP from the message with parse(_msg, "request from <ip>: done") transformation and then apply the ipv4_range() post-filter to the extracted ip field.

Hints:

  • If you need searching for log messages containing the given X.Y.Z.Q IPv4 address, then "X.Y.Z.Q" query can be used. See these docs for details.
  • If you need searching for log messages containing at least a single IPv4 address out of the given list, then "ip1" OR "ip2" ... OR "ipN" query can be used. See these docs for details.
  • If you need finding log entries with ip field in multiple ranges, then use ip:(ipv4_range(range1) OR ipv4_range(range2) ... OR ipv4_range(rangeN)) query. See these docs for details.

Performance tips:

See also:

String range filter

If you need to filter log message by some field with string values in some range, then string_range() filter can be used. For example, the following LogsQL query matches log entries with user.name field starting from A and B chars:

user.name:string_range(A, C)

The string_range() includes the lower bound, while excluding the upper bound. This simplifies querying distinct sets of logs. For example, the user.name:string_range(C, E) would match user.name fields, which start from C and D chars.

See also:

Length range filter

If you need to filter log message by its length, then len_range() filter can be used. For example, the following LogsQL query matches log messages with lengths in the range [5, 10] chars:

len_range(5, 10)

This query matches the following log messages, since their length is in the requested range:

  • foobar
  • foo bar

This query doesn't match the following log messages:

  • foo, since it is too short
  • foo bar baz abc, sinc it is too long

By default the len_range() is applied to the _msg field. Put the field name in front of the len_range() in order to apply the filter to the needed field. For example, the following query matches log entries with the foo field length in the range [10, 20] chars:

foo:len_range(10, 20)

See also:

Logical filter

Simpler LogsQL filters can be combined into more complex filters with the following logical operations:

  • q1 AND q2 - matches common log entries returned by both q1 and q2. Arbitrary number of filters can be combined with AND operation. For example, error AND file AND app matches log messages, which simultaneously contain error, file and app words. The AND operation is frequently used in LogsQL queries, so it is allowed to skip the AND word. For example, error file app is equivalent to error AND file AND app.

  • q1 OR q2 - merges log entries returned by both q1 and q2. Aribtrary number of filters can be combined with OR operation. For example, error OR warning OR info matches log messages, which contain at least one of error, warning or info words.

  • NOT q - returns all the log entries except of those which match q. For example, NOT info returns all the log messages, which do not contain info word. The NOT operation is frequently used in LogsQL queries, so it is allowed substituting NOT with ! in queries. For example, !info is equivalent to NOT info.

The NOT operation has the highest priority, AND has the middle priority and OR has the lowest priority. The priority order can be changed with parentheses. For example, NOT info OR debug is interpreted as (NOT info) OR debug, so it matches log messages, which do not contain info word, while it also matches messages with debug word (which may contain the info word). This is not what most users expect. In this case the query can be rewritten to NOT (info OR debug), which correctly returns log messages without info and debug words.

LogsQL supports arbitrary complex logical queries with arbitrary mix of AND, OR and NOT operations and parentheses.

By default logical filters apply to the _msg field unless the inner filters explicitly specify the needed log field via field_name:filter syntax. For example, (error OR warn) AND host.hostname:host123 is interpreted as (_msg:error OR _msg:warn) AND host.hostname:host123.

It is possible to specify a single log field for multiple filters with the following syntax:

field_name:(q1 OR q2 OR ... qN)

For example, log.level:error OR log.level:warning OR log.level:info can be substituted with the shorter query: log.level:(error OR warning OR info).

Performance tips:

  • VictoriaLogs executes logical operations from the left to the right, so it is recommended moving the most specific and the fastest filters (such as word filter and phrase filter) to the left, while moving less specific and the slowest filters (such as regexp filter and case-insensitive filter) to the right. For example, if you need to find log messages with the error word, which match some /foo/(bar|baz) regexp, it is better from performance PoV to use the query error re("/foo/(bar|baz)") instead of re("/foo/(bar|baz)") error.

    The most specific filter means that it matches the lowest number of log entries comparing to other filters.

  • See other performance tips.

Stream context

LogsQL will support the ability to select the given number of surrounding log lines for the selected log lines on a per-stream basis.

See the Roadmap for details.

Transformations

It is possible to perform various transformations on the selected log entries at client side with jq, awk, cut, etc. Unix commands according to these docs.

LogsQL will support the following transformations for the selected log entries:

  • Extracting the specified fields from text log fields according to the provided pattern.
  • Extracting the specified fields from JSON strings stored inside log fields.
  • Extracting the specified fields from logfmt strings stored inside log fields.
  • Creating a new field from existing log fields according to the provided format.
  • Creating a new field according to math calculations over existing log fields.
  • Copying of the existing log fields.
  • Parsing duration strings into floating-point seconds for further stats calculations.
  • Creating a boolean field with the result of arbitrary post-filters applied to the current fields. Boolean fields may be useful for conditional stats calculation.
  • Creating an integer field with the length of the given field value. This can be useful for stats calculations.

See the Roadmap for details.

Post-filters

It is possible to perform post-filtering on the selected log entries at client side with grep or similar Unix commands according to these docs.

LogsQL will support post-filtering on the original log fields and fields created by various transformations. The following post-filters will be supported:

See the Roadmap for details.

Stats

It is possible to perform stats calculations on the selected log entries at client side with sort, uniq, etc. Unix commands according to these docs.

LogsQL will support calculating the following stats based on the log fields and fields created by transformations:

  • The number of selected logs.
  • The number of non-empty values for the given field.
  • The number of unique values for the given field.
  • The min, max, avg, and sum for the given field.
  • The median and percentile for the given field.

It will be possible specifying an optional condition filter when calculating the stats. For example, sumIf(response_size, is_admin:true) calculates the total response size for admins only.

It will be possible to group stats by the specified fields and by the specified time buckets.

See the Roadmap for details.

Sorting

By default VictoriaLogs sorts the returned results by _time field if their total size doesn't exceed -select.maxSortBufferSize command-line value (by default it is set to one megabytes). Otherwise sorting is skipped because of performance and efficiency concerns described here.

It is possible to sort the selected log entries at client side with sort Unix command according to these docs.

LogsQL will support results' sorting by the given set of log fields.

See the Roadmap for details.

Limiters

It is possible to limit the returned results with head, tail, less, etc. Unix commands according to these docs.

LogsQL will support the ability to limit the number of returned results alongside the ability to page the returned results. Additionally, LogsQL will provide the ability to select fields, which must be returned in the response.

See the Roadmap for details.

Querying specific fields

By default VictoriaLogs query response contains _msg, _stream and _time fields.

If you want selecting other fields from the ingested structured logs, then they must be mentioned in query filters. For example, if you want selecting log.level field, and this field isn't mentioned in the query yet, then add log.level:* filter filter to the end of the query. The field_name:* filter doesn't return log entries with empty or missing field_name. If you want returning log entries with and without the given field, then (field_name:* OR field_name:"") filter can be used. See the following docs for details:

In the future LogsQL will support | fields field1, field2, ... fieldN syntax for selecting the listed fields. It will also support the ability to select all the fields for the matching log entries with | fields * syntax. See the Roadmap for details.

Performance tips

  • It is highly recommended specifying time filter in order to narrow down the search to specific time range.
  • It is highly recommended specifying stream filter in order to narrow down the search to specific log streams.
  • Move faster filters such as word filter and phrase filter to the beginning of the query. This rule doesn't apply to time filter and stream filter, which can be put at any place of the query.
  • Move more specific filters, which match lower number of log entries, to the beginning of the query. This rule doesn't apply to time filter and stream filter, which can be put at any place of the query.