diff --git a/docs/VictoriaLogs/LogsQL.md b/docs/VictoriaLogs/LogsQL.md index 29c8cf127..32725a5c0 100644 --- a/docs/VictoriaLogs/LogsQL.md +++ b/docs/VictoriaLogs/LogsQL.md @@ -13,7 +13,10 @@ aliases: # LogsQL LogsQL is a simple yet powerful query language for [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/). -It provides the following features: +See [examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [tutorial](#logsql-tutorial) +in order to feel the language. + +LogsQL provides the following features: - Full-text search across [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). See [word filter](#word-filter), [phrase filter](#phrase-filter) and [prefix filter](#prefix-filter). @@ -194,7 +197,8 @@ _time:5m error | stats count() logs_with_error Finally, it is recommended reading [performance tips](#performance-tips). -Now you are familiar with LogsQL basics. Read [query syntax](#query-syntax) if you want to continue learning LogsQL. +Now you are familiar with LogsQL basics. See [LogsQL examples](https://docs.victoriametrics.com/victorialogs/logsql-examples/) and [query syntax](#query-syntax) +if you want to continue learning LogsQL. ### Key concepts diff --git a/docs/VictoriaLogs/README.md b/docs/VictoriaLogs/README.md index 6eae9414c..4737c30a6 100644 --- a/docs/VictoriaLogs/README.md +++ b/docs/VictoriaLogs/README.md @@ -10,7 +10,7 @@ aliases: VictoriaLogs is [open source](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/victoria-logs) user-friendly database for logs from [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/). -VictoriaLogs provides the following key features: +VictoriaLogs provides the following features: - VictoriaLogs can accept logs from popular log collectors. See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/). - VictoriaLogs is much easier to set up and operate compared to Elasticsearch and Grafana Loki. diff --git a/docs/VictoriaLogs/logsql-examples.md b/docs/VictoriaLogs/logsql-examples.md new file mode 100644 index 000000000..4feb1e117 --- /dev/null +++ b/docs/VictoriaLogs/logsql-examples.md @@ -0,0 +1,345 @@ +--- +sort: 100 +weight: 100 +title: LogsQL examples +menu: + docs: + parent: "victorialogs" + weight: 100 +--- + +# LogsQL examples + +## How to select recently ingested logs? + +[Run](https://docs.victoriametrics.com/victorialogs/querying/) the following query: + +```logsql +_time:5m +``` + +It returns logs over the last 5 minutes by using [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter). +The logs are returned in arbitrary order because of performance reasons. +Add [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) to the query if you need sorting +the returned logs by some field (usually [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field)): + +```logsql +_time:5m | sort by (_time) + +If the number of returned logs is too big, it may be limited with the [`limit` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe). +For example, the following query returns 10 most recent logs, which were ingested during the last 5 minutes: + +```logsql +_time:5m | sort by (_time desc) | limit 10 +``` + +See also: + +- [How to count the number of matching logs](#how-to-count-the-number-of-matching-logs) + +## How to select logs with the given word in log message? + +Just put the needed [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) in the query. +For example, the following query returns all the logs with the `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) +in [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): + +```logsql +error +``` + +If the number of returned logs is too big, then add [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) +for limiting the time range for the selected logs. For example, the following query returns logs with `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) +over the last hour: + +```logsql +error _time:1h +``` + +If the number of returned logs is still too big, then consider adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) +to the query. For example, the following query selects logs with `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word), +which do not contain `kubernetes` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word), over the last hour: + +```logsql +error !kubernetes _time:1h +``` + +The logs are returned in arbitrary order because of performance reasons. Add [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) +for sorting logs by the needed [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, the following query +sorts the selected logs by [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): + +```logsql +error _time:1h | sort by (_time) +``` + +See also: + +- [How to select logs with all the given words in log message?](#how-to-select-logs-with-all-the-given-words-in-log-message) +- [How to select logs with some of the given words in log message?](#how-to-select-logs-with-some-of-the-given-words-in-log-message) +- [How to select logs without the given word in log message?](#how-to-select-logs-without-the-given-word-in-log-message) +- [Filtering by phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter) +- [Filtering by prefix](https://docs.victoriametrics.com/victorialogs/logsql/#prefix-filter) +- [Filtering by regular expression](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) +- [Filtering by substring](https://docs.victoriametrics.com/victorialogs/logsql/#substring-filter) + + +## How to select logs without the given word in log message? + +Use [`NOT` logical filter](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter). For example, the following query returns all the logs +without the `INFO` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): + +```logsq +!INFO +``` + +If the number of returned logs is too big, then add [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) +for limiting the time range for the selected logs. For example, the following query returns matching logs over the last hour: + +```logsql +!INFO _time:1h +``` + +If the number of returned logs is still too big, then consider adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) +to the query. For example, the following query selects logs without `INFO` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word), +which contain `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word), over the last hour: + +```logsql +!INFO error _time:1h +``` + +The logs are returned in arbitrary order because of performance reasons. Add [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) +for sorting logs by the needed [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, the following query +sorts the selected logs by [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): + +```logsql +!INFO _time:1h | sort by (_time) +``` + +See also: + +- [How to select logs with all the given words in log message?](#how-to-select-logs-with-all-the-given-words-in-log-message) +- [How to select logs with some of given words in log message?](#how-to-select-logs-with-some-of-the-given-words-in-log-message) +- [Filtering by phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter) +- [Filtering by prefix](https://docs.victoriametrics.com/victorialogs/logsql/#prefix-filter) +- [Filtering by regular expression](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) +- [Filtering by substring](https://docs.victoriametrics.com/victorialogs/logsql/#substring-filter) + + +## How to select logs with all the given words in log message? + +Just enumerate the needed [words](https://docs.victoriametrics.com/victorialogs/logsql/#word) in the query, by deliming them with whitespace. +For example, the following query selects logs containing both `error` and `kubernetes` [words](https://docs.victoriametrics.com/victorialogs/logsql/#word) +in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): + +```logsql +error kubernetes +``` + +This query uses [`AND` logical filter](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter). + +If the number of returned logs is too big, then add [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) +for limiting the time range for the selected logs. For example, the following query returns matching logs over the last hour: + +```logsql +error kubernetes _time:1h +``` + +If the number of returned logs is still too big, then consider adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) +to the query. For example, the following query selects logs with `error` and `kubernetes` [words](https://docs.victoriametrics.com/victorialogs/logsql/#word) +from [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) containing `container="my-app"` field, over the last hour: + +```logsql +error kubernetes _stream:{container="my-app"} _time:1h +``` + +The logs are returned in arbitrary order because of performance reasons. Add [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) +for sorting logs by the needed [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, the following query +sorts the selected logs by [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): + +```logsql +error kubernetes _time:1h | sort by (_time) +``` + +See also: + +- [How to select logs with some of given words in log message?](#how-to-select-logs-with-some-of-the-given-words-in-log-message) +- [How to select logs without the given word in log message?](#how-to-select-logs-without-the-given-word-in-log-message) +- [Filtering by phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter) +- [Filtering by prefix](https://docs.victoriametrics.com/victorialogs/logsql/#prefix-filter) +- [Filtering by regular expression](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) +- [Filtering by substring](https://docs.victoriametrics.com/victorialogs/logsql/#substring-filter) + + +## How to select logs with some of the given words in log message? + +Put the needed [words](https://docs.victoriametrics.com/victorialogs/logsql/#word) into `(...)`, by delimiting them with ` or `. +For example, the following query selects logs with `error`, `ERROR` or `Error` [words](https://docs.victoriametrics.com/victorialogs/logsql/#word) +in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): + +```logsql +(error or ERROR or Error) +``` + +This query uses [`OR` logical filter](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter). + +If the number of returned logs is too big, then add [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) +for limiting the time range for the selected logs. For example, the following query returns matching logs over the last hour: + +```logsql +(error or ERROR or Error) _time:1h +``` + +If the number of returned logs is still too big, then consider adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) +to the query. For example, the following query selects logs without `error`, `ERROR` or `Error` [words](https://docs.victoriametrics.com/victorialogs/logsql/#word), +which do not contain `kubernetes` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word), over the last hour: + +```logsql +(error or ERROR or Error) !kubernetes _time:1h +``` + +The logs are returned in arbitrary order because of performance reasons. Add [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) +for sorting logs by the needed [fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). For example, the following query +sorts the selected logs by [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): + +```logsql +(error or ERROR or Error) _time:1h | sort by (_time) +``` + +See also: + +- [How to select logs with all the given words in log message?](#how-to-select-logs-with-all-the-given-words-in-log-message) +- [How to select logs without the given word in log message?](#how-to-select-logs-without-the-given-word-in-log-message) +- [Filtering by phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter) +- [Filtering by prefix](https://docs.victoriametrics.com/victorialogs/logsql/#prefix-filter) +- [Filtering by regular expression](https://docs.victoriametrics.com/victorialogs/logsql/#regexp-filter) +- [Filtering by substring](https://docs.victoriametrics.com/victorialogs/logsql/#substring-filter) + + +## How to select logs from the given application instance? + +Make sure the application is properly configured with [stream-level log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). +Then just use [`_stream` filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter) for selecting logs for the given application instance. +For example, if the application contains `job="app-42"` and `instance="host-123:5678"` [stream fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields), +then the following query selects all the logs from this application: + +```logsql +_stream:{job="app-42",instance="host-123:5678"} +``` + +If the number of returned logs is too big, it is recommended adding [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) +to the query in order to reduce the number of matching logs. For example, the following query returns logs for the given application for the last day: + +```logsql +_stream:{job="app-42",instance="host-123:5678"} _time:1d +``` + +If the number of returned logs is still too big, then consider adding more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) +to the query. For example, the following query selects logs from the given [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields), +which contain `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) in the [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field), +over the last day: + +```logsql +_stream:{job="app-42",instance="host-123:5678"} error _time:1d +``` + +The logs are returned in arbitrary order because of performance reasons. Use [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) +for sorting the returned logs by the needed fields. For example, the following query sorts the selected logs +by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field): + +```logsql +_stream:{job="app-42",instance="host-123:5678"} _time:1d | sort by (_time) +``` + +See also: + +- [How to determine applications with the most logs?](#how-to-determine-applications-with-the-most-logs) +- [How to select logs without the given word in log message?](#how-to-select-logs-without-the-given-word-in-log-message) + + +## How to count the number of matching logs? + +Use [`count()` stats function](https://docs.victoriametrics.com/victorialogs/logsql/#count-stats). For example, the following query returns +the number of results returned by `your_query_here`: + +```logsql +your_query_here | count() +``` + +## How to determine applications with the most logs? + +[Run](https://docs.victoriametrics.com/victorialogs/querying/) the following query: + +```logsql +_time:5m | stats by (_stream) count() as logs | sort by (logs desc) | limit 10 +``` + +This query returns top 10 application instances (aka [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)) +with the most logs over the last 5 minutes. + +This query uses the following [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) features: + +- [`_time` filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for selecting logs on the given time range (5 minutes in the query above). +- [`stats` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#stats-pipe) for calculating the number of logs. + per each [`_stream`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields). [`count` stats function](https://docs.victoriametrics.com/victorialogs/logsql/#count-stats) + is used for calculating the needed stats. +- [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) for sorting the stats by `logs` field in descending order. +- [`limit` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#limit-pipe) for limiting the number of returned results to 10. + +See also: + +- [How to filter out data after stats calculation?](#how-to-filter-out-data-after-stats-calculation) +- [How to calculate the number of logs per some interval?](#how-to-calculate-the-number-of-logs-per-some-interval) +- [How to select logs from the given application instance?](#how-to-select-logs-from-the-given-application-instance) + + +## How to parse JSON inside log message? + +It is better from performance and resource usage PoV to avoid storing JSON inside [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field). +It is recommended storing individual JSON fields and log fields instead according to [VictoriaLogs data model](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model). + +If you have to store JSON inside log message or inside any other [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model), +then the stored JSON can be parsed during query time via [`unpack_json` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unpack_json-pipe). +For example, the following query unpacks JSON from the [`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) +across all the logs for the last 5 minutes: + +```logsql +_time:5m | unpack_json +``` + +If you need to parse JSON array, then take a look at [`unroll` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#unroll-pipe). + + +## How to extract some data from text log message? + +Use [`extract`](https://docs.victoriametrics.com/victorialogs/logsql/#extract-pipe) or [`extract_regexp`](https://docs.victoriametrics.com/victorialogs/logsql/#extract_regexp-pipe) pipe. +For example, the following query extracts `username` and `user_id` fields from text [log message](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field): + +```logsql +_time:5m | extract "username=, user_id=," +``` + +See also: + +- [Replacing substrings in text fields](https://docs.victoriametrics.com/victorialogs/logsql/#replace-pipe) + + +## How to filter out data after stats calculation? + +Use [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe). For example, the following query +returns only [log streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with more than 1000 logs +over the last 5 minutes: + +```logsql +_time:5m | stats by (_stream) count() rows | filter rows:>1000 +``` + +## How to calculate the number of logs per some interval? + +Use [`stats` by time bucket](https://docs.victoriametrics.com/victorialogs/logsql/#stats-by-time-buckets). For example, the following query +returns per-hour number of logs with the `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word) for the last day: + +```logsq +_time:1d error | stats by (_time:1h) count() rows | sort by (_time) +``` + +This query uses [`sort` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe) in order to sort per-hour stats +by [`_time`](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field).