mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-31 15:06:26 +00:00
wip
This commit is contained in:
parent
a15a56def0
commit
2f49685080
8 changed files with 44 additions and 44 deletions
|
@ -70,7 +70,7 @@ const ExploreLogsHeader: FC<ExploreLogHeaderProps> = ({ query, limit, error, onC
|
||||||
<a
|
<a
|
||||||
className="vm-link vm-link_with-icon"
|
className="vm-link vm-link_with-icon"
|
||||||
target="_blank"
|
target="_blank"
|
||||||
href="https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html"
|
href="https://docs.victoriametrics.com/victorialogs/logsql/"
|
||||||
rel="help noreferrer"
|
rel="help noreferrer"
|
||||||
>
|
>
|
||||||
<InfoIcon/>
|
<InfoIcon/>
|
||||||
|
|
|
@ -151,7 +151,7 @@ Released at 2024-03-01
|
||||||
|
|
||||||
Released at 2023-11-15
|
Released at 2023-11-15
|
||||||
|
|
||||||
* BUGFIX: properly locate logs for the [requested streams](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-filter). Previously logs for some streams may be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856). Thanks to @XLONG96 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295)!
|
* BUGFIX: properly locate logs for the [requested streams](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter). Previously logs for some streams may be missing in query results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4856). Thanks to @XLONG96 for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5295)!
|
||||||
* BUGFIX: [web UI](https://docs.victoriametrics.com/VictoriaLogs/querying/#web-ui): properly sort found logs by time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300).
|
* BUGFIX: [web UI](https://docs.victoriametrics.com/VictoriaLogs/querying/#web-ui): properly sort found logs by time. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5300).
|
||||||
|
|
||||||
## [v0.4.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.4.1-victorialogs)
|
## [v0.4.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.4.1-victorialogs)
|
||||||
|
@ -186,9 +186,9 @@ Released at 2023-07-20
|
||||||
|
|
||||||
Released at 2023-07-17
|
Released at 2023-07-17
|
||||||
|
|
||||||
* FEATURE: support short form of `_time` filters over the last X minutes/hours/days/etc. For example, `_time:5m` is a short form for `_time:(now-5m, now]`, which matches logs with [timestamps](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) for the last 5 minutes. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#time-filter) for details.
|
* FEATURE: support short form of `_time` filters over the last X minutes/hours/days/etc. For example, `_time:5m` is a short form for `_time:(now-5m, now]`, which matches logs with [timestamps](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) for the last 5 minutes. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for details.
|
||||||
* FEATURE: add ability to specify offset for the selected time range. For example, `_time:5m offset 1h` is equivalent to `_time:(now-5m-1h, now-1h]`. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#time-filter) for details.
|
* FEATURE: add ability to specify offset for the selected time range. For example, `_time:5m offset 1h` is equivalent to `_time:(now-5m-1h, now-1h]`. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for details.
|
||||||
* FEATURE: [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html): replace `exact_prefix("...")` with `exact("..."*)`. This makes it consistent with [i()](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#case-insensitive-filter) filter, which can accept phrases and prefixes, e.g. `i("phrase")` and `i("phrase"*)`. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#exact-prefix-filter).
|
* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): replace `exact_prefix("...")` with `exact("..."*)`. This makes it consistent with [i()](https://docs.victoriametrics.com/victorialogs/logsql/#case-insensitive-filter) filter, which can accept phrases and prefixes, e.g. `i("phrase")` and `i("phrase"*)`. See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#exact-prefix-filter).
|
||||||
|
|
||||||
## [v0.1.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.1.0-victorialogs)
|
## [v0.1.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.1.0-victorialogs)
|
||||||
|
|
||||||
|
|
|
@ -35,7 +35,7 @@ VictoriaLogs is optimized specifically for logs. So it provides the following fe
|
||||||
- Up to 30x less RAM usage than Elasticsearch for the same workload.
|
- Up to 30x less RAM usage than Elasticsearch for the same workload.
|
||||||
- Up to 15x less disk space usage than Elasticsearch for the same amounts of stored logs.
|
- Up to 15x less disk space usage than Elasticsearch for the same amounts of stored logs.
|
||||||
- Ability to work with hundreds of terabytes of logs on a single node.
|
- Ability to work with hundreds of terabytes of logs on a single node.
|
||||||
- Very easy to use query language optimized for typical log analysis tasks - [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html).
|
- Very easy to use query language optimized for typical log analysis tasks - [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||||
- Fast full-text search over all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) out of the box.
|
- Fast full-text search over all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) out of the box.
|
||||||
- Good integration with traditional command-line tools for log analysis. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line).
|
- Good integration with traditional command-line tools for log analysis. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line).
|
||||||
|
|
||||||
|
@ -57,7 +57,7 @@ VictoriaLogs and Grafana Loki have the following differences:
|
||||||
- Grafana Loki provides very inconvenient query language - [LogQL](https://grafana.com/docs/loki/latest/logql/).
|
- Grafana Loki provides very inconvenient query language - [LogQL](https://grafana.com/docs/loki/latest/logql/).
|
||||||
This query language is hard to use for typical log analysis tasks.
|
This query language is hard to use for typical log analysis tasks.
|
||||||
|
|
||||||
VictoriaMetrics provides easy to use query language for typical log analysis tasks - [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html).
|
VictoriaMetrics provides easy to use query language for typical log analysis tasks - [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||||
|
|
||||||
- VictoriaLogs performs typical full-text queries up to 1000x faster than Grafana Loki.
|
- VictoriaLogs performs typical full-text queries up to 1000x faster than Grafana Loki.
|
||||||
|
|
||||||
|
@ -88,7 +88,7 @@ VictoriaLogs is designed solely for logs. VictoriaLogs uses [similar design idea
|
||||||
over the stored logs.
|
over the stored logs.
|
||||||
|
|
||||||
VictoriaLogs provides easy to use query language with full-text search specifically optimized
|
VictoriaLogs provides easy to use query language with full-text search specifically optimized
|
||||||
for log analysis - [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html).
|
for log analysis - [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||||
LogsQL is usually much easier to use than SQL for typical log analysis tasks, while some
|
LogsQL is usually much easier to use than SQL for typical log analysis tasks, while some
|
||||||
non-trivial analytics may require SQL power.
|
non-trivial analytics may require SQL power.
|
||||||
|
|
||||||
|
@ -119,11 +119,11 @@ This architecture is inspired by [ClickHouse architecture](https://clickhouse.co
|
||||||
On top of this, VictoriaLogs employs additional optimizations for achieving high query performance:
|
On top of this, VictoriaLogs employs additional optimizations for achieving high query performance:
|
||||||
|
|
||||||
- It uses [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) for skipping blocks without the given
|
- It uses [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) for skipping blocks without the given
|
||||||
[word](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#word-filter) or [phrase](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#phrase-filter).
|
[word](https://docs.victoriametrics.com/victorialogs/logsql/#word-filter) or [phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter).
|
||||||
- It uses custom encoding and compression for fields with different data types.
|
- It uses custom encoding and compression for fields with different data types.
|
||||||
For example, it encodes IP addresses as 4-byte tuples. Custom fields' encoding reduces data size on disk and improves query performance.
|
For example, it encodes IP addresses as 4-byte tuples. Custom fields' encoding reduces data size on disk and improves query performance.
|
||||||
- It physically groups logs for the same [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)
|
- It physically groups logs for the same [log stream](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields)
|
||||||
close to each other. This improves compression ratio, which helps reducing disk space usage. This also improves query performance
|
close to each other. This improves compression ratio, which helps reducing disk space usage. This also improves query performance
|
||||||
by skipping blocks for unneeded streams when [stream filter](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-filter) is used.
|
by skipping blocks for unneeded streams when [stream filter](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter) is used.
|
||||||
- It maintains sparse index for [log timestamps](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field),
|
- It maintains sparse index for [log timestamps](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field),
|
||||||
which allow improving query performance when [time filter](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#time-filter) is used.
|
which allow improving query performance when [time filter](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) is used.
|
||||||
|
|
|
@ -17,7 +17,7 @@ VictoriaLogs provides the following key features:
|
||||||
See [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/).
|
See [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/).
|
||||||
- VictoriaLogs provides easy yet powerful query language with full-text search capabilities across
|
- VictoriaLogs provides easy yet powerful query language with full-text search capabilities across
|
||||||
all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) -
|
all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) -
|
||||||
see [LogsQL docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html).
|
see [LogsQL docs](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||||
- VictoriaLogs can be seamlessly combined with good old Unix tools for log analysis such as `grep`, `less`, `sort`, `jq`, etc.
|
- VictoriaLogs can be seamlessly combined with good old Unix tools for log analysis such as `grep`, `less`, `sort`, `jq`, etc.
|
||||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line) for details.
|
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line) for details.
|
||||||
- VictoriaLogs capacity and performance scales linearly with the available resources (CPU, RAM, disk IO, disk space).
|
- VictoriaLogs capacity and performance scales linearly with the available resources (CPU, RAM, disk IO, disk space).
|
||||||
|
|
|
@ -34,10 +34,10 @@ The following functionality is planned in the future versions of VictoriaLogs:
|
||||||
- Fluentd
|
- Fluentd
|
||||||
- Syslog
|
- Syslog
|
||||||
- Journald (systemd)
|
- Journald (systemd)
|
||||||
- Add missing functionality to [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html):
|
- Add missing functionality to [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/):
|
||||||
- [Transformation functions](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#transformations).
|
- [Transformation functions](https://docs.victoriametrics.com/victorialogs/logsql/#transformations).
|
||||||
- [Stream context](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-context).
|
- [Stream context](https://docs.victoriametrics.com/victorialogs/logsql/#stream-context).
|
||||||
- Live tailing for [LogsQL filters](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#filters) aka `tail -f`.
|
- Live tailing for [LogsQL filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) aka `tail -f`.
|
||||||
- Web UI with the following abilities:
|
- Web UI with the following abilities:
|
||||||
- Explore the ingested logs ([partially done](https://docs.victoriametrics.com/VictoriaLogs/querying/#web-ui)).
|
- Explore the ingested logs ([partially done](https://docs.victoriametrics.com/VictoriaLogs/querying/#web-ui)).
|
||||||
- Build graphs over time for the ingested logs via [hits HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-hits-stats).
|
- Build graphs over time for the ingested logs via [hits HTTP API](https://docs.victoriametrics.com/victorialogs/querying/#querying-hits-stats).
|
||||||
|
|
|
@ -80,7 +80,7 @@ The command should return the following response:
|
||||||
```
|
```
|
||||||
|
|
||||||
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
||||||
See [how to query specific fields](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields).
|
See [how to query specific fields](https://docs.victoriametrics.com/victorialogs/logsql/#querying-specific-fields).
|
||||||
|
|
||||||
The duration of requests to `/insert/elasticsearch/_bulk` can be monitored with `vl_http_request_duration_seconds{path="/insert/elasticsearch/_bulk"}` metric.
|
The duration of requests to `/insert/elasticsearch/_bulk` can be monitored with `vl_http_request_duration_seconds{path="/insert/elasticsearch/_bulk"}` metric.
|
||||||
|
|
||||||
|
@ -132,7 +132,7 @@ The command should return the following response:
|
||||||
```
|
```
|
||||||
|
|
||||||
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
||||||
See [how to query specific fields](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields).
|
See [how to query specific fields](https://docs.victoriametrics.com/victorialogs/logsql/#querying-specific-fields).
|
||||||
|
|
||||||
The duration of requests to `/insert/jsonline` can be monitored with `vl_http_request_duration_seconds{path="/insert/jsonline"}` metric.
|
The duration of requests to `/insert/jsonline` can be monitored with `vl_http_request_duration_seconds{path="/insert/jsonline"}` metric.
|
||||||
|
|
||||||
|
@ -171,7 +171,7 @@ The command should return the following response:
|
||||||
```
|
```
|
||||||
|
|
||||||
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
||||||
See [how to query specific fields](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields).
|
See [how to query specific fields](https://docs.victoriametrics.com/victorialogs/logsql/#querying-specific-fields).
|
||||||
|
|
||||||
The duration of requests to `/insert/loki/api/v1/push` can be monitored with `vl_http_request_duration_seconds{path="/insert/loki/api/v1/push"}` metric.
|
The duration of requests to `/insert/loki/api/v1/push` can be monitored with `vl_http_request_duration_seconds{path="/insert/loki/api/v1/push"}` metric.
|
||||||
|
|
||||||
|
@ -222,12 +222,12 @@ curl http://localhost:9428/select/logsql/query -d 'query=*' | head
|
||||||
```
|
```
|
||||||
|
|
||||||
This command selects all the data ingested into VictoriaLogs via [HTTP query API](https://docs.victoriametrics.com/VictoriaLogs/querying/#http-api)
|
This command selects all the data ingested into VictoriaLogs via [HTTP query API](https://docs.victoriametrics.com/VictoriaLogs/querying/#http-api)
|
||||||
using [any value filter](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#any-value-filter),
|
using [any value filter](https://docs.victoriametrics.com/victorialogs/logsql/#any-value-filter),
|
||||||
while `head` cancels query execution after reading the first 10 log lines. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line)
|
while `head` cancels query execution after reading the first 10 log lines. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line)
|
||||||
for more details on how `head` integrates with VictoriaLogs.
|
for more details on how `head` integrates with VictoriaLogs.
|
||||||
|
|
||||||
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
||||||
See [how to query specific fields](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields).
|
See [how to query specific fields](https://docs.victoriametrics.com/victorialogs/logsql/#querying-specific-fields).
|
||||||
|
|
||||||
VictoriaLogs provides the following command-line flags, which can help debugging data ingestion issues:
|
VictoriaLogs provides the following command-line flags, which can help debugging data ingestion issues:
|
||||||
|
|
||||||
|
|
|
@ -55,7 +55,7 @@ during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-inges
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
- Arrays, numbers and boolean values are converted into strings. This simplifies [full-text search](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) over such values.
|
- Arrays, numbers and boolean values are converted into strings. This simplifies [full-text search](https://docs.victoriametrics.com/victorialogs/logsql/) over such values.
|
||||||
For example, the following JSON with an array, a number and a boolean value is converted into the following JSON with string values:
|
For example, the following JSON with an array, a number and a boolean value is converted into the following JSON with string values:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
|
@ -87,7 +87,7 @@ Unicode chars must be encoded with [UTF-8](https://en.wikipedia.org/wiki/UTF-8)
|
||||||
```
|
```
|
||||||
|
|
||||||
VictoriaLogs automatically indexes all the fields in all the [ingested](https://docs.victoriametrics.com/victorialogs/data-ingestion/) logs.
|
VictoriaLogs automatically indexes all the fields in all the [ingested](https://docs.victoriametrics.com/victorialogs/data-ingestion/) logs.
|
||||||
This enables [full-text search](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) across all the fields.
|
This enables [full-text search](https://docs.victoriametrics.com/victorialogs/logsql/) across all the fields.
|
||||||
|
|
||||||
VictoriaLogs supports the following field types:
|
VictoriaLogs supports the following field types:
|
||||||
|
|
||||||
|
@ -133,7 +133,7 @@ during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-inges
|
||||||
If `_time` field is missing, then the data ingestion time is used as log entry timestamp.
|
If `_time` field is missing, then the data ingestion time is used as log entry timestamp.
|
||||||
|
|
||||||
The log entry timestamp allows quickly narrowing down the search to a particular time range.
|
The log entry timestamp allows quickly narrowing down the search to a particular time range.
|
||||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#time-filter) for details.
|
See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) for details.
|
||||||
|
|
||||||
### Stream fields
|
### Stream fields
|
||||||
|
|
||||||
|
@ -149,7 +149,7 @@ VictoriaLogs optimizes storing and querying of individual log streams. This prov
|
||||||
than a mixed log stream from multiple distinct applications.
|
than a mixed log stream from multiple distinct applications.
|
||||||
|
|
||||||
- Increased query performance, since VictoriaLogs needs to scan lower amounts of data
|
- Increased query performance, since VictoriaLogs needs to scan lower amounts of data
|
||||||
when [searching by stream labels](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-filter).
|
when [searching by stream labels](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter).
|
||||||
|
|
||||||
VictoriaLogs cannot determine automatically, which fields uniquely identify every log stream,
|
VictoriaLogs cannot determine automatically, which fields uniquely identify every log stream,
|
||||||
so it stores all the received log entries in a single default stream - `{}`.
|
so it stores all the received log entries in a single default stream - `{}`.
|
||||||
|
@ -186,7 +186,7 @@ In this case it is OK to associate the log stream with filepath fields such as `
|
||||||
Structured logs may contain big number of fields, which do not change across log entries received from a single application instance.
|
Structured logs may contain big number of fields, which do not change across log entries received from a single application instance.
|
||||||
There is no need in associating all these fields with log stream - it is enough to associate only those fields, which uniquely identify
|
There is no need in associating all these fields with log stream - it is enough to associate only those fields, which uniquely identify
|
||||||
the application instance across all the ingested logs. Additionally, some fields such as `datacenter`, `environment`, `namespace`, `job` or `app`,
|
the application instance across all the ingested logs. Additionally, some fields such as `datacenter`, `environment`, `namespace`, `job` or `app`,
|
||||||
can be associated with log stream in order to optimize searching by these fields with [stream filtering](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-filter).
|
can be associated with log stream in order to optimize searching by these fields with [stream filtering](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter).
|
||||||
|
|
||||||
Never associate log streams with fields, which may change across log entries of the same application instance. See [these docs](#high-cardinality) for details.
|
Never associate log streams with fields, which may change across log entries of the same application instance. See [these docs](#high-cardinality) for details.
|
||||||
|
|
||||||
|
@ -218,5 +218,5 @@ The rest of [structured logging](#data-model) fields are optional. They can be u
|
||||||
For example, it is usually faster to search over a dedicated `trace_id` field instead of searching for the `trace_id` inside long log message.
|
For example, it is usually faster to search over a dedicated `trace_id` field instead of searching for the `trace_id` inside long log message.
|
||||||
E.g. the `trace_id:XXXX-YYYY-ZZZZ` query usually works faster than the `_msg:"trace_id=XXXX-YYYY-ZZZZ"` query.
|
E.g. the `trace_id:XXXX-YYYY-ZZZZ` query usually works faster than the `_msg:"trace_id=XXXX-YYYY-ZZZZ"` query.
|
||||||
|
|
||||||
See [LogsQL docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) for more details.
|
See [LogsQL docs](https://docs.victoriametrics.com/victorialogs/logsql/) for more details.
|
||||||
|
|
||||||
|
|
|
@ -14,7 +14,7 @@ aliases:
|
||||||
|
|
||||||
# Querying
|
# Querying
|
||||||
|
|
||||||
[VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can be queried with [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html)
|
[VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can be queried with [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/)
|
||||||
via the following ways:
|
via the following ways:
|
||||||
|
|
||||||
- [Web UI](#web-ui) - a web-based UI for querying logs
|
- [Web UI](#web-ui) - a web-based UI for querying logs
|
||||||
|
@ -36,7 +36,7 @@ VictoriaLogs provides the following HTTP endpoints:
|
||||||
### Querying logs
|
### Querying logs
|
||||||
|
|
||||||
Logs stored in VictoriaLogs can be queried at the `/select/logsql/query` HTTP endpoint.
|
Logs stored in VictoriaLogs can be queried at the `/select/logsql/query` HTTP endpoint.
|
||||||
The [LogsQL](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) query must be passed via `query` argument.
|
The [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/) query must be passed via `query` argument.
|
||||||
For example, the following query returns all the log entries with the `error` word:
|
For example, the following query returns all the log entries with the `error` word:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
@ -44,13 +44,13 @@ curl http://localhost:9428/select/logsql/query -d 'query=error'
|
||||||
```
|
```
|
||||||
|
|
||||||
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
The response by default contains all the [log fields](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).
|
||||||
See [how to query specific fields](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#querying-specific-fields).
|
See [how to query specific fields](https://docs.victoriametrics.com/victorialogs/logsql/#querying-specific-fields).
|
||||||
|
|
||||||
The `query` argument can be passed either in the request url itself (aka HTTP GET request) or via request body
|
The `query` argument can be passed either in the request url itself (aka HTTP GET request) or via request body
|
||||||
with the `x-www-form-urlencoded` encoding (aka HTTP POST request). The HTTP POST is useful for sending long queries
|
with the `x-www-form-urlencoded` encoding (aka HTTP POST request). The HTTP POST is useful for sending long queries
|
||||||
when they do not fit the maximum url length of the used clients and proxies.
|
when they do not fit the maximum url length of the used clients and proxies.
|
||||||
|
|
||||||
See [LogsQL docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) for details on what can be passed to the `query` arg.
|
See [LogsQL docs](https://docs.victoriametrics.com/victorialogs/logsql/) for details on what can be passed to the `query` arg.
|
||||||
The `query` arg must be properly encoded with [percent encoding](https://en.wikipedia.org/wiki/URL_encoding) when passing it to `curl`
|
The `query` arg must be properly encoded with [percent encoding](https://en.wikipedia.org/wiki/URL_encoding) when passing it to `curl`
|
||||||
or similar tools.
|
or similar tools.
|
||||||
|
|
||||||
|
@ -86,7 +86,7 @@ This allows post-processing the returned lines at the client side with the usual
|
||||||
See [these docs](#command-line) for more details.
|
See [these docs](#command-line) for more details.
|
||||||
|
|
||||||
The returned lines aren't sorted, since sorting disables the ability to send matching log entries to response stream as soon as they are found.
|
The returned lines aren't sorted, since sorting disables the ability to send matching log entries to response stream as soon as they are found.
|
||||||
Query results can be sorted either at VictoriaLogs side according [to these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#sort-pipe)
|
Query results can be sorted either at VictoriaLogs side according [to these docs](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe)
|
||||||
or at client side with the usual `sort` command according to [these docs](#command-line).
|
or at client side with the usual `sort` command according to [these docs](#command-line).
|
||||||
|
|
||||||
By default the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy) is queried.
|
By default the `(AccountID=0, ProjectID=0)` [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy) is queried.
|
||||||
|
@ -443,7 +443,7 @@ See also:
|
||||||
|
|
||||||
## Web UI
|
## Web UI
|
||||||
|
|
||||||
VictoriaLogs provides a simple Web UI for logs [querying](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) and exploration
|
VictoriaLogs provides a simple Web UI for logs [querying](https://docs.victoriametrics.com/victorialogs/logsql/) and exploration
|
||||||
at `http://localhost:9428/select/vmui`. The UI allows exploring query results:
|
at `http://localhost:9428/select/vmui`. The UI allows exploring query results:
|
||||||
|
|
||||||
<img src="vmui.webp" />
|
<img src="vmui.webp" />
|
||||||
|
@ -457,7 +457,7 @@ There are three modes of displaying query results:
|
||||||
This is the first version that has minimal functionality. It comes with the following limitations:
|
This is the first version that has minimal functionality. It comes with the following limitations:
|
||||||
|
|
||||||
- The number of query results is always limited to 1000 lines. Iteratively add
|
- The number of query results is always limited to 1000 lines. Iteratively add
|
||||||
more specific [filters](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#filters) to the query
|
more specific [filters](https://docs.victoriametrics.com/victorialogs/logsql/#filters) to the query
|
||||||
in order to get full response with less than 1000 lines.
|
in order to get full response with less than 1000 lines.
|
||||||
- Queries are always executed against [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy) `0`.
|
- Queries are always executed against [tenant](https://docs.victoriametrics.com/VictoriaLogs/#multitenancy) `0`.
|
||||||
|
|
||||||
|
@ -482,7 +482,7 @@ These features allow executing queries at command-line interface, which potentia
|
||||||
without the risk of high resource usage (CPU, RAM, disk IO) at VictoriaLogs server.
|
without the risk of high resource usage (CPU, RAM, disk IO) at VictoriaLogs server.
|
||||||
|
|
||||||
For example, the following query can return very big number of matching log entries (e.g. billions) if VictoriaLogs contains
|
For example, the following query can return very big number of matching log entries (e.g. billions) if VictoriaLogs contains
|
||||||
many log messages with the `error` [word](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#word):
|
many log messages with the `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl http://localhost:9428/select/logsql/query -d 'query=error'
|
curl http://localhost:9428/select/logsql/query -d 'query=error'
|
||||||
|
@ -512,9 +512,9 @@ It doesn't consume CPU and disk IO resources during this time. It resumes query
|
||||||
when the `less` continues reading the response stream.
|
when the `less` continues reading the response stream.
|
||||||
|
|
||||||
Suppose that the initial investigation of the returned query results helped determining that the needed log messages contain
|
Suppose that the initial investigation of the returned query results helped determining that the needed log messages contain
|
||||||
`cannot open file` [phrase](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#phrase-filter).
|
`cannot open file` [phrase](https://docs.victoriametrics.com/victorialogs/logsql/#phrase-filter).
|
||||||
Then the query can be narrowed down to `error AND "cannot open file"`
|
Then the query can be narrowed down to `error AND "cannot open file"`
|
||||||
(see [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#logical-filter) about `AND` operator).
|
(see [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter) about `AND` operator).
|
||||||
Then run the updated command in order to continue the investigation:
|
Then run the updated command in order to continue the investigation:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
@ -531,7 +531,7 @@ The returned VictoriaLogs query response can be post-processed with any combinat
|
||||||
which are usually used for log analysis - `grep`, `jq`, `awk`, `sort`, `uniq`, `wc`, etc.
|
which are usually used for log analysis - `grep`, `jq`, `awk`, `sort`, `uniq`, `wc`, etc.
|
||||||
|
|
||||||
For example, the following command uses `wc -l` Unix command for counting the number of log messages
|
For example, the following command uses `wc -l` Unix command for counting the number of log messages
|
||||||
with the `error` [word](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#word)
|
with the `error` [word](https://docs.victoriametrics.com/victorialogs/logsql/#word)
|
||||||
received from [streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with `app="nginx"` field
|
received from [streams](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) with `app="nginx"` field
|
||||||
during the last 5 minutes:
|
during the last 5 minutes:
|
||||||
|
|
||||||
|
@ -539,9 +539,9 @@ during the last 5 minutes:
|
||||||
curl http://localhost:9428/select/logsql/query -d 'query=_stream:{app="nginx"} AND _time:5m AND error' | wc -l
|
curl http://localhost:9428/select/logsql/query -d 'query=_stream:{app="nginx"} AND _time:5m AND error' | wc -l
|
||||||
```
|
```
|
||||||
|
|
||||||
See [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#stream-filter) about `_stream` filter,
|
See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#stream-filter) about `_stream` filter,
|
||||||
[these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#time-filter) about `_time` filter
|
[these docs](https://docs.victoriametrics.com/victorialogs/logsql/#time-filter) about `_time` filter
|
||||||
and [these docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#logical-filter) about `AND` operator.
|
and [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#logical-filter) about `AND` operator.
|
||||||
|
|
||||||
The following example shows how to sort query results by the [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field):
|
The following example shows how to sort query results by the [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field):
|
||||||
|
|
||||||
|
@ -555,7 +555,7 @@ and piping them to `sort` command.
|
||||||
|
|
||||||
Note that the `sort` command needs to read all the response stream before returning the sorted results. So the command above
|
Note that the `sort` command needs to read all the response stream before returning the sorted results. So the command above
|
||||||
can take non-trivial amounts of time if the `query` returns too many results. The solution is to narrow down the `query`
|
can take non-trivial amounts of time if the `query` returns too many results. The solution is to narrow down the `query`
|
||||||
before sorting the results. See [these tips](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#performance-tips)
|
before sorting the results. See [these tips](https://docs.victoriametrics.com/victorialogs/logsql/#performance-tips)
|
||||||
on how to narrow down query results.
|
on how to narrow down query results.
|
||||||
|
|
||||||
The following example calculates stats on the number of log messages received during the last 5 minutes
|
The following example calculates stats on the number of log messages received during the last 5 minutes
|
||||||
|
@ -565,11 +565,11 @@ grouped by `log.level` [field](https://docs.victoriametrics.com/victorialogs/key
|
||||||
curl http://localhost:9428/select/logsql/query -d 'query=_time:5m log.level:*' | jq -r '."log.level"' | sort | uniq -c
|
curl http://localhost:9428/select/logsql/query -d 'query=_time:5m log.level:*' | jq -r '."log.level"' | sort | uniq -c
|
||||||
```
|
```
|
||||||
|
|
||||||
The query selects all the log messages with non-empty `log.level` field via ["any value" filter](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#any-value-filter),
|
The query selects all the log messages with non-empty `log.level` field via ["any value" filter](https://docs.victoriametrics.com/victorialogs/logsql/#any-value-filter),
|
||||||
then pipes them to `jq` command, which extracts the `log.level` field value from the returned JSON stream, then the extracted `log.level` values
|
then pipes them to `jq` command, which extracts the `log.level` field value from the returned JSON stream, then the extracted `log.level` values
|
||||||
are sorted with `sort` command and, finally, they are passed to `uniq -c` command for calculating the needed stats.
|
are sorted with `sort` command and, finally, they are passed to `uniq -c` command for calculating the needed stats.
|
||||||
|
|
||||||
See also:
|
See also:
|
||||||
|
|
||||||
- [Key concepts](https://docs.victoriametrics.com/victorialogs/keyconcepts/).
|
- [Key concepts](https://docs.victoriametrics.com/victorialogs/keyconcepts/).
|
||||||
- [LogsQL docs](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html).
|
- [LogsQL docs](https://docs.victoriametrics.com/victorialogs/logsql/).
|
||||||
|
|
Loading…
Reference in a new issue