diff --git a/app/vlinsert/insertutils/common_params.go b/app/vlinsert/insertutils/common_params.go index 23f100775..389bfa0c9 100644 --- a/app/vlinsert/insertutils/common_params.go +++ b/app/vlinsert/insertutils/common_params.go @@ -14,7 +14,7 @@ import ( // CommonParams contains common HTTP parameters used by log ingestion APIs. // -// See https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters +// See https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters type CommonParams struct { TenantID logstorage.TenantID TimeField string diff --git a/app/vlstorage/main.go b/app/vlstorage/main.go index e36966888..ddfc6afd0 100644 --- a/app/vlstorage/main.go +++ b/app/vlstorage/main.go @@ -32,7 +32,7 @@ var ( logNewStreams = flag.Bool("logNewStreams", false, "Whether to log creation of new streams; this can be useful for debugging of high cardinality issues with log streams; "+ "see https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields ; see also -logIngestedRows") logIngestedRows = flag.Bool("logIngestedRows", false, "Whether to log all the ingested log entries; this can be useful for debugging of data ingestion; "+ - "see https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/ ; see also -logNewStreams") + "see https://docs.victoriametrics.com/victorialogs/data-ingestion/ ; see also -logNewStreams") minFreeDiskSpaceBytes = flagutil.NewBytes("storage.minFreeDiskSpaceBytes", 10e6, "The minimum free disk space at -storageDataPath after which "+ "the storage stops accepting new data") ) diff --git a/docs/VictoriaLogs/CHANGELOG.md b/docs/VictoriaLogs/CHANGELOG.md index e8e25d81b..3103f63cc 100644 --- a/docs/VictoriaLogs/CHANGELOG.md +++ b/docs/VictoriaLogs/CHANGELOG.md @@ -164,7 +164,7 @@ Released at 2023-10-04 Released at 2023-10-03 -* FEATURE: add `-elasticsearch.version` command-line flag, which can be used for specifying Elasticsearch version returned by VictoriaLogs to Filebeat at [elasticsearch bulk API](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#elasticsearch-bulk-api). This helps resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777). +* FEATURE: add `-elasticsearch.version` command-line flag, which can be used for specifying Elasticsearch version returned by VictoriaLogs to Filebeat at [elasticsearch bulk API](https://docs.victoriametrics.com/victorialogs/data-ingestion/#elasticsearch-bulk-api). This helps resolving [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777). * FEATURE: expose the following metrics at [/metrics](https://docs.victoriametrics.com/VictoriaLogs/#monitoring) page: * `vl_data_size_bytes{type="storage"}` - on-disk size for data excluding [log stream](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) indexes. * `vl_data_size_bytes{type="indexdb"}` - on-disk size for [log stream](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) indexes. @@ -180,7 +180,7 @@ Released at 2023-10-03 Released at 2023-07-20 -* FEATURE: add support for data ingestion via Promtail (aka default log shipper for Grafana Loki). See [these](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Promtail.html) and [these](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#loki-json-api) docs. +* FEATURE: add support for data ingestion via Promtail (aka default log shipper for Grafana Loki). See [these](https://docs.victoriametrics.com/victorialogs/data-ingestion/Promtail.html) and [these](https://docs.victoriametrics.com/victorialogs/data-ingestion/#loki-json-api) docs. ## [v0.2.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.2.0-victorialogs) diff --git a/docs/VictoriaLogs/FAQ.md b/docs/VictoriaLogs/FAQ.md index bd6da3e92..caea49ecb 100644 --- a/docs/VictoriaLogs/FAQ.md +++ b/docs/VictoriaLogs/FAQ.md @@ -92,7 +92,7 @@ VictoriaLogs is designed solely for logs. VictoriaLogs uses [similar design idea LogsQL is usually much easier to use than SQL for typical log analysis tasks, while some non-trivial analytics may require SQL power. -- VictoriaLogs accepts logs from popular log shippers out of the box - see [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +- VictoriaLogs accepts logs from popular log shippers out of the box - see [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/). ClickHouse needs an intermediate applications for converting the ingested logs into `INSERT` SQL statements for the particular database schema. This may increase the complexity of the system and, subsequently, increase its' maintenance costs. diff --git a/docs/VictoriaLogs/LogsQL.md b/docs/VictoriaLogs/LogsQL.md index 0e7721c7c..9e3c6ba72 100644 --- a/docs/VictoriaLogs/LogsQL.md +++ b/docs/VictoriaLogs/LogsQL.md @@ -27,7 +27,7 @@ If you aren't familiar with VictoriaLogs, then start with [key concepts docs](ht Then follow these docs: - [How to run VictoriaLogs](https://docs.victoriametrics.com/victorialogs/quickstart/). -- [how to ingest data into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +- [how to ingest data into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/). - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). The simplest LogsQL query is just a [word](#word), which must be found in the [log message](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#message-field). @@ -150,7 +150,7 @@ _time:5m log.level:error !app:(buggy_app OR foobar) The `app` field uniquely identifies the application instance if a single instance runs per each unique `app`. In this case it is recommended associating the `app` field with [log stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). This usually improves both compression rate +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). This usually improves both compression rate and query performance when querying the needed streams via [`_stream` filter](#stream-filter). If the `app` field is associated with the log stream, then the query above can be rewritten to more performant one: diff --git a/docs/VictoriaLogs/QuickStart.md b/docs/VictoriaLogs/QuickStart.md index 52bcfd56c..3a1df37f6 100644 --- a/docs/VictoriaLogs/QuickStart.md +++ b/docs/VictoriaLogs/QuickStart.md @@ -39,16 +39,16 @@ tar xzf victoria-logs-linux-amd64-v0.10.0-victorialogs.tar.gz ./victoria-logs-prod ``` -VictoriaLogs is ready for [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +VictoriaLogs is ready for [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) at the TCP port `9428` now! It has no any external dependencies, so it may run in various environments without additional setup and configuration. VictoriaLogs automatically adapts to the available CPU and RAM resources. It also automatically setups and creates -the needed indexes during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +the needed indexes during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). See also: - [How to configure VictoriaLogs](#how-to-configure-victorialogs) -- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) @@ -65,7 +65,7 @@ docker run --rm -it -p 9428:9428 -v ./victoria-logs-data:/victoria-logs-data \ See also: - [How to configure VictoriaLogs](#how-to-configure-victorialogs) -- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) ### Helm charts @@ -96,16 +96,16 @@ Follow the following steps in order to build VictoriaLogs from source code: bin/victoria-logs ``` -VictoriaLogs is ready for [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +VictoriaLogs is ready for [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) at the TCP port `9428` now! It has no any external dependencies, so it may run in various environments without additional setup and configuration. VictoriaLogs automatically adapts to the available CPU and RAM resources. It also automatically setups and creates -the needed indexes during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +the needed indexes during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). See also: - [How to configure VictoriaLogs](#how-to-configure-victorialogs) -- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) @@ -127,13 +127,13 @@ via `-storageDataPath` command-line flag. See [these docs](https://docs.victoria By default VictoriaLogs stores [log entries](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html) with timestamps in the time range `[now-7d, now]`, while dropping logs outside the given time range. E.g. it uses the retention of 7 days. Read [these docs](https://docs.victoriametrics.com/VictoriaLogs/#retention) on how to control the retention -for the [ingested](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) logs. +for the [ingested](https://docs.victoriametrics.com/victorialogs/data-ingestion/) logs. It is recommended setting up monitoring of VictoriaLogs according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/#monitoring). See also: -- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- [How to ingest logs into VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/) - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/) ## Docker demos diff --git a/docs/VictoriaLogs/README.md b/docs/VictoriaLogs/README.md index 985e849c8..dd313ab8c 100644 --- a/docs/VictoriaLogs/README.md +++ b/docs/VictoriaLogs/README.md @@ -12,7 +12,7 @@ from [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/). VictoriaLogs provides the following key features: -- VictoriaLogs can accept logs from popular log collectors. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +- VictoriaLogs can accept logs from popular log collectors. See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/). - VictoriaLogs is much easier to set up and operate compared to Elasticsearch and Grafana Loki. See [these docs](https://docs.victoriametrics.com/victorialogs/quickstart/). - VictoriaLogs provides easy yet powerful query language with full-text search capabilities across @@ -77,10 +77,10 @@ For example, the following command starts VictoriaLogs with the retention of 8 w /path/to/victoria-logs -retentionPeriod=8w ``` -VictoriaLogs stores the [ingested](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) logs in per-day partition directories. +VictoriaLogs stores the [ingested](https://docs.victoriametrics.com/victorialogs/data-ingestion/) logs in per-day partition directories. It automatically drops partition directories outside the configured retention. -VictoriaLogs automatically drops logs at [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) stage +VictoriaLogs automatically drops logs at [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) stage if they have timestamps outside the configured retention. A sample of dropped logs is logged with `WARN` message in order to simplify troubleshooting. The `vl_rows_dropped_total` [metric](#monitoring) is incremented each time an ingested log entry is dropped because of timestamp outside the retention. It is recommended to set up the following alerting rule at [vmalert](https://docs.victoriametrics.com/vmalert/) in order to be notified @@ -115,7 +115,7 @@ VictoriaLogs automatically creates the `-storageDataPath` directory on the first ## Multitenancy VictoriaLogs supports multitenancy. A tenant is identified by `(AccountID, ProjectID)` pair, where `AccountID` and `ProjectID` are arbitrary 32-bit unsigned integers. -The `AccountID` and `ProjectID` fields can be set during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +The `AccountID` and `ProjectID` fields can be set during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) via `AccountID` and `ProjectID` request headers. If `AccountID` and/or `ProjectID` request headers aren't set, then the default `0` value is used. @@ -201,7 +201,7 @@ Pass `-help` to VictoriaLogs in order to see the list of supported command-line -internStringMaxLen int The maximum length for strings to intern. A lower limit may save memory at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringDisableCache and -internStringCacheExpireDuration (default 500) -logIngestedRows - Whether to log all the ingested log entries; this can be useful for debugging of data ingestion; see https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/ ; see also -logNewStreams + Whether to log all the ingested log entries; this can be useful for debugging of data ingestion; see https://docs.victoriametrics.com/victorialogs/data-ingestion/ ; see also -logNewStreams -logNewStreams Whether to log creation of new streams; this can be useful for debugging of high cardinality issues with log streams; see https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields ; see also -logIngestedRows -loggerDisableTimestamps diff --git a/docs/VictoriaLogs/Roadmap.md b/docs/VictoriaLogs/Roadmap.md index 67bd62b8e..a8dade3d9 100644 --- a/docs/VictoriaLogs/Roadmap.md +++ b/docs/VictoriaLogs/Roadmap.md @@ -21,7 +21,7 @@ It isn't recommended migrating from existing solutions to VictoriaLogs Preview y The following functionality is available in VictoriaLogs Preview: -- [Data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +- [Data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). - [Querying](https://docs.victoriametrics.com/VictoriaLogs/querying/). - [Querying via command-line](https://docs.victoriametrics.com/VictoriaLogs/querying/#command-line). @@ -29,7 +29,7 @@ See [these docs](https://docs.victoriametrics.com/VictoriaLogs/) for details. The following functionality is planned in the future versions of VictoriaLogs: -- Support for [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) from popular log collectors and formats: +- Support for [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) from popular log collectors and formats: - OpenTelemetry for logs - Fluentd - Syslog diff --git a/docs/VictoriaLogs/data-ingestion/Filebeat.md b/docs/VictoriaLogs/data-ingestion/Filebeat.md index 1185a635a..c9aa6cdf6 100644 --- a/docs/VictoriaLogs/data-ingestion/Filebeat.md +++ b/docs/VictoriaLogs/data-ingestion/Filebeat.md @@ -26,11 +26,11 @@ output.elasticsearch: Substitute the `localhost:9428` address inside `hosts` section with the real TCP address of VictoriaLogs. -See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the `parameters` section. +See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on the `parameters` section. It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). -This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) and inspecting VictoriaLogs logs then: ```yaml @@ -44,7 +44,7 @@ output.elasticsearch: ``` If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped -during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: ```yaml @@ -118,7 +118,7 @@ command-line flag. See also: -- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting). - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). - [Filebeat `output.elasticsearch` docs](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html). - [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/filebeat-docker). diff --git a/docs/VictoriaLogs/data-ingestion/Fluentbit.md b/docs/VictoriaLogs/data-ingestion/Fluentbit.md index 410387469..6c53fa349 100644 --- a/docs/VictoriaLogs/data-ingestion/Fluentbit.md +++ b/docs/VictoriaLogs/data-ingestion/Fluentbit.md @@ -28,11 +28,11 @@ for sending the collected logs to [VictoriaLogs](https://docs.victoriametrics.co Substitute the host (`localhost`) and port (`9428`) with the real TCP address of VictoriaLogs. -See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the query args specified in the `uri`. +See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on the query args specified in the `uri`. It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). -This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) in the `uri` +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) in the `uri` and inspecting VictoriaLogs logs then: ```conf @@ -47,7 +47,7 @@ and inspecting VictoriaLogs logs then: ``` If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped -during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: ```conf @@ -95,7 +95,7 @@ For example, the following `fluentbit.conf` config instructs Fluentbit to store See also: -- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting). - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). - [Fluentbit HTTP output config docs](https://docs.fluentbit.io/manual/pipeline/outputs/http). - [Docker-compose demo for Fluentbit integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-docker). diff --git a/docs/VictoriaLogs/data-ingestion/Logstash.md b/docs/VictoriaLogs/data-ingestion/Logstash.md index 3d60377c1..cb1a938ef 100644 --- a/docs/VictoriaLogs/data-ingestion/Logstash.md +++ b/docs/VictoriaLogs/data-ingestion/Logstash.md @@ -30,11 +30,11 @@ output { Substitute `localhost:9428` address inside `hosts` with the real TCP address of VictoriaLogs. -See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on the `parameters` section. +See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on the `parameters` section. It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). -This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) and inspecting VictoriaLogs logs then: ```conf @@ -52,7 +52,7 @@ output { ``` If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped -during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: ```conf @@ -109,7 +109,7 @@ output { See also: -- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting). - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). - [Logstash `output.elasticsearch` docs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html). - [Docker-compose demo for Logstash integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash). diff --git a/docs/VictoriaLogs/data-ingestion/Promtail.md b/docs/VictoriaLogs/data-ingestion/Promtail.md index 3ee401f78..69453d045 100644 --- a/docs/VictoriaLogs/data-ingestion/Promtail.md +++ b/docs/VictoriaLogs/data-ingestion/Promtail.md @@ -30,12 +30,12 @@ with the list of labels, which uniquely identify log streams. There is no need i it is usually enough specifying `instance` and `job` labels. See [these docs](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields) for details. -See also [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on other supported query args. +See also [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on other supported query args. There is no need in specifying `_msg_field` and `_time_field` query args, since VictoriaLogs automatically extracts log message and timestamp from the ingested Loki data. It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). -This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) and inspecting VictoriaLogs logs then: ```yaml @@ -44,7 +44,7 @@ clients: ``` If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped -during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). For example, the following config instructs VictoriaLogs to ignore `filename` and `stream` fields in the ingested logs: ```yaml @@ -66,4 +66,4 @@ clients: The ingested log entries can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/). -See also [data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting) docs. +See also [data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting) docs. diff --git a/docs/VictoriaLogs/data-ingestion/README.md b/docs/VictoriaLogs/data-ingestion/README.md index 8ec0e8b3c..963394a8b 100644 --- a/docs/VictoriaLogs/data-ingestion/README.md +++ b/docs/VictoriaLogs/data-ingestion/README.md @@ -16,11 +16,11 @@ aliases: [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) can accept logs from the following log collectors: -- Filebeat. See [how to setup Filebeat for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html). -- Fluentbit. See [how to setup Fluentbit for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html). -- Logstash. See [how to setup Logstash for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html). -- Vector. See [how to setup Vector for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html). -- Promtail (aka Grafana Loki). See [how to setup Promtail for sending logs to VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Promtail.html). +- Filebeat. See [how to setup Filebeat for sending logs to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/Filebeat.html). +- Fluentbit. See [how to setup Fluentbit for sending logs to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/Fluentbit.html). +- Logstash. See [how to setup Logstash for sending logs to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/Logstash.html). +- Vector. See [how to setup Vector for sending logs to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/Vector.html). +- Promtail (aka Grafana Loki). See [how to setup Promtail for sending logs to VictoriaLogs](https://docs.victoriametrics.com/victorialogs/data-ingestion/Promtail.html). The ingested logs can be queried according to [these docs](https://docs.victoriametrics.com/VictoriaLogs/querying/). @@ -258,8 +258,8 @@ Here is the list of log collectors and their ingestion formats supported by Vict | How to setup the collector | Format: Elasticsearch | Format: JSON Stream | Format: Loki | |------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------------------------------------------------------|-------------------------------------------------------------------------------------| -| [Filebeat](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Filebeat.html) | [Yes](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) | No | No | -| [Fluentbit](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Fluentbit.html) | No | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/http) | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/loki) | -| [Logstash](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Logstash.html) | [Yes](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) | No | No | -| [Vector](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Vector.html) | [Yes](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) | [Yes](https://vector.dev/docs/reference/configuration/sinks/http/) | [Yes](https://vector.dev/docs/reference/configuration/sinks/loki/) | -| [Promtail](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/Promtail.html) | No | No | [Yes](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#clients) | +| [Filebeat](https://docs.victoriametrics.com/victorialogs/data-ingestion/Filebeat.html) | [Yes](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html) | No | No | +| [Fluentbit](https://docs.victoriametrics.com/victorialogs/data-ingestion/Fluentbit.html) | No | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/http) | [Yes](https://docs.fluentbit.io/manual/pipeline/outputs/loki) | +| [Logstash](https://docs.victoriametrics.com/victorialogs/data-ingestion/Logstash.html) | [Yes](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) | No | No | +| [Vector](https://docs.victoriametrics.com/victorialogs/data-ingestion/Vector.html) | [Yes](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/) | [Yes](https://vector.dev/docs/reference/configuration/sinks/http/) | [Yes](https://vector.dev/docs/reference/configuration/sinks/loki/) | +| [Promtail](https://docs.victoriametrics.com/victorialogs/data-ingestion/Promtail.html) | No | No | [Yes](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#clients) | diff --git a/docs/VictoriaLogs/data-ingestion/Vector.md b/docs/VictoriaLogs/data-ingestion/Vector.md index 2b81a3a22..cc9286780 100644 --- a/docs/VictoriaLogs/data-ingestion/Vector.md +++ b/docs/VictoriaLogs/data-ingestion/Vector.md @@ -36,12 +36,12 @@ Substitute the `localhost:9428` address inside `endpoints` section with the real Replace `your_input` with the name of the `inputs` section, which collects logs. See [these docs](https://vector.dev/docs/reference/configuration/sources/) for details. -See [these docs](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) for details on parameters specified +See [these docs](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) for details on parameters specified in the `[sinks.vlogs.query]` section. It is recommended verifying whether the initial setup generates the needed [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) and uses the correct [stream fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#stream-fields). -This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters) +This can be done by specifying `debug` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters) in the `[sinks.vlogs.query]` section and inspecting VictoriaLogs logs then: ```toml @@ -61,7 +61,7 @@ in the `[sinks.vlogs.query]` section and inspecting VictoriaLogs logs then: ``` If some [log fields](https://docs.victoriametrics.com/VictoriaLogs/keyConcepts.html#data-model) must be skipped -during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#http-parameters). +during data ingestion, then they can be put into `ignore_fields` [parameter](https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters). For example, the following config instructs VictoriaLogs to ignore `log.offset` and `event.original` fields in the ingested logs: ```toml @@ -164,7 +164,7 @@ for sending data to [JSON stream API](https://docs.victoriametrics.com/victorial See also: -- [Data ingestion troubleshooting](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#troubleshooting). +- [Data ingestion troubleshooting](https://docs.victoriametrics.com/victorialogs/data-ingestion/#troubleshooting). - [How to query VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/querying/). - [Elasticsearch output docs for Vector](https://vector.dev/docs/reference/configuration/sinks/elasticsearch/). - [Docker-compose demo for Filebeat integration with VictoriaLogs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-docker). diff --git a/docs/VictoriaLogs/keyConcepts.md b/docs/VictoriaLogs/keyConcepts.md index d0380f31f..22e14e654 100644 --- a/docs/VictoriaLogs/keyConcepts.md +++ b/docs/VictoriaLogs/keyConcepts.md @@ -32,7 +32,7 @@ For example: ``` VictoriaLogs automatically transforms multi-level JSON (aka nested JSON) into single-level JSON -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) according to the following rules: +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) according to the following rules: - Nested dictionaries are flattened by concatenating dictionary keys with `.` char. For example, the following multi-level JSON is transformed into the following single-level JSON: @@ -75,7 +75,7 @@ during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-inges ``` Both label name and label value may contain arbitrary chars. Such chars must be encoded -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) according to [JSON string encoding](https://www.rfc-editor.org/rfc/rfc7159.html#section-7). Unicode chars must be encoded with [UTF-8](https://en.wikipedia.org/wiki/UTF-8) encoding: @@ -86,7 +86,7 @@ Unicode chars must be encoded with [UTF-8](https://en.wikipedia.org/wiki/UTF-8) } ``` -VictoriaLogs automatically indexes all the fields in all the [ingested](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) logs. +VictoriaLogs automatically indexes all the fields in all the [ingested](https://docs.victoriametrics.com/victorialogs/data-ingestion/) logs. This enables [full-text search](https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html) across all the fields. VictoriaLogs supports the following field types: @@ -109,9 +109,9 @@ log entry, which can be ingested into VictoriaLogs: ``` If the actual log message has other than `_msg` field name, then it is possible to specify the real log message field -via `_msg_field` query arg during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +via `_msg_field` query arg during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). For example, if log message is located in the `event.original` field, then specify `_msg_field=event.original` query arg -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). ### Time field @@ -126,9 +126,9 @@ For example: ``` If the actual timestamp has other than `_time` field name, then it is possible to specify the real timestamp -field via `_time_field` query arg during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +field via `_time_field` query arg during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). For example, if timestamp is located in the `event.created` field, then specify `_time_field=event.created` query arg -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). If `_time` field is missing, then the data ingestion time is used as log entry timestamp. @@ -156,7 +156,7 @@ so it stores all the received log entries in a single default stream - `{}`. This may lead to not-so-optimal resource usage and query performance. Therefore it is recommended specifying stream-level fields via `_stream_fields` query arg -during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/). +during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/). For example, if logs from Kubernetes containers have the following fields: ```json @@ -170,7 +170,7 @@ For example, if logs from Kubernetes containers have the following fields: ``` then sepcify `_stream_fields=kubernetes.namespace,kubernetes.node.name,kubernetes.pod.name,kubernetes.container.name` -query arg during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) in order to properly store +query arg during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) in order to properly store per-container logs into distinct streams. #### How to determine which fields must be associated with log streams? @@ -199,7 +199,7 @@ VictoriaLogs works perfectly with such fields unless they are associated with [l Never associate high-cardinality fields with [log streams](#stream-fields), since this may result to the following issues: -- Performance degradation during [data ingestion](https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/) +- Performance degradation during [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/VictoriaLogs/querying/) - Increased memory usage - Increased CPU usage