VictoriaMetrics/docs/VictoriaLogs

VictoriaLogs

VictoriaLogs is log management and log analytics system from VictoriaMetrics.

It provides the following key features:

• VictoriaLogs can accept logs from popular log collectors. See these docs.
• VictoriaLogs is much easier to setup and operate comparing to ElasticSearch and Grafana Loki. See these docs.
• VictoriaLogs provides easy yet powerful query language with full-text search capabilities across all the log fields - see LogsQL docs.
• VictoriaLogs can be seamlessly combined with good old Unix tools for log analysis such as grep, less, sort, jq, etc. See these docs for details.
• VictoriaLogs capacity and performance scales linearly with the available resources (CPU, RAM, disk IO, disk space). It runs smoothly on both Raspberry PI and a server with hundreds of CPU cores and terabytes of RAM.
• VictoriaLogs can handle much bigger data volumes than ElasticSearch and Grafana Loki when running on comparable hardware. See these docs.
• VictoriaLogs supports multitenancy - see these docs.
• VictoriaLogs supports out of order logs' ingestion aka backfilling.
• VictoriaLogs provides simple web UI for querying logs - see these docs.

VictoriaLogs is at Preview stage now. It is ready for evaluation in production and verifying claims given above. It isn't recommended migrating from existing logging solutions to VictoriaLogs Preview in general case yet. See the Roadmap for details.

If you have questions about VictoriaLogs, then feel free asking them at VictoriaMetrics community Slack chat.

See Quick start docs for start working with VictoriaLogs.

Monitoring

VictoriaLogs exposes internal metrics in Prometheus exposition format at http://localhost:9428/metrics page. It is recommended to set up monitoring of these metrics via VictoriaMetrics (see these docs), vmagent (see these docs) or via Prometheus.

VictoriaLogs emits own logs to stdout. It is recommended investigating these logs during troubleshooting.

Upgrading

It is safe upgrading VictoriaLogs to new versions unless release notes say otherwise. It is safe skipping multiple versions during the upgrade unless release notes say otherwise. It is recommended performing regular upgrades to the latest version, since it may contain important bug fixes, performance optimizations or new features.

It is also safe downgrading to older versions unless release notes say otherwise.

The following steps must be performed during the upgrade / downgrade procedure:

• Send SIGINT signal to VictoriaLogs process in order to gracefully stop it. See how to send signals to processes.
• Wait until the process stops. This can take a few seconds.
• Start the upgraded VictoriaMetrics.

Retention

By default VictoriaLogs stores log entries with timestamps in the time range [now-7d, now], while dropping logs outside the given time range. E.g. it uses the retention of 7 days. The retention can be configured with -retentionPeriod command-line flag. This flag accepts values starting from 1d (one day) up to 100y (100 years). See these docs for the supported duration formats.

For example, the following command starts VictoriaLogs with the retention of 8 weeks:

/path/to/victoria-logs -retentionPeriod=8w

VictoriaLogs stores the ingested logs in per-day partition directories. It automatically drops partition directories outside the configured retention.

VictoriaLogs automatically drops logs at data ingestion stage if they have timestamps outside the configured retention. A sample of dropped logs is logged with WARN message in order to simplify troubleshooting. The vl_rows_dropped_total metric is incremented each time an ingested log entry is dropped because of timestamp outside the retention. It is recommended setting up the following alerting rule at vmalert in order to be notified when logs with wrong timestamps are ingested into VictoriaLogs:

rate(vl_rows_dropped_total[5m]) > 0

By default VictoriaLogs doesn't accept log entries with timestamps bigger than now+2d, e.g. 2 days in the future. If you need accepting logs with bigger timestamps, then specify the desired "future retention" via -futureRetention command-line flag. This flag accepts values starting from 1d. See these docs for the supported duration formats.

For example, the following command starts VictoriaLogs, which accepts logs with timestamps up to a year in the future:

/path/to/victoria-logs -futureRetention=1y

Storage

VictoriaLogs stores all its data in a single directory - victoria-logs-data. The path to the directory can be changed via -storageDataPath command-line flag. For example, the following command starts VictoriaLogs, which stores the data at /var/lib/victoria-logs:

/path/to/victoria-logs -storageDataPath=/var/lib/victoria-logs

VictoriaLogs automatically creates the -storageDataPath directory on the first run if it is missing.

Multitenancy

VictoriaLogs supports multitenancy. A tenant is identified by (AccountID, ProjectID) pair, where AccountID and ProjectID are arbitrary 32-bit unsigned integeres. The AccountID and ProjectID fields can be set during data ingestion and querying via AccountID and ProjectID request headers.

If AccountID and/or ProjectID request headers aren't set, then the default 0 value is used.

VictoriaLogs has very low overhead for per-tenant management, so it is OK to have thousands of tenants in a single VictoriaLogs instance.

VictoriaLogs doesn't perform per-tenant authorization. Use vmauth or similar tools for per-tenant authorization.

Benchmarks

Here is a benchmark suite for comparing data ingestion performance and resource usage between VictoriaLogs and Elasticsearch.

It is recommended setting up VictoriaLogs in production alongside the existing log management systems and comparing resource usage + query performance between VictoriaLogs and your system such as ElasticSearch or Grafana Loki.

Please share benchmark results and ideas on how to improve benchmarks / VictoriaLogs via VictoriaMetrics community channels.