github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/deployment/logs-benchmark
History
Zakhar Bessarab b308088b3c
deployment/logs-benchmark: update dashboard (#4500) 
- remove second panel for disk usage. It is not very useful for users and brings more confusion than profit from having it.
- update CPU graph to show number of used CPUs to make it less ambiguous

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:42:08 -07:00
..
du deployment/logs-benchmark: add benchmark setup (#4470) 2023-07-06 21:24:08 -07:00
elk deployment/logs-benchmark: add benchmark setup (#4470) 2023-07-06 21:24:08 -07:00
generator fix linter for logs benchmark 2023-07-06 21:31:03 -07:00
grafana deployment/logs-benchmark: update dashboard (#4500) 2023-07-06 21:42:08 -07:00
source_logs deployment/logs-benchmark: add benchmark setup (#4470) 2023-07-06 21:24:08 -07:00
vmsingle deployment/logs-benchmark: add benchmark setup (#4470) 2023-07-06 21:24:08 -07:00
docker-compose.yml all: update VictoriaLogs docker tag to v0.1.0-victorialogs 2023-07-06 21:39:12 -07:00
grafana-dashboard.png deployment/logs-benchmark: add benchmark setup (#4470) 2023-07-06 21:24:08 -07:00
readme.md all: update VictoriaLogs docker tag to v0.1.0-victorialogs 2023-07-06 21:39:12 -07:00

readme.md

Benchmark for VictoriaLogs

Benchmark is based on:

How it works

docker-compose.yml contains:

  • 2 filebeat instances - one for elastic and one for VictoriaLogs.
  • elastic instance
  • VictoriaLogs instance
  • kibana instance - port forwarded to localhost:5601 to see UI
  • vmsingle - port forwarded to localhost:8428 to see UI
  • logs generator
  • exporters for filebeat/system

Logs generator generates logs based on logs located at ./source_logs/logs and sends them to filebeat instances via syslog. Logs are generated by reading files line by line, adding randomized suffix to each line and sending them to filebeat via syslog. By default, generator will exit once all files are read. docker-compose will restart it and files will be read again generating new logs.

Each filebeat than writes logs to elastic and VictoriaLogs via elasticsearch-compatible API.

How to run

  1. Download and unarchive logs by running:
cd source_logs 
bash download.sh

Note that with logs listed in download.sh it will require 49GB of free space:

  • 3GB for archives
  • 46GB for unarchived logs

If it is needed to minimize disk footprint, you can download only some of them by commenting out lines in download.sh. Unarchived logs size per file for reference:

2.3M Linux.log
 73M SSH.log
 32G Thunderbird.log
5.1M Apache.log
 13G hadoop-*.log

  1. (optional) If needed, adjust amount of logs sent by generator by modifying -outputRateLimitItems and outputRateLimitPeriod parameters in docker-compose.yml. By default, it is configured to send 10000 logs per second.

  2. Build victoria-logs image and adjust image parameter in docker-compose.yml:

make package-victoria-logs

Image name should be replaced at vlogs service in docker-compose.yml.

It is also possible to configure filebeat to send logs to VictoriaLogs running on local machine. To do this modify filebeat config for vlogs and replace vlogs address with address of local VictoriaLogs instance:

output.elasticsearch:
  hosts: [ "http://vlogs:9428/insert/elasticsearch/" ]

  1. Run docker-compose up -d to start benchmark.

  2. Navigate to http://localhost:3000/d/hkm6P6_4z/elastic-vs-vlogs to see Grafana dashboard with resource usage comparison.

grafana-dashboard.png