VictoriaMetrics/app/vlogsgenerator

vlogsgenerator

Logs generator for VictoriaLogs.

How to build vlogsgenerator?

Run make vlogsgenerator from the repository root. This builds bin/vlogsgenerator binary.

How to generate logs?

vlogsgenerator generates logs in JSON line format suitable for the ingestion via /insert/jsonline endpoint at VictoriaLogs.

By default it writes the generated logs into stdout. For example, the following command writes generated logs to stdout:

bin/vlogsgenerator

It is possible to redirect the generated logs to file. For example, the following command writes the generated logs to logs.json file:

bin/vlogsgenerator > logs.json

The generated logs at logs.json file can be inspected with the following command:

head logs.json | jq .

Below is an example output:

{
  "_time": "2024-05-08T14:34:00.854Z",
  "_msg": "message for the stream 8 and worker 0; ip=185.69.136.129; uuid=b4fe8f1a-c93c-dea3-ba11-5b9f0509291e; u64=8996587920687045253",
  "host": "host_8",
  "worker_id": "0",
  "const_0": "some value 0 8",
  "const_1": "some value 1 8",
  "const_2": "some value 2 8",
  "var_0": "some value 0 12752539384823438260",
  "dict_0": "warn",
  "dict_1": "info",
  "u8_0": "6",
  "u16_0": "35202",
  "u32_0": "1964973739",
  "u64_0": "4810489083243239145",
  "float_0": "1.868",
  "ip_0": "250.34.75.125",
  "timestamp_0": "1799-03-16T01:34:18.311Z"
}
{
  "_time": "2024-05-08T14:34:00.854Z",
  "_msg": "message for the stream 9 and worker 0; ip=164.244.254.194; uuid=7e8373b1-ce0d-1ce7-8e96-4bcab8955598; u64=13949903463741076522",
  "host": "host_9",
  "worker_id": "0",
  "const_0": "some value 0 9",
  "const_1": "some value 1 9",
  "const_2": "some value 2 9",
  "var_0": "some value 0 5371555382075206134",
  "dict_0": "INFO",
  "dict_1": "FATAL",
  "u8_0": "219",
  "u16_0": "31459",
  "u32_0": "3918836777",
  "u64_0": "6593354256620219850",
  "float_0": "1.085",
  "ip_0": "253.151.88.158",
  "timestamp_0": "2042-10-05T16:42:57.082Z"
}

The generated logs can be written directly to VictoriaLogs by passing the address of /insert/jsonline endpoint to -addr command-line flag. For example, the following command writes the generated logs to VictoriaLogs running at localhost:

bin/vlogsgenerator -addr=http://localhost:9428/insert/jsonline

vlogsgenerator accepts various command-line flags, which can be used for configuring the number and the shape of the generated logs. These flags can be inspected by running vlogsgenerator -help. Below are the most interesting flags:

• -start - starting timestamp for generating logs. Logs are evenly generated on the [-start ... -end] interval.
• -end - ending timestamp for generating logs. Logs are evenly generated on the [-start ... -end] interval.
• -activeStreams - the number of active log streams to generate.
• -logsPerStream - the number of log entries to generate per each log stream. Log entries are evenly distributed on the [-start ... -end] interval.

The total number of generated logs can be calculated as -activeStreams * -logsPerStream.

For example, the following command generates 1_000_000 log entries on the time range [2024-01-01 - 2024-02-01] across 100 log streams, where every logs stream contains 10_000 log entries, and writes them to http://localhost:9428/insert/jsonline:

bin/vlogsgenerator \
  -start=2024-01-01 -end=2024-02-01 \
  -activeStreams=100 \
  -logsPerStream=10_000 \
  -addr=http://localhost:9428/insert/jsonline

By default vlogsgenerator generates and writes logs by a single worker. This may limit the maximum data ingestion rate during benchmarks. The number of workers can be changed via -workers command-line flag. For example, the following command generates and writes logs with 16 workers:

bin/vlogsgenerator \
  -start=2024-01-01 -end=2024-02-01 \
  -activeStreams=100 \
  -logsPerStream=10_000 \
  -addr=http://localhost:9428/insert/jsonline \
  -workers=16

Every 10 seconds vlogsgenerator writes statistics about the generated logs into stderr. The frequency of the generated statistics can be adjusted via -statInterval command-line flag. For example, the following command writes statistics every 2 seconds:

bin/vlogsgenerator \
  -start=2024-01-01 -end=2024-02-01 \
  -activeStreams=100 \
  -logsPerStream=10_000 \
  -addr=http://localhost:9428/insert/jsonline \
  -statInterval=2s