![]() Previously columns with negative int64 values were stored either as float64 or string depending on whether the negative int64 values are bigger or smaller than -2^53. If the integer values are smaller than -2^53, then they are stored as string, since float64 cannot hold such values without precision loss. Now such values are stored as int64. This should improve compression ratio and query performance over columns with negative int64 values. |
||
---|---|---|
.. | ||
main.go | ||
Makefile | ||
README.md |
vlogsgenerator
Logs generator for VictoriaLogs.
How to build vlogsgenerator?
Run make vlogsgenerator
from the repository root. This builds bin/vlogsgenerator
binary.
How run vlogsgenerator?
vlogsgenerator
generates logs in JSON line format suitable for the ingestion
via /insert/jsonline
endpoint at VictoriaLogs.
By default it writes the generated logs into stdout
. For example, the following command writes generated logs to stdout
:
bin/vlogsgenerator
It is possible to redirect the generated logs to file. For example, the following command writes the generated logs to logs.json
file:
bin/vlogsgenerator > logs.json
The generated logs at logs.json
file can be inspected with the following command:
head logs.json | jq .
Below is an example output:
{
"_time": "2024-05-08T14:34:00.854Z",
"_msg": "message for the stream 8 and worker 0; ip=185.69.136.129; uuid=b4fe8f1a-c93c-dea3-ba11-5b9f0509291e; u64=8996587920687045253",
"host": "host_8",
"worker_id": "0",
"run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
"const_0": "some value 0 8",
"const_1": "some value 1 8",
"const_2": "some value 2 8",
"var_0": "some value 0 12752539384823438260",
"dict_0": "warn",
"dict_1": "info",
"u8_0": "6",
"u16_0": "35202",
"u32_0": "1964973739",
"u64_0": "4810489083243239145",
"float_0": "1.868",
"ip_0": "250.34.75.125",
"timestamp_0": "1799-03-16T01:34:18.311Z",
"json_0": "{\"foo\":\"bar_3\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":32}"
}
{
"_time": "2024-05-08T14:34:00.854Z",
"_msg": "message for the stream 9 and worker 0; ip=164.244.254.194; uuid=7e8373b1-ce0d-1ce7-8e96-4bcab8955598; u64=13949903463741076522",
"host": "host_9",
"worker_id": "0",
"run_id": "f9b3deee-e6b6-7f56-5deb-1586e4e81725",
"const_0": "some value 0 9",
"const_1": "some value 1 9",
"const_2": "some value 2 9",
"var_0": "some value 0 5371555382075206134",
"dict_0": "INFO",
"dict_1": "FATAL",
"u8_0": "219",
"u16_0": "31459",
"u32_0": "3918836777",
"u64_0": "6593354256620219850",
"float_0": "1.085",
"ip_0": "253.151.88.158",
"timestamp_0": "2042-10-05T16:42:57.082Z",
"json_0": "{\"foo\":\"bar_5\",\"baz\":{\"a\":[\"x\",\"y\"]},\"f3\":NaN,\"f4\":27}"
}
The run_id
field uniquely identifies every vlogsgenerator
invocation.
How to write logs to VictoriaLogs?
The generated logs can be written directly to VictoriaLogs by passing the address of /insert/jsonline
endpoint
to -addr
command-line flag. For example, the following command writes the generated logs to VictoriaLogs running at localhost
:
bin/vlogsgenerator -addr=http://localhost:9428/insert/jsonline
Configuration
vlogsgenerator
accepts various command-line flags, which can be used for configuring the number and the shape of the generated logs.
These flags can be inspected by running vlogsgenerator -help
. Below are the most interesting flags:
-start
- starting timestamp for generating logs. Logs are evenly generated on the [-start
...-end
] interval.-end
- ending timestamp for generating logs. Logs are evenly generated on the [-start
...-end
] interval.-activeStreams
- the number of active log streams to generate.-logsPerStream
- the number of log entries to generate per each log stream. Log entries are evenly distributed on the [-start
...-end
] interval.
The total number of generated logs can be calculated as -activeStreams
* -logsPerStream
.
For example, the following command generates 1_000_000
log entries on the time range [2024-01-01 - 2024-02-01]
across 100
log streams, where every logs stream contains 10_000
log entries,
and writes them to http://localhost:9428/insert/jsonline
:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
Churn rate
It is possible to generate churn rate for active log streams
by specifying -totalStreams
command-line flag bigger than -activeStreams
. For example, the following command generates
logs for 1000
total streams, while the number of active streams equals to 100
. This means that at every time there are logs for 100
streams,
but these streams change over the given [-start
... -end
] time range, so the total number of streams on the given time range becomes 1000
:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-totalStreams=1_000 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline
In this case the total number of generated logs equals to -totalStreams
* -logsPerStream
= 10_000_000
.
Benchmark tuning
By default vlogsgenerator
generates and writes logs by a single worker. This may limit the maximum data ingestion rate during benchmarks.
The number of workers can be changed via -workers
command-line flag. For example, the following command generates and writes logs with 16
workers:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-workers=16
Output statistics
Every 10 seconds vlogsgenerator
writes statistics about the generated logs into stderr
. The frequency of the generated statistics can be adjusted via -statInterval
command-line flag.
For example, the following command writes statistics every 2 seconds:
bin/vlogsgenerator \
-start=2024-01-01 -end=2024-02-01 \
-activeStreams=100 \
-logsPerStream=10_000 \
-addr=http://localhost:9428/insert/jsonline \
-statInterval=2s