2023-06-20 05:55:12 +00:00
|
|
|
package logstorage
|
|
|
|
|
|
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".
The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.
Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.
The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.
The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.
Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 14:18:28 +00:00
|
|
|
// partFormatLatestVersion is the latest format version for parts.
|
|
|
|
|
//
|
|
|
|
|
// See partHeader.FormatVersion for details.
|
2024-10-29 10:28:41 +00:00
|
|
|
const partFormatLatestVersion = 2
|
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".
The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.
Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.
The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.
The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.
Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 14:18:28 +00:00
|
|
|
|
2024-10-29 10:28:41 +00:00
|
|
|
// bloomValuesMaxShardsCount is the number of shards for bloomFilename and valuesFilename files.
|
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".
The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.
Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.
The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.
The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.
Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 14:18:28 +00:00
|
|
|
//
|
2024-10-29 10:28:41 +00:00
|
|
|
// The partHeader.FormatVersion and partFormatLatestVersion must be updated when this number changes.
|
|
|
|
|
const bloomValuesMaxShardsCount = 128
|
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".
The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.
Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.
The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.
The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.
Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 14:18:28 +00:00
|
|
|
|
2023-06-20 05:55:12 +00:00
|
|
|
// maxUncompressedIndexBlockSize contains the maximum length of uncompressed block with blockHeader entries aka index block.
|
|
|
|
|
//
|
|
|
|
|
// The real block length can exceed this value by a small percentage because of the block write details.
|
|
|
|
|
const maxUncompressedIndexBlockSize = 128 * 1024
|
|
|
|
|
|
|
|
|
|
// maxUncompressedBlockSize is the maximum size of uncompressed block in bytes.
|
|
|
|
|
//
|
|
|
|
|
// The real uncompressed block can exceed this value by up to 2 times because of block merge details.
|
|
|
|
|
const maxUncompressedBlockSize = 2 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxRowsPerBlock is the maximum number of log entries a single block can contain.
|
|
|
|
|
const maxRowsPerBlock = 8 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxColumnsPerBlock is the maximum number of columns per block.
|
2024-09-24 16:51:23 +00:00
|
|
|
//
|
|
|
|
|
// It isn't recommended setting this value to too big value, because this may result
|
2024-11-01 15:46:49 +00:00
|
|
|
// in excess memory usage during data ingestion and significant slowdown during query execution.
|
|
|
|
|
const maxColumnsPerBlock = 2_000
|
2023-06-20 05:55:12 +00:00
|
|
|
|
2023-10-02 17:01:17 +00:00
|
|
|
// MaxFieldNameSize is the maximum size in bytes for field name.
|
|
|
|
|
//
|
|
|
|
|
// Longer field names are truncated during data ingestion to MaxFieldNameSize length.
|
|
|
|
|
const MaxFieldNameSize = 128
|
|
|
|
|
|
|
|
|
|
// maxConstColumnValueSize is the maximum size in bytes for const column value.
|
|
|
|
|
//
|
|
|
|
|
// Const column values are stored in columnsHeader, which is read every time the corresponding block is scanned during search queries.
|
|
|
|
|
// So it is better to store bigger values in regular columns in order to speed up search speed.
|
|
|
|
|
const maxConstColumnValueSize = 256
|
|
|
|
|
|
2023-06-20 05:55:12 +00:00
|
|
|
// maxIndexBlockSize is the maximum size of the block with blockHeader entries (aka indexBlock)
|
|
|
|
|
const maxIndexBlockSize = 8 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxTimestampsBlockSize is the maximum size of timestamps block
|
|
|
|
|
const maxTimestampsBlockSize = 8 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxValuesBlockSize is the maximum size of values block
|
|
|
|
|
const maxValuesBlockSize = 8 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxBloomFilterBlockSize is the maximum size of bloom filter block
|
|
|
|
|
const maxBloomFilterBlockSize = 8 * 1024 * 1024
|
|
|
|
|
|
|
|
|
|
// maxColumnsHeaderSize is the maximum size of columnsHeader block
|
|
|
|
|
const maxColumnsHeaderSize = 8 * 1024 * 1024
|
2023-10-02 17:01:17 +00:00
|
|
|
|
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".
The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.
Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.
The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.
The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.
Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 14:18:28 +00:00
|
|
|
// maxColumnsHeaderIndexSize is the maximum size of columnsHeaderIndex block
|
|
|
|
|
const maxColumnsHeaderIndexSize = 8 * 1024 * 1024
|
|
|
|
|
|
2023-10-02 17:01:17 +00:00
|
|
|
// maxDictSizeBytes is the maximum length of all the keys in the valuesDict.
|
|
|
|
|
//
|
|
|
|
|
// Dict is stored in columnsHeader, which is read every time the corresponding block is scanned during search qieries.
|
|
|
|
|
// So it is better to store bigger values in regular columns in order to speed up search speed.
|
|
|
|
|
const maxDictSizeBytes = 256
|
|
|
|
|
|
|
|
|
|
// maxDictLen is the maximum number of entries in the valuesDict.
|
|
|
|
|
//
|
|
|
|
|
// it shouldn't exceed 255, since the dict len is marshaled into a single byte.
|
|
|
|
|
const maxDictLen = 8
|