Aliaksandr Valialkin
3d1cb011b6
lib/encoding: make deterministic tests which rely on math/rand
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
2023-01-23 18:41:09 -08:00
Aliaksandr Valialkin
99d67ac8ad
lib/storage: validate timestamps in the block only if they use encoding, which needs validation
...
This reduces CPU usage when there is no sense in validating timestamps.
This is a follow-up for 5fa9525498https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
2022-10-21 00:52:32 +03:00
Aliaksandr Valialkin
5fa9525498
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block
...
This should reduce chances of unnoticed on-disk data corruption.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.
This is OK, since the native export format is undocumented.
2022-09-06 13:08:09 +03:00
Aliaksandr Valialkin
c6eb404c69
lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint*
...
This allows Go complier to generate more optimal code without bound checks
2022-04-12 12:55:21 +03:00
Aliaksandr Valialkin
e13ce2ee98
lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64()
2022-01-31 23:36:48 +02:00
Aliaksandr Valialkin
8b7917cd81
all: add go:build lines for Go1.17
...
See https://tip.golang.org/doc/go1.17#gofmt for more details
2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin
00b1659dde
lib: dump compressed block contents on error during decompression
...
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
2020-08-15 14:44:33 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
303905cd84
lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting
2020-04-28 00:01:55 +03:00
Aliaksandr Valialkin
36fa3078c2
lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize
2020-04-28 00:01:54 +03:00
Aliaksandr Valialkin
4ba4abe666
lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled
...
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:48:31 +02:00
Aliaksandr Valialkin
5bb39e757b
lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta*
...
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:41:52 +02:00
Aliaksandr Valialkin
d5c9841220
lib/encoding: mention unpacked block size in the error message if unparsed tail left
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:35:13 +02:00
Aliaksandr Valialkin
6a1499efa3
lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel
...
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
2019-12-23 18:05:41 +02:00
Aliaksandr Valialkin
55d728c849
lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones
...
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
808fc0971f
lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones
...
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
68722c3c74
lib/encoding: optimize UnmarshalUint* and UnmarshalInt*
2019-09-20 13:08:16 +03:00
Aliaksandr Valialkin
b339890dca
lib/encoding/zstd: go fmt
2019-07-25 01:37:16 +03:00
Aliaksandr Valialkin
6c4ca89d75
lib/encoding/zstd: disable CRC checks in pure Go build
...
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
2019-07-24 19:17:16 +03:00
Aliaksandr Valialkin
d07f616609
lib/encoding: small fixes in tests after the PR #114
2019-07-23 19:37:51 +03:00
Roman Khavronenko
5bf4e5ffb5
all: add Pure Go build (pull request #114 )
...
Updates #94
2019-07-23 19:26:39 +03:00
Aliaksandr Valialkin
0f2ecde0e6
lib/encoding: improve gauge series detection
...
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets
This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
2019-07-20 14:05:09 +03:00
Aliaksandr Valialkin
54fb8b21f9
all: fix misspellings
2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
6b0ae332f8
lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls
2019-05-24 13:01:02 +03:00
Aliaksandr Valialkin
6fb9dd09f5
lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio
2019-05-24 12:34:32 +03:00
Aliaksandr Valialkin
19b6643e5c
lib/encoding: substitute CompressZSTD with CompressZSTDLevel
2019-05-24 12:32:55 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
