github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
8398 commits 150 branches 748 tags 347 MiB
Commit graph

44 commits

Author SHA1 Message Date
Aliaksandr Valialkin
32193b6059
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.

This improves performance of these functions in hot loops of VictoriaLogs a bit.
 2024-05-14 01:30:25 +02:00
Aliaksandr Valialkin
147704aab0
lib/logstorage: initial implementation of pipes in LogsQL 
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
 2024-05-12 16:36:01 +02:00
Aliaksandr Valialkin
9dc9c892b7
lib/encoding: optimizing UnmarshalVarUint64 and UnmarshalVarInt64 a bit 2024-05-12 16:35:24 +02:00
Aliaksandr Valialkin
87338633b1
lib/slicesutil: add helper functions for setting slice length and extending its capacity 
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
 2024-05-12 11:33:49 +02:00
Aliaksandr Valialkin
d845edc24b
lib: consistently use atomic.* types instead of atomic.* functions 
See ea9e2b19a5
 2024-02-24 02:10:04 +02:00
Aliaksandr Valialkin
faeabfc730
lib/encoding: follow-up for 49e3665d6d 
Improve performance for typical cases of varint marshaling / unmarshaling further.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
 2024-02-01 05:38:58 +02:00
Fuchun Zhang
78af9b3e30
make encoding.MarshalVarInt64s faster (#5721) 
* make encoding.MarshalVarInt64s faster

* add fast path for MarshalVarInt64s

* make UnmarshalVarUint64s faster

* remove comment
 2024-02-01 03:33:59 +00:00
Aliaksandr Valialkin
eee210810e
lib/encoding: added benchmarks for marshaling / unmarshaling of varints 
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
 2024-02-01 05:11:35 +02:00
Aliaksandr Valialkin
8dd73574ca
lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint* 2024-01-23 22:50:11 +02:00
Aliaksandr Valialkin
a45cbc101f
all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
Aliaksandr Valialkin
dcf5b42670
lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 
The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer.

This is a follow-up for 140e7b6b74
 2023-07-20 21:54:51 -07:00
Aliaksandr Valialkin
992c300ce9
all: replace atomic.Value with atomic.Pointer[T] 
This eliminates the need in .(*T) casting for results obtained from Load()

Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map,
because map is already a pointer type.
 2023-07-19 17:48:26 -07:00
Aliaksandr Valialkin
4c80193a86
lib/encoding: add MarshalBool/UnmarshalBool and GetUint32s/PutUint32s functions 
These functions are going to be used by VictoriaLogs
 2023-07-06 17:24:52 -07:00
Aliaksandr Valialkin
bf545fcc14
lib/encoding: fix test after 4725549cb2 2023-04-05 21:38:48 -07:00
Aliaksandr Valialkin
dfb1d1ead1
lib/encoding: make deterministic tests which rely on math/rand 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
 2023-01-23 18:43:49 -08:00
Aliaksandr Valialkin
af648279ce
lib/storage: validate timestamps in the block only if they use encoding, which needs validation 
This reduces CPU usage when there is no sense in validating timestamps.

This is a follow-up for 5fa9525498

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
 2022-10-21 00:54:37 +03:00
Aliaksandr Valialkin
051e722112
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block 
This should reduce chances of unnoticed on-disk data corruption.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011

This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.

This is OK, since the native export format is undocumented.
 2022-09-06 13:07:49 +03:00
Aliaksandr Valialkin
4a3172f150
lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* 
This allows Go complier to generate more optimal code without bound checks
 2022-04-12 12:56:52 +03:00
Aliaksandr Valialkin
084664d780
lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64() 2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
5d255846ac all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin
edf3ee2a9b lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
2aecf7c37c lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting 2020-04-28 00:02:19 +03:00
Aliaksandr Valialkin
806dc73d8a lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize 2020-04-28 00:02:18 +03:00
Aliaksandr Valialkin
d3c185f0ca lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled 
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin
091e35cf0c lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* 
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin
0e51058a0d lib/encoding: mention unpacked block size in the error message if unparsed tail left 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin
44f886cc9c lib/encoding/zstd: typo fix 2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin
108a60d69e lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines 
This should fix goroutine leak for https://github.com/klauspost/compress/issues/195
 2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin
335bd0ac0a lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel 
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
 2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin
5d439cc6f2 lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones 
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
 2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
1037053fed lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones 
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
 2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
272e2f77c9 lib/encoding: optimize UnmarshalUint* and UnmarshalInt* 2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
f2e8d54fb0 lib/encoding/zstd: go fmt 2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin
97b5dc7122 lib/encoding/zstd: disable CRC checks in pure Go build 
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
 2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin
54f035d4ce all: small updates after PR #114 2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin
7a133567fb lib/encoding: small fixes in tests after the PR #114 2019-07-24 17:43:39 +03:00
Roman Khavronenko
fcf09aaa3c all: add Pure Go build (pull request #114) 
Updates #94
 2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin
3fae34eeb4 lib/encoding: improve gauge series detection 
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets

This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
 2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin
bdf696ef18 all: fix misspellings 2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
9b64dfee4b lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls 2019-05-24 13:01:15 +03:00
Aliaksandr Valialkin
b23352dc9e lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio 2019-05-24 12:34:51 +03:00
Aliaksandr Valialkin
f67f40d63a lib/encoding: substitute CompressZSTD with CompressZSTDLevel 2019-05-24 12:32:49 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00