github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
9487 commits 150 branches 748 tags 347 MiB
Commit graph

41 commits

Author SHA1 Message Date
Aliaksandr Valialkin
cc2647d212
lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 
Change the return values for these functions - now they return the unmarshaled result plus
the size of the unmarshaled result in bytes, so the caller could re-slice the src for further unmarshaling.

This improves performance of these functions in hot loops of VictoriaLogs a bit.
 2024-05-14 01:23:54 +02:00
Aliaksandr Valialkin
9dbd0f9085
lib/logstorage: initial implementation of pipes in LogsQL 
See https://docs.victoriametrics.com/victorialogs/logsql/#pipes
 2024-05-12 16:33:31 +02:00
Aliaksandr Valialkin
e66465cb03
lib/encoding: optimizing UnmarshalVarUint64 and UnmarshalVarInt64 a bit 2024-05-12 16:32:11 +02:00
Aliaksandr Valialkin
590160ddbb
lib/slicesutil: add helper functions for setting slice length and extending its capacity 
The added helper functions - SetLength() and ExtendCapacity() - replace error-prone code with simple function calls.
 2024-05-12 11:32:17 +02:00
Aliaksandr Valialkin
7e1dd8ab9d
lib: consistently use atomic.* types instead of atomic.* functions 
See ea9e2b19a5
 2024-02-24 02:07:53 +02:00
Aliaksandr Valialkin
55b5c13839
lib/encoding: follow-up for 49e3665d6d 
Improve performance for typical cases of varint marshaling / unmarshaling further.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
 2024-02-01 05:37:40 +02:00
Fuchun Zhang
49e3665d6d
make encoding.MarshalVarInt64s faster (#5721) 
* make encoding.MarshalVarInt64s faster

* add fast path for MarshalVarInt64s

* make UnmarshalVarUint64s faster

* remove comment
 2024-02-01 05:34:37 +02:00
Aliaksandr Valialkin
c91614b626
lib/encoding: added benchmarks for marshaling / unmarshaling of varints 
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5721
 2024-02-01 05:11:01 +02:00
Aliaksandr Valialkin
8fb8b71295
lib/encoding: remove uneeded re-slicing of byte slice before passing it to binary.BigEndian.Uint* 2024-01-23 22:50:29 +02:00
Aliaksandr Valialkin
cf23dc6480
all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:12:51 +01:00
Aliaksandr Valialkin
4cae725edf
lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 
The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer.

This is a follow-up for 140e7b6b74
 2023-07-20 20:56:11 -07:00
Aliaksandr Valialkin
140e7b6b74
all: replace atomic.Value with atomic.Pointer[T] 
This eliminates the need in .(*T) casting for results obtained from Load()

Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map,
because map is already a pointer type.
 2023-07-19 17:42:06 -07:00
Aliaksandr Valialkin
497ec3f3e6
lib/encoding: add MarshalBool/UnmarshalBool and GetUint32s/PutUint32s functions 
These functions are going to be used by VictoriaLogs
 2023-06-19 22:40:55 -07:00
Aliaksandr Valialkin
593c151831
lib/encoding: fix test after 4725549cb2 2023-04-05 21:38:37 -07:00
Aliaksandr Valialkin
3d1cb011b6
lib/encoding: make deterministic tests which rely on math/rand 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3683
 2023-01-23 18:41:09 -08:00
Aliaksandr Valialkin
99d67ac8ad
lib/storage: validate timestamps in the block only if they use encoding, which needs validation 
This reduces CPU usage when there is no sense in validating timestamps.

This is a follow-up for 5fa9525498

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011
 2022-10-21 00:52:32 +03:00
Aliaksandr Valialkin
5fa9525498
lib/storage: verify that timestamps in block are in the range specified by blockHeader.{Min,Max}Timestamp when upacking the block 
This should reduce chances of unnoticed on-disk data corruption.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011

This change modifies the format for data exported via /api/v1/export/native -
now this data contains MaxTimestamp and PrecisionBits fields from blockHeader.

This is OK, since the native export format is undocumented.
 2022-09-06 13:08:09 +03:00
Aliaksandr Valialkin
c6eb404c69
lib/encoding: explicitly set slice length passed to binary.BigEndian.Uint* 
This allows Go complier to generate more optimal code without bound checks
 2022-04-12 12:55:21 +03:00
Aliaksandr Valialkin
e13ce2ee98
lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64() 2022-01-31 23:36:48 +02:00
Aliaksandr Valialkin
8b7917cd81 all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin
00b1659dde lib: dump compressed block contents on error during decompression 
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
 2020-08-15 14:44:33 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
303905cd84 lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting 2020-04-28 00:01:55 +03:00
Aliaksandr Valialkin
36fa3078c2 lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize 2020-04-28 00:01:54 +03:00
Aliaksandr Valialkin
4ba4abe666 lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled 
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:48:31 +02:00
Aliaksandr Valialkin
5bb39e757b lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* 
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:41:52 +02:00
Aliaksandr Valialkin
d5c9841220 lib/encoding: mention unpacked block size in the error message if unparsed tail left 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:35:13 +02:00
Aliaksandr Valialkin
6a1499efa3 lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel 
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
 2019-12-23 18:05:41 +02:00
Aliaksandr Valialkin
55d728c849 lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones 
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
 2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
808fc0971f lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones 
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
 2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
68722c3c74 lib/encoding: optimize UnmarshalUint* and UnmarshalInt* 2019-09-20 13:08:16 +03:00
Aliaksandr Valialkin
b339890dca lib/encoding/zstd: go fmt 2019-07-25 01:37:16 +03:00
Aliaksandr Valialkin
6c4ca89d75 lib/encoding/zstd: disable CRC checks in pure Go build 
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
 2019-07-24 19:17:16 +03:00
Aliaksandr Valialkin
d07f616609 lib/encoding: small fixes in tests after the PR #114 2019-07-23 19:37:51 +03:00
Roman Khavronenko
5bf4e5ffb5 all: add Pure Go build (pull request #114) 
Updates #94
 2019-07-23 19:26:39 +03:00
Aliaksandr Valialkin
0f2ecde0e6 lib/encoding: improve gauge series detection 
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets

This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
 2019-07-20 14:05:09 +03:00
Aliaksandr Valialkin
54fb8b21f9 all: fix misspellings 2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
6b0ae332f8 lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total for determining the number CompressZSTD / DecompressZSTD calls 2019-05-24 13:01:02 +03:00
Aliaksandr Valialkin
6fb9dd09f5 lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total metrics for rough estimation of block compression ratio 2019-05-24 12:34:32 +03:00
Aliaksandr Valialkin
19b6643e5c lib/encoding: substitute CompressZSTD with CompressZSTDLevel 2019-05-24 12:32:55 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00