Commit graph

3103 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8e131ee4d7
use new canonical urls to sd_configs docs: https://docs.victoriametrics.com/victoriametrics/sd_configs/
This avoids a redirect from the old link https://docs.victoriametrics.com/sd_configs/ to https://docs.victoriametrics.com/victoriametrics/sd_configs/ ,
and fixes `backwards` navigation for these links across VictoriaMetrics docs.

This is a follow-up for f152021521
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595#issuecomment-2831598274
2025-04-30 17:41:15 +02:00
Aliaksandr Valialkin
afd3cfe982
all: use new canonical urls to vmbackup docs: https://docs.victoriametrics.com/victoriametrics/vmbackup/
This avoids a redirect from the old link https://docs.victoriametrics.com/vmbackup/ to https://docs.victoriametrics.com/victoriametrics/vmbackup/ ,
and fixes `backwards` navigation for these links across VictoriaMetrics docs.

This is a follow-up for f152021521
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595#issuecomment-2831598274
2025-04-30 16:51:57 +02:00
Aliaksandr Valialkin
997e5aba15
lib/atomicutil: rename Slice.GetSlice to Slice.All for the sake of better readability 2025-04-30 16:14:10 +02:00
Aliaksandr Valialkin
1746687df1
all: use new canonical urls to vmagent docs: https://docs.victoriametrics.com/victoriametrics/vmagent/
This avoids a redirect from the old link https://docs.victoriametrics.com/vmagent/ to https://docs.victoriametrics.com/victoriametrics/vmagent/ ,
and fixes `backwards` navigation for these links across VictoriaMetrics docs.

This is a follow-up for f152021521
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8595#issuecomment-2831598274
2025-04-30 16:03:46 +02:00
Aliaksandr Valialkin
623375fbe4
lib/logstorage: prevent from slow memory leak at datadb.rb
datadb.rb contains logRows shards, which weren't freed up after the data ingestion
for the given per-day datadb is stopped. This leads to slow memory leak when VictoriaLogs runs
for multiple days without restarts. Avoid this memory leak by freeing up the logRows shards
after converting them to in-memory parts. Re-use the freed up logRows shards via a pool in order
to reduce the pressure on GC.

(cherry picked from commit ec6f33f526)
2025-04-27 17:51:24 +02:00
Aliaksandr Valialkin
9b2f63055d
lib/logstorage: make golangc-lint happy by substituting unused function arg with _
(cherry picked from commit 3b7039679f)
2025-04-27 17:51:23 +02:00
Aliaksandr Valialkin
301249cbaa
lib/logstorage: increase scalability of datadb.mustAddRows() on hosts with many CPU cores
Use multiple independent logRows shards for storing the pending log entries before converting them to searchable parts.
Every shard is protected by its own mutex, so multiple CPU cores may add multiple log rows into datadb at the same time.

This increases the performance of BenchmarkStorageMustAddRows/rowsPerInsert-1, which ingests log rows own-by-one
from concurrently running goroutines, by 2x.

(cherry picked from commit 8ad81220d3)
2025-04-27 17:51:22 +02:00
Aliaksandr Valialkin
2db53dcbb9
lib/logstorage: re-use newTestLogRows() for creating LogRows inside BenchmarkStorageMustAddRows
(cherry picked from commit 7455e6c0a5)
2025-04-27 17:51:21 +02:00
f41gh7
234bc82f6c
lib/handshake: log client network errors during handshake as warnings
This commit modifies the logging behavior for client network errors
(e.g., EOFs, timeouts) during the handshake process. They are now logged
as warnings instead of errors, as they are not actionable from the
server’s perspective. Here's some examples of such errors.

Timeouts during the initial read phase:

2025-04-09T07:08:59.323Z	error
VictoriaMetrics/lib/vmselectapi/server.go:204	cannot perform
vmselect handshake with client "<REDACTED>": cannot read hello: cannot
read message with size 11: read tcp4 <REDACTED>-><REDACTED>: i/o
timeout; read only 0 bytes

EOFs occurring later in the handshake process:

2025-04-08T18:01:30.783Z	error
VictoriaMetrics/lib/vmselectapi/server.go:204	cannot perform
vmselect handshake with client "<REDACTED>": cannot read isCompressed
flag: cannot read message with size 1: EOF; read only 0 bytes

By logging these as warnings, we reduce noise in error logs while
preserving valuble information for debug.
2025-04-25 12:02:39 +03:00
Nikolay
f97bb8aca3
lib/cgroup: properly parse cpu limit
Previously, if `cpu.max` file has only `max` resource defined without
`period`, it was parsed incorrectly and silently drop error. While this
syntax is valid and actually used by some container runtimes. If period
is not defined, default value for it 100_000 must be used.

 This commit fixes parsing function by using default value for period.
In addition, it adds zero value check, which fixes possible panic if
period has 0 value.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8808
2025-04-25 11:49:12 +03:00
Max Kotliar
9c8a96fd5a
docs\stream-aggregation: Describe dropping unneeded labels in more details
Follow-up on
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8715 and
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8681#issuecomment-2796127921
2025-04-24 21:05:34 +03:00
Zhu Jiekun
40c1cf306b
app/vmagent: add consistent hashing for the remote write sharding
This commit adds the following changes:
* use consistent hashing  for the remote write sharding.
* properly count metric of remote write samples drop rate  when `shardByURL` was
enabled.

Related issues:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8546
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8702
2025-04-24 21:05:33 +03:00
Aliaksandr Valialkin
f02a03bd13
Revert "ci: temporary disable vlogs tests for i386 "
This reverts commit fa6a32a39d.

Reason for revert: the broken tests were fixed on GOARCH=386 by skipping the check for the state size
after improting the state of stats function, since the state size depends on the hardware architecture.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710
2025-04-24 18:18:29 +02:00
Zakhar Bessarab
e44c0d2a3d
lib/backup/s3remote: enable HTTP/2 for S3 connections
### Describe Your Changes

HTTP/2 support is used by some S3-compatible storage providers, so
disabling it default leads to unexpected errors when trying to connect
to S3 endpoint.
For example, using MinIO as S3 storage backend: `net/http: HTTP/1.x
transport connection broken: malformed HTTP response`.

HTTP/2 was enabled by default previously, but while fixing inconsistency
e5f4826 commit disabled this by default.
cc: @valyala 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-23 09:46:23 +04:00
Zakhar Bessarab
224a85cf0f
fix: compatibility for FIPS builds
### Describe Your Changes

Fixes which are required in order to build FIPS-compliant binaries.
These changes were originally added for enterprise version and synced to
opensource for consistency and easier maintenance.

- consistently use `hash/fnv` at `app/vmalert` when calculating
checksums. Usage of md5 is not allowed in FIPS mode.
- increase encryption keys size used in testing in order to allow tests
to successfully run in FIPS mode

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-04-23 09:46:22 +04:00
Aliaksandr Valialkin
7c4e8a5a0c
lib/logstorage: add sample N for returning a random 1/Nth sample of matching logs 2025-04-22 16:41:03 +02:00
Zakhar Bessarab
f86c297cc6
lib/promrelabel/debug: use stricter format for labels ()
### Describe Your Changes

Previously, it was possible to use any UTF-8 string to specify list of
labels. While this makes it easier to use it also leads to unexpected
parsing results in some cases (see
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8584 as an
example).

Enforce specifying metric in format {label="value"...} in order to avoid
issues with unexpected parsing results.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b1523f650d)
2025-04-22 14:14:25 +02:00
Aliaksandr Valialkin
99291e9265
lib/logstorage: buffer the ingested log entries before converting them into searchable parts
This reduces the overhead needed for converting the ingested log entries to searchable in-memory parts
when small number of log entries are passed to Storage.MustAddRows().

The BenchmarkStorageMustAddRows shows up to 10x performance increase for rowsPerInsert=1,
up to 5x performance increase for rowsPerInsert=10 and up to 2x performance increase for rowsPerInsert=100.

This should reduce CPU usage during data ingestion when every request contains small number of rows.

(cherry picked from commit 5491d54c11)
2025-04-22 14:14:24 +02:00
Aliaksandr Valialkin
b8c73811f5
lib/logstorage: add a benchmark for different number of rows added to the storage via Storage.MustAddRows()
(cherry picked from commit 14561a7ed3)
2025-04-22 14:14:24 +02:00
Nikolay
d9a407392e
app/vmselect/netstorage: properly set max read size for metric name
Previously, metric names stats API had a false assumption, that max
size of metric name is 256 byte. But this is configurable parameter with
4096 bytes max size. It triggered errors during API requests.

 This commit replaces hard-coded 256 byte limit with common constant:
maxLabelValueSize. It has 16 MB limit.

 In addition, this commit adds check for metric name stats tracker,
if metric name size exceeds default buffer limit, it will be allocated
directly on heap. It must be rare case, since most metric names has
16-64 byte size.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8759
2025-04-22 11:31:48 +03:00
Nikolay
689a7567c9
lib/storage/metricnamestats: allow regex for match_pattern
This commit allows regex syntax for match_pattern query param.
It improves API usability.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
2025-04-22 11:30:59 +03:00
Phuong Le
f83947665b
lib/storage: put the unused in-memory part back into the pool 2025-04-22 11:29:52 +03:00
Aliaksandr Valialkin
e31995b436
lib/promscrape: prevent from excess memory allocation during scrapes when sample_limit is exceeded
Do not reset wc.labels in order to properly keep track of the number of used labels for the scrape,
and properly re-use the same number of wc.labels on subsequent scrapes.

See 12f26668a6 (r155481168)
2025-04-22 10:55:55 +03:00
Georgy Torquemada
a8dba30cdd
fix: add missed severity levels (warn) for protobuff parser
Closes
[8647](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8647)

### Describe Your Changes

Added missed OTEL severities levels, added test for severity, fix some
severity in given tests

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-21 16:41:56 +04:00
Andrii Chubatiuk
e2f3ce9bb7
lib/logstorage: expect message in a field with empty and _msg name ()
### Describe Your Changes

fixes 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-04-17 19:58:02 +02:00
Artem Fetishev
0bddd6ad7c
lib/storage: Followup for 2acc6c1106
Fix tests by adding accountID and projectID.

The tests were cherry-picked from master and were failing to build because
cluster version requires accountID and projectID.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-17 17:10:33 +02:00
Artem Fetishev
2acc6c1106
lib/storage: test wasMetricIDsMissingBefore with "testing/synctest" ()
Using this package lets to manipulate time. In this particular case, it
lets to advance the time 61 second forward instantly.

A few side changes were necessary:

- Do not use fasttime in unit tests. The fasttime package starts a
goroutine outside the test bubble which causes the clock to be real, not
fake.
- Stop the time.Ticker explicitly and also stop idbNext. These two
create goroutines with infinite loops which causes the unit tests that
use synctest to hang forever. All goroutines created inside the bubble
must exit in order for the syntest to finish.
- synctest is an experimental package and requires an environment
variable to be set. The Makefile was changed to set it.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-17 17:02:25 +02:00
Max Kotliar
82c7501952
lib/protoparser/protoparserutil: restore write concurrency limiter in ReadUncompressedData due to performance regressions ()
### Describe Your Changes

The write concurrency limiter in ReadUncompressedData was previously
removed in

22d1b916bf
to avoid suboptimal behavior in certain scenarios. However, follow-up
reports—including issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8674 and
production feedback from VictoriaMetrics Cloud—indicated a noticeable
degradation in performance after its removal.

To mitigate these regressions, this commit reintroduces the concurrency
limiter. A long-term, more optimal solution will be explored separately
in issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8728.

TODO:

* [x] Changelog

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 231810fe49)
2025-04-17 14:11:45 +02:00
Nikolay
07d0593076
lib/storage: enhance TSDB status response
This commit adds new fields - `requestsCount` and `lastRequestTimestamp`
to series count be metric names stats.
It allows to display an additional stats at explore cardinality page.
Stats will only be added if `storage.trackMetricNameStats` flag is set.

 This change requires an update to RPC protocol in order to properly
marshal data.

 In addition, this commit adds integration tests to TSDB stats API.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
2025-04-16 19:56:46 +02:00
Roman Khavronenko
c254a08578
ci: temporary disable vlogs tests for i386
This change unblocks testing pipelines in CI for other contributions.
The tests are commented because I don't have full understanding of
fixing them.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710

---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-14 11:13:23 +02:00
Max Kotliar
491fcc3279
lib/writeconcurrencylimiter: add some hints to unexpected EOF error message. ()
### Describe Your Changes

Under heavy load, vmagent's wirte concurrency limiter

(2ab53acce4/lib/writeconcurrencylimiter/concurrencylimiter.go (L111))
queues incoming requests. If a client's timeout is shorter than the wait
time in the
queue, the client may close the connection before vmagent starts
processing it. When vmagent then tries to read the request body, it
encounters an ambiguous `unexpected EOF` error
(https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8675).

This commit adds more context to such errors to help users diagnose and
resolve
the issue when it's related to vmagent's own load and queuing behavior.

Possible user actions include:
- Lowering `-insert.maxQueueDuration` below the client's timeout.
- Increasing the client-side timeout, if applicable.
- Scaling up vmagent (e.g., adding more CPU resources).
- Increasing `-maxConcurrentInserts` if CPU capacity allows.

Steps to reproduce:
https://gist.github.com/makasim/6984e20f57bfd944411f56a7ebe5b6bf

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-14 11:13:22 +02:00
Max Kotliar
01fff44a8d
vmagent/client: Use VictoriaMetrics remote write protocol by default, downgrade to Prometheus if needed ()
This commit improves how vmagent selects the remote write protocol.
Previously, vmagent [performed a handshake
probe](0ff1a3b154/lib/protoparser/protoparserutil/vmproto_handshake.go (L11))
at
[startup](0ff1a3b154/app/vmagent/remotewrite/client.go (L173)):

- If the probe succeeded, it used the VictoriaMetrics (VM) protocol.

- If the probe failed, it downgraded to the Prometheus protocol.

- No protocol changes occurred after the initial probe at runtime.

However, this approach had limitations:

- If vmstorage was unavailable during vmagent startup, vmagent would
immediately downgrade to the Prometheus protocol, leading to higher
network usage unitl vmagent restarted. This case has been reported in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615.

- If the remote write server was updated or downgraded (e.g., during a
fallback or migration), vmagent would not detect the protocol change. It
would continue retrying failed requests and eventually drop them.
Require a restart of vmagent to pick up the new protocol.

This commit introduces a more adaptive mechanism.
vmagent always starts with the VM protocol and downgrades to the
Prometheus protocol only if an unsupported media type or bad request
response is received.
When this happens, the protocol is downgraded for all future requests.
In-flight requests are re-packed from Zstd to Snappy and retried
immediately.
Snappy-encoded requests are dropped if an unsupported media type or bad
request is received (no retrying).

Additionally, the in-memory and persisted queues could mix snappy and
zstd encoded blocks. The proper encoding is decided before sending by
encoding.IsZstd function.

TODO:
* [x] Add tests
* [x] Update documentation
* [x] Changelog
* [x] Research on
[content-type](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054),
[accept-encoding](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786923382)

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615#top
issue.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-14 11:13:22 +02:00
Aliaksandr Valialkin
f28013894f
lib/logstorage: add support for <duration_seconds:field> formatting option for format pipe
This option formats duration values as floating-point seconds.
2025-04-10 22:57:22 +02:00
Aliaksandr Valialkin
04fb337752
victorialogs: add cluster mode
Cluster mode is enabled when -storageNode command-line flag is passed to VictoriaLogs.
In this mode it spreads the ingested logs among storage nodes specified in the -storageNode flag.
It also queries storage nodes during `select` queries.

Cluster mode allows building multi-level cluster setup when top-level select node can query multiple lower-level clusters
and get global querying view.

See https://docs.victoriametrics.com/victorialogs/cluster/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5077
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7950
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8223
2025-04-10 16:57:15 +02:00
Aliaksandr Valialkin
b61d8059b8
lib/protoparser: support for identity encoding in a generic way inside protoparserutil.GetUncompressedReader
This should help avoiding future issues when `identity` encoding isn't replaced to `` encoding
by the caller of protoparserutil.GetUncompressedReader().

This is a follow-up for 303b425fa3

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8652
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8649
2025-04-10 13:52:31 +02:00
Andrii Chubatiuk
61ab2e14b4
lib/protoparser/datadog*: support Content-Encoding: identity value
introduction of common decompression logic in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416 removed
ability to treat unsupported compression algorithms as uncompressed data
for datadog v1 endpoint. This PR adds support of `identity`
Content-Encoding header value, though according to RFC 2616 this value
is only expected in `Accept-Encoding` header

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8649
2025-04-08 17:45:22 +02:00
Nikolay
d5522e7c15
lib/httpserver: mask authKey at PostFrom
'authKey' is well-known url and form param for VictoriaMetrics
components authorization. Previously, it could be printed into stdout
via httpserver error logger. It makes this authKey insecure and hard to
use.

This commit prevents from logging authKey defined at PostForm or as part
of url.Query.

It's recommneded to transfer authKey via PostForm and it should be
implemented at separate PRs.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5973

---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-04-08 17:45:22 +02:00
Nikolay
ebe15e0c7b
lib/backup/s3: properly set ProfileName
Previously, if ProfileName is set to empty value (as default). AWS s3
lib ignored any profile config defined with `-configProfilePath`.

This commit correctly configure client options and set profile name only
if it's set to non-empty value.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8668
2025-04-08 17:45:22 +02:00
Max Kotliar
79254126f1
vmagent/remotewrite: set content encoding header based on actual body
Improve remote write handling in vmagent by setting the
`Content-Encoding` header based on the actual request body, rather than
relying on configuration.

- Detects Zstd compression via the Zstd magic number.
- Falls back to Snappy if Zstd is not detected.
- Persistent queue may now contain mixed-encoding content.
- Add basic vmagent integration tests

Follow up on
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5344 and
12cd32fd75.

Extracted from
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5301
2025-04-08 17:45:20 +02:00
Andrii Chubatiuk
d339f75159
lib/streamaggr: fix panic in rate output
This commit properly reset aggregator state. Previously, it was not checked for `nil` and it lead to the panic on access.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8634
2025-04-04 14:17:17 +02:00
Aliaksandr Valialkin
0bcb5194b7
lib/logstorage: pad pipeStatsProcessorShard.groupMapShards in order to avoid false sharing when merging these shards in parallel on many CPU cores 2025-04-03 22:21:35 +02:00
Aliaksandr Valialkin
1525a93f21
lib/logstorage: add padding between hitsMap items at hitsMapAdaptive.shards in order to avoid false sharing when processing the hitsMapAdaptive.shards on multiple CPU cores 2025-04-03 20:15:19 +02:00
Artem Fetishev
4b3d7627f7
lib/storage: When creating and listing snapshots, panic instead of returning an error ()
When creating and listing snapshots, panic instead of returning an error
since errors are not recoverable anyway.
Also do not cleanup the filesystem on panic. Leave as is for further
manual inspection.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-02 16:00:25 +02:00
Artem Fetishev
cf340f6e76
lib/storage: Pass the partition time range during the partition creation and opening ()
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-02 15:04:07 +02:00
Artem Fetishev
bc0e651fd2
lib/storage: mergeBlockStreams(): replace the dependency on Storage with dependency on the set of deleted metricIDs ()
This should narrow down the function dependencies and simplify testing.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-02 13:20:32 +02:00
Aliaksandr Valialkin
984d294b9c
lib/promscrape: do not clutter logs with cannot scrape target ...: context canceled errors when vmagent is stopped 2025-04-01 23:22:42 +02:00
Aliaksandr Valialkin
f632ab8763
lib/promscrape: use chunkedbuffer.Buffer instead of bytesutil.ByteBuffer for reading response body from scrape targets
This reduces memory usage when reading large response bodies because the underlying buffer
doesn't need to be re-allocated during the read of large response body in the buffer.

Also decompress response body under the processScrapedDataConcurrencyLimitCh .
This reduces CPU usage and RAM usage a bit when scraping thousands of targets.
2025-04-01 20:48:14 +02:00
Aliaksandr Valialkin
f6bb26cd08
lib/promscrape: always store the last response per every scrape target in compressed form
This reduces memory usage for vmagent when scraping big number of targets at the cost of slightly higher CPU usage.

The increased CPU usage can be decreased by disabling tracking of stale markers either via -promscrape.noStaleMarkers
command-line flag or via `no_stale_markers: true` option at the scrape config pointed by -promscrape.config command-line flag.
See https://docs.victoriametrics.com/vmagent/#prometheus-staleness-markers
2025-04-01 16:00:30 +02:00
Aliaksandr Valialkin
94f89b7898
lib/leveledbytebufferpool: start with the pools[0] for byte slices up to 256 bytes
The pool is used mostly for obtaining byte buffers for responses from scrape targets.
There are no responses smaller than 256 bytes in practice, so there is no sense in maintaining
pools for byte slices up to 64 and 128 bytes.
2025-04-01 12:05:19 +02:00
Aliaksandr Valialkin
c80025bbfd
lib/promscrape: make sure that the maxLabelsLen contains really the maximum len(wc.labels) among concurrently running callbacks at stream.Parse
Previously the maxLabelsLen could be updated with smaller value after it is updated to bigger value by concurrently running goroutines.
Prevent this by loading the latest maxLabelsLen value and updating it only if it is smaller than the current len(wc.labels)
before the exit from callback passed to stream.Parse.

While at it, return early from the callback on the sample_limit exceeding error,
since the rest of the code in the callback becomes no-op after wc.reset().
This simplifies following the logic in the code a bit.

Also remove outdated misleading comment in front of sw.pushData() call inside callbacks passed to stream.Parse.
This comment has no sense after every callback start working with its own goroutine-local wc.
2025-04-01 11:53:45 +02:00