There are three possible cases for blockResultColumn.getValuesEncoded():
- The blockResultColumn.valuesEncoded is already set. This is the case for manually constructed blockResultColumn,
or if it is cloned via blockResult.clone().
- The blockResultColumn.chSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.bs,
by applying br.bm filter.
- The blockResultColumn.cSrc is non-nil. In this case the valuesEncoded must be read from the corresponding br.brSrc,
by applying br.bm filter.
It is better from maintainability and debuggability PoV to write this logic in a single getValuesEncoded() function
instead of indirecting it via valuesEncodedCreator.
VictoriaLogs stores min and max column values per every data block.
These values were incorrectly used by min() and max() stats functions
inside updateStatsForAllRows() function. It was assumed that this function
could use min / max values stored in the block, since all the rows in the blockResult
must be processed. But the blockResult contains _filtered_ rows,
e.g. it may have less rows than the number of rows in the original block.
In this case it is unsafe assuming that the min / max values from the original block
exist in the filtered rows inside blockResult.
Add blockResult.isFull() function, which returns true if the blockResult contains all rows
from the original block (e.g. they aren't filtered). Use this function in fast path,
while fall back to slow path, which triggers reading the column values and iterating over them.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a23c277b04)
### Describe Your Changes
fixed typos in docs and code
fixed collision in cloud docs
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/victoriametrics/contributing/).
(cherry picked from commit ac414d8b93)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* {lib/backup,app/}: gracefully cancel currently running operation during graceful shutdown
Make backup/restore process interruptable by passing global context from the operation caller.
This is needed in order to reduce shutdown delays in case backup/restore cancellation is requested.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8554
Adds a hint to check for errors on the client side when a network error
occurs during the handshake.
Follow-up on commit 53170abdccd2ca3f5952a916c5f544e0e77b5596
This PR fixes two related bugs in the `replace_regexp` pipe:
1. **Infinite loop on empty matches when `limit` is not set**
[#8625](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625)
When a regex pattern like `\d*`, `()`, or `\b` was used, the
implementation could repeatedly match the same zero-width position
without advancing the string, causing unbounded memory usage and
eventual OOM. This is now fixed by collecting all matches up front,
respecting the `limit`, and applying replacements in a single pass.
2. **Incorrect handling of anchors (`^` and `$`)**
The previous implementation applied regex matching to progressively
sliced substrings (`s = s[end:]`), which unintentionally caused anchor
patterns like `^` (start-of-string) to match at every new substring's
start. As a result, patterns that should have matched only once (e.g.,
`^|$`) ended up matching multiple times.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8625
### Changes
Updated `lib/httpserver/httpserver.go` to include a flag that can toggle
CORS (defaults to true to keep the current behavior).
This PR relates to
[this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8680#issue-2983786438)
feature request
### Checklist
The following checks are **mandatory**:
- [x] My change does not break backwards compatibility (i.e., preserves
CORS being enabled unless specified otherwise via the
`-http.cors.disabled=true` flag & value)
---------
Co-authored-by: Jai Mehra <jai.mehra@nav-timing.safrangroup.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 84163a56eb)
These were added while working on paritition index (#8134). Submitting
the separately in order to:
1. Make sure partition index will not break anything and
2. Be able to compare performance before and after swiching to parition
index.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
use the built-in max/min to simplify the code
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: pkucode <cssjtu@163.com>
(cherry picked from commit 41abe16adc)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8466
This pull request:
1. Add `GraphiteWrite`, `CSVImport`, `OpenTSDBHTTPImport` data import
methods for vmsingle and vminsert.
2. Add TCP `Write` method to `apptest.Client` to make it capable of
writing data in TCP-based protocol.
3. Add test cases:
1. for new import methods.
2. for corner cases placed under `app/victoria-metrics/main_test`.
4. Removed all test cases under `app/victoria-metrics/main_test`.
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
datadb.rb contains logRows shards, which weren't freed up after the data ingestion
for the given per-day datadb is stopped. This leads to slow memory leak when VictoriaLogs runs
for multiple days without restarts. Avoid this memory leak by freeing up the logRows shards
after converting them to in-memory parts. Re-use the freed up logRows shards via a pool in order
to reduce the pressure on GC.
(cherry picked from commit ec6f33f526)
Use multiple independent logRows shards for storing the pending log entries before converting them to searchable parts.
Every shard is protected by its own mutex, so multiple CPU cores may add multiple log rows into datadb at the same time.
This increases the performance of BenchmarkStorageMustAddRows/rowsPerInsert-1, which ingests log rows own-by-one
from concurrently running goroutines, by 2x.
(cherry picked from commit 8ad81220d3)
This commit modifies the logging behavior for client network errors
(e.g., EOFs, timeouts) during the handshake process. They are now logged
as warnings instead of errors, as they are not actionable from the
server’s perspective. Here's some examples of such errors.
Timeouts during the initial read phase:
2025-04-09T07:08:59.323Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read hello: cannot
read message with size 11: read tcp4 <REDACTED>-><REDACTED>: i/o
timeout; read only 0 bytes
EOFs occurring later in the handshake process:
2025-04-08T18:01:30.783Z error
VictoriaMetrics/lib/vmselectapi/server.go:204 cannot perform
vmselect handshake with client "<REDACTED>": cannot read isCompressed
flag: cannot read message with size 1: EOF; read only 0 bytes
By logging these as warnings, we reduce noise in error logs while
preserving valuble information for debug.
Previously, if `cpu.max` file has only `max` resource defined without
`period`, it was parsed incorrectly and silently drop error. While this
syntax is valid and actually used by some container runtimes. If period
is not defined, default value for it 100_000 must be used.
This commit fixes parsing function by using default value for period.
In addition, it adds zero value check, which fixes possible panic if
period has 0 value.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8808
This reverts commit fa6a32a39d.
Reason for revert: the broken tests were fixed on GOARCH=386 by skipping the check for the state size
after improting the state of stats function, since the state size depends on the hardware architecture.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710
### Describe Your Changes
HTTP/2 support is used by some S3-compatible storage providers, so
disabling it default leads to unexpected errors when trying to connect
to S3 endpoint.
For example, using MinIO as S3 storage backend: `net/http: HTTP/1.x
transport connection broken: malformed HTTP response`.
HTTP/2 was enabled by default previously, but while fixing inconsistency
e5f4826 commit disabled this by default.
cc: @valyala
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Fixes which are required in order to build FIPS-compliant binaries.
These changes were originally added for enterprise version and synced to
opensource for consistency and easier maintenance.
- consistently use `hash/fnv` at `app/vmalert` when calculating
checksums. Usage of md5 is not allowed in FIPS mode.
- increase encryption keys size used in testing in order to allow tests
to successfully run in FIPS mode
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Previously, it was possible to use any UTF-8 string to specify list of
labels. While this makes it easier to use it also leads to unexpected
parsing results in some cases (see
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8584 as an
example).
Enforce specifying metric in format {label="value"...} in order to avoid
issues with unexpected parsing results.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b1523f650d)
This reduces the overhead needed for converting the ingested log entries to searchable in-memory parts
when small number of log entries are passed to Storage.MustAddRows().
The BenchmarkStorageMustAddRows shows up to 10x performance increase for rowsPerInsert=1,
up to 5x performance increase for rowsPerInsert=10 and up to 2x performance increase for rowsPerInsert=100.
This should reduce CPU usage during data ingestion when every request contains small number of rows.
(cherry picked from commit 5491d54c11)
Previously, metric names stats API had a false assumption, that max
size of metric name is 256 byte. But this is configurable parameter with
4096 bytes max size. It triggered errors during API requests.
This commit replaces hard-coded 256 byte limit with common constant:
maxLabelValueSize. It has 16 MB limit.
In addition, this commit adds check for metric name stats tracker,
if metric name size exceeds default buffer limit, it will be allocated
directly on heap. It must be rare case, since most metric names has
16-64 byte size.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8759
Do not reset wc.labels in order to properly keep track of the number of used labels for the scrape,
and properly re-use the same number of wc.labels on subsequent scrapes.
See 12f26668a6 (r155481168)
