Commit graph

3103 commits

Author SHA1 Message Date
Aliaksandr Valialkin
0217198d5c
lib/prompb: use clear() function instead of loops for clearing WriteRequest fields inside WriteRequest.Reset
This makes the code shorter without lossing the clarity.
2025-03-17 14:32:02 +01:00
Aliaksandr Valialkin
d0cbf0ab9c
app/vlinsert/opentelemetry: follow-up for a884949aba
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8511
2025-03-16 01:09:38 +01:00
Devops
4fd2cb9102
fix:Fixed an issue where and were incorrectly displayed ()
### Describe Your Changes

Fixed an issue where and were incorrectly displayed when sent from
OpenTelemetry Collector to Victoria Logs

Fixes 
2025-03-16 01:09:38 +01:00
Aliaksandr Valialkin
6f9d70ae89
lib/{mergeset,storage,logstorage}: use chunked buffer instead of bytesutil.ByteBuffer as a storage for in-memory parts
This commit adds lib/chunkedbuffer.Buffer - an in-memory chunked buffer
optimized for random access via MustReadAt() function.
It is better than bytesutil.ByteBuffer for storing large volumes of data,
since it stores the data in chunks of a fixed size (4KiB at the moment)
instead of using a contiguous memory region. This has the following benefits over bytesutil.ByteBuffer:

- reduced memory fragmentation
- reduced memory re-allocations when new data is written to the buffer
- reduced memory usage, since the allocated chunks can be re-used
  by other Buffer instances after Buffer.Reset() call

Performance tests show up to 2x memory reduction for VictoriaLogs
when ingesting logs with big number of fields (aka wide events) under high speed.
2025-03-15 21:20:04 +01:00
Aliaksandr Valialkin
9ef0d7002e
lib/logstorage: pre-allocate buffers for fields and rows inside block.appendRowsTo()
This reduces the number of memory re-allocations inside the loop, which copies the rows.
2025-03-15 21:20:03 +01:00
Aliaksandr Valialkin
22eec97422
lib/logstorage: pre-allocated buffers for fields and rows inside rows.appendRows()
This should reduce the number of memory re-allocations inside the loop, which copies the rows.
2025-03-15 21:20:03 +01:00
Aliaksandr Valialkin
0019621d38
lib/logstorage: pre-allocate the buffer needed for marshaling a block of strings inside marshalStringsBlock
This reduces the number of memory re-allocations when appending the strings to the buffer in the loop.
2025-03-15 21:20:02 +01:00
Aliaksandr Valialkin
2f3e55f41f
lib/logstorage: optimize copying dict values inside valuesDict.copyFrom a bit
Pre-allocate the needed slice of strings and then assign items to it by index
instead of appending them. This reduces the number of memory allocations
and improves performance a bit.
2025-03-15 21:20:02 +01:00
Aliaksandr Valialkin
b0ac8c1f35
lib/logstorage: intern column names instead of cloning them during data ingestion
This reduces the number of memory allocations when ingesting logs with big number of fields (aka wide events)
2025-03-15 21:20:01 +01:00
Aliaksandr Valialkin
619c9a4eeb
lib/protoparser/common: properly decode snappy-encoded requests
Snappy-encoded requests are encoded in block mode instead of stream mode.
Stream mode is incompatible with block mode. See https://pkg.go.dev/github.com/golang/snappy
That's why Snappy-encoded requests must be read in block mode.

Also add a protection against passing invalid readers to PutUncompressedReader().

This is a follow-up for 0451a1c9e0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
2025-03-15 14:45:33 +01:00
Roman Khavronenko
53904f8816
lib/bytesutil: don't drop ByteBuffer.B when its capacity is bigger th… ()
…an 64KB at Reset

This commit reverts
b58e2ab214
as it has negative impacts when ByteBuffer is used for workloads that
always exceed 64KiB size. This significantly slows down affected
components because:
* buffers aren't beign reused;
* growing new buffers to >64KiB is very slow.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8501

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-03-15 01:39:01 +01:00
Aliaksandr Valialkin
32128e5d4b
lib/logstorage: support for {label in (v1,...,vN)} and {label not_in (v1, ..., vN)} syntax 2025-03-15 01:36:41 +01:00
Aliaksandr Valialkin
f8aeb0e7fc
app/vlinsert: follow-up for 37ed1842ab
- Properly decode protobuf-encoded Loki request if it has no Content-Encoding header.
  Protobuf Loki message is snappy-encoded by default, so snappy decoding must be used
  when Content-Encoding header is missing.

- Return back the previous signatures of parseJSONRequest and parseProtobufRequest functions.
  This eliminates the churn in tests for these functions. This also fixes broken
  benchmarks BenchmarkParseJSONRequest and BenchmarkParseProtobufRequest, which consume
  the whole request body on the first iteration and do nothing on subsequent iterations.

- Put the CHANGELOG entries into correct places, since they were incorrectly put into already released
  versions of VictoriaMetrics and VictoriaLogs.

- Add support for reading zstd-compressed data ingestion requests into the remaining protocols
  at VictoriaLogs and VictoriaMetrics.

- Remove the `encoding` arg from PutUncompressedReader() - it has enough information about
  the passed reader arg in order to properly deal with it.

- Add ReadUncompressedData to lib/protoparser/common for reading uncompressed data from the reader until EOF.
  This allows removing repeated code across request-based protocol parsers without streaming mode.

- Consistently limit data ingestion request sizes, which can be read by ReadUncompressedData function.
  Previously this wasn't the case for all the supported protocols.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8416
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300
2025-03-15 00:11:58 +01:00
Aliaksandr Valialkin
cd7be54731
app/vlinsert: add an ability to ignore log fields starting with the given prefixes
The `ignore_fields` HTTTP query args can contain prefixes ending with '*'.
For example, `ignore_fields=foo.*,bar` skips all the fields starting with `foo.`
during data ingestion.
2025-03-15 00:06:16 +01:00
Aliaksandr Valialkin
5069b253de
lib/logstorage: show a link to query options docs in the error message emitted during failure to parse query options
This should help figuring out and fixing the error by the user.
2025-03-15 00:06:15 +01:00
Aliaksandr Valialkin
4b8f2ee5d4
lib/logstorage: optimize handling long constant fields
Long constant fields cannot be stored in columnsHeader as a const column,
because their size exceeds maxConstColumnValueSize, so they are stored as regular values.
This commit optimizes storing such fields by storing only a single value
across the field values in a block instead of storing multiple values.
This should improve data ingestion performance a bit. This also should improve query
performance when the query accesses such fields because of better cache locality.

Also improve persisting of constant string lengths by storing them only once.
2025-03-14 03:17:18 +01:00
Aliaksandr Valialkin
46b408d054
lib/logstorage: add a test for marshalUint64Block / unmarshalUint64Block 2025-03-14 03:17:18 +01:00
Aliaksandr Valialkin
375c86b077
lib/logstorage: newTestLogRows: create a const column, which cannot be stored in the column header because its length exceeds maxConstColumnValueSize 2025-03-14 03:17:17 +01:00
f41gh7
dd32d2f99d
lib/protoparser: support zstd in all logs http ingestion, datadog and otel metrics protocols ()
This commit introduces common readers for multiple compression encoding algorithms.

Currently, supported encodings are:
* zstd
* gzip
* deflat
* snappy

 It adds new common reader to the all VictoriaLogs ingestion protocols.
And updates opentelemetry metrics parsing for VictoriaMetrics components.

Also, it ports zstd stream parses from cluster branch.

Related issues:
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8380
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8300

---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-14 00:44:50 +01:00
Zhu Jiekun
bcd775098f
app/vmagent: prevent dropping persistent queue if -remoteWrite.showURL changed
Previously, if the command-line flag value `-remoteWrite.showURL` changed, vmagent dropped content of persistent queues. It's not expected behavior and may lead to data-loss at queue.
 Further more if command-line flag value `-remoteWrite.showURL` is set to `true`, any changes to url query arguments will lead to persistent queue drop. The most common uses is kafka and gcp pub-sub integration. It uses url query arguments for client configuration.
 Also, it complicates copy content of persistent queue between vmagents. Since it requires to properly change name inside metainfo.json.

 This commit removes persistent queue name equality check from `lib/persistentqueue`. This check was added as an additional protection from on-disk data corruption.
 It's safe to skip this check for vmagent, because vmagent encodes remoteWrite.url as part of path to the queue. It guarantees that there will be no collision. 

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8477.


### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2025-03-14 00:16:52 +01:00
Andrii Chubatiuk
7c2874ff39
lib/awsapi: add EKS Pod Identity auth method
AWS introduced a new secure way for Kubernetes Pod authorization at AWS API.
The feature is called Pod Identity.
 It adds the following env variables to the Pod:
* AWS_CONTAINER_CREDENTIALS_FULL_URI -  endpoint URI served by the EKS Pod Identity Agent running on the worker node.
* AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE - projected JWT token that is used to exchange for IAM credentials.

See related blog post https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-a-new-way-for-applications-on-eks-to-obtain-iam-credentials/

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5780
2025-03-14 00:16:52 +01:00
Zakhar Bessarab
a43c317e8f
lib/httputils: always set up TLS config
Previously, TLS config was only created for URLs with `https` scheme.
This could lead to unexpected errors when original URL was redirecting
to `https` one as TLS config is not applied.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8494
2025-03-14 00:16:52 +01:00
Artem Fetishev
415f1a1527
lib/storage: Deduplication integration test ()
Add an integration test to confirm that deduplication works for the
current month. See .

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-13 17:09:33 +01:00
Artem Fetishev
ca49ac9c8c
lib/storage: increment indexdb refcount during data ingestion and retrieval ()
Almost all storage API operations, both ingestion and retrieval, involve
writing and/or reading the indexdb. However, during these operations,
the indexdb refcount is not incremented. This may lead to panics if
indexdb is rotated more than once during these operations.

This commit increments the refcount before using indexdb and decrements it
after use.

Note that rotating indexdb more than once during some operation is an
impossible case under normal circumstances as the min retention period
is 1 day (i.e. the indexdb will be rotated once per day). However, we
want the storage to behave correctly in all cases.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-13 12:26:52 +01:00
Artem Fetishev
4c853c1dd3
lib/storage: fix active timeseries collection when per-day index is disabled ()
Fix metric that shows number of active time series when per-day index is disabled. Previously, once per-day index was disabled, the active time series metric would stop being populated and the `Active time series` chart would show 0.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8411.
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-03-12 17:54:14 +01:00
Aliaksandr Valialkin
ca65aa1cce
lib/logstorage: properly parse floating-point numbers with leading zeroes in fractional part
Parsing for floating-point numbers with leading zeroes such as 1.023, 1.00234 has been broken
in the commit ae5e28524e .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8464
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-03-12 15:29:21 +01:00
Evgeny
c223aade17
lib/promscrape: use original job name as scrapePool value in targets api ()
### Fix scrapePool name

If in the scrape file, I do some magic and manipulate the job name then
Prometheus will show scrapePool as the original job name in the targets
API, but vmagent will set it to the final value which is wrong.
example
```
job: consul-targets
...

- source_labels: [ __meta_consul_service ]
      regex: (\w+)[_-]exporter
      target_label: job
      replacement: $1
```

curl to prom API will show
`"scrapePool": "consul-targets",`
vmagent:
`""scrapePool": "node",`

before changes:
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
pgbackrest
postgres
```
after changes
```
curl -s 'http://localhost:8429/api/v1/targets' | jq -r '.data.activeTargets[].scrapePool'| sort|uniq
blackbox
consul-targets
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 486b9e1c64)
2025-03-11 13:13:41 +01:00
Andrii Chubatiuk
394654c127
lib/streamaggr: fixed streamaggr panic ()
### Describe Your Changes

fixes 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit c174a046e2)
2025-03-10 13:54:23 +01:00
f41gh7
e24a9d3053
lib/metricnamestats: follow-up after b85b28d30a
* properly save state for cross-device mount points
* properly check empty state for tracker

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-03-06 23:18:42 +01:00
Nikolay
773b8b0b28
lib/storage: add tracker for time series metric names statistics
This feature allows to track query requests by metric names. Tracker
state is stored in-memory, capped by 1/100 of allocated memory to the
storage. If cap exceeds, tracker rejects any new items add and instead
registers query requests for already observed metric names.

This feature is disable by default and new flag:
`-storage.trackMetricNamesStats` enables it.

  New API added to the select component:

* /api/v1/status/metric_names_stats - which returns a JSON
object
    with usage statistics.
* /admin/api/v1/status/metric_names_stats/reset - which resets internal
    state of the tracker and reset tsid/cache.

   New metrics were added for this feature:

  * vm_cache_size_bytes{type="storage/metricNamesUsageTracker"}
  * vm_cache_size{type="storage/metricNamesUsageTracker"}
  * vm_cache_size_max_bytes{type="storage/metricNamesUsageTracker"}

  Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4458
---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-03-06 22:10:41 +01:00
Andrii Chubatiuk
c72d5690cc
lib/protoparser/opentelemetry: properly marshal nested attributes into JSON
Previously, opentelemetry attribute parsed added extra field names according to 
golang JSON parser spec for structs:

```
struct AnyValue{
 StringValue string
}
```
 Was serialized into:
```
{"StringValue": "some-string"}
```
 While opentelemetry-collector serializes it as
```
"some-string"
```

 This commit changes this behaviour it makes parses compatible with opentelemetry-collector format. See test cases for examples.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8384
2025-03-05 18:38:25 +01:00
hagen1778
a0501d01fd
lib/timeutil: add test for ParseDuration
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8403#discussion_r1976110052

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6db97d6f79)
2025-03-03 10:46:23 +01:00
Roman Khavronenko
d5d143f849
lib/promutils: move time-related funcs from promutils to timeutil ()
Since funcs `ParseDuration` and `ParseTimeMsec` are used in vlogs,
vmalert, victoriametrics and other components, importing promutils only
for this reason makes them to export irrelevant
`vm_rows_invalid_total{type="prometheus"}` metric.

This change removes `vm_rows_invalid_total{type="prometheus"}` metric
from /metrics page for these components.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 63f6ac3ff8)
2025-03-03 10:28:07 +01:00
Zakhar Bessarab
04b6939c34
lib/promrelabel/scrape_url: properly parse IPv6 address from __address__ label
Fix parsing of IPv6 addresses after discovery. Previously, it could lead
to target being discovered and discarded afterwards.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8374

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 99de272b72)
2025-02-28 14:20:24 +04:00
Aliaksandr Valialkin
c8a12435ec
lib/logstorage: add ability to specify field name prefixes inside fields (...) lists passed to pack_json and pack_logfmt pipes 2025-02-27 22:56:14 +01:00
Roman Khavronenko
3ec0247ee3
lib/prompbmarshal: move MustParsePromMetrics to protoparser/prometheus ()
`MustParsePromMetrics` imports `lib/protoparser/prometheus`, and this
package exposes the following metrics:
```
vm_protoparser_rows_read_total{type="promscrape"}
vm_rows_invalid_total{type="prometheus"}
```

It means every package that uses `lib/prompbmarshal` will start exposing
these metrics. For example, vlogs imports `lib/protoparser/common` which
uses `lib/prompbmarshal.Label`. And only because of this vlogs starts
exposing unrelated prometheus metrics on /metrics page.

Moving `MustParsePromMetrics` to `lib/protoparser/prometheus` seems like
the leas intrusive change.


-----------

Depends on another change
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8403

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-27 22:55:32 +01:00
Aliaksandr Valialkin
a1aa4b7aa9
lib/logstorage: allow passing * at in(*), contains_any(*) and contains_all(*)
Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for Grafana.

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238#issuecomment-2685447673
2025-02-27 11:41:39 +01:00
Zhu Jiekun
6631899ead
lib/storage: properly cache extDB metricsID on search error
Previously, if indexDB search failed for some reason during search at previous indexDB (aka extDB), VictoriaMetrics stored empty search result at cache. It could cause incorrect search results at subsequent requests.

 This commit checks search error and stores request results only on success. 

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8345
2025-02-26 16:07:48 +01:00
Aliaksandr Valialkin
a3ff49def0
lib/logstorage: do not treat a string with leading zeros as a number at tryParseUint64
The "00123" string shouldn't be treated as 123 number.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8361
2025-02-26 16:07:47 +01:00
Aliaksandr Valialkin
dd1c0e3bb7
lib/logstorage: optimize common regex filters generated by Grafana
For example, `field:~".+"`, `field:~".*"` or `field:""`

Replace such filters to faster ones. For example, `field:~".*"` is replaced with `*`,
while `field:~".+"` is replaced with `field:*`.
2025-02-25 20:35:04 +01:00
Aliaksandr Valialkin
e36e28a2b0
lib/regexutil: speed up Regex.MatchString for ".*" 2025-02-25 20:35:03 +01:00
Aliaksandr Valialkin
14a5ccdc83
lib/logstorage: run make fmt after 30974e7f3f
(cherry picked from commit 82cdcec6c6)
2025-02-25 19:13:31 +01:00
Aliaksandr Valialkin
9e0581533c
lib/logstorage: add le_field and lt_field filters
These filters can be used for selecting logs where one field value is less than another field value.
These filter complement `<=` and `<` filters for constant literals.

(cherry picked from commit 30974e7f3f)
2025-02-25 19:13:31 +01:00
Aliaksandr Valialkin
3bc89226bb
lib/logstorage: optimize eq_filter when it is applied to fields of the same type
(cherry picked from commit edc750dd55)
2025-02-25 19:13:30 +01:00
Aliaksandr Valialkin
dc09d0bff4
lib/mergeset: explicitly pass the interval for flushing in-memory data to disk at MustOpenTable()
This allows using different intervals for flushing in-memory data among different mergeset.Table instances.

The initial user of this feature is lib/logstorage.Storage, which explicitly passes Storage.flushInterval
to every created mereset.Table instance. Previously mergeset.Table instances were using 5 seconds
flush interval, which didn't depend on the Storage.flushInterval.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
2025-02-24 15:34:59 +01:00
Aliaksandr Valialkin
a964cc7a0c
lib/logstorage: properly use datadb.flushInterval as an interval between flushes for the in-memory parts
The dataFlushInterval variable has been mistakenly introduced in the commit 9dbd0f9085

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
2025-02-24 15:34:59 +01:00
Aliaksandr Valialkin
d56f9327ec
lib/logstorage: limit the maximum log field name length, which can be generated by JSONParser.ParseLogMessage
Make sure that the maximum log field name, which can be generated by JSONParser.ParseLogMessage,
doesn't exceed the hardcoded limit maxFieldNameSize. Stop flattening of nested JSON objects
when the resulting field name becomes longer than maxFieldNameSize, and return the nested JSON object
as a string instead.

This should prevent from parse errors when ingesting deeply nested JSON logs with long field names.
2025-02-24 15:34:59 +01:00
Aliaksandr Valialkin
dc536d5626
lib/logstorage: add a benchmark for JSONParser.ParseLogMessage 2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
0d3ee707ba
lib/encoding/zstd: reduce the number of cached zstd.Encoder instances
Use the real compression level supported by github.com/klauspost/compress/zstd as a cache map key.
The number of real compression levels is smaller than the number of zstd compression levels.
This should reduce the number of cached zstd.Encoder instances.

See https://github.com/klauspost/compress/discussions/1025
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7503#issuecomment-2500088591
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
3ee4b3ef24
lib/logstorage: add contains_any and contains_all filters
- `contains_any` selects logs with fields containing at least one word/phrase from the provided list.
  The provided list can be generated by a subquery.

- `contains_all` selects logs with fields containing all the words and phrases from the provided list.
  The provided list can be generated by a subquery.
2025-02-24 15:34:58 +01:00
Aliaksandr Valialkin
3e941920f6
lib/logstorage: do not spend CPU time on preparing values for already filtered out rows according to bm at filterEqField.applyToBlockSearch 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
6975352d5a
lib/logstorage: avoid extra memory allocations at getEmptyStrings() 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
a2d0846e86
lib/logstorage: add an ability to drop duplicate words at unpack_words pipe 2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
518ed87a3a
lib/logstorage: rename unpack_tokens to unpack_words pipe
The LogsQL defines a word at https://docs.victoriametrics.com/victorialogs/logsql/#word ,
so it is more natural to use unpack_words instead of unpack_tokens name for the pipe.
2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
4beceb67ab
lib/logstorage: optimize OR filter a bit for many inner filters
Use two operations on bitmaps per each inner filter instead of three operations.
2025-02-24 15:34:57 +01:00
Aliaksandr Valialkin
bff5551ba5
lib/logstorage: use clear() for clearing bitmap bits at resetBits() instead of a loop
The clear() call is easier to read and understand than the loop.
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
4dfd1407ba
lib/logstorage: avoid calling bitmap.reset() at getBitmap()
The bitmap at getBitamp() must be already reset when it was returned to the pool via putBitamp().
Thise saves CPU a bit.
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
bc3e557f02
lib/logstorage: improve error logging for improperly escaped backslashes inside quoted strings
This should simplify debugging LogsQL queries by users
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
1f11bc948e
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
504c034cbf
lib/logstorage: optimize len, hash and json_array_len pipes for repeated values
Re-use the previous result instead of calculating new result for repated input values
2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
959282090a
lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-24 15:34:56 +01:00
Aliaksandr Valialkin
aef939dc20
lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-24 15:34:55 +01:00
Aliaksandr Valialkin
afd74d82db
lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-24 15:34:55 +01:00
Aliaksandr Valialkin
2dfd6bb689
lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
061fd098b5
lib/logstorage: properly handle _time:<=max_time filter
_time:<=max_time filter must include logs with timestamps matching max_time.
For example, _time:<=2025-02-24Z must include logs with timestamps until the end of February 24, 2025.
2025-02-21 12:43:26 +01:00
Aliaksandr Valialkin
80d173471f
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-21 12:43:26 +01:00
Hui Wang
93bbe10074
app/vmselect: add query resource limits priority
This commit adds support for overriding vmstorage `maxUniqueTimeseries` with specific
resource limits:
1. `-search.maxLabelsAPISeries` for
[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
[/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
2. `-search. maxSeries` for
[/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series)
3. `-search.maxTSDBStatusSeries` for
[/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats)
4. `-search.maxDeleteSeries` for
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)

Currently, this limit priority logic cannot be applied to flags
`-search.maxFederateSeries` and `-search.maxExportSeries`, because they
share the same RPC `search_v7` with the /api/v1/query and
/api/v1/query_range APIs, preventing vmstorage from identifying the
actual API of the request. To address that, we need to add additional
information to the protocol between vmstorage and vmselect, which should
be introduced in the future when possible.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7857
2025-02-19 18:14:54 +01:00
Andrii Chubatiuk
94bf90842a
app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:12:40 +01:00
Andrii Chubatiuk
99de7456c3
lib/protoparser/influx: add -influx.forceStreamMode flag to force parsing all Influx data in stream mode ()
Addresses 

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 17:40:32 +01:00
Andrii Chubatiuk
a041488786
lib/streamaggr: added aggregation windows ()
### Describe Your Changes

By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.

For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.

Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:

- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
  It allows enabling aggregation windows for a specific aggregator.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit c8fc903669)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:31:37 +01:00
hagen1778
bb302df170
lib/logstorage: adjust expected compression ratio in tests
A follow-up after 9bb5ba5d2f
that impacted compression ratio for data compressed with native GO zstd lib (`make test-pure`).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 38bded4e58)
2025-02-19 13:30:05 +01:00
Aliaksandr Valialkin
697b775a46
lib/logstorage: remove optimizations from LogRows.sortFieldsInRows
It has been appeared these optimizatios do not give measurable performance improvements,
while they complicate the code too much and may result in slowdown when the ingested logs have
different sets of fields.

This is a follow-up for 630601488e

(cherry picked from commit dce5eb88d3)
2025-02-19 13:30:04 +01:00
Aliaksandr Valialkin
d0d9fb2818
lib/logstorage: return back the maximum number of files for log fields data from 256 to 128
It has been appeared that 256 files increase RAM usage too much comparing to 128 files
when ingesting logs with hundreds of fields (aka wide events). So let's return back 128 files
limit for now.

This is a follow-up for 9bb5ba5d2f

(cherry picked from commit a50ab10998)
2025-02-19 13:30:04 +01:00
Aliaksandr Valialkin
0a8d52376e
lib/bytesutil: drop ByteBuffer.B when its capacity is bigger than 64KB at Reset
There is little sense in keeping too big buffers - they just waste RAM and do not reduce
the load on GC too much. So it is better dropping such buffers at Reset instead of keeping them around.

(cherry picked from commit b58e2ab214)
2025-02-19 13:30:03 +01:00
Aliaksandr Valialkin
53849c95b7
lib/filestream: use smaller sizes for read buffers than for write buffers
The number of filestream readers is proportional to the number of parts to be merged,
while the number of filestream writers is proportional to the number of concurrent merges.
Usually around 4-16 parts are merged at once, so the number of active filestream readers is ~8x
bigger than the number of active filestream writers.

So it is a good idea to use smaller size of read buffers comparing to the size of write buffers.
Limit read buffer size by 64Kb, while write buffer size is limited by 128Kb.
This should reduce the overall memory usage when merging parts with big number of files.
This is the case for VictoriaLogs, which works with logs containing hundreds of fields (aka wide events).

(cherry picked from commit 659251beaa)
2025-02-19 13:30:03 +01:00
Aliaksandr Valialkin
a842114070
lib/logstorage: make sure that the data for every log field is stored in a separate file until the number of files is smaller than 256
This should improve query performance for logs with hundreds of fields (aka wide events).
Previously there was a high chance that the data for multiple log fields is stored in the same file.
This could result in query performance slowdown and/or increased disk read IO,
since the operating system could read unnecessary data for the fields, which aren't used in the query.

Now log fields are guaranteed to be stored in separate files until the number of fields exceeds 256.
After that multiple log fields start sharing files.

(cherry picked from commit 9bb5ba5d2f)
2025-02-19 13:30:02 +01:00
Aliaksandr Valialkin
0cd8591700
lib/filestream: reduce the maximum size of the buffered data per every stream from 512Kb to 256Kb
This reduces memory usage when many filestreams are processed simultaneously.
This is the case for VictoriaLogs when it processes logs with hundreds of fields.

(cherry picked from commit 2a681f2e8d)
2025-02-19 13:30:02 +01:00
Aliaksandr Valialkin
6a590de86f
lib/logstorage: LogRows.mustAddInternal a bit
- Re-use column names and values from the previously added rows if possible.
  This increases locality of reference for field names and values, while improving
  access speed for the field names and values.

- Postpone sorting fields in the added rows until creating inmemory part from them.
  This allows optimizing the sorting for log fields with the same set of fields.
  This is usually the case for logs, which belong to the same logs stream.

(cherry picked from commit 630601488e)
2025-02-19 13:30:02 +01:00
Aliaksandr Valialkin
893241b280
lib/logstorage: log the path to metadata file on errors at partHeader.mustReadMetadata
This should simplify troubleshooting

(cherry picked from commit f4ca5d3b1a)
2025-02-19 13:30:01 +01:00
Aliaksandr Valialkin
00d8e7a373
lib/logstorage: allow calling visitSubqueries on nil Query
This makes the code, which calls Query.visitSubquery, less error prone

(cherry picked from commit 910f307ca2)
2025-02-19 13:30:01 +01:00
Aliaksandr Valialkin
3ba095a875
lib/logstorage: remove needExecuteQuery from filterIn and filterStreamID, since it isn't needed
(cherry picked from commit 6afd66dcc8)
2025-02-19 13:30:01 +01:00
Nikolay
46b66626c8
lib/httpserver: properly check basic authorization
Commit 68791f9ccc introduced regression.
It performed basicAuth check before built-in routes. It made impossible
to bypass basic authorization with `authKey` param.

This commit fixeds that issue and removes unneeded check. It also adds
integration tests for this case.

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-17 16:08:50 +01:00
Aliaksandr Valialkin
88363b46b5
lib/logstorage: consistently use Query.cloneShallow() for shallow cloning of the original query 2025-02-17 15:36:38 +01:00
Aliaksandr Valialkin
5e4b5f9969
lib/logstorage: move common code for parsing a query inside parens into a separate function 2025-02-17 15:36:37 +01:00
Aliaksandr Valialkin
6155b85a13
lib/logstorage: make sure that chunkedAllocator is isn't used from concurrently running goroutines
This is needed in order to avoid data races
2025-02-17 15:36:37 +01:00
Aliaksandr Valialkin
7458aa392a
lib/logstorage: ensure that statsProcessor.updateStatsForAllRows() is called on non-empty blockResult
This eliminates a class of potential bugs with incorrect stats calculations when an additional filter
is applied to the blockResult before passing it to the stats function, and this filter removes
all the rows from blockResult.
2025-02-17 15:36:37 +01:00
Aliaksandr Valialkin
71636e922a
lib/logstorage: properly initialize minValue and maxValue at pipeLenProcessorShard and pipeHashProcessorShard
Previously this could result in incorrect 0 result of min() stats function applied to the len() results.

This is a follow-up for eddeccfcfb
2025-02-17 15:36:36 +01:00
Artem Fetishev
e17ca8bcc0
Allow disabling per-day index ()
Allow disabling the per-day index using the `-disablePerDayIndex` flag.
This should significantly improve the ingestion rate and decrease the
disk space usage for the use cases that assume small or no churn rate.
See the docs added to `docs/README.md` for details.

Both improvements are due to no data written to the per-day index.
Benchmark results:

```shell
rm -Rf ./lib/storage/Benchmark*; go test ./lib/storage -run=NONE -bench=BenchmarkStorageInsertWithAndWithoutPerDayIndex --loggerLevel=ERROR
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: 13th Gen Intel(R) Core(TM) i7-1355U
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/perDayIndexes-12                 1        3850268120 ns/op                39.56 data-MiB          28.20 indexdb-MiB           259722 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/HighChurnRate/noPerDayIndexes-12               1        2916865725 ns/op                39.57 data-MiB          25.73 indexdb-MiB           342834 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/perDayIndexes-12                   1        2218073474 ns/op                 9.772 data-MiB         13.73 indexdb-MiB           450842 rows/s
BenchmarkStorageInsertWithAndWithoutPerDayIndex/NoChurnRate/noPerDayIndexes-12                 1        1295140898 ns/op                 9.771 data-MiB          0.3566 indexdb-MiB         772119 rows/s
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/storage  11.421s
```

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-02-17 15:36:23 +01:00
Roman Khavronenko
c1861bdf8b
bump golangci-lint to v1.64.4
See https://github.com/golangci/golangci-lint/releases/tag/v1.64.4

* address linting errors

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-13 11:18:09 +01:00
f41gh7
dd6ea01c73
lib/storage: refactoring introduce OpenOptions
MustOpenStorage function may accept variable number of optional
arguments. This commit combines optional arguments into dedicated OpenOptions
struct. It reduces complexity of adding new optional arguments.

Related PR:
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8118
2025-02-13 11:18:03 +01:00
Aliaksandr Valialkin
59e9426068
lib/logstorage: attempt to use int64 bucketing before trying float64 bucketing at blockResult.getbucketedValue()
int64 bucketing is lossless and faster than float64 bucketing, so it is preferred over float64 bucketing
2025-02-13 00:02:20 +01:00
Aliaksandr Valialkin
7b38f7b5ef
lib/logstorage: refactor bucketing code
1. Use distinct code paths for blockResult.getValues() and blockResult.getValuesBucketed().
   This should simplify debugging and maintenance of the resulting code.

2. Do not load column values if all the values in the block fit the same bucket.
   Use blockResultColumn.minValue and blockResultColumn.maxValue for determining whether
   column values must be loaded via blockResultColumn.getValuesEncoded().
   This signiciantly improves performance for big buckets, which cover all the column
   values in a block.

3. Properly calculate buckets for negative values.

4. Properly adjust weekly buckets by Monday.
2025-02-12 21:47:46 +01:00
Zhu Jiekun
e25a5a3d43
docs: [all] fix typo for description of flag -pprofAuthKey ()
### Describe Your Changes

fix typo for description of flag  -pprofAuthKey

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 5fad3c8492)
2025-02-12 12:47:43 +01:00
Andrii Chubatiuk
e9ca6eaaf0
app/vlinsert: add OpenTelemetry ingested logs trace_id and span_id
This commit parses additional optional fields from OpenTelemetry logs protocol.

Related issue:

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8255
(cherry picked from commit 3a27073634)
2025-02-12 12:47:42 +01:00
Phuong Le
c2f3aa291a
lib/protoparser/clusternative: properly return error from parsing callback
Previously the `callbackErr` is silently ignored in clusternative parser, which is used at vminsert for parsing clusterNative requests and at vmstorage for parsing vminsert requests.

 This commit  fixes that by properly return callbackError after reading all block metrics. This aligns
with other parsers in `lib/protoparser`.
2025-02-12 10:32:28 +01:00
Aliaksandr Valialkin
8d76c1c2c0
lib/logstorage: improve performance of stats by (...) bucketing a bit 2025-02-12 03:26:16 +01:00
Aliaksandr Valialkin
c6b3899c86
lib/logstorage/pipe_sort_topk.go: do not read _time field values if they aren't referred in the sort by(...)
This improves performance for queries, which use `sort by (...) limit N` without mentioning _time field.
For example, the following query must work faster now

    _time:1d | rm _time | sort by (request_duration desc) limit 10

(cherry picked from commit 422caf6bd7)
2025-02-11 23:02:22 +01:00
Aliaksandr Valialkin
22591df851
lib/logstorage/block_result.go: remove misleading comment left after the commit eddeccfcfb
(cherry picked from commit 33c55d7a22)
2025-02-11 23:02:21 +01:00
Aliaksandr Valialkin
404901d7e8
lib/logstorage: optimize parsing timezone offset at TryParseTimestampRFC3339Nano()
- Add a fast path for timestamps ending with 'Z'
- Use strings.LastIndexAny instead of strings.IndexAny for searching
  for timezone offset at the end of the string. This works faster
  for timestamps with sub-second precision.

(cherry picked from commit 335071cf3d)
2025-02-11 23:02:21 +01:00
Aliaksandr Valialkin
4760df3e04
lib/logstorage: optimize pipe pipe for repeated strings, uint8 values and tuples
Update the pipe state only once per each series of repeated strings, uint8 values and tuples.
This improves performance a bit for the following `top` pipes:

- top (string_field)
- top (uint8_field)
- top (field1, ..., fieldN)

Do not apply the optimization for uint16, uint32, uint64 and int64 fields, since they
usually contain big number of unique values, which do not repeat most of the time.
2025-02-11 17:01:22 +01:00
Aliaksandr Valialkin
2d7263dd5f
lib/logstorage: properly compare RFC3339 timestamps with sub-second precision in lessString()
Previously RFC3339 timestamps with sub-second precision could be incorrectly compared by lessString().
For example, 2025-01-20T10:20:30.1Z was incorrectly treated as smaller than 2025-01-20T10:20:30.09Z,
because the first timestamp has smaller decimal number after the last dot than the second timestamp.

(cherry picked from commit 81d359507d)
2025-02-10 21:32:45 +04:00
Aliaksandr Valialkin
a6171ca7e2
lib/logstorage: optimize performance for stats, top and uniq pipes a bit
Split unique values (groups) into shards according to the configured concurrency
during processing of the matching rows if the number of unique values exceeds the hardcoded threshold.
Previously this splitting was performed unconditionally at the merge stage when merging independently
calculated per-CPU states into a single state. It is faster to perform the split during rows processing
if the number of unique values is big.

This gives up to 30% perfromance improvements when these pipes are applied to big number of unique values (groups).

(cherry picked from commit 48602a1ae8)
2025-02-07 18:32:30 +04:00
f41gh7
a98163a9e0
app/vmselect/netstorage: stop exposing `vm_index_search_duration_seconds metric
This metric records time spent on search operations in the index.
It was introduced in
[v1.56.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.56.0).
However, this metric was used neither in dashboards nor in alerting
rules.
It also has high cardinality because index search operations latency can
differ by 3 orders of magnitude.

See
[example](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/cardinality?date=2025-02-05&match=vm_index_search_duration_seconds_bucket&topN=10&focusLabel=).

 Hence, dropping it as unused.

---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-06 13:48:32 +01:00
Aliaksandr Valialkin
851a5636aa
lib/logstorage: properly limit the number of concurrent workers at stats, top and uniq pipes according to the provided options(concurrency=N)
The number of worker shards per each pipe processor is created during query initialization.
This number equals to the `options(concurrency=N)` if this option is set or to the number of available CPU cores.
This means that all the pipes must adhere the given concurrency when passing data blocks
to the next pipe.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8201

The bug has been introduced in 0214aa328e
2025-02-06 13:44:35 +01:00
Nikolay
295d2b9721
app/vmauth: allow to serve internal API and different address
vmauth uses 'lib/httpserver' for serving HTTP requests. This server
unconditionally defines built-in routes (such as '/metrics',
'/health', etc). It makes impossible to proxy `HTTP` requests to  backends with the same routes.
Since vmauth's httpserver matches built-in route and return local
response.

 This commit adds new flag `httpInternalListenAddr` with
default empty value. Which removes internal API routes from public
router and exposes it at separate http server.

For example given configuration disables private routes at `0.0.0.0:8427` address and serves it at `0.0.0.0:8426`:

`./bin/vmauth --auth.config=config.yaml --httpListenAddr=:8427 --httpInternalListenAddr=127.0.0.1:8426`

Related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6468
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7345
2025-02-06 13:44:34 +01:00
Artem Fetishev
fd0bd8a67f
lib/storage: fix cardinality limiting for cases when insertion takes fast path ()
### Describe Your Changes

The cardinality limiter in this case does not receive the actual
metricID but some other value found in r.TSID.MetricID and is not
initialized. Depending on the system and/or go runtime implementation,
this value can be 0 or some garbage value (which shouldn't have too wide
a range). Thus, there basically no limit for inserted metricIDs.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 631b736bc2)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-05 15:29:07 +01:00
f41gh7
39e69b103e
app/vmselect: properly cancel long running requests on client connection close
At this time `bufferedwriter` [silently ignores connection close
errors](78eaa056c0/lib/bufferedwriter/bufferedwriter.go (L67)).
It may be very convenient in some situations (to not log such
unimportant errors), but it's too implicit and unsafe for the others.
For example, if you close [export
API](https://docs.victoriametrics.com/#how-to-export-time-series) client
connection in the middle of communication, VictoriaMetrics won't notice
it and will start to hog CPU by exporting all the data into nowhere
until it process all of them. If you'll make a few retries, it will be
effectively a DoS on the server.

This commit replaces this implicit error suppressing with explicit error
handling which fixes the issue with export API.

Issue was introduced at e78f3ac8ac
2025-01-29 16:38:38 +01:00
Roman Khavronenko
6c596d2f4d
lib/cgroup: warn users about using fractional CPU quotas ()
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7988

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 13c4324bb5)
2025-01-29 13:29:23 +01:00
Aliaksandr Valialkin
bacf58de76
lib/logstorage: remove unnecesary abstraction - RowsFormatter
It is better to use the AppendFieldsToJSON function directly
instead of hiding it under RowsFormatter abstraction.

(cherry picked from commit 95f182053b)
2025-01-29 13:29:23 +01:00
Aliaksandr Valialkin
40646a125e
lib/logstorage: ignore logs with too long field names during data ingestion
Previously too long field names were silently truncated. This is not what most users expect.
It is better ignoring the whole log entry in this case and logging it with the WARNING message,
so human operator could notice and fix the ingestion of incorrect logs ASAP.

The commit also adds and updates the following entries to VictoriaLogs faq:

- https://docs.victoriametrics.com/victorialogs/faq/#how-many-fields-a-single-log-entry-may-contain
- https://docs.victoriametrics.com/victorialogs/faq/#what-is-the-maximum-supported-field-name-length
- https://docs.victoriametrics.com/victorialogs/faq/#what-length-a-log-record-is-expected-to-have

These entries are referred at `-insert.maxLineSizeBytes` and `-insert.maxFieldsPerLine` command-line descriptions
and at the WARNING messages, which are emitted when log entries are ignored because of some of these limits
are exceeded.

(cherry picked from commit 3c036e0d31)
2025-01-29 13:29:22 +01:00
Aliaksandr Valialkin
585ff968f4
lib/fs/fsutil: move lib/envutil to the more appropriate place at lib/fs/fsutil
This is a follow-up for 043d066133
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6871
2025-01-27 18:47:53 +01:00
Aliaksandr Valialkin
d60238d917
lib/storage: open per-month partitions in parallel
This should reduce the time needed for opening the storage with retentions exceeding a few months.

While at at, limit the concurrency of opening partitions in parallel to the number of available CPU cores,
since higher concurrency may increase RAM usage and CPU usage without performance improvements
if opening a single partition is CPU-bound task.

This is a follow-up for 17988942ab
2025-01-27 16:12:25 +01:00
Aliaksandr Valialkin
bca61bdd35
lib/filestream: use correct formatting option for error type in the error message 2025-01-27 15:24:11 +01:00
Aliaksandr Valialkin
c51282d698
lib/logstorage: open per-day partitions in parallel during startup
This significantly reduces startup times when the storage contains large partitions over many days.
2025-01-27 00:35:35 +01:00
Aliaksandr Valialkin
b01b02b40c
lib/logstorage: optimize unmarshalColumnNames a bit
This should reduce the time needed for opening a large storage with many partitions,
which contain logs with big number of fields (aka wide events).

Thanks to @kiriklo for the initial idea at the pull request https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8061

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7937
2025-01-27 00:14:42 +01:00
Aliaksandr Valialkin
dbe8e9df09
lib/logstorage: improve error message by adding a link with the explanation why VictoriaLogs ignores logs with the size exceeding 2MB
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7972
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7984
2025-01-26 22:53:15 +01:00
Aliaksandr Valialkin
4f9e51e106
lib/logstorage: block_stat pipe: return the path to the part where the block is stored 2025-01-26 22:53:15 +01:00
Aliaksandr Valialkin
ee69dd028a
lib/{fs,filestream}: unconditionally disable fsync in tests
Use the testing.Testing() function in order to determine whether the code runs in test.
This allows running tests and fast speed without the need to specify DISABLE_FSYNC_FOR_TESTING
environment variable.

This is a follow-up for the commit 334cd92a6c
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6871
2025-01-26 22:07:44 +01:00
Aliaksandr Valialkin
4908f98ee2
lib/logstorage: remove unneeded code after 202eb429a7
readerWithStats isn't used when reading column names from file
2025-01-26 22:07:44 +01:00
Aliaksandr Valialkin
fea934936b
lib/logstorage: properly propagate extra filters to all the subqueries
The purpose of extra filters ( https://docs.victoriametrics.com/victorialogs/querying/#extra-filters )
is to limit the subset of logs, which can be queried. For example, it is expected that all the queries
with `extra_filters={tenant=123}` can access only logs, which contain `123` value for the `tenant` field.

Previously this wasn't the case, since the provided extra filters weren't applied to subqueries.
For example, the following query could be used to select all the logs outside `tenant=123`, for any `extra_filters` arg:

    * | union({tenant!=123})

This commit fixes this by propagating extra filters to all the subqueries.

While at it, this commit also properly propagates [start, end] time range filter from HTTP querying APIs
into all the subqueries, since this is what most users expect. This behaviour can be overriden on per-subquery
basis with the `options(ignore_global_time_filter=true)` option - see https://docs.victoriametrics.com/victorialogs/logsql/#query-options

Also properly apply apply optimizations across all the subqueries. Previously the optimizations at Query.optimize()
function were applied only to the top-level query.
2025-01-26 22:05:05 +01:00
Aliaksandr Valialkin
7b62086609
lib: consistently use logger.Panicf("BUG: ...") for logging programming bugs
logger.Fatalf("BUG: ...") complicates investigating the bug, since it doesn't show the call stack,
which led to the bug. So it is better to consistently use logger.Panicf("BUG: ...") for logging programming bugs.
2025-01-24 16:40:50 +01:00
Nikolay
cab5cf3c4c
app/vmselect: fixes panic data race at query tracing
Previously, NewChild elements of querytracer could be referenced by concurrent
storageNode goroutines. After earlier return ( if search.skipSlowReplicas is set), it is
possible, that tracer objects could be still in-use by concurrent workers.
  It may cause panics and data races. Most probable case is when parent tracer is finished, but children
still could write data to itself via Donef() method. It triggers read-write data race at trace
formatting.

This commit adds a new methods to the querytracer package, that allows to
create children not referenced by parent and add it to the parent later.

 Orphaned child must be registered at the parent, when goroutine returns. It's done synchronously by the single caller  via finishQueryTracer call. 
If child didn't finished work and reference for it is used by concurrent goroutine, new child must be created instead with
context message.
 It prevents panics and possible data races.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8114

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-01-24 13:55:32 +01:00
Phuong Le
3ada13dd48
lib/logstorage: remove redundant error check 2025-01-24 07:52:52 +01:00
Aliaksandr Valialkin
45cc9974ab
lib/logstorage: inherit query options by nested queries
This is a follow-up for b620b5cff5
2025-01-24 07:52:51 +01:00
Aliaksandr Valialkin
bfd198a4d5
lib/logstorage: add hash pipe for calculating hash over the given log field
This pipe may be useful for sharding log entries among hash buckets.
2025-01-24 07:52:51 +01:00
Aliaksandr Valialkin
0a586ecfd8
lib/logstorage: add an ability to set query concurrency on a per-query basis
This is done via 'options(concurrency=N)' prefix for the query.
For example, the following query is executed on at most 4 CPU cores:

    options(concurrency=4) _time:1d | count_uniq(user_id)

This allows reducing RAM and CPU usage at the cost of longer query execution times,
since by default every query is executed in parallel on all the available CPU cores.

See https://docs.victoriametrics.com/victorialogs/logsql/#query-options
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
fb311d3ad5
lib/logstorage: always pass the current timestamp to newLexer()
Also always initialize Query.timestamp with the timestamp from the lexer.

This should avoid potential problems with relative timestamps inside inner queries.
For example, the `_time:1h` filter in the following query is correctly executed
relative to the current timestamp:

   foo:in(_time:1h | keep foo)
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
159b821a52
lib/logstorage: simplify the caller side of addNewItem() function 2025-01-24 07:52:50 +01:00
Andrii Chubatiuk
49694efc02
lib/protoparser/opentelemetry: do not drop histogram buckets, when sum is absent ()
Despite requirement in OpenTelemetry spec that histograms should contain
sum, [OpenTelemetry collector promremotewrite
translator](37c8044abf/pkg/translator/prometheusremotewrite/helper.go (L222))
and [Prometheus OpenTelemetry
parsing](d52e689a20/storage/remote/otlptranslator/prometheusremotewrite/helper.go (L264))
skip only sum if it's absent. Our current implementation drops buckets
if sum is absent, which causes issues for users, that are expecting a
similar to Prometheus behaviour

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 2adb5fe014)
2025-01-21 12:51:36 +01:00
Aliaksandr Valialkin
5095833357
lib/logstorage: add ability to execute INNER JOIN with join pipe
(cherry picked from commit bfbe06e912)
2025-01-21 12:51:33 +01:00
Zhu Jiekun
21f6d84b4b
docs: update docs for *authKey, add authKey to HTTP 401 resp body ()
### Describe Your Changes

optimize for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6226

for user who set `*AuthKey` flag, they will receive new response in
body:
```go
// query arg not set
The provided authKey '' doesn't match -search.resetCacheAuthKey

// incorrect query arg
The provided authKey '5dxd71hsz==' doesn't match -search.resetCacheAuthKey
```

previously, they receive:
```
The provided authKey doesn't match -search.resetCacheAuthKey
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1f0b03aebe)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-20 13:49:12 +01:00
Aliaksandr Valialkin
cb6f69b3ee
lib/logstorage: merge top-level _stream:{...} filters in the query
This should improve performance of queries, which contain multiple top-level _stream:{...} filters.

This should help the case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8037#issuecomment-2595854592

(cherry picked from commit 2eb15cf30c)
2025-01-17 13:26:52 +04:00
Aliaksandr Valialkin
9f0ce93433
lib/logstorage: add a test for union pipe
This is a follow-up for f27e120aeb

(cherry picked from commit 499f0b9588)
2025-01-17 13:26:50 +04:00
Aliaksandr Valialkin
403dbd6047
lib/logstorage: properly pass tenantIDs list to initStreamFilters
Previously an empty tenantIDs list was mistakenly passed to initStreamFilters
when the query already contained top-level stream filter.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8037

(cherry picked from commit 43d615ae87)
2025-01-17 13:26:48 +04:00
Aliaksandr Valialkin
faf1bcdbdf
lib/logstorage: make golangci-lint happy after f27e120aeb
(cherry picked from commit e5b4cf33bf)
2025-01-16 17:07:35 +01:00
Aliaksandr Valialkin
7cbfe32d7e
lib/logstorage: add union pipe, which allows uniting results from multiple queries
(cherry picked from commit f27e120aeb)
2025-01-16 17:07:34 +01:00
Aliaksandr Valialkin
cb454aca03
lib/logstorage: properly drop temporary directories created by filter* tests
(cherry picked from commit ee1ce90501)
2025-01-16 17:07:34 +01:00
Aliaksandr Valialkin
0deab437e9
lib/logstorage: math pipe: add rand() function
(cherry picked from commit 47fe8cf3be)
2025-01-16 17:07:33 +01:00
Aliaksandr Valialkin
f4b08b70d2
lib/logstorage: improve performance of unique pipe for integer columns with big number of unique values
(cherry picked from commit b4f4ece162)
2025-01-16 17:07:32 +01:00
Aliaksandr Valialkin
bc9ff8e40e
lib/logstorage: improve performance when applying math calculations for _time, const and dict values
(cherry picked from commit bb00f7529f)
2025-01-16 17:07:32 +01:00
Nikolay
158f78976e
lib/storage: reduce test suite batch size ()
Commit eef6943084 added new test
functions. Which checks various cases for metricName registration at
data ingestion.
Initial dataset size had 4 batches with 100 rows each. It works fine at
machines with 5GB+ memory.
But i386 architecture supports only 4GB of memory per process.

Due to given limitations, batch size should be reduced to 3 batches and
30 rows. It keeps the same
test funtionality, but reduces overall memory usage to ~3GB.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 277fdd1070)
2025-01-14 14:29:50 +01:00
Roman Khavronenko
2406480722
lib/opentlemetry: throttle log messages during parsing ()
Samples parsing is a hot path. Bad client could easily overwhelm
receiver with bad or unsupported data. So it is better to throttle such
messages.

Follow-up after
b26a68641c

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d290efb849)
2025-01-14 14:29:50 +01:00
chenlujjj
dc10b26306
lib/opentelemetry: log the metric name of unsupported metrics ()
To resolve:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8009
Log the name of unsupported metrics.

(cherry picked from commit b26a68641c)
2025-01-14 14:29:49 +01:00
Aliaksandr Valialkin
0c7b7c9404
lib/logstorage: make golangci-lint happy after the commit d2a791bef3
(cherry picked from commit b88cda5c41)
2025-01-14 14:29:49 +01:00
Aliaksandr Valialkin
e1c044f326
lib/logstorage: add histogram stats function for calculating histogram buckets over numeric fields
(cherry picked from commit d2a791bef3)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
46ee68683b
lib/logstorage: top pipe: allow mixing the order of hits and rank suffixes
(cherry picked from commit 99516a5730)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
a944ce3f34
lib/logstorage: do not copy pipeTopkProcessorShard when obtaining parition keys
(cherry picked from commit aecc86c390)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
dbb1007b43
lib/logstorage: track integer field values in integer map for top N (int_field)
This reduces memory usage by up to 2x for the map used for tracking hits.
This also reduces CPU usage for tracking integer fields.

(cherry picked from commit cc29692e27)
2025-01-14 14:29:47 +01:00
Aliaksandr Valialkin
3775998a82
lib/logstorage: avoid callback overhead at visitValuesReadonly
Process values in batches instead of passing every value in the callback.
This improves performance of reading the encoded values from storage by up to 50%.

(cherry picked from commit f018aa33cb)
2025-01-14 14:29:47 +01:00
Roman Khavronenko
9de0b8a165
make: bump golangci-lint to v1.63.4 (
New version has additional checks and reduced resource consumption, so
it doesn't timeout for our internal repos.

To make linter happy, I addressed "redefinition of the built-in
function" lint error.

----
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-13 07:23:21 +01:00