Commit graph

2977 commits

Author SHA1 Message Date
Aliaksandr Valialkin
fb311d3ad5
lib/logstorage: always pass the current timestamp to newLexer()
Also always initialize Query.timestamp with the timestamp from the lexer.

This should avoid potential problems with relative timestamps inside inner queries.
For example, the `_time:1h` filter in the following query is correctly executed
relative to the current timestamp:

   foo:in(_time:1h | keep foo)
2025-01-24 07:52:50 +01:00
Aliaksandr Valialkin
159b821a52
lib/logstorage: simplify the caller side of addNewItem() function 2025-01-24 07:52:50 +01:00
Andrii Chubatiuk
49694efc02
lib/protoparser/opentelemetry: do not drop histogram buckets, when sum is absent (#8054)
Despite requirement in OpenTelemetry spec that histograms should contain
sum, [OpenTelemetry collector promremotewrite
translator](37c8044abf/pkg/translator/prometheusremotewrite/helper.go (L222))
and [Prometheus OpenTelemetry
parsing](d52e689a20/storage/remote/otlptranslator/prometheusremotewrite/helper.go (L264))
skip only sum if it's absent. Our current implementation drops buckets
if sum is absent, which causes issues for users, that are expecting a
similar to Prometheus behaviour

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 2adb5fe014)
2025-01-21 12:51:36 +01:00
Aliaksandr Valialkin
5095833357
lib/logstorage: add ability to execute INNER JOIN with join pipe
(cherry picked from commit bfbe06e912)
2025-01-21 12:51:33 +01:00
Zhu Jiekun
21f6d84b4b
docs: update docs for *authKey, add authKey to HTTP 401 resp body (#7971)
### Describe Your Changes

optimize for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6226

for user who set `*AuthKey` flag, they will receive new response in
body:
```go
// query arg not set
The provided authKey '' doesn't match -search.resetCacheAuthKey

// incorrect query arg
The provided authKey '5dxd71hsz==' doesn't match -search.resetCacheAuthKey
```

previously, they receive:
```
The provided authKey doesn't match -search.resetCacheAuthKey
```

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1f0b03aebe)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-20 13:49:12 +01:00
Aliaksandr Valialkin
cb6f69b3ee
lib/logstorage: merge top-level _stream:{...} filters in the query
This should improve performance of queries, which contain multiple top-level _stream:{...} filters.

This should help the case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8037#issuecomment-2595854592

(cherry picked from commit 2eb15cf30c)
2025-01-17 13:26:52 +04:00
Aliaksandr Valialkin
9f0ce93433
lib/logstorage: add a test for union pipe
This is a follow-up for f27e120aeb

(cherry picked from commit 499f0b9588)
2025-01-17 13:26:50 +04:00
Aliaksandr Valialkin
403dbd6047
lib/logstorage: properly pass tenantIDs list to initStreamFilters
Previously an empty tenantIDs list was mistakenly passed to initStreamFilters
when the query already contained top-level stream filter.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8037

(cherry picked from commit 43d615ae87)
2025-01-17 13:26:48 +04:00
Aliaksandr Valialkin
faf1bcdbdf
lib/logstorage: make golangci-lint happy after f27e120aeb
(cherry picked from commit e5b4cf33bf)
2025-01-16 17:07:35 +01:00
Aliaksandr Valialkin
7cbfe32d7e
lib/logstorage: add union pipe, which allows uniting results from multiple queries
(cherry picked from commit f27e120aeb)
2025-01-16 17:07:34 +01:00
Aliaksandr Valialkin
cb454aca03
lib/logstorage: properly drop temporary directories created by filter* tests
(cherry picked from commit ee1ce90501)
2025-01-16 17:07:34 +01:00
Aliaksandr Valialkin
0deab437e9
lib/logstorage: math pipe: add rand() function
(cherry picked from commit 47fe8cf3be)
2025-01-16 17:07:33 +01:00
Aliaksandr Valialkin
f4b08b70d2
lib/logstorage: improve performance of unique pipe for integer columns with big number of unique values
(cherry picked from commit b4f4ece162)
2025-01-16 17:07:32 +01:00
Aliaksandr Valialkin
bc9ff8e40e
lib/logstorage: improve performance when applying math calculations for _time, const and dict values
(cherry picked from commit bb00f7529f)
2025-01-16 17:07:32 +01:00
Nikolay
158f78976e
lib/storage: reduce test suite batch size (#8022)
Commit eef6943084 added new test
functions. Which checks various cases for metricName registration at
data ingestion.
Initial dataset size had 4 batches with 100 rows each. It works fine at
machines with 5GB+ memory.
But i386 architecture supports only 4GB of memory per process.

Due to given limitations, batch size should be reduced to 3 batches and
30 rows. It keeps the same
test funtionality, but reduces overall memory usage to ~3GB.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit 277fdd1070)
2025-01-14 14:29:50 +01:00
Roman Khavronenko
2406480722
lib/opentlemetry: throttle log messages during parsing (#8021)
Samples parsing is a hot path. Bad client could easily overwhelm
receiver with bad or unsupported data. So it is better to throttle such
messages.

Follow-up after
b26a68641c

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d290efb849)
2025-01-14 14:29:50 +01:00
chenlujjj
dc10b26306
lib/opentelemetry: log the metric name of unsupported metrics (#8018)
To resolve:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8009
Log the name of unsupported metrics.

(cherry picked from commit b26a68641c)
2025-01-14 14:29:49 +01:00
Aliaksandr Valialkin
0c7b7c9404
lib/logstorage: make golangci-lint happy after the commit d2a791bef3
(cherry picked from commit b88cda5c41)
2025-01-14 14:29:49 +01:00
Aliaksandr Valialkin
e1c044f326
lib/logstorage: add histogram stats function for calculating histogram buckets over numeric fields
(cherry picked from commit d2a791bef3)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
46ee68683b
lib/logstorage: top pipe: allow mixing the order of hits and rank suffixes
(cherry picked from commit 99516a5730)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
a944ce3f34
lib/logstorage: do not copy pipeTopkProcessorShard when obtaining parition keys
(cherry picked from commit aecc86c390)
2025-01-14 14:29:48 +01:00
Aliaksandr Valialkin
dbb1007b43
lib/logstorage: track integer field values in integer map for top N (int_field)
This reduces memory usage by up to 2x for the map used for tracking hits.
This also reduces CPU usage for tracking integer fields.

(cherry picked from commit cc29692e27)
2025-01-14 14:29:47 +01:00
Aliaksandr Valialkin
3775998a82
lib/logstorage: avoid callback overhead at visitValuesReadonly
Process values in batches instead of passing every value in the callback.
This improves performance of reading the encoded values from storage by up to 50%.

(cherry picked from commit f018aa33cb)
2025-01-14 14:29:47 +01:00
Roman Khavronenko
9de0b8a165
make: bump golangci-lint to v1.63.4 (
New version has additional checks and reduced resource consumption, so
it doesn't timeout for our internal repos.

To make linter happy, I addressed "redefinition of the built-in
function" lint error.

----
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-01-13 07:23:21 +01:00
Aliaksandr Valialkin
c4b3def5b0
lib/logstorage: format pipe: add frequently used formatters
- url encoding / decoding with <urlencode:field> and <urldecode:field>
- base64 encoding / decoding with <base64encode:field> and <base64decode:field>
- hex encoding / decoding with <hexencode:field> and <hexdecode:field>
- hex encoding for integers with <hexnumencode:field> and <hexnumdecode:field>
2025-01-13 07:23:20 +01:00
Aliaksandr Valialkin
262ccf7963
lib/logstorage: explicitly pass statsFunc to statsProcessor methods
This allows reducing the state of every statsProcessor by removing pointer to the corresponding statsFunc.
For example, this reduces statsCountProcessor size by 2x.
2025-01-13 07:23:20 +01:00
Aliaksandr Valialkin
a2003de567
lib/logstorage: stats pipe: stop finalizeStats() as soon as the query is canceled
Previoysly finalizeStats() for some functions such as count_uniq() could run for long periods
of time after the query is canceled, since stopCh wan't propagated to finalizeStats().
2025-01-13 07:23:20 +01:00
Aliaksandr Valialkin
1ac4516919
lib/logstorage: stats pipe: use integer group keys if stats by(...) contains a single field with integer values
This reduces memory usage and improves performance, since access to a map with integer keys
is faster than access to a map with string keys.
2025-01-13 07:23:20 +01:00
Aliaksandr Valialkin
c17149cb0c
lib/logstorage: count_uniq and count_uniq_hash stats functions: avoid converting integer values to strings
Prevsiously integer values were converted to strings before being passed to `updateState()` function at `count_uniq`
and `count_uniq_hash`. Later such values are converted back to integers in order to track them via integer map of unique values.

This commit avoids the int -> string -> int conversion. Instead, it passes integers directly to the integer map of unique values.
This improves performance of `count_uniq` and `count_uniq_hash` functions even further.
2025-01-13 07:23:20 +01:00
Aliaksandr Valialkin
9ff6128102
lib/logstorage: add value_type filter to LogsQL
This filter can be used when debugging and exploring logs in order to understand better
which value types are used for storing the particular log fields.

The `value_type` filter complements `block_stats` pipe.
2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
8efd21519d
lib/logstorage: run make fmt after e610edf045 2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
fcc692d00e
lib/logstorage: improve performance for math pipe
- Pass the calculated results to the next pipe in float64 columns.
  Previously the results were converted to string columns. This could slow down further calculations.

- Use custom optimized logic for processing numeric columns, which are passed to math pipe.
  Previously all the input columns were converted to string and then converted to float64
  before math pipe calculations.

- Initialize the newly added columns at blockResult as soon as they are added.
  This improves performance when big number of columns are calculated by math pipe.
2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
6b59561f96
lib/logstorage: track integer values in integer maps when counting the number of unique values at count_uniq stats function
Previously integer values were tracked in string maps. Now every input value is parsed as integer.
On success the parsed integer is tracked via specialized maps, which hold only integers.
This reduces CPU usage and memory usage in general case.
2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
2e8c678e19
lib/logstorage: avoid copying column name inside blockSearch.getColumnHeader() and blockSearch.getConstColumnValue()
Use the column name attached to the corresponding part. The lifetime of this column name exceed the blockSearch lifetime,
so it is safe using it here.

This is a follow-up for 8d968acd0a
2025-01-13 07:23:19 +01:00
Aliaksandr Valialkin
0b2b29c3d1
lib/logstorage: automatically detect columns with int64 values and store them as packed 8-byte int64 values
Previously columns with negative int64 values were stored either as float64 or string
depending on whether the negative int64 values are bigger or smaller than -2^53.
If the integer values are smaller than -2^53, then they are stored as string, since float64 cannot
hold such values without precision loss. Now such values are stored as int64.
This should improve compression ratio and query performance over columns with negative int64 values.
2025-01-13 07:23:18 +01:00
Aliaksandr Valialkin
e95b491221
lib/logstorage: make sure that the automatic conversion of field values to float64 is lossless
Previously field values could be automatically converted to float64 with precision loss.
This could lead to unexpected results when querying such field values.
For example, "10007199254740992" was incorrectly represented as 10007199254740993.
This commit prevents from such lossy conversions when storing field values.

While at it, prevent from int64 overflow at tryParseBytes and tryParseDuration functions,
which are used for parsing constants in queries for byte sizes and durations.
Now these functions return 1<<63-1 (the maximum int64 value) for constants exceeding
this value. Previously they could return arbitrary garbage for such constants.
2025-01-13 07:23:18 +01:00
Nikolay
ca5e105ada
lib/storage: add a hint for merge about type of parts in merge (#7998)
Hint allows to choose type of cache to be used for index search:
- in-memory parts are storing recently ingested samples and should use
main cache. This improves ingestion speed and cache hit ration for
queries accessing recently ingested samples.
- merges of file parts is performed in background, using a separate
cache allows avoiding pollution of the main cache with irrelevant
entries.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
(cherry picked from commit e9f86af7f5)
2025-01-10 17:45:20 +04:00
Nikolay
688f667919
lib/storage: make finalDedup schedule interval configurable
This commit makes configurable interval for checking if final dedup
process for the historical data should be started. It allows to spread
resource utilisation for multiple vmstorage/vmsingle instances in time.
Since final dedup may add additional preasure on disk, backup systems
and make cluster less stable. Storage unconditionally adds 25% jitter to
the provided value, it should simplify configuration management at
Kubernetes ecosystem. Because Kubernetes application pods must have the
same configuration.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7880

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 9ada784983)
2025-01-10 16:56:08 +04:00
Zhu Jiekun
3c1282ae7c
lib/promscrape: add Marathon service discovery
This commit adds support for [Marathon](https://mesosphere.github.io/marathon/)
service discovery to the scrape configuration.

The following flag is introduced:
```
  -promscrape.marathonSDCheckInterval duration
          Interval for checking for changes in Marathon service discovery. This works only if marathon_sd_configs is configured in '-promscrape.config' file. See https://docs.victoriametrics.com/sd_configs.html#marathon_sd_configs for details  (default 30s)
```

The service discovery could be config like:
```yaml
scrape_configs:
- job_name: marathon_job
  marathon_sd_configs:
      servers:
        - "..."
        - "..."
```
See:
[b555d94d1a/docs/sd_configs.md (marathon_sd_configs))

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6642

---------

Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 276989716f)
2025-01-10 16:50:39 +04:00
cuiweiyuan
56f98e74f5
chore: fix function name in comment (#7926)
### Describe Your Changes

 fix function name in comment

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>
(cherry picked from commit d064e14933)
2025-01-10 16:50:32 +04:00
Hui Wang
92c8049647
app/vmstorage: allow to override the default unique time series limit
previously vmstorage ignored limit values from vmselect component.

This behavior is prohibited starting from v1.105.0, with
85f60237e2.

This breaks the original intent of the -search.maxUniqueTimeseries command-line flag, which has been added at vmselect nodes in the commit b843f0e : to be able to override the default limit at vmstorage on the number of unique time series, at different subsets of vmselect nodes.

The behavior should be the following:

*    If -search.maxUniqueTimeseries command-line flag isn't set at both vmselect and vmstorage nodes, then the limit on  the number of unique time series must be automatically detected at vmstorage nodes according to 

* vmstorage: automatically adjust -search.maxUniqueTimeseries max value   . This simplifies configuration of VictoriaMetrics cluster for the typical case.

* If -search.maxUniqueTimeseries command-line flag is explicitly set at vmstorage node, then it must be used as the limit on the number of unique time series, without automatic detection of the limit. Explicitly set limit at vmstorage node cannot be exceeded by the limit from vmselect nodes.
* If the -search.maxUniqueTimeseries command-line flag is explicitly set at vmselect node, then it must override the automatically detected limit at vmstorage node. For example, if vmselect node provides the limit, which exceeds the automatically detected limit at vmstorage node, then the limit from the vmselect node must be applied during query execution at vmstorage node. This will allow properly executing queries from the subset of vmselect nodes for reporting queries described above.

related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7852
2024-12-30 15:19:03 +01:00
Andrii Chubatiuk
761d72a17d
vlinsert: take into account order of msgfields to have predictable _msg field selection in case of multiple matches (#7784)
### Describe Your Changes

Currently if multiple msgFields are present in a log row it's not
obvious which field is selected as a _msg field. With this PR and order
of msgfield values defined either via headers or query arg params
defines a priority of these values

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-12-23 19:45:04 +01:00
Andrii Chubatiuk
4b59f5e351
datadog-serverless: fixed metrics and logs ingestion from Datadog serverless extensions for AWS and GCP (#7769)
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7761

### Describe Your Changes

- datadog /api/v2/logs api supports message field in json format, which
is not documented and is used by serverless extension. This PR allows
message field to be both string and object type. Also added support of
not documented timestamp field
- added `-datadog.streamFields` and `-datadog.ignoreFields` flags to
configure default stream fields for datadog logs, where there's no
alternative option to pass extra headers and query args
- added ingest `max` and `min` values of data, which are ingested using
`datadogsketches` API, which is also actively used by serverless
extensions
- use default `.` separator instead of `_` for sketches metric names
until metrics are not sanitized
2024-12-23 19:45:04 +01:00
Aliaksandr Valialkin
bb62d3fe65
lib/logstorage: limit the maximum number of logs and/or log streams, which can be passed to stream_context pipe
This should prevent from excess usage of CPU, RAM and other resources when too many logs
are passed to 'stream_context' pipe.

It is expected that 'stream_context' pipe results are investigated by humans, who cannot inspect
surrounding logs for millions of initial logs. That's why it is OK to limit the number of logs
and/or log streams, which can be passed to 'stream_context' pipe.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7766
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7903
2024-12-23 19:45:03 +01:00
Aliaksandr Valialkin
a326a4747e
lib/logstorage: reduce memory allocations when splitting in(...) values into tokens and calculating hashes for these tokens
While at it, reduce memory allocations at Storage.getFieldValuesNoHits and make it more scalable on multi-CPU systems.

This improves performance of in(<query>) filter when the <query> returns big number of values.
2024-12-23 19:45:03 +01:00
Aliaksandr Valialkin
bb4dbbab7c
lib/logstorage: allow specifying hits column name in the top pipe via top ... hits as <column_name> syntax 2024-12-23 19:45:03 +01:00
Aliaksandr Valialkin
6c0ec98e4e
lib/logstorage: uncommend accidentally commented tests at 60f9f44150 2024-12-23 19:45:02 +01:00
Aliaksandr Valialkin
6b0da64b30
lib/logstorage: reduce memory allocations at stats and top pipes
Use chunked allocator in order to reduce memory allocations. It allocates objects from slices of up to 64Kb size.
This improves performance for `stats` and `top` pipes by up to 2x when they are applied to big number of `by (...)` groups.

Also parallelize execution of `count_uniq`, `count_uniq_hash` and `uniq_values` stats functions,
so they are executed faster on hosts with many CPU cores when applied to fields with big number
of unique values.
2024-12-23 19:45:02 +01:00
Aliaksandr Valialkin
4a406fe3fd
lib/logstorage: fixed a typo in blockResult.reset()
The commit 4599429f51 improperly set br.cs to nil,
while it should set br.bs to nil instead. This resulted in excess memory allocations
at br.csInit() and br.csInitFast().
2024-12-23 19:45:02 +01:00
Aliaksandr Valialkin
97aaaf34fd
lib/cgroup: add missing initialization of gogc variable inside SetGOGC
This is a follow-up for 79c08ecac4

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7902
2024-12-23 19:45:00 +01:00
f41gh7
0b987a1534
lib/cgroup: use the default GOGC=100 for the most of VictoriaMetrics components
Historically some of VictoriaMetrics components were optimized for the low rate of memory allocations.
These are: vmagent, single-node VictoriaMetrics and vmstorage. These components benefit from the low
GOGC value, since this allow reducing their memory usage in steady state on typical workloads.

Other VictoriaMetrics components aren't optimized for the reduced rate of memory allocations.
This results in the increased CPU usage spent on garbage collection (GC) in these components,
since it must be triggered at higher rate. See https://tip.golang.org/doc/gc-guide#GOGC for details.

These components do not use too much memory, so it is OK increasing the GOGC for these components
from 30 to 100 - this won't affect the most users.

Keep GOGC to 30 only for vmagent, single-node VictoriaMetrics and vmstorage components.
See 077193d87c and 54b9e1d3cb .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7902
2024-12-23 19:44:55 +01:00
Aliaksandr Valialkin
e325619d26
lib/logstorage: fix imroper sorting of numeric fields when they are stored as const values at sort pipe
Numeric fields can be stored as const values in the block of logs. In this case the `sort` pipe
was incorrectly comparing such values as strings instead of numbers. This results in incorrect
sort results. For example, 123 was smaller than 2. Fix this by removing the incorrect case
for comparing const fields.

While at it, replace lessString() with strings.LessNatural() in the sortBlockLess.
This improves sorting performance a bit, since the sortBlockLess function already tried
comparing numeric values, and it doesn't need to spend CPU time on such a comparison again inside lessString() call.
The commit 42c9183281 wasn't correct by replacing strings.LessNatural() with lessString()
inside the sortBlockLess() function.
2024-12-23 19:42:30 +01:00
Aliaksandr Valialkin
95dfdc434a
lib/logstorage: eliminate memory allocations when finalizing per-group values calculated by stats pipe
This improves query performance a bit when `stats by (...)` returns millions of individual `by (...)` groups
2024-12-18 22:41:06 +01:00
Aliaksandr Valialkin
53d726eca0
app/vlselect: allow passing arbitrary LogsQL filters to extra_filters and extra_stream_filters query args
While at at, allow passing an array of string values per each JSON entry at extra_filters and extra_stream_filters.
For example, `extra_filters={"foo":["bar","baz"]}` is converted into `foo:in("bar", "baz")` extra filter,
while `extra_stream_fitlers={"foo":["bar","baz"]}` is converted into `{foo=~"bar|baz"}` extra filter.

This should simplify creating faceted search when multiple values per a single log field must be selected.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365#issuecomment-2447964259

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5542
2024-12-18 22:40:52 +01:00
Aliaksandr Valialkin
63b0f02878
lib/logstorage: do not return log fields with the same constant value across all the selected logs from facets pipe
Such log fields do not give any useful information during logs' exploration.
They just clutter the output of the `facets` pipe. So it is better to drop such fields by default.

If these fields are needed, then `keep_const_fields` option can be added to `facets` pipe.
2024-12-18 22:40:51 +01:00
Roman Khavronenko
fd39eb2b2a
lib/storage: update dedup tests
* update misleading comments about preferring NaNs on intervals. NaNs
are only preferred on timestamp conflicts
* add conflicting timestamps to the benchmark test. Previously,
benchmark wasn't checking the timestamp conflict code branch. The
updated results after
c0fcfd6b97
are the following:
```
benchstat old.txt new.txt

goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: Apple M4 Pro
                                                       │   old.txt    │               new.txt                │
                                                       │    sec/op    │    sec/op     vs base                │
DeduplicateSamples/minScrapeInterval=3s-14               889.7n ± ∞ ¹   904.3n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=4s-14               735.9n ± ∞ ¹   748.7n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=10s-14              637.7n ± ∞ ¹   659.3n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=3s-14    838.8n ± ∞ ¹   810.4n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=4s-14    765.2n ± ∞ ¹   735.1n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=10s-14   673.1n ± ∞ ¹   622.4n ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                  751.7n         741.0n        -1.42%
```

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

---
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-12-18 22:38:36 +01:00
Zhu Jiekun
87d805b5f6
discovery/dockerswarm: add missing service labels to tasks discovery role
Previously service labels won't be attached when `role: tasks` is set.
Because the `addServicesLabels` function is shared by `role: tasks` and
`role: services`, and it will return nothing when `vip.Addr` is invalid
or empty.

In Prometheus, even if `vip.Addr` is empty, it attach common service
labels with [a standalone
function](f10c3454e9/discovery/moby/services.go (L129)),
which offers:
- `__meta_dockerswarm_service_id`: the id of the service.
- `__meta_dockerswarm_service_name`: the name of the service.
- `__meta_dockerswarm_service_mode`: the mode of the service.
- `__meta_dockerswarm_service_label_<labelname>`: each label of the
service, with any unsupported characters converted to an underscore.

This PR add a `addServicesLabelsForTask`, to replace the usage of
`addServicesLabels` when `role: tasks` is set. This function offers
common service labels listed above.

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7800
2024-12-13 12:10:31 +01:00
Artem Fetishev
dd079eb8e6
app/vmselect: respect -search.skipSlowReplicas when -globalReplicationFactor > 1
Previously cluster with the following vmselect configuration:

./bin/vmselect 
  -storageNode=gr1/:8211,gr1/:8212 
  -storageNode=gr2/:8213,gr2/:8214 
  -search.skipSlowReplicas=true
  -globalReplicationFactor=2

Here we have two vmstorage groups and -globalReplicationFactor=2, which effectively means that "every ingested sample is replicated across multiple vmstorage groups". Hence, gr1 and gr2 contain identical data set. And when we set -search.skipSlowReplicas=true it is expected vmselect should return result as soon as at least one storage group returned the full result.
In current state, -search.skipSlowReplicas is ignored on the storage group level. It is only respected within the group (with -replicationFactor flag).
  

   This commit fixes global replication for skipSlowReplicas. 

 To ensure that the fix works and does not break
anything replication tests have been added. For checking the fix for
skipping slow replicas see `testGroupSkipSlowReplicas()`.

To emulate storage groups, the integration test creates a cluster with
multilevel vminsert. The L1 inserts are group-level inserts, each writes
to its own group of vmstorages. The L2 vminsert is a global vminsert
that writes replicated to the L1 vminserts.

To enable multilevel inserts changes in apptest framework and
`lib/ingestserver/clusternative/server.go` were necessary.

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6924

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-12-13 11:56:06 +01:00
Andrei Baidarov
33a012c225
lib/storage: prefer stale markers over other values on dedup interval
Previously, during de-duplication staleness markers could be removed due to incorrect logic at
values equality check.
 During the evaluation of read query vmselect deduplicates samples using dedupInterval option. It picks the highest value across all points with the same timestamp next to the border of dedupInterval. The issue is any comparison with NaN via <, > returns false. This means that the position of NaN in srcValues could affect the result.


 This commit changes this logic with additional step, that explicitly checks for staleness marker for the following cases:
 1. Deduplication on vmselect
2. Deduplication in vmstorage during merges
3. Deduplication in stream aggregation

check performed only for stale markers, because other NaNs are rejected on ingestion
by vmstorage or by stream aggregation.

Checking for stale markers in general slows down dedup speed by 3%:
```
 benchstat old.txt new.txt

goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: Apple M4 Pro
                                                       │   old.txt    │               new.txt                │
                                                       │    sec/op    │    sec/op     vs base                │
DeduplicateSamples/minScrapeInterval=1s-14               462.8n ± ∞ ¹   425.2n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=2s-14               905.6n ± ∞ ¹   903.3n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=5s-14               710.0n ± ∞ ¹   698.9n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=10s-14              632.7n ± ∞ ¹   638.5n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=1s-14    439.7n ± ∞ ¹   409.9n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=2s-14    908.9n ± ∞ ¹   882.2n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=5s-14    721.2n ± ∞ ¹   684.7n ± ∞ ¹       ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=10s-14   659.1n ± ∞ ¹   630.6n ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                  659.5n         636.0n        -3.56%
```

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7674
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-12-12 13:00:34 +01:00
Andrii Chubatiuk
fb8ff5e35c
app/vlinsert/syslog: fixed structured data parsing (#7801)
### Describe Your Changes

rfc5424 doesn't allow structured data to be started from whitespace, but
it can be present in the end of this section
related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7776

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit e0ab3fccaf)
2024-12-11 17:10:10 +01:00
f41gh7
f589d65a9b
lib/timeserieslimits: follow-up for 564e6ea024
Changed enabled limit condition to `or` instead of `and`. Since labels must checked if at least one of the limits is defined.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-12-11 11:01:10 +01:00
f41gh7
78ad858ff7
app/{vminsert,vmagent}: drop time series on exceeding labels limits.
Previously, time series with labels exceeding the configured limits were truncated and written to storage, potentially causing data inconsistency. This could lead to collisions between time series and make it difficult to identify the source due to truncated labels.

This commit changes the behavior:
*  Such time series are now rejected outright.
* Rejected time series are logged to stdout, and corresponding counters are incremented.
* removes `vm_too_long_label_values_total`, `vm_too_long_label_names_total`, `vm_metrics_with_dropped_labels_total` metrics.
* adds new values `[too_many_labels,too_long_label_name,too_long_label_value]`  to `reason` label of the `vm_rows_ignored_total` metric name

related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6928
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7661
2024-12-10 22:15:38 +01:00
Zhu Jiekun
8aacee27ac
discovery/docker: add match_first_network support for docker_sd_configs
This commit aligns behaviour of docker service discovery with Prometheus implementation.

It adds the following changes:
* introduce new config param `match_first_network` with default value of `true`. It uses the first network if the container has multiple networks
defined.  It should help to avoid collecting duplicate targets error with multi network setups.

* add `networks` for the containers with linked network to the other containers with `network_mode: container:id` setting. It resolve an issue with attached containers aka `pods` in Kubernetes.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7398
2024-12-10 21:43:40 +01:00
Aliaksandr Valialkin
09710d368f
lib/logstorage: add count_uniq_hash function to stats pipe
This function calculates the number of unique value hashes. This number is a good approximation
for the number of unique values. The `count_uniq_hash` function uses less memory and works faster
than `count_uniq` when applied to fields with big number of unique values.
2024-12-09 15:18:53 +01:00
Aliaksandr Valialkin
f469c7bc8a
lib/logstorage: fix possible panic in stream_context pipe
The panic may occur when the surrounding logs for some original log entry are empty.
This is possible when these logs were included into surrounding logs for the previous original log entry.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7762

(cherry picked from commit 843fae3419)
2024-12-09 12:23:37 +01:00
Aliaksandr Valialkin
c74d5becd4
lib/logstorage: add an ability to detect common patterns at collapse_nums pipe
The following patterns are detected:

- `<N>-<N>-<N>-<N>-<N>` is replaced with `<UUID>`.
- `<N>.<N>.<N>.<N>` is replaced with `<IP4>`.
- `<N>:<N>:<N>` is replaced with `<TIME>`. Optional fractional seconds after the time are treated as a part of `<TIME>`.
- `<N>-<N>-<N>` and `<N>/<N>/<N>` is replaced with `<DATE>`.
- `<N>-<N>-<N>T<N>:<N>:<N>` and `<N>-<N>-<N> <N>:<N>:<N>` is replaced with `<DATETIME>`. Optional timezone after the datetime is treated as a part of `<DATETIME>`.

(cherry picked from commit db961f8609)
2024-12-09 12:23:37 +01:00
Aliaksandr Valialkin
0a6f493089
lib/promutils: properly parse timestamps in microseconds and nanoseconds
This is needed for _time filter in VictoriaLogs, which supports timestamps with nanosecond precision

(cherry picked from commit c45451bf69)
2024-12-09 12:23:36 +01:00
Aliaksandr Valialkin
977c0d557c
lib/logstorage: fix assignment to entry in nil map panic at facets pipe
The panic has been introduced in the commit b4f3861690

(cherry picked from commit 48f395456e)
2024-12-09 12:23:35 +01:00
Aliaksandr Valialkin
5c17813522
lib/logstorage: add collapse_nums pipe, which replaces decimal and hexadecimal nums in the given log field with <N>
This is useful for detecting patterns across log messages, which differ by various numeric fields,
with the following query:

_time:1h | collapse_nums | top 10 by (_msg)

(cherry picked from commit 65d831a0ee)
2024-12-09 12:23:33 +01:00
Aliaksandr Valialkin
3e37e6c08e
app/vlselect: allow passing max_value_len query arg to /select/logsql/facets API
The max_value_len query arg allows controlling the maximum length of values
per every log field. If the length is exceeded, then the log field is dropped
from the results, since it contains incomplete (misleading) set of most frequently seen field values.

(cherry picked from commit 48540ac409)
2024-12-09 12:23:33 +01:00
Aliaksandr Valialkin
cf70b1daa6
lib/logstorage: facets pipe: return back ignoring empty values
It is impossible to count all the empty value per every seen field,
since they aren't counted for data blocks, which do not contain the given field.
So it is better ignoring empty values in order to reduce the level of confusion
when users see incorrect hits for empty per-field values.

(cherry picked from commit 3cef820cba)
2024-12-09 12:23:32 +01:00
Aliaksandr Valialkin
3da4ff7199
lib/logstorage: facets pipe: ignore fields, which contain at least a single value with too big length
It is very confusing to see incomplete set of values for fields, which contain a subset of short values,
while the rest of values are too long. It is better to ignore all the values in such fields.

It is also very confusing if the list of most frequently values has no an empty value.
So it is better counting hits for an empty value.

(cherry picked from commit b4f3861690)
2024-12-09 12:23:32 +01:00
Aliaksandr Valialkin
effdc8ba87
lib/logstorage: stream_context pipe: reduce the amounts of surrounding logs to check
Do not check surrounding logs before the selected log if `after N` in set,
and do not check logs after the selected log if `before N` is set

This is a follow-up for 08af80ebe0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7637

(cherry picked from commit 4c8691450a)
2024-12-09 12:23:31 +01:00
Aliaksandr Valialkin
45cc1ff72d
lib/logstorage: add an ability to change the time window for searching for surrounding logs in the stream_context pipe
Thanks to @worker24h for the idea at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7637#issuecomment-2523313740

(cherry picked from commit 08af80ebe0)
2024-12-09 12:23:31 +01:00
Aliaksandr Valialkin
33d2e316e5
lib/logstorage: optimize stream_context pipe over log streams with tens of millions of logs
`stream_context` is implemented in the way, which needs scanning all the logs for the selected log streams.
The scan performance is usually fast, since the majority of blocks are skipped, since they do not contain
rows with the needed timestamps. But there was a pathological case with `stream_context before N`:

VictoriaLogs usually scans blocks in chronological order. That means that the `before` context logs are constantly
updated with the new logs. This requires reading the actual data for the requested log fields from disk.
The workaround is to split the process of obtaining stream context logs into two phases:

1. Select only timestamps for the stream context logs, whithout selecting other log fields.
   This operation is usually much faster than reading the requested log fields.

2. Select stream context logs for the selected timestamps. This operation is usually fast,
   since the requested number of context logs is usually not so big.

Performance testing for the new algorithm shows up to 30x speed improvement for `stream_context before N`
and up to 5x speed improvement for `stream_context after N` when applied to log stream with 50M logs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7637

(cherry picked from commit bddb0e369f)
2024-12-09 12:23:29 +01:00
Aliaksandr Valialkin
80d4c7b50a
app/vlselect: add /select/logsql/facets endpoint
This endpoint returns the most frequent values per each field seen in the selected logs.
This endpoint is going to be used by VictoriaLogs web UI for faceted search.

(cherry picked from commit 740548ccfc)
2024-12-09 12:23:27 +01:00
Aliaksandr Valialkin
e71a8e3a6c
lib/logstorage: add facets pipe for returning the most frequent values across all the log fields seen in the selected logs
(cherry picked from commit dbec34bafc)
2024-12-09 12:23:27 +01:00
Aliaksandr Valialkin
81b9773234
lib/fs: suggest increasing the limit on the number of open files in the error message when the file cannot be opened by ReaderAt
This should simplify troubleshooting of too low limit on the number of open files

(cherry picked from commit 04796ba249)
2024-12-09 12:23:27 +01:00
Aliaksandr Valialkin
a701b70e15
lib/fs: suggest possible solutions inside cannot allocate memory errors during failed mmap attempt
This should improve troubleshooting of the such errors

(cherry picked from commit 5c7b044685)
2024-12-09 12:23:26 +01:00
Aliaksandr Valialkin
ac52894c9e
lib/logstorage: properly format math pipe expressions, which contain multiple binary operators with the same priority
Previously such expressions were improperly formatted, which could result
in incorrect calculations at vlogscli.

For example, 'x / (y / z)' was formatted as 'x / y / z',
while 'x - (y + z)' was formatted as 'x - y + z'.

(cherry picked from commit 80c5066ef3)
2024-12-09 12:23:26 +01:00
Aliaksandr Valialkin
659782ff75
lib/logstorage: add rate and rate_sum stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7415
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7646

(cherry picked from commit c3b8da81cd)
2024-12-09 12:23:26 +01:00
Aliaksandr Valialkin
13a25e7cd3
lib/logstorage: add tests, which verify that offset and limit pipes cannot be used in /select/logsql/stats_query_range
`offset` and `limit` pipes cannot be applied individually per every step on the [start ... end] time range,
so they must be disallowed at /select/logsql/stats_query_range.

This is a follow-up for 534371031e
2024-12-05 15:16:54 +01:00
Aliaksandr Valialkin
cef135f2e8
lib/logstorage: add first and last pipes
The `first N by (field)` pipe is a shorthand to `sort by (field) limit N`,
while the `last N by (field)` pipe is a shorthand to `sort by (field) desc limit N`.

While at it, add support for partitioning sort results by log groups and applying
individual limit per each group.

For example, the following query returns up to 3 logs per each host with the biggest value
for the `request_duration` field:

_time:5m | last 3 by (request_duration) partition by (host)

This query is equivalent to the following one:

_time:5m | sort by (request_duration) desc limit 3 partition by (host)

Automatically add the 'partition by (_time)` into `sort`, `first` and `last` pipes
used in the query to `/select/logsql/stats_query_range` API.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7699
2024-12-05 15:16:54 +01:00
Aliaksandr Valialkin
7e4b86ed40
lib/logstorage: properly ignore log fields when they are passed via streamFields arg to LogRows.MustAdd()
Previously streamFields were unconditionally added to log stream fields, even if they were listed in the ignoreFields.
Also do not add extraStreamFields to log stream fields if streamFields is non-nil, since this may confuse users.

This is a follow-up for 17b813ba28

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7554
2024-12-05 15:16:54 +01:00
Aliaksandr Valialkin
9a7e7ebb01
lib/logstorage: add support for uppercase/lowercase transformations for log fields in "| format ..." pipe
This is needed for consistent formatting of some log fields in the same case.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7620#issuecomment-2502170924
2024-12-05 15:16:53 +01:00
Aliaksandr Valialkin
bd37715c7b
app/vlinsert: use default set of log stream fields for Loki and OpenTelemetry protocols if _stream_fields query arg is empty
Loki protocol supports a list of log stream labels - see https://grafana.com/docs/loki/latest/get-started/labels/

OpenTelemetry protocol also supports a list of log stream labels, which are named resource attributes there.
See https://opentelemetry.io/docs/concepts/resources/#semantic-attributes-with-sdk-provided-default-value

Simplify logs' ingestion into VictoriaLogs for these protocols by allowing the data ingestion without
the need to specify _stream_fields query arg or VL-Stream-Fields HTTP header. In this case the upstream log stream fields
are used during data ingestion. The set of log stream fields can be overriden via _stream_fields query arg
and via VL-Stream-Fields HTTP header if needed.

Thanks to @AndrewChubatiuk for the initial idea and implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7554
2024-12-05 15:16:53 +01:00
Aliaksandr Valialkin
74a314ef77
lib/logstorage: ignore logs with too many fields instead of trying to store them
The storage isn't designed to work efficiently with logs containing too many log fields.
It is better to emit a warning to the user and ignore such logs instead of trying to store them.
This will allow fixing the issue by the user ASAP, and won't lead to excess resource usage
at VictoriaLogs side, such as RAM, CPU, disk IO and disk space.

While at it, ignore too long logs with the size exceeding the maximum block size during data ingestion.
This should prevent from possible issues when dealing with such long logs if they were stored in the storage.
Emit a warning in this case, so the user could identify and fix the issue ASAP.

This is a follow-up for 22e6385f56

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568
2024-12-05 15:16:53 +01:00
Aliaksandr Valialkin
27a50b9551
lib/logstorage: print column names to the error log message when the number of unique columns in a block exceeds the limit
This should simplify debugging issues related to too big number of columns per block in the future.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568
2024-11-30 18:25:57 +01:00
Aliaksandr Valialkin
191180a1b5
lib/logstorage: follow-up for 22e6385f56
Make variable names and comments more clear. This should simplify code maintenance in the future.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568
2024-11-30 18:06:59 +01:00
Nikolay
e45556fc05
lib/logstorage: fixes panic at Block.MustInitFromRows (#7695)
Previously Block columns wasn't properly limited by maxColumnsPerBlock.
And it was possible a case, when more columns per block added than
expected.
 For example, if ingested log stream has many unuqie fields
and it's sum exceed maxColumnsPerBlock.
 We only enforce fieldsPerBlock limit during row parsing, which limits
isn't enough to mitigate this issue. Also it
would be very expensive to apply maxColumnsPerBlock limit during
ingestion, since it requires to track all possible field tags
combinations.

 This commit adds check for maxColumnsPerBlock limit during
MustInitFromRows function call. And it returns offset of the rows and
timestamps added to the block.
 Function caller must create another block and ingest remaining rows
into it.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-11-30 18:06:58 +01:00
Aliaksandr Valialkin
4892d4d805
lib/logstorage: allow special chars in unquoted _stream tag names and values
This simplifies writing _stream filters. For example,

{foo-bar=abc:de}

can be written instead of

{"foo-bar"="abc:de"}
2024-11-30 17:27:58 +01:00
Zhu Jiekun
44d856922a
lib/promscrape/discovery: properly apply the resource_group filter for Azure service discovery
Previously, this filter did not apply to virtual
machine scale sets, causing all virtual machines to be discovered.

 This commit conditionally adds `resource_group` filter for Azure service discovery on virtual
machine scale sets. 

 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7630.
2024-11-26 19:08:31 +01:00
f41gh7
ed9ab2ea73
refactoring: changed prompb to prompbmarshal everythere where internal series transformations are happening (#7409)
doing similar changes for both vmagent and vminsert (like one in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7399) ends up
with almost same implementations for each of packages instead of having
this shared code in one place. one of the reasons is the same Timeseries
and Labels structure from different prompb and prompbmarshal packages.
My proposal is to use structures from prompb package only to
marshal/unmarshal sent/received data, but for internal transformations
use only structures from prompbmarshal package

Another example, where it already can help to simplify code is streaming
aggregation pipeline for vmsingle (now it first marshals
prompb.Timeseries to storage.MetricRow and then if streaming aggregation
or deduplication is enabled it unmarshals all the series back but to
prompbmarshal.Timeseries)

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-11-26 19:02:06 +01:00
Andrei Baidarov
728ceefca1
vmagent: set up a timeout for tcp connection establishment during k8s discovery
Previously, default dial timeout was used for kubernetes API server connection.

 This commit changes it for custom dialer used by the all VictoriaMetrics components. It has lower connection timeout (30s by default). 


 Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7127

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-11-25 18:08:32 +01:00
Ivan Yurochko
60ac4e0c00
lib/streamaggr: add ignore_first_sample_interval param for streamaggr cfg (#7313)
### Describe Your Changes

As of right now by default aggregated output in streaming aggregation
takes a staleness interval and only starts sending first samples after
the staleness interval passes. We have a use case where we prefer to
start sending data as soon as we have any. This adds the option to
configure when we start sending first samples

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7116

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-11-22 17:38:13 +01:00
Artem Fetishev
3ddaafa729
lib/storage: confirm that changing retention period can cause previous indexDB deletion (#7569)
### Describe Your Changes

Add test cases proving that it is possible to lose indexDB after
changing the retention period. See #7609

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 3383589fd1)
2024-11-21 15:23:48 +01:00
Will Jordan
209a5024ce
lib/tenantmetrics: improves CounterMap performance with large numbers of tenants
Previously, map for storing tenant metrics was re-created to each newly ingested tenant. It has significant performance impact for systems with large number of tenants. 

 This commit addresses this issue by changing algorithm of creating tenant metric records at map. Instead of map re-creation, it uses `sync.Map` primitive. 


Benchmark results:

```
goos: linux
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/tenantmetrics
cpu: AMD Ryzen 9 5900X 12-Core Processor            
                                            │ lib/tenantmetrics/orig.bench │     lib/tenantmetrics/new.bench     │
                                            │            sec/op            │    sec/op     vs base               │
CounterMapGrowth/n=100,nProcs=GOMAXPROCS-24                  1943.2µ ±  5%   248.0µ ± 11%  -87.24% (p=0.001 n=7)
CounterMapGrowth/n=100-24                                    434.63µ ±  5%   98.82µ ± 16%  -77.26% (p=0.001 n=7)
CounterMapGrowth/n=1000-24                                   32.719m ± 20%   1.425m ±  5%  -95.65% (p=0.001 n=7)
CounterMapGrowth/n=10000-24                                 3653.60m ±  5%   18.00m ±  2%  -99.51% (p=0.001 n=7)
geomean                                                       17.83m         890.4µ        -95.00%
```

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7482

---
Co-authored-by: Artem Fetishev <rtm@victoriametrics.com>
2024-11-20 18:36:06 +01:00
Nikolay
59d2b4c7fc
lib/storage: properly check for minMissingTimestamps
After changes at commit 787b9cd. Minimal timestamps for extDB check was performed without context of the index search prefix.
It worked fine for Single node version, but for cluster version a different prefix was used for
metricID search requests. It may lead to incomplete results, if minimal missing timestamp was cached
for the tenant with different ingestion patterns.

 Minimal reproducible case is:
- metrics were ingested for tenants 0 and 1
- at some point in time metrics ingestion for tenant 1 stopped
- index records have the following timestamps layout:
 tenant 0:  1,2,3,4,5,6
 tenant 1:  1,2,3,4
- after indexDB rotation, containsTimeRange lookups may produce
  incorrect results:
 time range request for tenant 1 - 5:6 caches 5 as min timestamp
 request for the same or smaller time range for tenant 0 now returns
empty results.

Second case:
- requests for the tenant without metrics always updates atomic value with incorrect minimal time range for other tenants.

 This commit replaces single atomic with map of search prefix keys. It should have slight performance overhead,
but work consistently for cluster version. minMissingTimestamp is cached by prefix search key, which included tenantID.

 Since it will be only populated at runtime, it doesn't hold unused tenants for queries.

Related issue: 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7417
2024-11-15 16:18:32 +01:00
Aliaksandr Valialkin
75e4a8e64b
lib/logstorage: properly skip filtered out dict values when calculating uniq_values, min, max, row_min and row_max stats functions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
8b287e8da4
lib/logstorage: properly clone field values at values stats function
Previously field values weren't properly cloned, which could lead to garbage output for `values` stats function

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7458
2024-11-14 17:21:28 +01:00
Aliaksandr Valialkin
a7e1575ebe
lib/logstorage: simplify the code for uniq_values stats function a bit
Move the repeated check for an empty value into statsUniqValuesProcessor.updateState() function.
This allow removing duplicate code for this check from statsUniqValuesProcessor.updateState() call sites.
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
5b0b7d509f
lib/logstorage: support for [label1=value1 ... labelN=valueN] syntax inside syslog messages for adding arbitrary labels (fields) to log entries 2024-11-14 17:21:26 +01:00
Aliaksandr Valialkin
a02d26e853
lib/logstorage: properly take into account the end query arg when calculating time range for _time:duration filters
(cherry picked from commit e5537bc64d)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
f82cfa16bf
lib/logstorage: allow specifying _time filter offset without time range
This is useful when builiding graphs on time ranges in the past.

(cherry picked from commit a98fb495c6)
2024-11-08 17:07:57 +01:00
Aliaksandr Valialkin
a4ea3b87d7
lib/logstorage: optimize query imeediately after its parsing
This eliminates possible bugs related to forgotten Query.Optimize() calls.

This also allows removing optimize() function from pipe interface.

While at it, drop filterNoop inside filterAnd.

(cherry picked from commit 66b2987f49)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
52929c060a
app/vlselect/logsql: call Query.Optimize() inside parseCommonArgs(), which is called et every /select/logsql/* endpoint.
This reduces the probability of forgotten call to Query.Optimize().

(cherry picked from commit 0550093802)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
7d078dd591
lib/logstorage: add an ability to add prefix to resulting query field names in join pipe
See https://docs.victoriametrics.com/victorialogs/logsql/#join-pipe

(cherry picked from commit 5a6531b329)
2024-11-08 17:07:56 +01:00
Aliaksandr Valialkin
364a2e3e1f
docs/VictoriaLogs: properly sort log fields with floating-point numbers
(cherry picked from commit 42c9183281)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
7a39f526ec
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
(cherry picked from commit 5ed54ebadf)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
83c9d42263
lib/logstorage: add join pipe for joining multiple query results
(cherry picked from commit f9e23bf8e3)
2024-11-07 13:00:19 +01:00
Zakhar Bessarab
718f8077a8
Revert "lib/mergeset: add sparse indexdb cache (#7269)"
This reverts commit 837d0d136d.
2024-11-04 10:33:22 -03:00
Aliaksandr Valialkin
fced48d540
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445

(cherry picked from commit 4478e48eb6)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
bcbaecd73f
lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000
This will allow storing wide events with up to 2K fields per event into VictoriaLogs.
While at it, remove the misleading comment that columnsHeader is read in full per each matching block.
This is no longer the case after the improvements made at 202eb429a7 .
Now only the needed columnHeader is read for the column mentioned in the query.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6425#issuecomment-2418337124
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762

(cherry picked from commit 9ba6be4179)
2024-11-04 10:23:15 -03:00
Nikolay
b6e7852eee
lib/promscrape: add relabel configs to global section
This commit adds `metric_relabel_configs` and `relabel_configs` fields
into the `global` section of scrape configuration file.

 New fields are used as global relabeling rules for the scrape targets.

 These relabel configs are prepended to the target relabel configs.
This feature is useful to:
* apply global rules to __meta labels from service discovery targets.
* drop noisy labels during scrapping.
* mutate labels without affecting metrics ingested via any of push
protocols. 

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6966

---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 20:08:31 +01:00
Aliaksandr Valialkin
1ea65d662f
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit c5d08d317c)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
b74bcb7886
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit 2e635a42d8)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
0c657a95dc
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365

(cherry picked from commit 7603446850)
2024-10-31 14:11:07 +01:00
Artem Fetishev
d7b3589dbd
tests: Initial version of integration tests (#7253)
### Describe Your Changes

Related issue: #7199 

This is the initial version of the integration tests for cluster. See
`README.md` for details.

Currently cluster only, but it can also be used for vm-single if needed.

The code has been added to the apptest package that resides in the root
directory of the VM codebase. This is done to exclude the integration
tests from regular testing build targets because:

- Most of the test variants do not apply to integration testing (such as
pure or race).
- The integtation tests may also be slow because each test must wait for
2 seconds so vmstorage flushes pending content). It may be okay when
there are a few tests but when there is a 100 of them running tests will
require much more time which will affect the developer wait time and CI
workflows.
- Finally, the integration tests may be flaky especially short term.

An alternative approach would be placing apptest under app package and
exclude apptest from packages under test, but that is not trivial.

The integration tests rely on retrieving some application runtime info
from the application logs, namely the application's host:port. Therefore
some changes to lib/httpserver/httpserver.go were necessary, such as
reporting the effective host:port instead the one from the flag.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-30 15:22:06 +01:00
Aliaksandr Valialkin
8baa5177aa
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.

(cherry picked from commit ed73f8350b)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
bf243df9ce
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.

(cherry picked from commit 102e9d4f4e)
2024-10-30 15:19:51 +01:00
cangqiaoyuzhuo
07cf3189f8
chore: fix function name (#7381)
### Describe Your Changes

 fix function name

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 45896fb477)
2024-10-30 13:13:05 +01:00
Aliaksandr Valialkin
1dd01b8a8f
lib/logstorage: follow-up for af831a6c906158f371f1b6810706fa0a54b78386
Sync the code between top and sort pipes regarding the code related to rank.

(cherry picked from commit 7a623c225f)
2024-10-30 09:52:52 +01:00
Aliaksandr Valialkin
329d9a46ee
lib/logstorage: add an ability to return rank from top pipe results
(cherry picked from commit 3c06d083ea)
2024-10-30 09:52:51 +01:00
Aliaksandr Valialkin
fe5f16b662
lib/logstorage: dynamically adjust the number of (bloom, values) shards in a part depending on the number of non-const columns
This allows reducing the amounts of data, which must be read during queries over logs with big number of fields (aka "wide events").
This, in turn, improves query performance when the data, which needs to be scanned during the query, doesn't fit OS page cache.

(cherry picked from commit 7a62eefa34)
2024-10-30 09:52:51 +01:00
Aliaksandr Valialkin
76b21c8560
lib/logstorage: avoid reading columnsHeader data when field_values pipe is applied directly to log filters
This improves performance of `field_values` pipe when it is applied to large number of data blocks.
This also improves performance of /select/logsql/field_values HTTP API.

(cherry picked from commit 8d968acd0a)
2024-10-30 09:52:50 +01:00
Hui Wang
9616814728
vmalert: integrate with victorialogs (#7255)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.

Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.

Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.

Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68bad22fd2)
2024-10-29 16:32:00 +01:00
Zakhar Bessarab
517bd9392c
lib/storage/partition: prevent panic in case resulting in-memory part is empty after merge (#7329)
It is possible for in-memory part to be empty if ingested samples are
removed by retention filters. In this case, data will not be discarded
due to retention before creating in memory part. After in-memory parts
merge samples will be removed resulting in creating completely empty
part at destination.

 This commit checks for resulting part and skips it, if it's empty.

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-27 20:42:42 +01:00
Zhu Jiekun
3d55605ae5
lib/promscrape: adds support for PuppetDB service discovery
This commit adds support for
[PuppetDB](https://www.puppet.com/docs/puppetdb/8/overview.html) service
discovery to the `vmagent` and `victoria-metrics-single` components.

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5744
2024-10-27 20:42:42 +01:00
Andrii Chubatiuk
f6f4884ba6
lib/promscrape/discovery/kubernetes: support kubernetes native sidecars (#7324)
This commit adds Kubernetes Native Sidecar support. 

It's the special type of init containers, that have restartPolicy == "Always" and continue to run after container initialization. 


related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7287
2024-10-27 20:25:15 +01:00
Zakhar Bessarab
8198e7241d
lib/mergeset: add sparse indexdb cache (#7269)
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182

- add a separate index cache for searches which might read through large
amounts of random entries. Primary use-case for this is retention and
downsampling filters, when applying filters background merge needs to
fetch large amount of random entries which pollutes an index cache.
Using different caches allows to reduce effect on memory usage and cache
efficiency of the main cache while still having high cache hit rate. A
separate cache size is 5% of allowed memory.

- reduce size of indexdb/dataBlocks cache in order to free memory for
new sparse cache. Reduced size by 5% and moved this to a separate cache.

- add a separate metricName search which does not cache metric names -
this is needed in order to allow disabling metric name caching when
applying downsampling/retention filters. Applying filters during
background merge accesses random entries, this fills up cache and does
not provide an actual improvement due to random access nature.

Merge performance and memory usage stats before and after the change:

- before

![image](https://github.com/user-attachments/assets/485fffbb-c225-47ae-b5c5-bc8a7c57b36e)

- after

![image](https://github.com/user-attachments/assets/f4ba3440-7c1c-4ec1-bc54-4d2ab431eef5)

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 837d0d136d)
2024-10-24 12:43:06 -03:00
Artem Fetishev
a722ddbdd9
lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267)
This commit fixes the TestStorageRotateIndexDB flaky test reported at:
#6977. Sample test failure: https://pastebin.com/bTSs8HP1

The test fails because one goroutine adds items to the indexDB table
while another goroutine is closing that table. This may happen if
indexDB rotation happens twice during one Storage.add() operation:
-  Storage.add() takes the current indexDB and adds index recods to it
- First index db rotation makes the current index DB a previous one
(still ok at this point)
- Second index db rotation removes the indexDB that was current two
rotations earlier. It does this by setting the mustDrop flag to true and
decrementing the ref counter. The ref counter reaches zero which cases
the underlying indexdb table to release its resources gracefully.
Graceful release assumes that the table is not written anymore. But
Storage.add() still adds items to it.

The solution is to increment the indexDB ref counters while it is used
inside add().
The unit test has been changed a little so that the test fails reliably.
The idea is to make add() function invocation to last much longer,
therefore the test inserts not just one record at a time but thouthands
of them.

To see the test fail, just replace the idbsLocked() func with:

```go
unc (s *Storage) idbsLocked2() (*indexDB, *indexDB, func()) {
       return s.idbCurr.Load(), s.idbNext.Load(), func() {}
}
```

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>

(cherry picked from commit 6b9f57e5f7)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-24 15:45:54 +02:00
Zhu Jiekun
85f60237e2
vmstorage: auto calculate maxUniqueTimeseries based on resources (#6961)
### Describe Your Changes

Add support for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6930

Calculate `-search.maxUniqueTimeseries` by
`-search.maxConcurrentRequests` and remaining memory if it's **not set**
or **less equal than 0**.

The remaining memory is affected by `-memory.allowedPercent`,
`-memory.allowedBytes` and cgroup memory limit.
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-10-18 13:41:43 +02:00
Andrii Chubatiuk
1d352b92c7
lib/promscrape: fixed reload on max_scrape_size change (#7282)
### Describe Your Changes

fixed reload on max_scrape_size change
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7260

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 965a33c893)
2024-10-18 11:42:47 +02:00
Aliaksandr Valialkin
62e4baf556
lib/logstorage: use simpler in-memory cache instead of workingsetcache for caching recently ingested _stream values and recently queried set of streams
These caches aren't expected to grow big, so it is OK to use the most simplest cache based on sync.Map.
The benefit of this cache compared to workingsetcache is better scalability on systems with many CPU cores,
since it doesn't use mutexes at fast path.
An additional benefit is lower memory usage on average, since the size of in-memory cache equals
working set for the last 3 minutes.

The downside is that there is no upper bound for the cache size, so it may grow big during workload spikes.
But this is very unlikely for typical workloads.

(cherry picked from commit 0f24078146)
2024-10-18 11:42:16 +02:00
Aliaksandr Valialkin
f9d86a913c
lib/logstorage: do not persist streamIDCache, since it may go out of sync with partition directories, which can be changed manually between VictoriaLogs restarts
Partition directories can be manually deleted and copied from another sources such as backups or other VitoriaLogs instances.
In this case the persisted cache becomes out of sync with partitions. This can result in missing index entries
during data ingestion or in incorrect results during querying. So it is better to do not persist caches.
This shouldn't hurt VictoriaLogs performance just after the restart too much, since its caches usually contain
small amounts of data, which can be quickly re-populated from the persisted data.

(cherry picked from commit 8aa144fa74)
2024-10-18 11:42:16 +02:00
Aliaksandr Valialkin
b9fae4378a
lib/logstorage: consistently use "pHits := m[..]" pattern
Consistency improves maintainability of the code a bit.

(cherry picked from commit 1892e357c3)
2024-10-18 11:42:16 +02:00
Aliaksandr Valialkin
92b9b13df1
lib/logstorage: optimize performance for queries, which select all the log fields for logs containing hundreds of log fields (aka "wide events")
Unpack the full columnsHeader block instead of unpacking meta-information per each individual column
when the query, which selects all the columns, is executed. This improves performance when scanning
logs with big number of fields.

(cherry picked from commit 2023f017b1)
2024-10-18 11:42:15 +02:00
Aliaksandr Valialkin
5d541322c6
lib/logstorage: improve performance of top and field_values pipes on systems with many CPU cores
- Parallelize mering of per-CPU results.
- Parallelize writing the results to the next pipe.

(cherry picked from commit 78c6fb0883)
2024-10-18 11:42:15 +02:00
Aliaksandr Valialkin
cd7823a310
lib/logstorage: optimize 'stats by(...)' calculations for by(...) fields with millions of unique values on multi-CPU systems
- Parallelize merging of per-CPU `stats by(...)` result shards.
- Parallelize writing `stats by(...)` results to the next pipe.

(cherry picked from commit c4b2fdff70)
2024-10-18 11:42:15 +02:00
Aliaksandr Valialkin
1000ae437c
lib/logstorage: optimize performance for top pipe when it is applied to a field with millions of unique values
- Use parallel merge of per-CPU shard results. This improves merge performance on multi-CPU systems.
- Use topN heap sort of per-shard results. This improves performance when results contain millions of entries.

(cherry picked from commit 192c07f76a)
2024-10-18 11:42:15 +02:00
hagen1778
d14cf4b679
docs: follow-up after f0d1db81dc
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:19:03 -03:00
Roman Khavronenko
4114301955
lib/flagutil: rename Duration to RetentionDuration (#7284)
The purpose of this change is to reduce confusion between using
`flag.Duration` and `flagutils.Duration`. The reason is that
`flagutils.Duration` was mistakenly used for cases that required `m`
support. See
ab0d31a7b0

The change in name should clearly indicate the purpose of this data
type.

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:18:45 -03:00
Alexander Frolov
4ed01e3ab8
lib/flagutil: rm misleading minutes support from flagutil.Duration docs (#7066)
### Describe Your Changes

`flagutil.Duration` docs state that `m` suffix stands for `minute`, but
in fact this suffix is not supported due to ambiguity with `month`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Alexander Frolov <winningpiece@gmail.com>
2024-10-17 11:11:46 -03:00
Zakhar Bessarab
efb7213f6b
lib/flagutil/dict: properly update default value in case there is no key value set (#7211)
### Describe Your Changes

If a dict flag has only one value without a prefix it is supposed to
replace default value.

Previously, when flag was set to `-flag=2` and the default value in
`NewDictInt` was set to 1 the resulting value for any `flag.Get()` call
would be 1 which is not expected.

This commit updates default value for the flag in case there is only one
entry for flag and the entry is a number without a key.

This affects cluster version and specifically `replicationFactor` flag
usage with vmstorage [node
groups](https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect).
Previously, the following configuration would effectively be ignored:
```
/path/to/vmselect \
 -replicationFactor=2 \
 -storageNode=g1/host1,g1/host2,g1/host3 \
 -storageNode=g2/host4,g2/host5,g2/host6 \
 -storageNode=g3/host7,g3/host8,g3/host9
```

Changes from this PR will force default value for `replicationFactor`
flag to be set to `2` which is expected as the result of this
configuration.


---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-17 11:11:45 -03:00
Aliaksandr Valialkin
54ccf09fdd
lib/logstorage: follow-up for 72941eac36
- Allow dropping metrics if the query result contains at least a single metric.
- Allow copying by(...) fields.
- Disallow overriding by(...) fields via `math` pipe.
- Allow using `format` pipe in stats query. This is useful for constructing some labels from the existing by(...) fields.
- Add more tests.
- Remove the check for time range in the query filter according to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254/files#r1803405826

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254
2024-10-17 11:09:16 -03:00
Hui Wang
21864de527
victorialogs: add more checks for stats query APIs (#7254)
1. Verify if field in [fields
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe)
exists. If not, it generates a metric with illegal float value "" for
prometheus metrics protocol.
2. check if multiple time range filters produce conflicted query time
range, for instance:
```
query: _time: 5m | stats count(), 
start:2024-10-08T10:00:00.806Z, 
end: 2024-10-08T12:00:00.806Z, 
time: 2024-10-10T10:02:59.806Z
```
must give no result due to invalid final time range.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-17 11:09:16 -03:00
Aliaksandr Valialkin
3346576a3a
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".

The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.

Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.

The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.

The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.

Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-17 11:09:16 -03:00
Nikolay
135e3ced8c
lib/storage: properly unmarshal SearchQuery (#7277)
After adding multitenant query feature at v1.104.0, searchQuery wasn't
properly unmarshalled at bottom vmselect in multi-level cluster setup.
It resulted into empty query responses.

This commit adds fallback to Unmarshal method of SearchQuery to fill
TenantTokens. It allows to properly execute search requests
at vmselect side.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7270

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-10-17 11:45:27 +02:00
Andrii Chubatiuk
019171fdfc
lib/protoparser/influx: enable batch processing by default (#7165)
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7090

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit daa7183749)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-15 11:51:48 +02:00
Aliaksandr Valialkin
0881e5fd5c
app/vlselect: do not show empty fields in query results
Empty fields are treated as non-existing fields by VictoriaLogs data model.
So there is no sense in returning empty fields in query results, since they may mislead and confuse users.

(cherry picked from commit bac193e50b)
2024-10-15 11:49:32 +02:00