### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Updated email images with new support email
### Checklist
The following checks are **mandatory**:
- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Cutting new changelod doc reduces the size of the current's year
changelog and improves navigation for users.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
- fix table rendering on writer and scheduler pages
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
added search page required for docs site
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
- url encoding / decoding with <urlencode:field> and <urldecode:field>
- base64 encoding / decoding with <base64encode:field> and <base64decode:field>
- hex encoding / decoding with <hexencode:field> and <hexdecode:field>
- hex encoding for integers with <hexnumencode:field> and <hexnumdecode:field>
Prevsiously integer values were converted to strings before being passed to `updateState()` function at `count_uniq`
and `count_uniq_hash`. Later such values are converted back to integers in order to track them via integer map of unique values.
This commit avoids the int -> string -> int conversion. Instead, it passes integers directly to the integer map of unique values.
This improves performance of `count_uniq` and `count_uniq_hash` functions even further.
This filter can be used when debugging and exploring logs in order to understand better
which value types are used for storing the particular log fields.
The `value_type` filter complements `block_stats` pipe.
- Pass the calculated results to the next pipe in float64 columns.
Previously the results were converted to string columns. This could slow down further calculations.
- Use custom optimized logic for processing numeric columns, which are passed to math pipe.
Previously all the input columns were converted to string and then converted to float64
before math pipe calculations.
- Initialize the newly added columns at blockResult as soon as they are added.
This improves performance when big number of columns are calculated by math pipe.
Previously integer values were tracked in string maps. Now every input value is parsed as integer.
On success the parsed integer is tracked via specialized maps, which hold only integers.
This reduces CPU usage and memory usage in general case.
When vmselect process a rollup function it fetches all the raw samples
on requested `start-end` interval of the query. It then loops through
the raw samples, picks the range of the samples based on provided `step`
interval and invokes a rollup function for each of the picked ranges of
samples.
During this processing, vmselect always populates the `realPrevValue`
field with the closest previous raw sample value before the picked range
of samples. This `realPrevValue` is used by rollup functions like
increase_pure or delta to decide whether the counter change happened or
not. For example, we get the counter value == 1. If we've seen this
counter before and its value was also 1 - then no change happened. If we
didn't see it before, then this counter should have started with value=0
and we need to account for `1-0=1` change. All this is required to deal
with situations when scrapes are missing or `step` is too small.
However, vmselect doesn't check how "old" is the `realPrevValue`. In
other words, it doesn't respect the staleness interval when picking it.
In result, depending on the `start` and `end` params, vmselect can use
`realPrevValue` which is a couple of hours old and is unlikely to be a
temporary scrape fail. In result, some increases can be incorrectly
ingnored by vmselect.
This change makes sure that vmselect doesn't populate `realPrevValue`
with samples that are older than staleness interval.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
-------------------
To reproduce, create a dataset with one metric `foo` which has samples
with value=1 on interval of couple of hours and resolution 15s, and a
gap for an hour in the middle:
<img width="769" alt="image"
src="https://github.com/user-attachments/assets/a39b2740-b741-45f8-ad18-093b7c57c3b3"
/>
Then run `increase(foo[1m])` expression on this time range (disable
cache):
<img width="1472" alt="image"
src="https://github.com/user-attachments/assets/463cece1-f359-4c75-a96c-60092a31cab2"
/>
In result, there will be one increase on the beginning of the series.
And no increase after the gap. Then change the time range so it starts
in the middle of the gap:
<img width="1505" alt="image"
src="https://github.com/user-attachments/assets/f4a460c3-9fd1-4ec7-ab47-15e716ec1019"
/>
Now, there is an increase>0 because the `realPrevValue` wasn't
populated. This is wrong, because it hides the increase of the series.
With the fix, the original increase query on full time range should show
2 increases:
<img width="1492" alt="image"
src="https://github.com/user-attachments/assets/aa9d8a6b-7b22-41f6-9eb9-83b3113a6982"
/>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
This chapter is needed for referring from Github issues when CPU or memory profiles are needed to be collected
in order to investigate issues with high CPU and/or RAM usage at VictoriaLogs.
Since
44b071296d
`evalNumber` function no longer updating MetricName tenancy information.
This leads to mismatch in metric names between the query result and
evaluated number for all tenants other than 0:0.
For example, query `count(up) or 0` will return different results for
tenants 0:0 and 1:1 (assuming up is present for both tenants):
- tenant 0:0 - will only contain result of `count(up)`
- tenant 1:1 - will return both `count(up)` and `0` since metric names
will not be matched
This restores setting of tenancy information for metric name for
single-tenant queries.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7987
---
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Hint allows to choose type of cache to be used for index search:
- in-memory parts are storing recently ingested samples and should use
main cache. This improves ingestion speed and cache hit ration for
queries accessing recently ingested samples.
- merges of file parts is performed in background, using a separate
cache allows avoiding pollution of the main cache with irrelevant
entries.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7182
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
This commit makes configurable interval for checking if final dedup
process for the historical data should be started. It allows to spread
resource utilisation for multiple vmstorage/vmsingle instances in time.
Since final dedup may add additional preasure on disk, backup systems
and make cluster less stable. Storage unconditionally adds 25% jitter to
the provided value, it should simplify configuration management at
Kubernetes ecosystem. Because Kubernetes application pods must have the
same configuration.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7880
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
commitL c7fc0d0d2f enabled skipping alerts
in case there is no labels present for an alert. This made clause which
was adding a comma for the JSON list incorrect as it is not possible to
determine if the next alert will be skipped or not.
This fix renders all alert labels in advance allowing properly format
JSON payload for Alertmanager notification.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7985
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
### Describe Your Changes
There has been a question in our public Slack on whether the length
limit of a log record is going to be changed. See:
https://victoriametrics.slack.com/archives/C05UNTPAEDN/p1736156255119689
This PR documents the max length and explains why it has been chosen.
This FAQ section could serve as an answer to more questions like this.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
Fixes error in `vmauth` when discovering ipv6 addresses.
`vmauth` attempts to [slice till
`:`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/auth_config.go#L397)
in the discovered addresses without accounting for ipv6. This causes it
to fail in ipv6 only environments.
```sh
$ nslookup vmselect.ns.svc.cluster.local
...
Name: vmselect.ns.svc.cluster.local
Address: 2600:dead:beef:dead:beef::8
```
```sh
$ kubectl logs -f vmauth
...
error: dial tcp: lookup 2600: no such host
```
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
This pull request adds support for autocomplete in LogsQL queries. The
new feature provides suggestions for field names, field values, and pipe
names as you type.
---------
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
### Describe Your Changes
added links to badges and made them clickable at
docs.victoriametrics.com
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
The point of the new section is to highlight publicly available
playgrounds for users. All of them were mentioned in other parts of the
documentation, but we didn't have all of them in one place before.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
clarify extra resource is needed when downsampling with filter(s) or
retention filter(s) is applied
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Binary operations like `exprFirst op exprSecond` in VictoriaMetrics are
performed in the following way:
1. Execute exprFirst.
2. Extract **common label filters** from the result of step 1.
3. Apply these common label filters to `exprSecond` and execute it, in
order to retrieve less time series from vmstorage nodes.
In step 2, only labels with less than `100` (hard-coded) value could be
used as **common label filter** (e.g. `{common_lb=~"v1|v2|...|v100"}`.
In our scenarios, a label, take `instance` label as an example, could
has thousands of candidate values. Regarding bring more pressure to
vmstorage node, it's still beneficial if labels with more than 100
values could be used as filter in `exprSecond`, with enough vmstorage
resources. After adjusting the value from `100` to `10000`, our query
round-trip time drops significantly from 5s to 2s.
This pull request change the hard-coded value into a configurable flag.
storageNode sorting should be BUGFIX, since previously vminsert performed sort and this behaviour was changed.
Also this change only affects OSS version
### Describe Your Changes
Parse cache is a pretty simple implementation of cache. It's just a
standard map with mutex.
Map with mutex overall has poor performance, plus when the cache
overflow occurs, the whole cache locks until 1k elements have been
deleted (now it's 10% of 10000 max elements in the cache). To avoid this
bottleneck and improve performance of cache on systems with many CPU
cores but keep it rather simple, we can implement cache with per bucket
locks like it's done in fastcache. The logic and API remain the same. So
now each bucket will have a map with approximately 78 elements (with 128
buckets), and overflow will occur now for each bucket, and only 7
elements need to be deleted.
Because exec_test.go has about 10k lines of code, it's better to move
the cache into a separate file to add tests and benchmarks for it,
because now it does not have them.
```
goos: windows
goarch: amd64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql
cpu: 11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz
Current cache implementation performance on 8 cores:
BenchmarkCachePutNoOverFlow-8 1932 618372 ns/op 253 B/op 0 allocs/op
BenchmarkCacheGetNoOverflow-8 6547 211527 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutGetNoOverflow-8 1873 621718 ns/op 261 B/op 0 allocs/op
BenchmarkCachePutOverflow-8 2262 464328 ns/op 32 B/op 0 allocs/op
BenchmarkCachePutGetOverflow-8 1764 655866 ns/op 38 B/op 0 allocs/op
New cache implementation performance on 8 cores:
BenchmarkCachePutNoOverFlow-8 10408 111412 ns/op 0 B/op 0 allocs/op
BenchmarkCacheGetNoOverflow-8 22407 52809 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutGetNoOverflow-8 6583 168088 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutOverflow-8 9822 117212 ns/op 2 B/op 0 allocs/op
BenchmarkCachePutGetOverflow-8 6481 175952 ns/op 3 B/op 0 allocs/op
Current cache implementation performance on 16 cores:
BenchmarkCachePutNoOverFlow-16 2331 475307 ns/op 218 B/op 0 allocs/op
BenchmarkCacheGetNoOverflow-16 6069 196905 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutGetNoOverflow-16 1870 644236 ns/op 262 B/op 0 allocs/op
BenchmarkCachePutOverflow-16 2296 509279 ns/op 34 B/op 0 allocs/op
BenchmarkCachePutGetOverflow-16 1726 671510 ns/op 45 B/op 0 allocs/op
New cache implementation performance on 16 cores:
BenchmarkCachePutNoOverFlow-16 13549 82413 ns/op 0 B/op 0 allocs/op
BenchmarkCacheGetNoOverflow-16 30274 38997 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutGetNoOverflow-16 8512 126239 ns/op 0 B/op 0 allocs/op
BenchmarkCachePutOverflow-16 13884 88124 ns/op 1 B/op 0 allocs/op
BenchmarkCachePutGetOverflow-16 7903 131299 ns/op 3 B/op 0 allocs/op
```
From the benchmarks above, we can see that the new implementation is ~5
times faster than the old one.
---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
consistently use `vmagent_remotewrite_pending_data_bytes` on vmagent dashboard to represent persistent queue size.
`vmagent_remotewrite_pending_data_bytes =
vm_persistentqueue_bytes_pending + pendingInmemoryBytes`
According to panel description, `vmagent_remotewrite_pending_data_bytes`
is more accurate.
>Persistent queue size shows size of pending samples in bytes which
hasn't been flushed to remote storage yet.
And we already use `vmagent_remotewrite_pending_data_bytes` in other two
panels.
44d2205136/dashboards/vmagent.json (L7132)
- removed absolute paths to run without docker
- set cspell to default entrypoint value
- set cspell config path instead of cspell.json copying and removal
Previously, since labels slice is reused for both `ALERTS` and
`ALERTS_FOR_STATE`, metrics might have incorrect labels and affect the
restore process. Tested the fix under `TestAlertingRule_Exec:
"for-pending=>empty"`.
The bug is introduced in
282f13cf11.
Affected versions: v1.106.1, v1.107...v1.108.x
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7796
At Enterprise version of the vmalert, `group` supports `tenant` field.
`tenant` field value must be added to the `datasource` as a part of the URL path prefix.
But VictoriaLogs can obtain tenant information only from `headers` and defined `tenant` breaks requests to the `VictoriaLogs` datasource.
This commit properly checks `datasourceType` and skips adding path prefix if `datasourceType` is `vlogs`.
---------
Co-authored-by: Nikolay <nik@victoriametrics.com>
previously vmstorage ignored limit values from vmselect component.
This behavior is prohibited starting from v1.105.0, with
85f60237e2.
This breaks the original intent of the -search.maxUniqueTimeseries command-line flag, which has been added at vmselect nodes in the commit b843f0e : to be able to override the default limit at vmstorage on the number of unique time series, at different subsets of vmselect nodes.
The behavior should be the following:
* If -search.maxUniqueTimeseries command-line flag isn't set at both vmselect and vmstorage nodes, then the limit on the number of unique time series must be automatically detected at vmstorage nodes according to
* vmstorage: automatically adjust -search.maxUniqueTimeseries max value . This simplifies configuration of VictoriaMetrics cluster for the typical case.
* If -search.maxUniqueTimeseries command-line flag is explicitly set at vmstorage node, then it must be used as the limit on the number of unique time series, without automatic detection of the limit. Explicitly set limit at vmstorage node cannot be exceeded by the limit from vmselect nodes.
* If the -search.maxUniqueTimeseries command-line flag is explicitly set at vmselect node, then it must override the automatically detected limit at vmstorage node. For example, if vmselect node provides the limit, which exceeds the automatically detected limit at vmstorage node, then the limit from the vmselect node must be applied during query execution at vmstorage node. This will allow properly executing queries from the subset of vmselect nodes for reporting queries described above.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7852
### Describe Your Changes
Previously, vmctl expect that tag must exist for each measurement, but
it's actually not necessary.
f16a58f14c/app/vmctl/influx/influx.go (L183-L186)
This pull request fix it by removing the check. For influx series
`measurement1_value1{}`, it will be represented as:
```go
Series{
Measurement: "measurement1",
Field: "value1",
LabelPairs: []LabelPair{},
EmptyTags: []string{},
}
```
and searched by the following query:
```sql
select "value1" from "measurement1"
```
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7921
Commit 71bb9fc0d0 introduced a regression.
If labels are empty and relabeling is not configured, influx ingestion hanlder
performed an earlier exit due to TryPrepareLabels call.
Due micro-optimisations for this procotol, this check was not valid.
Since it didn't take in account metircName, which added later and skip metrics line.
This commit removes `TryPrepareLabel` function call from this path and inline it instead.
It properly track empty labels path.
Adds initial tests implementation for data ingestion protocols.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7933
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Fixed typo in contributing.md (enterpriZe -> enterpriSe in the label
name)
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Currently if multiple msgFields are present in a log row it's not
obvious which field is selected as a _msg field. With this PR and order
of msgfield values defined either via headers or query arg params
defines a priority of these values
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7761
### Describe Your Changes
- datadog /api/v2/logs api supports message field in json format, which
is not documented and is used by serverless extension. This PR allows
message field to be both string and object type. Also added support of
not documented timestamp field
- added `-datadog.streamFields` and `-datadog.ignoreFields` flags to
configure default stream fields for datadog logs, where there's no
alternative option to pass extra headers and query args
- added ingest `max` and `min` values of data, which are ingested using
`datadogsketches` API, which is also actively used by serverless
extensions
- use default `.` separator instead of `_` for sketches metric names
until metrics are not sanitized
This should prevent from excess usage of CPU, RAM and other resources when too many logs
are passed to 'stream_context' pipe.
It is expected that 'stream_context' pipe results are investigated by humans, who cannot inspect
surrounding logs for millions of initial logs. That's why it is OK to limit the number of logs
and/or log streams, which can be passed to 'stream_context' pipe.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7766
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7903
While at it, reduce memory allocations at Storage.getFieldValuesNoHits and make it more scalable on multi-CPU systems.
This improves performance of in(<query>) filter when the <query> returns big number of values.
Use chunked allocator in order to reduce memory allocations. It allocates objects from slices of up to 64Kb size.
This improves performance for `stats` and `top` pipes by up to 2x when they are applied to big number of `by (...)` groups.
Also parallelize execution of `count_uniq`, `count_uniq_hash` and `uniq_values` stats functions,
so they are executed faster on hosts with many CPU cores when applied to fields with big number
of unique values.
### Describe Your Changes
added deprecated form and available from popups in vmanomaly docs
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
- Adds Headers to FAQ questions in vmalert for Victorialogs
- Adds FAQ for multitenant recording rules described in #7656
### Checklist
The following checks are **mandatory**:
- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Haley Wang <haley@victoriametrics.com>
Historically some of VictoriaMetrics components were optimized for the low rate of memory allocations.
These are: vmagent, single-node VictoriaMetrics and vmstorage. These components benefit from the low
GOGC value, since this allow reducing their memory usage in steady state on typical workloads.
Other VictoriaMetrics components aren't optimized for the reduced rate of memory allocations.
This results in the increased CPU usage spent on garbage collection (GC) in these components,
since it must be triggered at higher rate. See https://tip.golang.org/doc/gc-guide#GOGC for details.
These components do not use too much memory, so it is OK increasing the GOGC for these components
from 30 to 100 - this won't affect the most users.
Keep GOGC to 30 only for vmagent, single-node VictoriaMetrics and vmstorage components.
See 077193d87c and 54b9e1d3cb .
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7902
Numeric fields can be stored as const values in the block of logs. In this case the `sort` pipe
was incorrectly comparing such values as strings instead of numbers. This results in incorrect
sort results. For example, 123 was smaller than 2. Fix this by removing the incorrect case
for comparing const fields.
While at it, replace lessString() with strings.LessNatural() in the sortBlockLess.
This improves sorting performance a bit, since the sortBlockLess function already tried
comparing numeric values, and it doesn't need to spend CPU time on such a comparison again inside lessString() call.
The commit 42c9183281 wasn't correct by replacing strings.LessNatural() with lessString()
inside the sortBlockLess() function.
* simplify wording
* update styles
* remove extra info about go application details. The details are likely
not needed and we didn't have details for rolling-dice app anyway. So
keep it simple for consstency and brevity.
* update navigation for simplicity sake
* fix typos
follow-up after
40b47601d1
Signed-off-by: hagen1778 <roman@victoriametrics.com>
vmauth started to use request.Host after commit
f4776fec1b for`src_hosts` routing rules.
This commit adds http.Request.Host to the debugInfo output in order to
be consistent with routing logic.
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
- added VictoriaLogs to OpenTelemetry guide
- updated deprecated dependencies
- added deltatocumulative processor to example and deltatemporality
selector to one of examples to use for counters by default
- added exponential histograms to example
---
Signed-off-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Regression was introduced at 564e6ea024
after implementing:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6928
ctx.Labels array could be incorrectly updated and changes to it after
relabeling rules can be lost.
E.g. ctx.Labels passed to WriteDataPoint function as slice copy, but
results of relabeling only changed an actual slice at ctx.Labels.
This commit replaces implicit relabeling call with explicit
`TryPrepareLabels` function.
It also reduces code diffs with cluster version and adds integration tests
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7865
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Added `available_from` popup into documentation of vmanomaly
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
While at at, allow passing an array of string values per each JSON entry at extra_filters and extra_stream_filters.
For example, `extra_filters={"foo":["bar","baz"]}` is converted into `foo:in("bar", "baz")` extra filter,
while `extra_stream_fitlers={"foo":["bar","baz"]}` is converted into `{foo=~"bar|baz"}` extra filter.
This should simplify creating faceted search when multiple values per a single log field must be selected.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365#issuecomment-2447964259
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5542
### Describe Your Changes
-Deprecate Overview page in Anomaly Detection docs.
- Adding service description to `README.md`
- Moving Licensing information to Quickstart page
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Such log fields do not give any useful information during logs' exploration.
They just clutter the output of the `facets` pipe. So it is better to drop such fields by default.
If these fields are needed, then `keep_const_fields` option can be added to `facets` pipe.
### Describe Your Changes
fixes#7804 by adding alert for missing uptime metric in vmanomaly
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Fix a typo and simplify the statement
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Update docs to reflect the changes introduced in #7767 to fix#5796
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
- Remove Loki sink, since it brings more troubles when users try using it in Vector.
For example, it encodes all the log fields as a JSON string and puts it into "message" field.
This results in storing the "message" field with the JSON string containing all the log fields
in VictoriaLogs. This is not what expected - every log field must be stored as a separate field
according to https://docs.victoriametrics.com/victorialogs/keyconcepts/
- Remove 'mode: bulk' option from Elasticsearch sink configuration, since this option is set by default to this value,
so there is no need in explicit setting.
- Add 'compression: gzip' to all the config examples, since the compression reduces the used network bandwidth by 4-5 times,
while it doesn't increase CPU usage too much at both Vector and VictoriaLogs sides. So it is better to enable the compression in config examples.
- Mention about HTTP parameters accepted by VictoriaLogs data ingestion APIs in both examples for Elasticsearch and JSON line protocols.
### Describe Your Changes
This PR fixes#5796. See the points 6 and 7 in `Steps to reproduce`:
> Now let's set time to only 5ms past the timestamp of the first point,
since even 199ms worked for the second point. Surprise, the point isn't
returned 💥:
>
> ```curl -s $VMQURL -d 'query=series1' -d 'time=1707123456705' -d
'step=1ms' | grep 10 # nothing!```
>
> But, 4ms works: 🤨🤔
>
> ```curl -s $VMQURL -d 'query=series1' -d 'time=1707123456704' -d
'step=1ms' | grep 10 # found```
This happens so because the actual step becomes 5ms due to jitter being
applied. THe fix is to do not apply jitter if scrape interval was not
detected (the case when vmstorage returns only one result). In this case
the scrape interval is set to `5m+step`.
An integration test has been added to check the steps to reproduce and
then to confirm that fix works. Note that the cluster tests are
currently disabled because the fix is not in cluster branch yet.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Previously cluster with the following vmselect configuration:
./bin/vmselect
-storageNode=gr1/:8211,gr1/:8212
-storageNode=gr2/:8213,gr2/:8214
-search.skipSlowReplicas=true
-globalReplicationFactor=2
Here we have two vmstorage groups and -globalReplicationFactor=2, which effectively means that "every ingested sample is replicated across multiple vmstorage groups". Hence, gr1 and gr2 contain identical data set. And when we set -search.skipSlowReplicas=true it is expected vmselect should return result as soon as at least one storage group returned the full result.
In current state, -search.skipSlowReplicas is ignored on the storage group level. It is only respected within the group (with -replicationFactor flag).
This commit fixes global replication for skipSlowReplicas.
To ensure that the fix works and does not break
anything replication tests have been added. For checking the fix for
skipping slow replicas see `testGroupSkipSlowReplicas()`.
To emulate storage groups, the integration test creates a cluster with
multilevel vminsert. The L1 inserts are group-level inserts, each writes
to its own group of vmstorages. The L2 vminsert is a global vminsert
that writes replicated to the L1 vminserts.
To enable multilevel inserts changes in apptest framework and
`lib/ingestserver/clusternative/server.go` were necessary.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6924
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Previously service labels won't be attached when `role: tasks` is set.
Because the `addServicesLabels` function is shared by `role: tasks` and
`role: services`, and it will return nothing when `vip.Addr` is invalid
or empty.
In Prometheus, even if `vip.Addr` is empty, it attach common service
labels with [a standalone
function](f10c3454e9/discovery/moby/services.go (L129)),
which offers:
- `__meta_dockerswarm_service_id`: the id of the service.
- `__meta_dockerswarm_service_name`: the name of the service.
- `__meta_dockerswarm_service_mode`: the mode of the service.
- `__meta_dockerswarm_service_label_<labelname>`: each label of the
service, with any unsupported characters converted to an underscore.
This PR add a `addServicesLabelsForTask`, to replace the usage of
`addServicesLabels` when `role: tasks` is set. This function offers
common service labels listed above.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7800
Previously after configuration reload call `externalURL` templaing function defined at external templates could be lost. Since it was added only at initial `Load` call and never copied during template reload process.
External templates for vmalert could be defined via `-rule.templates` flag.
This commit properly reload external templates. It's no longer copies mutated templates and instead fully reloads it each time if there is any changes.
Previously, during de-duplication staleness markers could be removed due to incorrect logic at
values equality check.
During the evaluation of read query vmselect deduplicates samples using dedupInterval option. It picks the highest value across all points with the same timestamp next to the border of dedupInterval. The issue is any comparison with NaN via <, > returns false. This means that the position of NaN in srcValues could affect the result.
This commit changes this logic with additional step, that explicitly checks for staleness marker for the following cases:
1. Deduplication on vmselect
2. Deduplication in vmstorage during merges
3. Deduplication in stream aggregation
check performed only for stale markers, because other NaNs are rejected on ingestion
by vmstorage or by stream aggregation.
Checking for stale markers in general slows down dedup speed by 3%:
```
benchstat old.txt new.txt
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/storage
cpu: Apple M4 Pro
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
DeduplicateSamples/minScrapeInterval=1s-14 462.8n ± ∞ ¹ 425.2n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=2s-14 905.6n ± ∞ ¹ 903.3n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=5s-14 710.0n ± ∞ ¹ 698.9n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamples/minScrapeInterval=10s-14 632.7n ± ∞ ¹ 638.5n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=1s-14 439.7n ± ∞ ¹ 409.9n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=2s-14 908.9n ± ∞ ¹ 882.2n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=5s-14 721.2n ± ∞ ¹ 684.7n ± ∞ ¹ ~ (p=1.000 n=1) ²
DeduplicateSamplesDuringMerge/minScrapeInterval=10s-14 659.1n ± ∞ ¹ 630.6n ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 659.5n 636.0n -3.56%
```
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7674
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
rfc5424 doesn't allow structured data to be started from whitespace, but
it can be present in the end of this section
related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7776
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
See https://alpinelinux.org/posts/Alpine-3.21.0-released.html
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously, time series with labels exceeding the configured limits were truncated and written to storage, potentially causing data inconsistency. This could lead to collisions between time series and make it difficult to identify the source due to truncated labels.
This commit changes the behavior:
* Such time series are now rejected outright.
* Rejected time series are logged to stdout, and corresponding counters are incremented.
* removes `vm_too_long_label_values_total`, `vm_too_long_label_names_total`, `vm_metrics_with_dropped_labels_total` metrics.
* adds new values `[too_many_labels,too_long_label_name,too_long_label_value]` to `reason` label of the `vm_rows_ignored_total` metric name
related issues:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6928
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7661
This commit aligns behaviour of docker service discovery with Prometheus implementation.
It adds the following changes:
* introduce new config param `match_first_network` with default value of `true`. It uses the first network if the container has multiple networks
defined. It should help to avoid collecting duplicate targets error with multi network setups.
* add `networks` for the containers with linked network to the other containers with `network_mode: container:id` setting. It resolve an issue with attached containers aka `pods` in Kubernetes.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7398
Previous commit b09272ccac added regression, which could lead to the template
global state overwrites.
The issue related to the mechanism how `vmalert` inherits templates. It has global templates, that could be changed via `rule.templates` flag. And local templates defined per labels/annotations for rules and groups.
During labels/annotations templating state could be changed via `define` syntax.
This commit restores previous behavior with `Clone` call for templates before templating labels/annotations.
Affected releases:
- 1.106.1
- v1.102.7
- v1.97.12
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6894
This commit adds ability to launch vmauth without configuration file.
Which is possible use case for operator based installations.
Operator provides global resource `VMAuth` and allows to create
`VMUser` objects for it. Eventually operator creates configuration for
`VMAuth` based on user defined selectors for `VMUser`.
Since there is no direct relations between
those objects. And any object could be created in on-demand by
Kubernetes users. It's required to be able to start `vmauth` with empty
auth config file.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6467
* mention that `debug` messages require -loggerLevel=INFO
* rm version requirement, as mentioned version is pretty old
and it is liklely everyone is using a newer version
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously, if only `-dedup.minScrapeInterval` was set without
`downsampling.Period, function
getDownsamplingFilters returned empty result for
downsamplingPeriodFilters. Because it didn't take in
account globalDedup variable.
This commit adds fast path for this case and returns a single
downsampling filter with global interval value.
In addition, it adds the following changes:
* Removes global state modification at ParseDownsamplingPeriods
function. Which could lead to data races at vmselect
* simplifies logic of isDedupNeeded function. Since
donwsamplingPeriodsWithout filters is subset of
dowsamplingPeriodByFilters. There is no need for len check
* Improves tests by proper reset global state of downsampling
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7764
This function calculates the number of unique value hashes. This number is a good approximation
for the number of unique values. The `count_uniq_hash` function uses less memory and works faster
than `count_uniq` when applied to fields with big number of unique values.
* vmbackupmanager: increase min sleep time between scheduling cycles from 0 to 1s to avoid spammed logs.
* Update docs/changelog/CHANGELOG.md
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
The panic may occur when the surrounding logs for some original log entry are empty.
This is possible when these logs were included into surrounding logs for the previous original log entry.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7762
The following patterns are detected:
- `<N>-<N>-<N>-<N>-<N>` is replaced with `<UUID>`.
- `<N>.<N>.<N>.<N>` is replaced with `<IP4>`.
- `<N>:<N>:<N>` is replaced with `<TIME>`. Optional fractional seconds after the time are treated as a part of `<TIME>`.
- `<N>-<N>-<N>` and `<N>/<N>/<N>` is replaced with `<DATE>`.
- `<N>-<N>-<N>T<N>:<N>:<N>` and `<N>-<N>-<N> <N>:<N>:<N>` is replaced with `<DATETIME>`. Optional timezone after the datetime is treated as a part of `<DATETIME>`.
This is useful for detecting patterns across log messages, which differ by various numeric fields,
with the following query:
_time:1h | collapse_nums | top 10 by (_msg)
The max_value_len query arg allows controlling the maximum length of values
per every log field. If the length is exceeded, then the log field is dropped
from the results, since it contains incomplete (misleading) set of most frequently seen field values.
### Describe Your Changes
Many users are running k8s-stack in multiple kubernetes clusters and to
configure a proper routing in alertmanager it's required to support
`cluster` label in alerting rules. It's now implemented in helm-chart
hack scripts, but it's tricky part to define if cluster label should be
added or not, when functions has no `by` expression. Updated existing
alerts to provide later an ability to inject cluster label later
Also take into an account `storage.minFreeDiskSpaceBytes` in
`DiskRunsOutOfSpace` alerts
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
`stream_context` is implemented in the way, which needs scanning all the logs for the selected log streams.
The scan performance is usually fast, since the majority of blocks are skipped, since they do not contain
rows with the needed timestamps. But there was a pathological case with `stream_context before N`:
VictoriaLogs usually scans blocks in chronological order. That means that the `before` context logs are constantly
updated with the new logs. This requires reading the actual data for the requested log fields from disk.
The workaround is to split the process of obtaining stream context logs into two phases:
1. Select only timestamps for the stream context logs, whithout selecting other log fields.
This operation is usually much faster than reading the requested log fields.
2. Select stream context logs for the selected timestamps. This operation is usually fast,
since the requested number of context logs is usually not so big.
Performance testing for the new algorithm shows up to 30x speed improvement for `stream_context before N`
and up to 5x speed improvement for `stream_context after N` when applied to log stream with 50M logs.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7637
This endpoint returns the most frequent values per each field seen in the selected logs.
This endpoint is going to be used by VictoriaLogs web UI for faceted search.
Previously such expressions were improperly formatted, which could result
in incorrect calculations at vlogscli.
For example, 'x / (y / z)' was formatted as 'x / y / z',
while 'x - (y + z)' was formatted as 'x - y + z'.
`top` pipe can be confused with the `first` and `last` pipes, so add references to these pipes from the `top` pipe docs.
This should help users locating the needed pipes.
Requests processed by built-in HTTP server has the [origin
form](https://datatracker.ietf.org/doc/html/rfc7230#section-5.3) rather
than the absolute form.
So in[Request.URL](https://pkg.go.dev/net/http#Request), fields other than
Path and RawQuery will be empty.
> // For server requests, the URL is parsed from the URI
> // supplied on the Request-Line as stored in RequestURI. For
> // most requests, fields other than Path and RawQuery will be
> // empty. (See RFC 7230, Section 5.3)
Using `request.Host` field instead to match `src_hosts` fixes issue and allows to route requests properly.
An addition It allows user to route requests with customized `Host` header.
The `first N by (field)` pipe is a shorthand to `sort by (field) limit N`,
while the `last N by (field)` pipe is a shorthand to `sort by (field) desc limit N`.
While at it, add support for partitioning sort results by log groups and applying
individual limit per each group.
For example, the following query returns up to 3 logs per each host with the biggest value
for the `request_duration` field:
_time:5m | last 3 by (request_duration) partition by (host)
This query is equivalent to the following one:
_time:5m | sort by (request_duration) desc limit 3 partition by (host)
Automatically add the 'partition by (_time)` into `sort`, `first` and `last` pipes
used in the query to `/select/logsql/stats_query_range` API.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7699
### Describe Your Changes
docs/vmanomaly: release v1.18.8
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
* explicitly mention it is using HTTP protocol
* consistently use `victoriametrics_url` placeholder across the docs
* mention v2 influx format in docs
* consistently remove/add extra newlines for better formatting
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
- docs/vmanomaly - release v1.18.7
- modified table markdown for proper rendering on vmanomaly component
pages
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
docs/vmanomaly - patch release v1.18.6 docs
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Previously Block columns wasn't properly limited by maxColumnsPerBlock.
And it was possible a case, when more columns per block added than
expected.
For example, if ingested log stream has many unuqie fields
and it's sum exceed maxColumnsPerBlock.
We only enforce fieldsPerBlock limit during row parsing, which limits
isn't enough to mitigate this issue. Also it
would be very expensive to apply maxColumnsPerBlock limit during
ingestion, since it requires to track all possible field tags
combinations.
This commit adds check for maxColumnsPerBlock limit during
MustInitFromRows function call. And it returns offset of the rows and
timestamps added to the block.
Function caller must create another block and ingest remaining rows
into it.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
This metric tracks an approximate amounts of bytes processed when parsing the ingested logs.
The metric is exposed individually per every supported data ingestion protocol. The protocol name
is exposed via "type" label in order to be consistent with vl_rows_ingested_total metric.
Thanks to @tenmozes for the initial idea and implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7682
While at it, remove the unneeded "format" label from vl_rows_ingested_total metric.
The "type" label must be enough for encoding the data ingestion format.
Previously, vmagent produced parsing error for 'multitenant' auth token
value for the cases:
* data ingestion with enableMultitentEndpoints
* data scrapping at promscrape
It's inconsistent to the other VictoriaMetrics components.
Since 'multitenant' is well-known token value for multitenancy via
labels. And vmagent is intended to be compatible with vminsert ingestion
endpoints.
This commit replaces NewToken with NewTokenPossibleMultitenant function
for token parsing. It allows to use multitenant value for it. And it
makes token values consistent for the all components.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7694
Previously ip_filters wasn't properly inited for this part of config.
It resulted to bypass requests for this section.
This commit properly inits `ip_filter`.
Previously, all requests rejected by `ip_filter` were silently aborted.
This commit adds new metrics:
* vmauth_user_ip_denies_total
* vmauth_global_ip_denies_total
* vmauth_unauthorized_user_ip_denies_total
It adds observability to this feature and allow to measure rejected requests.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6883
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Both vmalert and vmalert-tool support multiple `rule_files` and use
directory as a file, so it's ok if some files don't contain any rule
group. But vmalert-tool should warn the user if no rule group is found
in any of the `rule_files`.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7663
Previously, there was no option to replace value of `X-Forwarded-For`
HTTP Header. It was only possible to completely remove it. It's not good
solution, since backend may require this information. But using direct
value of this header is insecure. And requires complex knowledge of
infrastruce at backend side (see spoofing X-Forwarded-For articles).
This commit adds new flag, that replaces content of `X-Forwarded-For`
HTTP Header value with current `RemoteAddress` of client that send
request.
It should be used if `vmauth` is directly attached to the internet.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6883
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
This commit allows vmauth to obtain client IP address from HTTP Headers.
Main scenario for it is vmauth located behind reverse-proxy.
It adds both global and per user configuration settings: -httpRealIPHeader and `real_ip_header` config option.
vmauth try to obtain IP from header if this setting is set. If header is not exists, vmauth fallbacks to `remoteAddress`.
Commit also updates incorrect benchmarks and align test package naming for ip_filters
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6883
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
- Fixes the handling of the `showLegend` flag.
- Fixes the handling of `alias`.
- Adds support for alias templates, allowing dynamic substitutions like
`{{label_name}}`.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7565
### Describe Your Changes
docs/vmanomaly - release 1.18.5 doc updates
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
- **Memory Optimization**: Reduced memory consumption on the "Group" and
"JSON" tabs by approximately 30%.
- **Table Pagination**: Added pagination to the "Table" view with an
option to select the number of rows displayed (from 10 to 1000 items per
page, with a default of 1000). This change significantly reduced memory
usage by approximately 75%.
Related to #7185
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
### Describe Your Changes
Fix wrong path to rules_file in vmalert-tool doc
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Viet Hung Nguyen <hvn@familug.org>
Co-authored-by: Hui Wang <haley@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Previously, this filter did not apply to virtual
machine scale sets, causing all virtual machines to be discovered.
This commit conditionally adds `resource_group` filter for Azure service discovery on virtual
machine scale sets.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7630.
Additional info from the dump can be used to debug rotuing rules.
https://pkg.go.dev/net/http/httputil#DumpRequest
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
LTS versions are out of date, linking to old releases. This updates them
to the latest.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Previously, default dial timeout was used for kubernetes API server connection.
This commit changes it for custom dialer used by the all VictoriaMetrics components. It has lower connection timeout (30s by default).
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7127
---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
* cross-link related doc chapters and guides about otel
* mention different URL format for cluster version of VM
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously multitenant cache was inited before flag.Parse call. It
didn't allow to change cache expiration value and default value was
always used.
This commit adds cache init at the first time cache was called.
Also this commit adds small cache improvements:
* chore for cleanup cache, it now uses common pattern for in-place items
filtering
* fail cache request fast if item is already expired
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
This is a follow-up after 3120dc2
- Consistently use key for rollupCache in multitenant mode cache keys use different authTokens. Previously it could lead to panic in rare cases when cache state was inconsistent.
- Do not share `err` variable across goroutines for `processBlock` function. It could lead to data races.
Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7549
---------
Signed-off-by: Andrei Baidarov <abaidarov@yandex.ru>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
It looks like users get confused with vmui availability in vmselect.
Adding a note about it to cluster docs should improve situation.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7631
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Previously, when the alert got resolved shortly before the vmalert
process shuts down, this could result in false alerts.
This change switches vmalert to use MetricsQL function during alerts state restore, which makes it
incompatible for state restoration with PromQL.
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
* mention stream aggregation
* rm statement that Prometheus can only pull data, which is not true anymore
* mention absence of backfilling limitations
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
As of right now by default aggregated output in streaming aggregation
takes a staleness interval and only starts sending first samples after
the staleness interval passes. We have a use case where we prefer to
start sending data as soon as we have any. This adds the option to
configure when we start sending first samples
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7116
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
1. **Add new `Raw Query` tab**
A new `Raw Query` tab has been added to the
[vmui](https://docs.victoriametrics.com/#vmui) interface for displaying
raw data. The tab uses the `/api/v1/export` API endpoint. Related issue:
[#7024](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7024)
2. **Fix rendering of isolated points on the graph**
Previously, isolated points (not connected to other points on the left
or right) were not visible on the graph. Now, they are rendered
correctly.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Previously, for `^` aka pow function calls, VictoriaMetrics returned `1`
if left arg was Nan. For example, given query=`(hour()==2)^1` returns 1
for NaN produced by hour() == 2 function. It added additional non-exist
datapoints to the timeseries.
This commit port bugfix from `metricql` package and adds test for it.
Now, VictoriaMetrics
correctly returns `NaN` for such cases.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7359
Signed-off-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
fixes#7579 7579
Previously, dedup was added as a downsampling rule with 0s offset to all downsmapling rules with filters. That enforced a metric name lookup even in cases it is not needed.
For example, the following configuration: `-dedup.minScrapeInterval=10s -downsampling.period={__name__=~"node.*"}:1h:1m` would be parsed as: `{__name__=~"node.*"}:1h:1m {}:0s:10s`
This commit changes this logic and treats dedup as a separate case. This allows to perform metric name lookups only in cases when timestamp of current partition can be eligible to use some of downsampling filters. Newer parts will not trigger metric name lookup and will apply deduplication directly.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7440
---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
After changes at commit 787b9cd. Minimal timestamps for extDB check was performed without context of the index search prefix.
It worked fine for Single node version, but for cluster version a different prefix was used for
metricID search requests. It may lead to incomplete results, if minimal missing timestamp was cached
for the tenant with different ingestion patterns.
Minimal reproducible case is:
- metrics were ingested for tenants 0 and 1
- at some point in time metrics ingestion for tenant 1 stopped
- index records have the following timestamps layout:
tenant 0: 1,2,3,4,5,6
tenant 1: 1,2,3,4
- after indexDB rotation, containsTimeRange lookups may produce
incorrect results:
time range request for tenant 1 - 5:6 caches 5 as min timestamp
request for the same or smaller time range for tenant 0 now returns
empty results.
Second case:
- requests for the tenant without metrics always updates atomic value with incorrect minimal time range for other tenants.
This commit replaces single atomic with map of search prefix keys. It should have slight performance overhead,
but work consistently for cluster version. minMissingTimestamp is cached by prefix search key, which included tenantID.
Since it will be only populated at runtime, it doesn't hold unused tenants for queries.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7417
This commit fixes panic for multitenant requests and empty storage node responses for tenants api.
It also optimizes `populateSqTenantTokensIfNeeded` function calls, by making it only once for query request. Previously it was incorrectly called multiple times per each storage node request.
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7549
---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
This commit makes vmauth respect the routing config for unauthorized
requests for requests that despite having Authorization header failed to
authorize successfully.
It covers the following use-cases:
- vmauth is used at load-balanacer and must forward requests as is. There is no any authorization configs.
- vmauth has authorization config, but it must forward requests with invalid credential tokens to some other backend.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7543
---------
Signed-off-by: Andrii <andriibeee@gmail.com>
### Describe Your Changes
`vmanomaly` docs update with patch release 1.18.3
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
### Describe Your Changes
docs update for `vmanomaly` release 1.18.2
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
The following user-level options must be unconditionally inherited by url_map, since this is what most users expect:
- retry_status_codes
- load_balancing_policy
- drop_src_path_prefix_parts
- discover_backend_ips
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7519
### Describe Your Changes
when migrating from cluster to single, the native export API attach
`vm_account_id` and `vm_project_id` labels.
`--vm-native-disable-binary-protocol` is a workaround to remove them,
which is available since v1.93.0 but did not documented in vmctl doc.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4716
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
More key concept tests
- Verify how the time range points are calculated
- Vefify that a range query is equivalent to many instant queries
Fix docs accordingly.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
doc updates for vmanomaly v1.18.1
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
…ides
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
- The --delete option is needed to be passed to rsync during backups,
Since otherwise the backup may contain superfluous files after the second run of rsync,
because these files can be already removed at the source because of background merge
- the --delete option is needed when restoring from backup in order to remove superfluous files
from the destination directory. Otherwise these files may lead to inconsistent data at VictoriaLogs.
By default the delay equals to 1 second.
While at it, document refresh_interval query arg at /select/logsql/tail endpoint.
Thanks to @Fusl for the idea and the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428
Update figures for the existing RELEX Oy case study.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
Previously, vmauth could have pick `buMin` as least loaded backend
without checking its status. In result, vmauth could have respond to the
user with an error even if there were healthy backends. That could
happen if healthy backends already had non-zero amount of concurrent
requests executing at the moment of least-loaded backend choosing logic.
Steps to reproduce:
1. Setup vmauth with two backends: healthy and non-healthy
2. Execute a bunch of concurrent requests against vmauth (i.e. Grafana
dash reload)
3. Observe that some requests will fail with message that all backends
are unavailable
Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3061
---
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
add sorting of logs by groups and within each group by time in desc
order. See #7184 and #7045
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Previously it incorrectly applied xFilesFactor, if it's value equal to 0.
This commit properly handles this case and returns result according to
the graphite documentation:
`xFilesFactor follows the same semantics as in Whisper storage schemas. Setting it to 0 (the default) means that only a single value in the series needs to be non-null for it to be considered non-empty, setting it to 1 means that all values in the series must be non-null. A setting of 0.5 means that at least half the values in the series must be non-null.`
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Evgeniy Negriy <einegriy@avito.ru>
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously, loki request json parsers rejected log line if tuple size != 2.
This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the log message stream.
related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431
---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>
### Describe Your Changes
I don't like this solution, but it works. Other possible solutions
described in an issue
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7068
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
User experience suggests that examples shouldn't have `-rule.defaultRuleType=vlogs` set,
as it may confuse users who run vmalert with their existing rules or only use
rules from examples for testing purposes.
This change is supposed to remove the confusion by removing `-rule.defaultRuleType=vlogs`
from default recommendations and explcitily specifying `type` on group level in examples.
Signed-off-by: hagen1778 <roman@victoriametrics.com>
* fix typos in rules definition. Otherwise, they can't pass validation
* add code types for rendered examples
### Describe Your Changes
Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.
### Checklist
The following checks are **mandatory**:
- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
Signed-off-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301
When querying with condition like `WHERE a=1` (looking for series A),
InfluxDB can return data with the tag `a=1` (series A) and data with the
tag `a=1,b=1` (series B).
However, series B is will be queried later and it's data should not be
combined into series A's data.
This PR filter those series that are not identical to the original query
condition.
For table `example`:
```
// time host region value
// ---- ---- ------ -----
// 2024-10-25T02:12:13.469720983Z serverA us_west 0.64
// 2024-10-25T02:12:21.832755213Z serverA us_west 0.75
// 2024-10-25T02:12:32.351876479Z serverA 0.88
// 2024-10-25T02:12:37.766320484Z serverA 0.95
```
The query for series A (`example_value{host="serverA"}`) and result will
be:
```SQL
SELECT * FROM example WHERE host = "serverA"
```
```json
{
"results": [{
"statement_id": 0,
"series": [{
"name": "cpu",
"columns": ["time", "host", "region", "value"],
"values": [
["2024-10-25T02:12:13.469720983Z", "serverA", "us_west", 0.64],
["2024-10-25T02:12:21.832755213Z", "serverA", "us_west", 0.75],
["2024-10-25T02:12:32.351876479Z", "serverA", null, 0.88],
["2024-10-25T02:12:37.766320484Z", "serverA", null, 0.95]
]
}]
}]
}
```
We need to abandon `values[0]` and `values[1]` because the value of
**unwanted** column `region` is not null.
As for series B (`example_value{host="serverA", region="us_west"}`), no
change needed since the query filter out unwanted rows already.
### Note
This is a draft PR for verifying the fix.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
### Describe Your Changes
This is a follow up for #7435. Images need to be updated too:
- The time is changed from 10 hrs to 08 hrs
- A missing data point is added to the range query image
- Source escalidraw has been updated as well
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
### Describe Your Changes
Add puppetdb sd to changelog of `v1.106.0` version.
### Checklist
The following checks are **mandatory**:
- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
---------
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
