Commit graph

10243 commits

Author SHA1 Message Date
Aliaksandr Valialkin
69f02f83ae
lib/logstorage: use clear() for clearing bitmap bits at resetBits() instead of a loop
The clear() call is easier to read and understand than the loop.
2025-02-22 21:55:55 +01:00
Aliaksandr Valialkin
3c1d738196
lib/logstorage: avoid calling bitmap.reset() at getBitmap()
The bitmap at getBitamp() must be already reset when it was returned to the pool via putBitamp().
Thise saves CPU a bit.
2025-02-22 21:55:55 +01:00
Aliaksandr Valialkin
35e1c35281
lib/logstorage: improve error logging for improperly escaped backslashes inside quoted strings
This should simplify debugging LogsQL queries by users
2025-02-22 21:55:54 +01:00
Aliaksandr Valialkin
dfcfaba374
lib/logstorage: add field1:eq_field(field2) filter, which returns logs with identical values at field1 and field2 2025-02-22 21:55:54 +01:00
Aliaksandr Valialkin
cd5b24b377
lib/logstorage: optimize len, hash and json_array_len pipes for repeated values
Re-use the previous result instead of calculating new result for repated input values
2025-02-22 21:55:54 +01:00
Aliaksandr Valialkin
d33e24ab9b
lib/logstorage: add json_array_len pipe for calculating the length of JSON arrays 2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
cd73c1bafb
lib/logstorage: refactor unroll_tokens into unpack_tokens pipe
unpack_tokens pipe generates a JSON array of unpacked tokens from the source field.
This composes better with other pipes such as unroll pipe.
2025-02-22 21:55:53 +01:00
Aliaksandr Valialkin
d32c697361
lib/logstorage: add unroll_tokens pipe for unrolling individual word tokens from the log field 2025-02-22 21:55:52 +01:00
Fred Navruzov
965e3dc699
docs/vmanomaly: fix 404 in dashboard guide (#8372)
### Describe Your Changes

- Fixed 404 links to anomaly score dashboard `.json` file on Github.
- Fixed broken markdown on QuickStart page.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-22 20:26:58 +02:00
Fred Navruzov
7c27538993
docs: vmanomaly - add anomaly score dashboard and guide (#8284)
### Describe Your Changes

> (currently in DRAFT) This dashboard and the docs may look differently
after recent review and the feedback received.

As requested by our customers, this PR introduces `anomaly score`-based
dashboard for `vmanomaly` to improve the visual experience and ease the
drill down debugging process for respective anomaly detection setups.
Accompanying guide (under `default` preset mode) is provided as well.

- [For additional
benefits](https://docs.victoriametrics.com/victoriametrics-datasource/#motivation),
the dashboard is based on [VictoriaMetrics
datasource](https://docs.victoriametrics.com/victoriametrics-datasource/)
rather than on `Prometheus` datasource
- Tested locally and on our https://play.victoriametrics.com/
`anomaly_score` data (tenant ID 0)
- To check the guide, build the docs locally (`vmdocs`, `make run
local`) and follow to the
http://localhost:1313/anomaly-detection/presets/#default section

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-22 20:09:56 +02:00
f41gh7
93001ac931
make vmui-update 2025-02-21 14:08:00 +01:00
f41gh7
7e33efdb6b
docs/changelog: follow-up after 9ca74d1fff
Move vmalert BUGFIX record into current release.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2025-02-21 13:24:24 +01:00
Zakhar Bessarab
9ca74d1fff
app/vmalert: properly unregister exposed metrics for alerting rules
Previously if rule group parameters were changed, alerting rules related metrics could be deleted due to bug at `utils/metrics` package. 

 This commit introduces `metrics.Set` per rule group. It holds group and alerting rules metrics. It properly unregister alerting rules metrics and addresses issue.
In addition:
- expose group metrics only once group is started - this helps to avoid
exposing metrics for groups which are created during YAML unmarshaling
and only used to update existing group.
- properly close rules which are discarded after updating existing rules
so that metrics are also correctly closed.
- detect file renames and properly recreate groups "moved" between files.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8229
2025-02-21 10:36:29 +01:00
Aliaksandr Valialkin
2318e57bbe
deployment/docker: update VictoriaLogs Docker image tag from v1.11.0-victorialogs to v1.12.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.12.0-victorialogs
2025-02-20 22:41:07 +01:00
Aliaksandr Valialkin
1ea3f72d50
lib/logstorage: simplify usage of top, uniq and unroll pipes by allowing comma-separated list of fields without parens
Examples:

   - `top 5 x, y` is equivalent to `top 5 by (x, y)`
   - `uniq foo, bar` is equivalent to `uniq by (foo, bar)`
   - `unroll foo, bar` is equivalent to `unroll (foo, bar)`
2025-02-20 22:36:09 +01:00
Aliaksandr Valialkin
384a757c3d
docs/VictoriaLogs/CHANGELOG.md: cut v1.12.0-victorialogs 2025-02-20 20:49:54 +01:00
Aliaksandr Valialkin
31e88a692d
lib/logstorage: properly handle _time:<=max_time filter
_time:<=max_time filter must include logs with timestamps matching max_time.
For example, _time:<=2025-02-24Z must include logs with timestamps until the end of February 24, 2025.
2025-02-20 19:15:37 +01:00
Aliaksandr Valialkin
ffbd0ebbae
lib/logstorage: allow using '>', '>=', '<' and '<=' in '_time:...' filter
Examples:

  _time:>=2025-02-24Z selects logs with timestamps bigger or equal to 2025-02-24 UTC
  _time:>1d selects logs with timestamps older than one day comparing to the current time

This simplifies writing queries with _time filters.
See https://docs.victoriametrics.com/victorialogs/logsql/#time-filter
2025-02-20 19:04:51 +01:00
Aliaksandr Valialkin
dabe9ff389
docs/VictoriaLogs/keyConcepts.md: small clarifications 2025-02-20 19:04:50 +01:00
Jose Gómez-Sellés
8d389a6628
docs/cloud: fix image routes in getstarted (#8367)
It looks like the version deployed works differently than in local, and
some paths are not taken correctly. This commit fixes that by adding
absolute paths to the 2 images included in this section.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 17:34:56 +01:00
Zakhar Bessarab
cf0604a103
docs/security: update LTS version listed in security docs (#8366)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 17:00:49 +01:00
Jose Gómez-Sellés
eabe3783c4
docs: fix vmanomaly image broken links (#8364)
It looks like when using html image insertion, the path is changed. This
commit fixes that by redirecting to the previous dir in path.

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 16:46:48 +01:00
Jose Gómez-Sellés
de94879515
docs/cloud: rework quickstart guide (#8363)
This guide was outdated. the intention of these changes is to:
* Ease maintainability
* Ease user experience
* Encourage the user to quickly set up the product

Some images are removed because they were obsolete or to improve
readability

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 16:27:40 +01:00
Aliaksandr Valialkin
531ecdfa0d
docs/VictoriaLogs/CHANGELOG.md: move the description of the change at ca1d1bc12b to the appropriate release
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8314
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8315
2025-02-20 16:14:19 +01:00
Andrii Chubatiuk
dc94ccf857
docs: set proper code syntax for snippets in cluster docs (#8362)
### Describe Your Changes

apply proper syntax for code blocks

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-02-20 14:10:50 +01:00
Roman Khavronenko
9a2213eaa8
apptest: add tests for querying utf-8 label names (#8359)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 11:04:01 +01:00
Roman Khavronenko
1d4713f989
metricsql: support auto-format (prettify) for expressions that use quoted metric or label names (#8358)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7703

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 10:25:55 +01:00
Roman Khavronenko
85e2307e7c
apptest/vmsingle: check for right status code in return (#8355)
follow-up for
855dfb324d

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 10:05:57 +01:00
Yury Molodov
086ca47884
vmui: improve Explore Cardinality page (#8351)
### Describe Your Changes

This PR includes improvements to the **Explore Cardinality** page:  

- Improve number formatting for better readability.
- Add full error message display.
- Move values representing changes relative to the previous day to a
separate column for easier sorting.
- Refine UI styles for a better visual experience.

<details>

<summary>Demo UI</summary>

| Before | After |
|--------|-------|
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/a188c94c-3a06-498d-994d-453080c436a7"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/9a42624c-2779-457f-9c98-920fca116eeb"
/> |
| <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/cb4973ba-d7a0-475b-a7a4-f649607dc7b5"
/> | <img width="1269" alt="image"
src="https://github.com/user-attachments/assets/b396516e-2468-4835-b300-88b7a82deec0"
/> |

</details>

Related issue: #8318


### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 09:51:49 +01:00
hagen1778
b5c21337d3
docs: fix typos in recent change lines
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-20 09:14:15 +01:00
f41gh7
855dfb324d
app/vmselect: add query resource limits priority
This commit adds support for overriding vmstorage `maxUniqueTimeseries` with specific
resource limits:
1. `-search.maxLabelsAPISeries` for
[/api/v1/labels](https://docs.victoriametrics.com/url-examples/#apiv1labels),
[/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples/#apiv1labelvalues)
2. `-search. maxSeries` for
[/api/v1/series](https://docs.victoriametrics.com/url-examples/#apiv1series)
3. `-search.maxTSDBStatusSeries` for
[/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats)
4. `-search.maxDeleteSeries` for
[/api/v1/admin/tsdb/delete_series](https://docs.victoriametrics.com/url-examples/#apiv1admintsdbdelete_series)

Currently, this limit priority logic cannot be applied to flags
`-search.maxFederateSeries` and `-search.maxExportSeries`, because they
share the same RPC `search_v7` with the /api/v1/query and
/api/v1/query_range APIs, preventing vmstorage from identifying the
actual API of the request. To address that, we need to add additional
information to the protocol between vmstorage and vmselect, which should
be introduced in the future when possible.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7857
2025-02-19 18:18:32 +01:00
Yury Molodov
a5ca5d370a
vmui: add legend customization options
Legend settings have been added to **Graph Settings**.  

#### **New Features:**  
1. **Table View** – Toggle to display the legend in a table format.  
2. **Hide Common Values** – Option to hide fields with identical values
across all series.
3. **Hide Min/Medium/Max** – Ability to hide min, median, and max values
from the legend.
4. **Custom Label Format** – Set a custom format for series labels
(applies only to the legend).
5. **Group by Label** – Group legend entries based on a selected label.


Related Issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8031
2025-02-19 18:09:14 +01:00
Gunju Kim
10d5e979a6
app/vmalert: add command line flag -notifier.sendTimeout
Currently, vmalert uses a fixed 10-second client timeout for notifiers,
which can prevent large sets of alerts from being sent successfully.

This introduces `-notifier.sendTimeout` flag to vmalert to control the
client timeout duration for the notifiers.


Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8287
2025-02-19 18:04:31 +01:00
Andrii Chubatiuk
58a80a9718
app/vlinsert/syslog: properly parse log line with characters escaped by rfc5424
Inside PARAM-VALUE, the characters '"' (ABNF %d34), '\' (ABNF %d92),
and ']' (ABNF %d93) MUST be escaped.  This is necessary to avoid
parsing errors.  Escaping ']' would not strictly be necessary but is
REQUIRED by this specification to avoid syslog application
implementation errors.  Each of these three characters MUST be
escaped as '\"', '\\', and '\]' respectively.  The backslash is used
for control character escaping for consistency with its use for
escaping in other parts of the syslog message as well as in traditional syslog.

 Related RFC:
https://datatracker.ietf.org/doc/html/rfc5424#section-6.3.3

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8282
2025-02-19 18:01:54 +01:00
Zakhar Bessarab
f4b7e2e811
app/vmselect/filtersdebug: properly validate retention configuration
This commit properly validate retention flag value input for the enterprise debug UI.
API properly checks:
- presence of global retention configuration
- validate that retention is at least 1 day, same as vmstorage enforcement

Related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8343

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 17:52:22 +01:00
Andrii Chubatiuk
e97c530fdc
lib/protoparser/influx: add -influx.forceStreamMode flag to force parsing all Influx data in stream mode (#8319)
Addresses #8269

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 17:05:02 +01:00
Roman Khavronenko
439af7927f
docs: update histogram recommendation for stream aggr (#8349)
Use `rate_sum` instead of `total` output for the following reasons:
* `rate_sum` is far less sensitive for data delays than `total`, since
it represents the speed of change instead of absolute values.
* `rate_sum` remove need in using `rate` function for calculating final
results on query time.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 15:12:55 +01:00
hagen1778
ab1b1cd6eb
deployment/docker: use default network for single-node installation
There is no point of using custom network as all components reside
in the same network anyway.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 14:03:28 +01:00
hagen1778
9049731766
docs: re-order items in VictoriaMetrics navigation tab
* make QuickStart first, similarly to vlogs docs
* push release guide to bottom, as it is not for users

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:59:09 +01:00
Aliaksandr Valialkin
8d79b28b37
deployment: update VictoriaLogs Docker image tag from v1.10.1-victorialogs to v1.11.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.11.0-victorialogs
2025-02-19 13:34:11 +01:00
hagen1778
f927b08298
docs: clarify changelog line
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:27:22 +01:00
hagen1778
1dac6fcdc4
docs: rm duplicated changelog line
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:22:53 +01:00
hagen1778
3afd956041
docs: fix another markdown typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:22:15 +01:00
hagen1778
c77306616e
docs: fix markdown typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:21:42 +01:00
Zakhar Bessarab
eccef5f6ec
docs/guides: remove unneeded IDs for security context (#8344)
### Describe Your Changes

Based on feedback at:
e4240e61c6 (r1961497609)

Thanks to @vrutkovs for the feedback.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2025-02-19 13:20:16 +01:00
Andrii Chubatiuk
c8fc903669
lib/streamaggr: added aggregation windows (#6314)
### Describe Your Changes

By default, stream aggregation and deduplication stores a single state
per each aggregation output result.
The data for each aggregator is flushed independently once per
aggregation interval. But there's no guarantee that
incoming samples with timestamps close to the aggregation interval's end
will get into it. For example, when aggregating
with `interval: 1m` a data sample with timestamp 1739473078 (18:57:59)
can fall into aggregation round `18:58:00` or `18:59:00`.
It depends on network lag, load, clock synchronization, etc. In most
scenarios it doesn't impact aggregation or
deduplication results, which are consistent within margin of error. But
for metrics represented as a collection of series,
like
[histograms](https://docs.victoriametrics.com/keyconcepts/#histogram),
such inaccuracy leads to invalid aggregation results.

For this case, streaming aggregation and deduplication support mode with
aggregation windows for current and previous state. With this mode,
flush doesn't happen immediately but is shifted by a calculated samples
lag that improves correctness for delayed data.

Enabling of this mode has increased resource usage: memory usage is
expected to double as aggregation will store two states
instead of one. However, this significantly improves accuracy of
calculations. Aggregation windows can be enabled via
the following settings:

- `-streamAggr.enableWindows` at [single-node
VictoriaMetrics](https://docs.victoriametrics.com/single-server-victoriametrics/)
and [vmagent](https://docs.victoriametrics.com/vmagent/). At
[vmagent](https://docs.victoriametrics.com/vmagent/)
`-remoteWrite.streamAggr.enableWindows` flag can be specified
individually per each `-remoteWrite.url`.
If one of these flags is set, then all aggregators will be using fixed
windows. In conjunction with `-remoteWrite.streamAggr.dedupInterval` or
`-streamAggr.dedupInterval` fixed aggregation windows are enabled on
deduplicator as well.
- `enable_windows` option in [aggregation
config](https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config).
  It allows enabling aggregation windows for a specific aggregator.



### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:19:33 +01:00
hagen1778
38bded4e58
lib/logstorage: adjust expected compression ratio in tests
A follow-up after 9bb5ba5d2f
that impacted compression ratio for data compressed with native GO zstd lib (`make test-pure`).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-02-19 13:11:37 +01:00
Aliaksandr Valialkin
77aec3fe77
docs/VictoriaLogs/CHANGELOG.md: cut v1.11.0-victorialogs 2025-02-19 12:46:25 +01:00
Aliaksandr Valialkin
eb1bceafb1
app/vlselect/vmui: run make vmui-logs-update after b2a99d7c6f 2025-02-19 12:44:58 +01:00
Yury Molodov
b2a99d7c6f
vmui: optimize group view (#8334)
### Describe Your Changes

This PR introduces several enhancements and optimizations for the Group
view:

1. **Disable hover effect:**  
Add the option to disable the hover effect. This can help reduce CPU
load when viewing a large number of logs.

2. **Limit entries per Group:**  
Add the ability to limit the number of records displayed per group. When
a limit is set, groups are rendered sequentially – the next group starts
only after the current group has finished. By default, there is no
limit.

3. **Display group info:**  
Include the group number in the title along with the total count of
groups to improve clarity and navigation.

4. **Performance improvement:**  
In addition to the features above, separate optimizations reduce CPU
load during hover interactions by approximately 5-10%.
  
Related issue: #8135  


<details>
  <summary>Demo UI</summary>
  
<img
src="https://github.com/user-attachments/assets/9c89066e-28af-4df2-b368-2380412b3c3f"/>
<img
src="https://github.com/user-attachments/assets/a2338c8d-558c-437c-969e-f825043eb35b"/>

</details>
  
### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-02-19 12:42:54 +01:00