Commit graph

10756 commits

Author SHA1 Message Date
Artur Minchukou
a7106040de
app/vmui/logs: add 0 label to the Y axis
Added 0 label to the Y axis on VictoriaLogs chart.


Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8409
2025-04-22 11:26:34 +03:00
Phuong Le
792d0f8bb8
vmagent: reduce log noise from remote write retries
Throttle remote write retry warning logs to one message per 5 seconds.
This reduces log noise during connectivity issues.

Users can monitor `vmagent_remotewrite_retries_count_total` and
`vmagent_remotewrite_errors_total` metrics for detailed retry rates per
destination if needed. This change prioritizes reducing log volume over
immediate destination-specific error logging in the retry warnings.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8723
2025-04-22 11:20:33 +03:00
Phuong Le
8628f90e6b
lib/storage: put the unused in-memory part back into the pool 2025-04-22 11:19:50 +03:00
Georgy Torquemada
89237a04b9
fix: add missed severity levels (warn) for protobuff parser
Closes
[8647](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8647)

### Describe Your Changes

Added missed OTEL severities levels, added test for severity, fix some
severity in given tests

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-21 16:39:10 +04:00
Aliaksandr Valialkin
d6c156727b
docs: remove misleading requirements for the minimum supported Go version needed for building VictoriaMetrics executables
VictoriaMetrics executables need the latest available stable release of Go (1.24 currently),
since they use its features. See, for example, GOEXPERIMENT=synctest from the commit 06c26315df .

There is no need in specifying the minimum supported Go version for building VictoriaMetrics products,
since Go automatically downloads and uses the version specified at `go ...` directive inside go.mod
starting from Go1.21. See https://go.dev/doc/toolchain for details.

So the actual minimum needed Go version is Go1.21, which has been released 1.5 years ago. It should automatically
install newer Go versions specified at `go ...` directive inside go.mod.
2025-04-21 13:06:09 +02:00
Aliaksandr Valialkin
f1fdc8610d
lib/promscrape: prevent from excess memory allocation during scrapes when sample_limit is exceeded
Do not reset wc.labels in order to properly keep track of the number of used labels for the scrape,
and properly re-use the same number of wc.labels on subsequent scrapes.

See 12f26668a6 (r155481168)
2025-04-21 13:06:08 +02:00
Artem Navoiev
611450a188
docs: kuma sd actualize link
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-19 20:37:44 +02:00
Artem Navoiev
3b1a41c014
docs: victorialogs fix link to external collectiors
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-19 10:53:17 +02:00
hagen1778
cd9bb6b315
docs: rm links to swiftstack
Swiftstack s3 service and docs seems unavailable anymore.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-19 07:35:27 +02:00
Artem Navoiev
e58d3f03c7
docs: victoriametrics home page use relative anchor
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 22:22:44 +02:00
Artem Navoiev
3e4c38c56c
docs: changelog fix link to grafana dashboard as far we rename it. Point to 2.53 (LTS) release of prometheus in vmalert as far link to 2.49 doesn't exist anymore
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 22:21:00 +02:00
Artem Navoiev
c3becae96b
docs: actualize link to timescale compression in faq
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 22:12:33 +02:00
Artem Navoiev
84ee713dc6
docs: home victoriametrics fix link to readme.md file
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 22:06:01 +02:00
Artem Navoiev
7dc58894d1
docs: changelog remove deadlink to mesosphere marathon as far project in achive since october 2024 see https://github.com/d2iq-archive/marathon
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 21:55:47 +02:00
Artem Navoiev
30376722c1
docs: fix wrong link in cluster documentation, point to the new era5 page in data examples
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 21:44:11 +02:00
Artem Navoiev
6674691a58
docs: victoriametrics list current urls for redict to simplify the feature changes. More urls
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 21:10:46 +02:00
Artem Navoiev
55e3ccb5f0
docs: victoriametrics list current urls for redict to simplify the feature changes
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2025-04-18 20:44:01 +02:00
hagen1778
2c97c8841c
deployment/docker: fix routing for vlogs vmui
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-18 13:55:25 +02:00
Andrii Chubatiuk
f38736343d
deployment/docker: added victorialogs cluster docker compose setup ()
### Describe Your Changes

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8694

additionally removed container_name, docker network, renamed all
compose, config files for consistency

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-18 13:47:53 +02:00
Aliaksandr Valialkin
33315f1ece
deployment/docker: update VictoriaLogs from v1.18.0-victorialogs to v1.19.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.19.0-victorialogs
2025-04-17 20:25:47 +02:00
Aliaksandr Valialkin
680cbef0cb
docs/victorialogs/CHANGELOG.md: cut v1.19.0-victorialogs release 2025-04-17 20:15:20 +02:00
Aliaksandr Valialkin
2d3e048f59
docs/victorialogs/CHANGELOG.md: mention @arturminchukov as the author of the recent Web UI changes 2025-04-17 20:13:18 +02:00
Aliaksandr Valialkin
024a40a799
app/{vmselect,vlselect} run make vmui-update vmui-logs-update after 5fa40ee631 2025-04-17 20:08:28 +02:00
Aliaksandr Valialkin
225c6b6f52
docs/victorialogs/CHANGELOG.md: clarify the description of the bugfix in the commit 0fee22e91a
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8743
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8707
2025-04-17 20:05:39 +02:00
Andrii Chubatiuk
0fee22e91a
lib/logstorage: expect message in a field with empty and _msg name ()
### Describe Your Changes

fixes 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2025-04-17 19:55:37 +02:00
Max Kotliar
96200a9ec4
vmagent: use tmp dir in integrations tests ()
Before the change, the vmagent integration tests created their directory
and files inside apptest/tests.
After the change, vmagent is instructed to store all files in a real
temporary directory, which is automatically deleted after the tests
complete.
2025-04-17 18:23:42 +02:00
Artem Fetishev
06c26315df
lib/storage: test wasMetricIDsMissingBefore with "testing/synctest" ()
Using this package lets to manipulate time. In this particular case, it
lets to advance the time 61 second forward instantly.

A few side changes were necessary:

- Do not use fasttime in unit tests. The fasttime package starts a
goroutine outside the test bubble which causes the clock to be real, not
fake.
- Stop the time.Ticker explicitly and also stop idbNext. These two
create goroutines with infinite loops which causes the unit tests that
use synctest to hang forever. All goroutines created inside the bubble
must exit in order for the syntest to finish.
- synctest is an experimental package and requires an environment
variable to be set. The Makefile was changed to set it.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2025-04-17 16:58:47 +02:00
Max Kotliar
231810fe49
lib/protoparser/protoparserutil: restore write concurrency limiter in ReadUncompressedData due to performance regressions ()
### Describe Your Changes

The write concurrency limiter in ReadUncompressedData was previously
removed in

22d1b916bf
to avoid suboptimal behavior in certain scenarios. However, follow-up
reports—including issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8674 and
production feedback from VictoriaMetrics Cloud—indicated a noticeable
degradation in performance after its removal.

To mitigate these regressions, this commit reintroduces the concurrency
limiter. A long-term, more optimal solution will be explored separately
in issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8728.

TODO:

* [x] Changelog


### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 14:05:41 +02:00
hagen1778
60ef715c79
docs: fix newline typo in query-stats.md
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 12:25:59 +02:00
hagen1778
8071dabe58
docs: update query-stats.md
* fix typos
* improve wording
* link grafana dashboard

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 12:23:15 +02:00
hagen1778
3d9b160fce
docs: make query-stats.md available in docs
The doc was incorrectly ported into wrong directory after
a113516588
This change moves it to the victoriametrics dir.

While there, updated the order of some pages to couple them by meaning.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 12:15:17 +02:00
Yury Molodov
5fa40ee631
vmui/logs: fix incorrect table sorting for numeric ()
### Describe Your Changes

Fixed table sorting logic and added unit tests for descendingComparator.
Values are now correctly sorted by type: number, date, or string.

Related issue: 

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:40:46 +02:00
Artur Minchukou
53814b1ea6
app/vmui: add query history to VictoriaLogs ()
### Describe Your Changes
Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8500

Added query history to VictoriaLogs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:34:15 +02:00
Artur Minchukou
9ca2a246a9
app/vmui: fix mobile layout on the VictoriaLogs page, fix width of groups and table settings modal ()
### Describe Your Changes
 - Fix mobile layout on the VictoriaLogs page
<img width="361" alt="image"
src="https://github.com/user-attachments/assets/2e09b999-34d5-4059-ba09-95a21b3e8ab3"
/>
<img width="353" alt="image"
src="https://github.com/user-attachments/assets/b9048d41-5972-4290-988f-e9b0a0472b99"
/>
 
- Fix width of groups settings modal
<img width="372" alt="image"
src="https://github.com/user-attachments/assets/e1cb1902-dc93-4bfd-8b32-eaf0d8e54253"
/>
<img width="352" alt="image"
src="https://github.com/user-attachments/assets/a7c7000f-2c4a-41d9-a3c1-a515fd077b9b"
/>

- Fix width of table settings modal
<img width="361" alt="image"
src="https://github.com/user-attachments/assets/12921936-6824-47e9-aff8-d0bb4de2e7f7"
/>
<img width="352" alt="image"
src="https://github.com/user-attachments/assets/77c857ee-85f4-4992-8113-4e252b40f1a6"
/>

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:27:10 +02:00
Artur Minchukou
4517425da8
app/vmui: add export logs button to VictoriaLogs ()
### Describe Your Changes
Related issue:
[](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8604)

Added a download logs button to VictoriaLogs, which allows you to export
logs in the following formats: csv, json.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2025-04-17 09:14:41 +02:00
Yury Molodov
953ed46680
vmui: update package.json
### Describe Your Changes

Bumped project dependencies in `package.json` to the latest stable
versions.
2025-04-16 20:19:17 +02:00
f41gh7
795d3fe722
lib/storage: enhance TSDB status response
This commit adds new fields - `requestsCount` and `lastRequestTimestamp`
to series count be metric names stats.
It allows to display an additional stats at explore cardinality page.
Stats will only be added if `storage.trackMetricNameStats` flag is set.

 This change requires an update to RPC protocol in order to properly
marshal data.

 In addition, this commit adds integration tests to TSDB stats API.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6145
2025-04-16 20:09:35 +02:00
f41gh7
536b40c06c
make linter happy after a113516588 2025-04-16 19:55:54 +02:00
Roman Khavronenko
a113516588
app/vmselect: log search query request stats
This commit adds `search.logSlowQueryStats=<duration>` cmd-line flag on vmselect.
It reads stats from eval function, and doesn't slow down the query execution.

 Log line has the following structure:

 vm_slow_query_stats type=%s query=%q query_hash=%d start_ms=%d end_ms=%d step_ms=%d range_ms=%d tenant=%q execution_duration_ms=%v series_fetched=%d samples_fetched=%d bytes=%d memory_estimated_bytes=%d

 This feature is only available for enterprise version.
2025-04-16 19:39:13 +02:00
Fred Navruzov
b68f9ea9e3
docs/vmanomaly - release 1.22.0 - experimental ()
### Describe Your Changes

Changelog note about experimental v1.22.0 release that solves deadlock
issue on multicore systems by complete parallelization turned off until
proper refactor is made to return it back w/o reintroducing the risk of
the deadlock in child processes.

P.s. other references and guides were not updated to experimental tag,
as long as it has downsides of dropped speed. The links will be updated
once we have parallelization properly turned on.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-14 15:56:10 +03:00
Roman Khavronenko
fa6a32a39d
ci: temporary disable vlogs tests for i386
This change unblocks testing pipelines in CI for other contributions.
The tests are commented because I don't have full understanding of
fixing them.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8710

---------
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2025-04-14 11:11:43 +02:00
Max Kotliar
477635e00f
Follow up to "vmagent/client: Use VictoriaMetrics remote write protocol by default, downgrade to Prometheus if needed" ()
### Describe Your Changes

Follow-up to
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462

Addressed review comments:
- Log panic with FATAL prefix to indicate possible on-disk data
corruption
- Moved version bump line to the tip block (v1.114.0 has already been
released) in changelog
- Removed duplicate vmagent entry from targets list from Makefile


### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2025-04-14 11:55:01 +03:00
Max Kotliar
8e92cd3b2d
lib/writeconcurrencylimiter: add some hints to unexpected EOF error message. ()
### Describe Your Changes

Under heavy load, vmagent's wirte concurrency limiter

(2ab53acce4/lib/writeconcurrencylimiter/concurrencylimiter.go (L111))
queues incoming requests. If a client's timeout is shorter than the wait
time in the
queue, the client may close the connection before vmagent starts
processing it. When vmagent then tries to read the request body, it
encounters an ambiguous `unexpected EOF` error
(https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8675).

This commit adds more context to such errors to help users diagnose and
resolve
the issue when it's related to vmagent's own load and queuing behavior.

Possible user actions include:
- Lowering `-insert.maxQueueDuration` below the client's timeout.
- Increasing the client-side timeout, if applicable.
- Scaling up vmagent (e.g., adding more CPU resources).
- Increasing `-maxConcurrentInserts` if CPU capacity allows.

Steps to reproduce:
https://gist.github.com/makasim/6984e20f57bfd944411f56a7ebe5b6bf

### Checklist

The following checks are **mandatory**:

- [x] My change adheres to [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-13 11:22:12 +03:00
Max Kotliar
2ab53acce4
vmagent/client: Use VictoriaMetrics remote write protocol by default, downgrade to Prometheus if needed ()
### Describe Your Changes

This commit improves how vmagent selects the remote write protocol.
Previously, vmagent [performed a handshake
probe](0ff1a3b154/lib/protoparser/protoparserutil/vmproto_handshake.go (L11))
at
[startup](0ff1a3b154/app/vmagent/remotewrite/client.go (L173)):

- If the probe succeeded, it used the VictoriaMetrics (VM) protocol.

- If the probe failed, it downgraded to the Prometheus protocol.

- No protocol changes occurred after the initial probe at runtime.

However, this approach had limitations:

- If vmstorage was unavailable during vmagent startup, vmagent would
immediately downgrade to the Prometheus protocol, leading to higher
network usage unitl vmagent restarted. This case has been reported in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615.

- If the remote write server was updated or downgraded (e.g., during a
fallback or migration), vmagent would not detect the protocol change. It
would continue retrying failed requests and eventually drop them.
Require a restart of vmagent to pick up the new protocol.

This commit introduces a more adaptive mechanism.
vmagent always starts with the VM protocol and downgrades to the
Prometheus protocol only if an unsupported media type or bad request
response is received.
When this happens, the protocol is downgraded for all future requests.
In-flight requests are re-packed from Zstd to Snappy and retried
immediately.
Snappy-encoded requests are dropped if an unsupported media type or bad
request is received (no retrying).

Additionally, the in-memory and persisted queues could mix snappy and
zstd encoded blocks. The proper encoding is decided before sending by
encoding.IsZstd function.

TODO:
* [x] Add tests
* [x] Update documentation
* [x] Changelog
* [x] Research on
[content-type](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786918054),
[accept-encoding](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8462#issuecomment-2786923382)

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7615#top
issue.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2025-04-12 10:39:47 +03:00
Aliaksandr Valialkin
2f4680d8f3
docs/victoriametrics/Articles.md: add a link to https://chronicles.mad-scientist.club/tales/grepping-logs-remains-terrible/ 2025-04-10 23:10:24 +02:00
Aliaksandr Valialkin
1f1afeb06e
lib/logstorage: add support for <duration_seconds:field> formatting option for format pipe
This option formats duration values as floating-point seconds.
2025-04-10 22:55:08 +02:00
Aliaksandr Valialkin
b2c075191e
docs/victorialogs/cluster.md: add an example on how to query every vmstorage node as a single-node VictoriaLogs 2025-04-10 22:16:31 +02:00
f41gh7
1191c6453b
app/vmagent: properly init kafka consumer
Previously, if vmagent was built with CGO_ENABLED=0, vmagent cannot start and reported runtime error:

 `Kafka client is not supported at systems without CGO`

 This error was trigger even if `-kafka.consumer.topic` was not
 provided. CGO_ENABLED=0 is default build option for linux/arm and some other archs.

 This commit properly inits kafka consumer by checking if `-kafka.consumer.topic` is set.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6019
2025-04-10 18:15:36 +02:00
Aliaksandr Valialkin
5d61122fd5
docs/victorialogs/cluster.md: add a link to the changelog for the latest available release 2025-04-10 17:09:23 +02:00
Aliaksandr Valialkin
e9c04879ce
docs/victorialogs/CHANGELOG.md: add release date for v1.18.0-victorialogs 2025-04-10 17:07:58 +02:00