Commit graph

9504 commits

Author SHA1 Message Date
Viet Hung Nguyen
cb85e6aef3
vmalert-tool: set default interval for unittest input_series (#7392)
### Describe Your Changes
Currently it is not optional option, but if user forgot to set, it
defaults to 0, which cause unexpected behavior.

This change sets default = evaluation_interval similar to promtool
behavior.

https://prometheus.io/docs/prometheus/2.55/configuration/unit_testing_rules/#test_group
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Viet Hung Nguyen <hvn@familug.org>

(cherry picked from commit 21d1385ae1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 14:11:16 +01:00
Roman Khavronenko
d71c745d19
deployment/alerts: add RemoteWriteDroppingData to vmalert rules (#7393)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3f0e2ab3b2)
2024-10-31 14:11:10 +01:00
Yury Molodov
a3c942d206
vmui: fix the display of the link to vmalert (#7380)
### Describe Your Changes

Fix the issue mentioned in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7088#issuecomment-2391360368

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit bfb55d5f2f)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
dd6781b179
deployment/docker: update VictoriaLogs from v0.39.0-victorialogs to v0.40.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.40.0-victorialogs

(cherry picked from commit 24b6e117dd)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
4895ec79c5
docs/VictoriaLogs/CHANGELOG.md: cut v0.40.0-victorialogs release
(cherry picked from commit 1a7b55009b)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
2d1bc39fe5
docs/VictoriaLogs/README.md: follow-up for 0f6b9e9490: consistently use canonical url for Grafana dashboards
See also 0a5ffb3bc1

(cherry picked from commit f88e2ae9fb)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
cce047c27a
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886

(cherry picked from commit 0f6b9e9490)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
1ea65d662f
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit c5d08d317c)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
b74bcb7886
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit 2e635a42d8)
2024-10-31 14:11:08 +01:00
Dmytro Kozlov
93b3010cb1
deployment/docker: update datasource versions to the latest releases (#7396)
### Describe Your Changes

Updated the versions of the data sources to the latest releases

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 6a6d08d03d)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
fb7f8dec60
docs/VictoriaLogs/querying/README.md: typo fix after f2cd284cf4a7429848fb4be1522483a9b93a43e7: Alls -> All
(cherry picked from commit f62502a943)
2024-10-31 14:11:07 +01:00
Aliaksandr Valialkin
0c657a95dc
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365

(cherry picked from commit 7603446850)
2024-10-31 14:11:07 +01:00
Aliaksandr Valialkin
0eaec9f376
app/vlinsert: typo fix after 16ee470da6
(cherry picked from commit d2dce13df6)
2024-10-31 14:11:07 +01:00
Artem Fetishev
5fc0c50f5b
dashboards: add Restarts panel (#7394)
Reopening PR #7373 from a branch in VictoriaMetrics repo in order to
enable edits and rebase.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 683f8c2780)
2024-10-30 16:44:51 +01:00
hagen1778
0e6ed4171b
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6494606924)
2024-10-30 16:44:51 +01:00
Artem Fetishev
d7b3589dbd
tests: Initial version of integration tests (#7253)
### Describe Your Changes

Related issue: #7199 

This is the initial version of the integration tests for cluster. See
`README.md` for details.

Currently cluster only, but it can also be used for vm-single if needed.

The code has been added to the apptest package that resides in the root
directory of the VM codebase. This is done to exclude the integration
tests from regular testing build targets because:

- Most of the test variants do not apply to integration testing (such as
pure or race).
- The integtation tests may also be slow because each test must wait for
2 seconds so vmstorage flushes pending content). It may be okay when
there are a few tests but when there is a 100 of them running tests will
require much more time which will affect the developer wait time and CI
workflows.
- Finally, the integration tests may be flaky especially short term.

An alternative approach would be placing apptest under app package and
exclude apptest from packages under test, but that is not trivial.

The integration tests rely on retrieving some application runtime info
from the application logs, namely the application's host:port. Therefore
some changes to lib/httpserver/httpserver.go were necessary, such as
reporting the effective host:port instead the one from the flag.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-30 15:22:06 +01:00
Aliaksandr Valialkin
50741d548e
deployment/docker: update VictoriaLogs from v0.38.0-victorialogs to v0.39.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.39.0-victorialogs

(cherry picked from commit 25bca94d04)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
310005f063
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release
(cherry picked from commit 13822f6d11)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
b4f5e5c1b8
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.

(cherry picked from commit 16ee470da6)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
44468e8d8b
docs/VictoriaLogs/data-ingestion/README.md: formatting fixes after 8b36529b32
- Remove leading whitespace from the first lines in 'HTTP parameters' chapter.
  This whitespace isn't needed for the markdown formatting.

- Add leading whitespace for the second sentence in the list bullet describing AccountID and ProjectID HTTP headers.
  This fixes markdown formatting for this list bullet.

(cherry picked from commit 96466562b6)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
8baa5177aa
app/vlinsert: allow specifying comma-separated list of fields containing log message via _msg_field query arg and VL-Msg-Field HTTP request header
This msy be useful when ingesting logs from different sources, which store the log message in different fields.
For example, `_msg_field=message,event.data,some_field` will get log message from the first non-empty field:
`message`, `event.data` and `some_field`.

(cherry picked from commit ed73f8350b)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
bf243df9ce
lib/logstorage: make sure that the number of output (bloom, values) shards is bigger than zero.
If the number of output (bloom, values) shards is zero, then this may lead to panic
as shown at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7391 .

This panic may happen when parts with only constant fields with distinct values are merged into
output part with non-constant fields, which should be written to (bloom, values) shards.

(cherry picked from commit 102e9d4f4e)
2024-10-30 15:19:51 +01:00
Dan Dascalescu
fd044a4ce0
docs: clarify "single" in Single-server-VictoriaMetrics.md (#7369)
### Describe Your Changes

"Single version" is unclear, since VM is also a single-executable. I
think "single-node" is clearer.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 258ee93fd1)
2024-10-30 13:13:05 +01:00
danish-mehmood
34c4b4ab0c
docs: fix typos and format in case study (#7374)
### Describe Your Changes

- made small typo fix in case studies

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 8dc4e2b5a5)
2024-10-30 13:13:05 +01:00
cangqiaoyuzhuo
07cf3189f8
chore: fix function name (#7381)
### Describe Your Changes

 fix function name

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 45896fb477)
2024-10-30 13:13:05 +01:00
hagen1778
97ff8a8fdf
docs: consistently update twitter.com to x.com
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7383

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cfba770c8e)
2024-10-30 13:13:05 +01:00
Lubov66
5330b473c7
docs: replace twitter.com with x.com (#7383)
Replacing the name and link of the social network

(cherry picked from commit 702ff923fe)
2024-10-30 13:13:04 +01:00
Github Actions
369cc724c7
Automatic update helm docs from VictoriaMetrics/helm-charts@87ea94d (#7385)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit b2cf8685e5)
2024-10-30 13:13:04 +01:00
hagen1778
bdfef9700d
docs: add frequently asked questions to vmalert integration with vlogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2c93353173)
2024-10-30 13:13:04 +01:00
hagen1778
8d64847f5e
docs: mark alerting feature as done in VictoriaLogs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b789a9dc83)
2024-10-30 13:13:04 +01:00
hagen1778
f923bcba5e
docs: update version placeholder to point to the next version
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 77b690ab26)
2024-10-30 13:13:03 +01:00
Artem Navoiev
19327643b4
docs: logs vmalert add identifier
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 3473440d26)
2024-10-30 09:52:54 +01:00
hagen1778
8d42bbcaa7
docs: rm unused vmanomaly assets
These images copies are present in vmanomaly subfolder.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2ac07aa813)
2024-10-30 09:52:54 +01:00
hagen1778
7c1b297772
docs: add step for updating VM version in docs
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a49eb7d4f5)
2024-10-30 09:52:54 +01:00
hagen1778
d7d5023acc
docs: update VM versions to the latest version
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7388

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b86b0dd910)
2024-10-30 09:52:53 +01:00
Aliaksandr Valialkin
d5b2ef3095
docs/VictoriaLogs/CHANGELOG.md: remove unnededed with prefix in front of rank at top pipe example
(cherry picked from commit c963d7d10d)
2024-10-30 09:52:53 +01:00
Aliaksandr Valialkin
16be99ccab
deployment/docker: update VictoriaLogs from v0.37.0-victorialogs to v0.38.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.38.0-victorialogs

(cherry picked from commit c43a6ce0eb)
2024-10-30 09:52:53 +01:00
Aliaksandr Valialkin
4266fdf145
docs/VictoriaLogs/CHANGELOG.md: cut v0.38.0 release
(cherry picked from commit 12223cf5d0)
2024-10-30 09:52:52 +01:00
Aliaksandr Valialkin
cadee3e254
app/vlselect/vmui: run make vmui-logs-update after dd89745a34
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7294
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7279
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7290

(cherry picked from commit 4f057e5669)
2024-10-30 09:52:52 +01:00
Aliaksandr Valialkin
1dd01b8a8f
lib/logstorage: follow-up for af831a6c906158f371f1b6810706fa0a54b78386
Sync the code between top and sort pipes regarding the code related to rank.

(cherry picked from commit 7a623c225f)
2024-10-30 09:52:52 +01:00
Aliaksandr Valialkin
48b707a929
app/vlogscli: print hint on how to see available commands when starting vlogscli
This should improve the first-time experience with vlogscli

(cherry picked from commit 8faee6b446)
2024-10-30 09:52:51 +01:00
Aliaksandr Valialkin
329d9a46ee
lib/logstorage: add an ability to return rank from top pipe results
(cherry picked from commit 3c06d083ea)
2024-10-30 09:52:51 +01:00
Aliaksandr Valialkin
fe5f16b662
lib/logstorage: dynamically adjust the number of (bloom, values) shards in a part depending on the number of non-const columns
This allows reducing the amounts of data, which must be read during queries over logs with big number of fields (aka "wide events").
This, in turn, improves query performance when the data, which needs to be scanned during the query, doesn't fit OS page cache.

(cherry picked from commit 7a62eefa34)
2024-10-30 09:52:51 +01:00
Aliaksandr Valialkin
40be393f5a
docs/VictoriaLogs/README.md: add tuning chapter
(cherry picked from commit 67b4059aa4)
2024-10-30 09:52:50 +01:00
Aliaksandr Valialkin
76b21c8560
lib/logstorage: avoid reading columnsHeader data when field_values pipe is applied directly to log filters
This improves performance of `field_values` pipe when it is applied to large number of data blocks.
This also improves performance of /select/logsql/field_values HTTP API.

(cherry picked from commit 8d968acd0a)
2024-10-30 09:52:50 +01:00
Hui Wang
9616814728
vmalert: integrate with victorialogs (#7255)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706.
See
https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md.

Related fix
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254.

Note: in this pull request, vmalert doesn't support
[backfilling](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/vmalert-support-vlog-ds/docs/VictoriaLogs/vmalert.md#rules-backfilling)
for rules with a customized time filter. It might be added in the
future, see [this
issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7289)
for details.

Feature can be tested with image
`victoriametrics/vmalert:heads-vmalert-support-vlog-ds-0-g420629c-scratch`.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 68bad22fd2)
2024-10-29 16:32:00 +01:00
Fred Navruzov
61657e56e8
docs/vmanomaly - release 1.18.0 (#7378)
### Describe Your Changes

docs/vmanomaly - release 1.18.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 5d73b8b866)
2024-10-29 16:31:59 +01:00
Zakhar Bessarab
517bd9392c
lib/storage/partition: prevent panic in case resulting in-memory part is empty after merge (#7329)
It is possible for in-memory part to be empty if ingested samples are
removed by retention filters. In this case, data will not be discarded
due to retention before creating in memory part. After in-memory parts
merge samples will be removed resulting in creating completely empty
part at destination.

 This commit checks for resulting part and skips it, if it's empty.

---------
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-27 20:42:42 +01:00
Zhu Jiekun
3d55605ae5
lib/promscrape: adds support for PuppetDB service discovery
This commit adds support for
[PuppetDB](https://www.puppet.com/docs/puppetdb/8/overview.html) service
discovery to the `vmagent` and `victoria-metrics-single` components.

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5744
2024-10-27 20:42:42 +01:00
Andrii Chubatiuk
da14c7b26e
app/vlinsert: adds journald ingestion support
This commit allows to ingest logs with journald format. 

https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-remote.service.html

related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4618
2024-10-27 20:42:41 +01:00