Commit graph

9436 commits

Author SHA1 Message Date
Zhu Jiekun
63df2205cf
vmctl: fixed import duplicate data when query result contains multiple series (#7330)
### Describe Your Changes

Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301

When querying with condition like `WHERE a=1` (looking for series A),
InfluxDB can return data with the tag `a=1` (series A) and data with the
tag `a=1,b=1` (series B).

However, series B is will be queried later and it's data should not be
combined into series A's data.

This PR filter those series that are not identical to the original query
condition.

For table `example`:
```
// time                           host    region  value
// ----                           ----    ------  -----
// 2024-10-25T02:12:13.469720983Z serverA us_west 0.64
// 2024-10-25T02:12:21.832755213Z serverA us_west 0.75
// 2024-10-25T02:12:32.351876479Z serverA         0.88
// 2024-10-25T02:12:37.766320484Z serverA         0.95
```

The query for series A (`example_value{host="serverA"}`) and result will
be:
```SQL
SELECT * FROM example WHERE host = "serverA"
```
```json
{
	"results": [{
		"statement_id": 0,
		"series": [{
			"name": "cpu",
			"columns": ["time", "host", "region", "value"],
			"values": [
				["2024-10-25T02:12:13.469720983Z", "serverA", "us_west", 0.64],
				["2024-10-25T02:12:21.832755213Z", "serverA", "us_west", 0.75],
				["2024-10-25T02:12:32.351876479Z", "serverA", null, 0.88],
				["2024-10-25T02:12:37.766320484Z", "serverA", null, 0.95]
			]
		}]
	}]
}
```

We need to abandon `values[0]` and `values[1]` because the value of
**unwanted** column `region` is not null.

As for series B (`example_value{host="serverA", region="us_west"}`), no
change needed since the query filter out unwanted rows already.

### Note
This is a draft PR for verifying the fix.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f16a58f14c)
2024-11-06 13:58:19 +01:00
Artem Navoiev
1f50f29e3f
docs: understand your setup size fix formula
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 115ac7d0d7)
2024-11-06 13:58:19 +01:00
Artem Fetishev
e946ec5121
docs: Fix images used in Query data docs (#7443)
### Describe Your Changes

This is a follow up for #7435. Images need to be updated too:

-   The time is changed from 10 hrs to 08 hrs
-   A missing data point is added to the range query image
-   Source escalidraw has been updated as well

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit d212243a0f)
2024-11-06 13:58:19 +01:00
Github Actions
0d2ece86dd
Automatic update helm docs from VictoriaMetrics/helm-charts@f598096 (#7452)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 021f514842)
2024-11-06 13:58:19 +01:00
Zhu Jiekun
9db7c74745
docs: [puppetdb] add changelog for puppetdb service discovery (#7455)
### Describe Your Changes

Add puppetdb sd to changelog of `v1.106.0` version.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 11d735a91f)
2024-11-06 13:58:18 +01:00
Github Actions
9eaf8d6136
Automatic update operator docs from VictoriaMetrics/operator@f8ca70f (#7444)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit fa7adcaeaf)
2024-11-06 13:58:18 +01:00
Github Actions
859602294c
Automatic update helm docs from VictoriaMetrics/helm-charts@76f3195 (#7447)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: zekker6 <1367798+zekker6@users.noreply.github.com>
(cherry picked from commit 74a9c6f91c)
2024-11-06 13:58:18 +01:00
Zakhar Bessarab
577a7a247d
{docker,docs}: update references to latest release
Update references to latest release - v1.106.0.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit be677065bc)
2024-11-06 13:58:17 +01:00
Smaine Kahlouch
64410bfdb8
docs(articles): add blog ogenki to third party articles (#7414)
### Describe Your Changes

Adding a blog post that introduces VictoriaMetrics to third party
articles

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Smaine Kahlouch <smainklh@gmail.com>
(cherry picked from commit 5b838b03ec)
2024-11-06 13:58:17 +01:00
Zakhar Bessarab
015991a5a6
dashboards: add dashboards with victoria-logs datasource (#7424)
### Describe Your Changes

Sync list of dashboards to be provided with Prometheus and
VictoriaMetrics' datasources.
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit d73e5bdb8b)
2024-11-06 13:58:17 +01:00
Andrii Chubatiuk
23dcec3911
vlinsert: support datadog logs
This commit adds the following changes:

- Added support to push datadog logs with examples of how to ingest data
using Vector and Fluentbit
- Updated VictoriaLogs examples directory structure to have single
container image for victorialogs, agent (fluentbit, vector, etc) but
multiple configurations for different protocols

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632

(cherry picked from commit e0930687f1)
2024-11-06 13:58:16 +01:00
Arie Heinrich
7a819ae5ce
docs: spelling fixes (#7420)
### Describe Your Changes

Christmas is early and you get the first present in the shape of
spelling fixes.
Sorry for the big amount :)

### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2e8f420d84)
2024-11-06 13:58:16 +01:00
Github Actions
d0d6c069a9
Automatic update helm docs from VictoriaMetrics/helm-charts@f8ad8eb (#7438)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 397997b2a8)
2024-11-06 13:58:15 +01:00
Artem Fetishev
9c390fd201
documentation: Fix query docs
This commit changes the following:
-   The datetime has been fixed so it corresponds to the timestamps in example samples. The datetime now also include the UTC time zone and is changed to adhere ISO format.
-   The data points in query range result have been fixed to match the inserted data.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit d311c12dce)
2024-11-06 13:58:15 +01:00
Zakhar Bessarab
36ca011fb4
docs/changelog: sync latest release
- remove reference to sparse cache as it was reverted in 9f9cc24e4c
- add reference to 1.102.6 and 1.97.11 LTS releases

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit a3401c8d64)
2024-11-06 13:58:15 +01:00
Roman Khavronenko
3542417757
docs: mention VM version when DS filters became available
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f52577a1a8)
2024-11-06 13:58:15 +01:00
Zakhar Bessarab
48d3052901
docs/CHANGELOG.md: cut v1.106.0
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 11:14:18 -03:00
Zakhar Bessarab
2bdede8ed5
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 10:59:58 -03:00
Zakhar Bessarab
718f8077a8
Revert "lib/mergeset: add sparse indexdb cache (#7269)"
This reverts commit 837d0d136d.
2024-11-04 10:33:22 -03:00
Aliaksandr Valialkin
cf8a971421
docs/VictoriaLogs/querying: refer to vlogscli from docs for /select/logsql/query and /select/logsql/tail HTTP endpoints
These endpoints are easier to query via vlogscli

(cherry picked from commit 5733e56e40)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
c8213b30af
docs/VictoriaLogs/data-ingestion: mention Grafana Agent and Grafana Alloy additionally to Promtail as log collectors for Grafana Loki
(cherry picked from commit 37a58677fa)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
5e7f0eb976
docs/VictoriaLogs/LogsQL.md: remove misleading information that min and max stats functions work only for numeric values
These functions work for both numeric and string values after ad505a7a9a

(cherry picked from commit c7e242b8ce)
2024-11-04 10:23:18 -03:00
Aliaksandr Valialkin
43998ff82b
docs/VictoriaLogs/LogsQL.md: typo fix in the link to description for sum_len stats function: sum-len-stats -> sum_len-stats
This is a follow-up for 364f084b43

(cherry picked from commit a86df52adf)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
fced48d540
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445

(cherry picked from commit 4478e48eb6)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
21d2c417da
docs/VictoriaLogs/keyConcepts.md: remove duplicate sentence about missing timezone information in the ingested _time field values
(cherry picked from commit b3e2db5647)
2024-11-04 10:23:15 -03:00
Aliaksandr Valialkin
bcbaecd73f
lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000
This will allow storing wide events with up to 2K fields per event into VictoriaLogs.
While at it, remove the misleading comment that columnsHeader is read in full per each matching block.
This is no longer the case after the improvements made at 202eb429a7 .
Now only the needed columnHeader is read for the column mentioned in the query.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6425#issuecomment-2418337124
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762

(cherry picked from commit 9ba6be4179)
2024-11-04 10:23:15 -03:00
Andrii Chubatiuk
a987096a1f
Updated k8s guides (#7411)
### Describe Your Changes

- updated dependencies versions
- removed Helm v2 from docs
- fixed VMAgent configuration

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 5e2cb78cce)
2024-11-04 10:23:15 -03:00
f41gh7
2c51fce7fe
app/vmgateway: fixes rate limit for multitenant requests
Previously vmgateway returned error for the requests with multitenant
tenant.

 This commit allows to rate limit multitenant requests and apply global
rate limit for it.

Currently it supports only queries for rate limiting.

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7201

This commit also addresses gateway start-up crash if datasource.url is not accessible.

Previously vmgateway could crash at start-up with enabled rate limiting if datasource for metrics
was not avaiable for any reason.  It seems, that crash is expected. But in fact it's not. For instance, datasource could be in restart phase.

Replaces crash with log message error. It increased availability of vmgateway component.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-10-31 20:20:52 +01:00
Nikolay
b6e7852eee
lib/promscrape: add relabel configs to global section
This commit adds `metric_relabel_configs` and `relabel_configs` fields
into the `global` section of scrape configuration file.

 New fields are used as global relabeling rules for the scrape targets.

 These relabel configs are prepended to the target relabel configs.
This feature is useful to:
* apply global rules to __meta labels from service discovery targets.
* drop noisy labels during scrapping.
* mutate labels without affecting metrics ingested via any of push
protocols. 

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6966

---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 20:08:31 +01:00
hagen1778
b543c8728c
docs: mention stats object in Prometheus API enhancements
The doc explains fields meaning in `stats` object.
It also clarifies their purpose.
See related ticket https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7170

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 20:07:58 +01:00
Github Actions
5ad662dd0f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3ecaebd (#7395)
(cherry picked from commit 955f3660de)
2024-10-31 14:11:17 +01:00
hagen1778
028e75348c
docs: mention #7392 in changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c5b36138e2)
2024-10-31 14:11:17 +01:00
Viet Hung Nguyen
cb85e6aef3
vmalert-tool: set default interval for unittest input_series (#7392)
### Describe Your Changes
Currently it is not optional option, but if user forgot to set, it
defaults to 0, which cause unexpected behavior.

This change sets default = evaluation_interval similar to promtool
behavior.

https://prometheus.io/docs/prometheus/2.55/configuration/unit_testing_rules/#test_group
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Viet Hung Nguyen <hvn@familug.org>

(cherry picked from commit 21d1385ae1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 14:11:16 +01:00
Roman Khavronenko
d71c745d19
deployment/alerts: add RemoteWriteDroppingData to vmalert rules (#7393)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3f0e2ab3b2)
2024-10-31 14:11:10 +01:00
Yury Molodov
a3c942d206
vmui: fix the display of the link to vmalert (#7380)
### Describe Your Changes

Fix the issue mentioned in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7088#issuecomment-2391360368

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit bfb55d5f2f)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
dd6781b179
deployment/docker: update VictoriaLogs from v0.39.0-victorialogs to v0.40.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.40.0-victorialogs

(cherry picked from commit 24b6e117dd)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
4895ec79c5
docs/VictoriaLogs/CHANGELOG.md: cut v0.40.0-victorialogs release
(cherry picked from commit 1a7b55009b)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
2d1bc39fe5
docs/VictoriaLogs/README.md: follow-up for 0f6b9e9490: consistently use canonical url for Grafana dashboards
See also 0a5ffb3bc1

(cherry picked from commit f88e2ae9fb)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
cce047c27a
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886

(cherry picked from commit 0f6b9e9490)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
1ea65d662f
lib/logstorage: properly reset cached output fields for extract and extract_regexp pipes after the log entry matches if(...) condition
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit c5d08d317c)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
b74bcb7886
lib/logstorage: properly cache replace() and replace_regexp() results for identical adjacent field values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7162

(cherry picked from commit 2e635a42d8)
2024-10-31 14:11:08 +01:00
Dmytro Kozlov
93b3010cb1
deployment/docker: update datasource versions to the latest releases (#7396)
### Describe Your Changes

Updated the versions of the data sources to the latest releases

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 6a6d08d03d)
2024-10-31 14:11:08 +01:00
Aliaksandr Valialkin
fb7f8dec60
docs/VictoriaLogs/querying/README.md: typo fix after f2cd284cf4a7429848fb4be1522483a9b93a43e7: Alls -> All
(cherry picked from commit f62502a943)
2024-10-31 14:11:07 +01:00
Aliaksandr Valialkin
0c657a95dc
app/vlselect: add support for extra_filters and extra_stream_filters query args across all the HTTP querying APIs
These query args are going to be used for quick filtering on field values at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7365

(cherry picked from commit 7603446850)
2024-10-31 14:11:07 +01:00
Aliaksandr Valialkin
0eaec9f376
app/vlinsert: typo fix after 16ee470da6
(cherry picked from commit d2dce13df6)
2024-10-31 14:11:07 +01:00
Artem Fetishev
5fc0c50f5b
dashboards: add Restarts panel (#7394)
Reopening PR #7373 from a branch in VictoriaMetrics repo in order to
enable edits and rebase.

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 683f8c2780)
2024-10-30 16:44:51 +01:00
hagen1778
0e6ed4171b
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6494606924)
2024-10-30 16:44:51 +01:00
Artem Fetishev
d7b3589dbd
tests: Initial version of integration tests (#7253)
### Describe Your Changes

Related issue: #7199 

This is the initial version of the integration tests for cluster. See
`README.md` for details.

Currently cluster only, but it can also be used for vm-single if needed.

The code has been added to the apptest package that resides in the root
directory of the VM codebase. This is done to exclude the integration
tests from regular testing build targets because:

- Most of the test variants do not apply to integration testing (such as
pure or race).
- The integtation tests may also be slow because each test must wait for
2 seconds so vmstorage flushes pending content). It may be okay when
there are a few tests but when there is a 100 of them running tests will
require much more time which will affect the developer wait time and CI
workflows.
- Finally, the integration tests may be flaky especially short term.

An alternative approach would be placing apptest under app package and
exclude apptest from packages under test, but that is not trivial.

The integration tests rely on retrieving some application runtime info
from the application logs, namely the application's host:port. Therefore
some changes to lib/httpserver/httpserver.go were necessary, such as
reporting the effective host:port instead the one from the flag.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-30 15:22:06 +01:00
Aliaksandr Valialkin
50741d548e
deployment/docker: update VictoriaLogs from v0.38.0-victorialogs to v0.39.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.39.0-victorialogs

(cherry picked from commit 25bca94d04)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
310005f063
docs/VictoriaLogs/CHANGELOG.md: cut v0.39.0-victorialogs release
(cherry picked from commit 13822f6d11)
2024-10-30 15:19:52 +01:00