Commit graph

9552 commits

Author SHA1 Message Date
Aliaksandr Valialkin
4d5f364ad9
docs/VictoriaLogs/LogsQL.md: add missing reference to pipes docs from join pipe docs
This is needed for consistency with other pipe docs

(cherry picked from commit d1af84a49d)
2024-11-07 13:00:22 +01:00
Aliaksandr Valialkin
722873e18b
deployment/docker: update VictoriaLogs from v0.40.0-victorialogs to v0.41.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.41.0-victorialogs

(cherry picked from commit a549240852)
2024-11-07 13:00:21 +01:00
Aliaksandr Valialkin
bf81ef667b
docs/VictoriaLogs/CHANGELOG.md: cut v0.41.0-victorialogs
(cherry picked from commit a4ce80b4ea)
2024-11-07 13:00:21 +01:00
Aliaksandr Valialkin
bcf8a788eb
app/vlselect/vmui: run make vmui-logs-update after 1e1952acf5
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7344
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7184
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7045

(cherry picked from commit bd64c7e3ca)
2024-11-07 13:00:21 +01:00
Yury Molodov
200926c2bc
vmui/logs: add log sorting (#7344)
### Describe Your Changes

add sorting of logs by groups and within each group by time in desc
order. See #7184 and #7045

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
(cherry picked from commit 1e1952acf5)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
1c7d8adc65
app/vlinsert/loki: follow-up for 3aeb1b96a2
- Disallow more than 3 items in Loki line entry, since it must contain two mandatory entries: timestamp and message,
  plus one optional entry - structured metadata. See https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs

- Update references to structured metadata docs in Loki, in order to simplify further maintenance of the code

- Move the change from bugfix to feature at docs/VictoriaLogs/CHANGELOG.md, since VictoriaLogs never supported
  structured metadata over JSON Loki protocol. The support for structured metadata in protobuf Loki protocol
  has been added in ac06569c49 , which has been included in v0.28.0-victorialogs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7432

(cherry picked from commit 3d75c39ff4)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
364a2e3e1f
docs/VictoriaLogs: properly sort log fields with floating-point numbers
(cherry picked from commit 42c9183281)
2024-11-07 13:00:20 +01:00
Aliaksandr Valialkin
7a39f526ec
lib/logstorage: add block_stats pipe for analyzing per-block storage stats
(cherry picked from commit 5ed54ebadf)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
83c9d42263
lib/logstorage: add join pipe for joining multiple query results
(cherry picked from commit f9e23bf8e3)
2024-11-07 13:00:19 +01:00
Aliaksandr Valialkin
ab3b4020b4
app/vlselect: add start_offset query arg for /select/logsql/tail endpoint for returning historical logs before live tailing
(cherry picked from commit 3a5f1019ba)
2024-11-07 13:00:19 +01:00
Evgeniy Negriy
98fe1950a1
app/vmselect: fixes graphite function transformRemoveEmptySeries
Previously it incorrectly applied xFilesFactor, if it's value equal to 0.

 This commit properly handles this case and returns result according to
the graphite documentation:

`xFilesFactor follows the same semantics as in Whisper storage schemas. Setting it to 0 (the default) means that only a single value in the series needs to be non-null for it to be considered non-empty, setting it to 1 means that all values ​​in the series must be non-null. A setting of 0.5 means that at least half the values ​​in the series must be non-null.`

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Evgeniy Negriy <einegriy@avito.ru>
(cherry picked from commit d27dfac5c6)
2024-11-07 13:00:18 +01:00
Zhu Jiekun
e113a40142
app/vlinisert/loki: properly parse json logs with structured metadata
Loki protocol supports optional `metadata` object for each ingested line. It's added as 3rd field at the (ts,msg,metadata) tuple. Previously,  loki request json parsers rejected log line if tuple size != 2.

This commit allows optional tuple field. It parses it as json object and adds it as log metadata fields to the  log message stream.

related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7431

---------
Co-authored-by: f41gh7 <nik@victoriametrics.com>

(cherry picked from commit 3aeb1b96a2)
2024-11-07 13:00:18 +01:00
Andrii Chubatiuk
93bc205e05
promql: exclude limit_offset from default by metric name sorting (#7402)
### Describe Your Changes

I don't like this solution, but it works. Other possible solutions
described in an issue

fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7068

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a88f896b43)
2024-11-06 15:27:29 +01:00
hagen1778
1cc066a1b2
docs/victorialogs: recommend using separate installations of vmalert for vm and vl
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 05f2e9548d)
2024-11-06 15:27:28 +01:00
hagen1778
a0779a86fa
docs/victorialogs: clarify usage of -rule.defaultRuleType=vlogs
User experience suggests that examples shouldn't have `-rule.defaultRuleType=vlogs` set,
as it may confuse users who run vmalert with their existing rules or only use
rules from examples for testing purposes.

This change is supposed to remove the confusion by removing `-rule.defaultRuleType=vlogs`
from default recommendations and explcitily specifying `type` on group level in examples.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit a5f1764171)
2024-11-06 15:27:28 +01:00
Roman Khavronenko
d6eefb86af
docs: fix typos in vlogs rules examples (#7457)
* fix typos in rules definition. Otherwise, they can't pass validation
* add code types for rendered examples

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0390d58a34)
2024-11-06 13:58:20 +01:00
Zhu Jiekun
63df2205cf
vmctl: fixed import duplicate data when query result contains multiple series (#7330)
### Describe Your Changes

Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301

When querying with condition like `WHERE a=1` (looking for series A),
InfluxDB can return data with the tag `a=1` (series A) and data with the
tag `a=1,b=1` (series B).

However, series B is will be queried later and it's data should not be
combined into series A's data.

This PR filter those series that are not identical to the original query
condition.

For table `example`:
```
// time                           host    region  value
// ----                           ----    ------  -----
// 2024-10-25T02:12:13.469720983Z serverA us_west 0.64
// 2024-10-25T02:12:21.832755213Z serverA us_west 0.75
// 2024-10-25T02:12:32.351876479Z serverA         0.88
// 2024-10-25T02:12:37.766320484Z serverA         0.95
```

The query for series A (`example_value{host="serverA"}`) and result will
be:
```SQL
SELECT * FROM example WHERE host = "serverA"
```
```json
{
	"results": [{
		"statement_id": 0,
		"series": [{
			"name": "cpu",
			"columns": ["time", "host", "region", "value"],
			"values": [
				["2024-10-25T02:12:13.469720983Z", "serverA", "us_west", 0.64],
				["2024-10-25T02:12:21.832755213Z", "serverA", "us_west", 0.75],
				["2024-10-25T02:12:32.351876479Z", "serverA", null, 0.88],
				["2024-10-25T02:12:37.766320484Z", "serverA", null, 0.95]
			]
		}]
	}]
}
```

We need to abandon `values[0]` and `values[1]` because the value of
**unwanted** column `region` is not null.

As for series B (`example_value{host="serverA", region="us_west"}`), no
change needed since the query filter out unwanted rows already.

### Note
This is a draft PR for verifying the fix.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f16a58f14c)
2024-11-06 13:58:19 +01:00
Artem Navoiev
1f50f29e3f
docs: understand your setup size fix formula
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 115ac7d0d7)
2024-11-06 13:58:19 +01:00
Artem Fetishev
e946ec5121
docs: Fix images used in Query data docs (#7443)
### Describe Your Changes

This is a follow up for #7435. Images need to be updated too:

-   The time is changed from 10 hrs to 08 hrs
-   A missing data point is added to the range query image
-   Source escalidraw has been updated as well

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit d212243a0f)
2024-11-06 13:58:19 +01:00
Github Actions
0d2ece86dd
Automatic update helm docs from VictoriaMetrics/helm-charts@f598096 (#7452)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 021f514842)
2024-11-06 13:58:19 +01:00
Zhu Jiekun
9db7c74745
docs: [puppetdb] add changelog for puppetdb service discovery (#7455)
### Describe Your Changes

Add puppetdb sd to changelog of `v1.106.0` version.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 11d735a91f)
2024-11-06 13:58:18 +01:00
Github Actions
9eaf8d6136
Automatic update operator docs from VictoriaMetrics/operator@f8ca70f (#7444)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
(cherry picked from commit fa7adcaeaf)
2024-11-06 13:58:18 +01:00
Github Actions
859602294c
Automatic update helm docs from VictoriaMetrics/helm-charts@76f3195 (#7447)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: zekker6 <1367798+zekker6@users.noreply.github.com>
(cherry picked from commit 74a9c6f91c)
2024-11-06 13:58:18 +01:00
Zakhar Bessarab
577a7a247d
{docker,docs}: update references to latest release
Update references to latest release - v1.106.0.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit be677065bc)
2024-11-06 13:58:17 +01:00
Smaine Kahlouch
64410bfdb8
docs(articles): add blog ogenki to third party articles (#7414)
### Describe Your Changes

Adding a blog post that introduces VictoriaMetrics to third party
articles

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Smaine Kahlouch <smainklh@gmail.com>
(cherry picked from commit 5b838b03ec)
2024-11-06 13:58:17 +01:00
Zakhar Bessarab
015991a5a6
dashboards: add dashboards with victoria-logs datasource (#7424)
### Describe Your Changes

Sync list of dashboards to be provided with Prometheus and
VictoriaMetrics' datasources.
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit d73e5bdb8b)
2024-11-06 13:58:17 +01:00
Andrii Chubatiuk
23dcec3911
vlinsert: support datadog logs
This commit adds the following changes:

- Added support to push datadog logs with examples of how to ingest data
using Vector and Fluentbit
- Updated VictoriaLogs examples directory structure to have single
container image for victorialogs, agent (fluentbit, vector, etc) but
multiple configurations for different protocols

Related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6632

(cherry picked from commit e0930687f1)
2024-11-06 13:58:16 +01:00
Arie Heinrich
7a819ae5ce
docs: spelling fixes (#7420)
### Describe Your Changes

Christmas is early and you get the first present in the shape of
spelling fixes.
Sorry for the big amount :)

### Checklist

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 2e8f420d84)
2024-11-06 13:58:16 +01:00
Github Actions
d0d6c069a9
Automatic update helm docs from VictoriaMetrics/helm-charts@f8ad8eb (#7438)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: AndrewChubatiuk <3162380+AndrewChubatiuk@users.noreply.github.com>
(cherry picked from commit 397997b2a8)
2024-11-06 13:58:15 +01:00
Artem Fetishev
9c390fd201
documentation: Fix query docs
This commit changes the following:
-   The datetime has been fixed so it corresponds to the timestamps in example samples. The datetime now also include the UTC time zone and is changed to adhere ISO format.
-   The data points in query range result have been fixed to match the inserted data.

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
(cherry picked from commit d311c12dce)
2024-11-06 13:58:15 +01:00
Zakhar Bessarab
36ca011fb4
docs/changelog: sync latest release
- remove reference to sparse cache as it was reverted in 9f9cc24e4c
- add reference to 1.102.6 and 1.97.11 LTS releases

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit a3401c8d64)
2024-11-06 13:58:15 +01:00
Roman Khavronenko
3542417757
docs: mention VM version when DS filters became available
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit f52577a1a8)
2024-11-06 13:58:15 +01:00
Zakhar Bessarab
48d3052901
docs/CHANGELOG.md: cut v1.106.0
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 11:14:18 -03:00
Zakhar Bessarab
2bdede8ed5
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-11-04 10:59:58 -03:00
Zakhar Bessarab
718f8077a8
Revert "lib/mergeset: add sparse indexdb cache (#7269)"
This reverts commit 837d0d136d.
2024-11-04 10:33:22 -03:00
Aliaksandr Valialkin
cf8a971421
docs/VictoriaLogs/querying: refer to vlogscli from docs for /select/logsql/query and /select/logsql/tail HTTP endpoints
These endpoints are easier to query via vlogscli

(cherry picked from commit 5733e56e40)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
c8213b30af
docs/VictoriaLogs/data-ingestion: mention Grafana Agent and Grafana Alloy additionally to Promtail as log collectors for Grafana Loki
(cherry picked from commit 37a58677fa)
2024-11-04 10:23:19 -03:00
Aliaksandr Valialkin
5e7f0eb976
docs/VictoriaLogs/LogsQL.md: remove misleading information that min and max stats functions work only for numeric values
These functions work for both numeric and string values after ad505a7a9a

(cherry picked from commit c7e242b8ce)
2024-11-04 10:23:18 -03:00
Aliaksandr Valialkin
43998ff82b
docs/VictoriaLogs/LogsQL.md: typo fix in the link to description for sum_len stats function: sum-len-stats -> sum_len-stats
This is a follow-up for 364f084b43

(cherry picked from commit a86df52adf)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
fced48d540
app/vlinsert: implement the ability to add extra fields to the ingested logs
This can be done via extra_fields query arg or via VL-Extra-Fields HTTP header.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7354#issuecomment-2448671445

(cherry picked from commit 4478e48eb6)
2024-11-04 10:23:16 -03:00
Aliaksandr Valialkin
21d2c417da
docs/VictoriaLogs/keyConcepts.md: remove duplicate sentence about missing timezone information in the ingested _time field values
(cherry picked from commit b3e2db5647)
2024-11-04 10:23:15 -03:00
Aliaksandr Valialkin
bcbaecd73f
lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000
This will allow storing wide events with up to 2K fields per event into VictoriaLogs.
While at it, remove the misleading comment that columnsHeader is read in full per each matching block.
This is no longer the case after the improvements made at 202eb429a7 .
Now only the needed columnHeader is read for the column mentioned in the query.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6425#issuecomment-2418337124
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762

(cherry picked from commit 9ba6be4179)
2024-11-04 10:23:15 -03:00
Andrii Chubatiuk
a987096a1f
Updated k8s guides (#7411)
### Describe Your Changes

- updated dependencies versions
- removed Helm v2 from docs
- fixed VMAgent configuration

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 5e2cb78cce)
2024-11-04 10:23:15 -03:00
f41gh7
2c51fce7fe
app/vmgateway: fixes rate limit for multitenant requests
Previously vmgateway returned error for the requests with multitenant
tenant.

 This commit allows to rate limit multitenant requests and apply global
rate limit for it.

Currently it supports only queries for rate limiting.

Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7201

This commit also addresses gateway start-up crash if datasource.url is not accessible.

Previously vmgateway could crash at start-up with enabled rate limiting if datasource for metrics
was not avaiable for any reason.  It seems, that crash is expected. But in fact it's not. For instance, datasource could be in restart phase.

Replaces crash with log message error. It increased availability of vmgateway component.

Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-10-31 20:20:52 +01:00
Nikolay
b6e7852eee
lib/promscrape: add relabel configs to global section
This commit adds `metric_relabel_configs` and `relabel_configs` fields
into the `global` section of scrape configuration file.

 New fields are used as global relabeling rules for the scrape targets.

 These relabel configs are prepended to the target relabel configs.
This feature is useful to:
* apply global rules to __meta labels from service discovery targets.
* drop noisy labels during scrapping.
* mutate labels without affecting metrics ingested via any of push
protocols. 

Related issue
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6966

---------
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Zhu Jiekun <jiekun@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 20:08:31 +01:00
hagen1778
b543c8728c
docs: mention stats object in Prometheus API enhancements
The doc explains fields meaning in `stats` object.
It also clarifies their purpose.
See related ticket https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7170

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 20:07:58 +01:00
Github Actions
5ad662dd0f
Automatic update Grafana datasource docs from VictoriaMetrics/victorialogs-datasource@3ecaebd (#7395)
(cherry picked from commit 955f3660de)
2024-10-31 14:11:17 +01:00
hagen1778
028e75348c
docs: mention #7392 in changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit c5b36138e2)
2024-10-31 14:11:17 +01:00
Viet Hung Nguyen
cb85e6aef3
vmalert-tool: set default interval for unittest input_series (#7392)
### Describe Your Changes
Currently it is not optional option, but if user forgot to set, it
defaults to 0, which cause unexpected behavior.

This change sets default = evaluation_interval similar to promtool
behavior.

https://prometheus.io/docs/prometheus/2.55/configuration/unit_testing_rules/#test_group
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Viet Hung Nguyen <hvn@familug.org>

(cherry picked from commit 21d1385ae1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-31 14:11:16 +01:00
Roman Khavronenko
d71c745d19
deployment/alerts: add RemoteWriteDroppingData to vmalert rules (#7393)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3f0e2ab3b2)
2024-10-31 14:11:10 +01:00