Commit graph

4203 commits

Author SHA1 Message Date
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
c9fc8079c4
app/vlinsert/syslog: properly skip empty lines in Syslog protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6548
2024-06-28 14:09:45 +02:00
Aliaksandr Valialkin
bb6424aeca
app/vlselect/logsql: add optional fields_limit query arg to /select/logsql/hits HTTP endpoint
This query arg is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6545
in order to return top N groups with the biggest number of hits.
2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
3eecc3de8c
docs/VictoriaLogs: typo fixes 2024-06-28 03:10:05 +02:00
Aliaksandr Valialkin
5a24ea6cc3
docs/VictoriaLogs: document how to export logs from VictoriaLogs
While at it, mention vmctl-like tool for migrating logs from other systems to VictoriaLogs at docs/VictoriaLogs/Roadmap.md

Thanks to the question from Xavier Pestel at https://www.linkedin.com/feed/update/urn:li:activity:7212093021301927937?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7212093021301927937%2C7212171550651731969%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287212171550651731969%2Curn%3Ali%3Aactivity%3A7212093021301927937%29
2024-06-28 02:06:24 +02:00
Aliaksandr Valialkin
05365999ea
docs/VictoriaLogs/querying/README.md: clarify live tailing docs 2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin
4391ef25e9
docs/VictoriaLogs/Roadmap.md: remove Web UI from roadmap, since it is already implemented
See https://docs.victoriametrics.com/victorialogs/querying/#web-ui

While at it, add DataDog protocol for logs into 'data ingestion protocols' roadmap.
2024-06-27 19:12:54 +02:00
Aliaksandr Valialkin
00899506f7
docs/VictoriaLogs/querying: remove unused and outdated vmui.webp image
The image has been deleted from the docs/VictoriaLogs/querying/README.md file
in the commit 87f1c8bd6c
2024-06-27 15:16:43 +02:00
Aliaksandr Valialkin
13df75a28b
docs/VictoriaLogs/querying/README.md: add a link to live tailing docs from querying log docs 2024-06-27 15:12:59 +02:00
Aliaksandr Valialkin
b26acec9a8
app/vlselect: properly return live tailing results 2024-06-27 15:06:15 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
0352d98d74
docs/goals.md: clarify that the usability of the existing functionality must be improved the first 2024-06-27 09:55:40 +02:00
Josh Keife
bdebc128f7
Update vmalert.md (#6539)
Fix typo

### Describe Your Changes

Fix typo in the vmalert docs. In the docs it states rules at `VMAgent`'s
namespace when it should be `VMAlert`'s namespace.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 8d01dd0514)
2024-06-27 09:32:07 +02:00
Aliaksandr Valialkin
911891da95
docs/goals.md: clarify that the listed goals are related to development and support, not sales and marketing 2024-06-26 19:12:36 +02:00
Fred Navruzov
ed41f18c5a
docs/vmanomaly: quickstart page improvements (#6538)
### Describe Your Changes

Small improvements to a QuickStart guide of `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
Andrii Chubatiuk
580d02c3f8
added IMDSv2 for YC SD (#6524)
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
Andrii Chubatiuk
f79df2aa8b
app/vmauth: allow dropping host header (#6525)
### Describe Your Changes

Fixes #6453

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-26 19:12:35 +02:00
rtm0
48a5c4cb01
Fix Date metricid cache consistency under concurrent use (#6534)
### Describe Your Changes

Fix Date metricid cache consistency under concurrent use.
When one goroutine calls Has() and does not find the cache entry in the
immutable map it will acquire a lock and check the mutable map. And it
is possible that before that lock is acquired, the entry is moved from
the mutable map to the immutable map by another goroutine causing a
cache miss.

The fix is to check the immutable map again once the lock is acquired. 

### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-06-26 19:12:34 +02:00
Aliaksandr Valialkin
3a2c0b0a61
docs/goals.md: heading fixes 2024-06-26 16:41:21 +02:00
Aliaksandr Valialkin
d2be74b4a1
docs/goals.md: add VictoriaMetrics proverbs 2024-06-26 14:01:15 +02:00
Aliaksandr Valialkin
75d45068a5
docs/VictoriaLogs/README.md: document the relation between -retention.maxDiskSpaceUsageBytes and -retentionPeriod command-line flags 2024-06-26 12:03:29 +02:00
Aliaksandr Valialkin
db5a5a4f6e
docs/goals.md: add additional non-goals, which frequently become false goals according to the development history of VictoriaMetrics 2024-06-26 12:03:28 +02:00
Yury Molodov
6bde0196d8
vmui/logs: fix the update of the relative time range (#6517)
### Describe Your Changes

- Fixed the update of the relative time range when `Execute Query` is
clicked
- Optimized server requests: now, if an error occurs in the `/query`
request, the `/hits` request will not be executed.

#6345 (duplicates: #6440, #6312)

(cherry picked from commit 43342745ac)
2024-06-26 11:26:08 +02:00
James Rhoat
8249de986e
Updating documentation around OTEL (#6519)
Updating documentation around the opentelemetry endpoint for metrics and
the "How to use OpenTelemetry metrics with VictoriaMetrics" guide so
that it shows not only how to directly write but also how to write to
the otel collector and view metrics in vmui.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit 6652fb630f)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-26 11:26:05 +02:00
Yury Molodov
904ec020ed
vmui: fix input cursor position reset (#6530)
### Describe Your Changes

This PR addresses the issue where the cursor jumps to the end of the
input fields in the modal settings window after each keystroke.

### Before fix:

![ezgif-7-4c69805cea](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/29711459/2e99e833-09e3-4b44-89aa-fc1bd3c4346d)

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit e9b71a2883)
2024-06-26 11:25:47 +02:00
hagen1778
6ad66b3317
docs: rm mentions of vmrecover as it still doesn't exist
While there, add a hint about truncated error messages.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 6775a00622)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-26 11:25:42 +02:00
Daria Karavaieva
affde1794f
docs/vmanomaly:quickstart example fix (#6535)
- Fixed config example in QuickStart vmanomaly docs for 1.13 version
compatibility

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 4b77778ff5)
2024-06-26 11:25:15 +02:00
Aliaksandr Valialkin
2dfbc40563
docs/goals.md: add VictoriaMetrics goals and non-goals page
Everybody who works on VictoriaMetrics must be aware of these goals and non-goals
2024-06-25 18:00:55 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
252ff71a49
docs/VictoriaLogs/data-ingestion/README.md: document that Rsyslog and Syslog-ng can be used for ingesting logs into VictoriaLogs 2024-06-25 15:15:53 +02:00
Aliaksandr Valialkin
0a88a55ae0
docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release 2024-06-25 14:59:31 +02:00
Aliaksandr Valialkin
f24123a776
lib/logstorage: parse syslog structured data into separate fields in order to simplify further querying of this data 2024-06-25 14:54:25 +02:00
Aliaksandr Valialkin
1716c4e609
lib/logstorage: properly parse timezone offset at TryParseTimestampRFC3339Nano()
The TryParseTimestampRFC3339Nano() must properly parse RFC3339 timestamps with timezone offsets.

While at it, make tryParseTimestampISO8601 function private in order to prevent
from improper usage of this function from outside the lib/logstorage package.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6508
2024-06-25 14:54:24 +02:00
hagen1778
26d58f8768
docs: add missing ticket links to the recent releases
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 14:54:24 +02:00
hagen1778
841305da8a
deployment: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 0dcb3acbf7)
2024-06-25 13:58:59 +02:00
Nikolay
bf1464fc33
dashboards: add dashboard and alerts for vmauth (#6491)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
(cherry picked from commit 14b9ef1e4d)
2024-06-25 11:17:04 +02:00
hagen1778
8237b63fe3
docs: make docs-sync
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cc6a1da0d7)
2024-06-25 11:17:04 +02:00
Roman Khavronenko
e3da10a7c1
docs: add section for docs modification (#6528)
While there, remove the obsolete files from the previous version of
docs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 87edfa61c1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 11:17:03 +02:00
Aliaksandr Valialkin
a5445e09c2
Revert "app/vmselect: fix the way of counting raw samples in single query (#6464)"
This reverts commit 5ecf439078.

Reason for revert: the previous logic was correct.

The purpose of `-search.maxSamplesPerQuery` command-line flag is to limit the amounts of CPU resources,
which could be taken by a single query - see https://docs.victoriametrics.com/#resource-usage-limits .
VictoriaMetrics processes samples in blocks during querying - it reads the block, then unpacks it,
then filters out samples outside the selected time range. This means that it _spends CPU time_
on reading and unpacking of _all the samples_ in every block on the requested time range,
even if only a single sample per each block matches the given time range.

The previous logic was effectively limiting CPU time a single query could take.
The new logic fails limiting CPU time a single query could take in some pathological cases
when only a small fraction of samples per each requested block fit the requested time range.
This allows performing multiplication DoS-attacks by querying very narrow time ranges over historical blocks,
which tend to be full. For example, if the `-search.maxSamplesPerQuery` equals to a billion,
and the query requests a single sample out of 8K samples per each block, this means that the query
may unpack a billion of such blocks without exceeding the limit, e.g. it may unpack and process 8K*1e9=8e12 samples.
This is not what the resource usage limits were created for originally - see https://docs.victoriametrics.com/#resource-usage-limits

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5851
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6464
2024-06-25 02:55:43 +02:00
Aliaksandr Valialkin
813a7aec6c
docs: return back spellcheck fixes from the commit 92b22581e6, which were accidentally reverted in 45cf83cd8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6470
2024-06-25 02:33:59 +02:00
Aliaksandr Valialkin
7f2b491748
Revert "docs: Add spellcheck workflow (#6470)"
This reverts commit 92b22581e6.

Reason for revert: too complex and slow approach for spellchecking task.

This approach may significantly slow down development pace. It also may take non-trivial
amounts of additional time and resources at CI/CD because of all this npm shit at cspell directory.

Note to @arkid15r : the idea with the ability to run spellchecker on all the VictoriaMetrics
codebase is great. But this shouldn't be mandatory pre-commit check. It is enough to have
a Makefile rule like `make spellcheck`, which could be run manually whenever spellcheck is needed
(e.g. once per month or once per quarter).
2024-06-25 02:28:38 +02:00
Aliaksandr Valialkin
7026498359
lib/httpserver: revert 9b7e532172
Reason for revert: this commit doesn't resolve real security issues,
while it complicates the resulting code in subtle ways (aka security circus).

Comparison of two strings (passwords, auth keys) takes a few nanoseconds.
This comparison is performed in non-trivial http handler, which takes thousands
of nanoseconds, and the request handler timing is non-deterministic because of Go runtime,
Go GC and other concurrently executed goroutines. The request handler timing is even
more non-deterministic when the application is executed in shared environments
such as Kubernetes, where many other applications may run on the same host and use
shared resources of this host (CPU, RAM bandwidth, network bandwidth).

Additionally, it is expected that the passwords and auth keys are passed via TLS-encrypted connections.
Establishing TLS connections takes additional non-trivial time (millions of nanoseconds),
which depends on many factors such as network latency, network congestion, etc.

This makes impossible to conduct timing attack on passwords and auth keys in VictoriaMetrics components.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6423/files
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6392
2024-06-25 01:51:06 +02:00
Aliaksandr Valialkin
7de6f5b4ce
lib/logstorage: work-in-progress 2024-06-25 00:44:57 +02:00
Aliaksandr Valialkin
8dd69fd7ae
docs/CHANGELOG.md: cut v1.102.0-rc2 release 2024-06-24 23:22:45 +02:00
Nikolay
7ab046cf88
vendor update: updates metrics to v1.34.0 (#6523)
mention change for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6457

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1af13208c2)
2024-06-24 14:52:32 +02:00
Ivan Yatskevich
50343768fd
Publish Alertmanager configuration docs for Managed VictoriaMetrics (#6428)
### Describe Your Changes

With the recent release of Managed VictoriaMetrics users are able to
create and execute Alerting & Recording rules and send notifications via
hosted Alertmanager.

So, we're publishing Alertmanager configuration docs for Managed
VictoriaMetrics.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 1106d91741)
2024-06-24 12:50:15 +02:00
Arkadii Yakovets
c454084eb1
docs: Add spellcheck workflow (#6470)
### Describe Your Changes

This is a follow-up PR containing workflow related part of the initial
[PR#6362](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6362).
It adds spell checking control based on
[cspell](https://github.com/streetsidesoftware/cspell). The related
configs are consolidated under `cspell` directory.

You can add VictoriaMetrics specific terms into `cspell/custom-dict.txt`
(it contains ~30 items atm). All other absent commonly used terms should
be added directly to respective
[cspell-dicts](https://github.com/streetsidesoftware/cspell-dicts/blob/main/CONTRIBUTING.md).

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
(cherry picked from commit 92b22581e6)
2024-06-24 12:50:13 +02:00
Andrii Chubatiuk
50783fca4d
app/vmagent: add max_scrape_size to scrape config (#6434)
Related to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6429

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1e83598be3)
2024-06-20 14:00:22 +02:00
Hui Wang
a393b993d6
vmalert: exit replay mode with non-zero code if generated samples are… (#6513)
… not successfully written into remoteWrite url

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6512

(cherry picked from commit bc37b279aa)
2024-06-20 14:00:19 +02:00
Aliaksandr Valialkin
d5224f3363
lib/logstorage: work-in-progress 2024-06-20 03:10:37 +02:00