Commit graph

8841 commits

Author SHA1 Message Date
Hui Wang
b515a7b69b
security: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2 (#6684)
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fix for:
OpenSSL CVE-2024-5535
2024-07-23 13:20:06 +02:00
Zakhar Bessarab
d88d0f382b
app/vmauth: change response code when all backend are not available (#6676)
### Describe Your Changes

Change response code to 502 to align it with behaviour of other existing
reverse proxies. Currently, the following reverse proxies will return
502 in case an upstream is not available: nginx, traefik, caddy, apache.


Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-22 17:31:18 +02:00
Zhu Jiekun
d297ac0e8f
docs: [vmagent] Add docs for remote write protocol control in Kafka integration (#6682)
### Describe Your Changes

Improve documentation to help:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6670

Currently, the documentation of Kafka Integration did not describe:
- How to switch between different remote write protocols in producer
side and consumer side.
2024-07-22 16:51:45 +02:00
Zakhar Bessarab
cb0c4e6cfb
docs/vmbackup: add a note for azure identity (#6675)
### Describe Your Changes

Add a note to clarify usage of azure credentials when multiple
credentials are available.

The user is required to specify AZURE_CLIENT_ID as otherwise Azure API
will return an error: "Multiple user assigned identities exist, please
specify the clientId / resourceId of the identity in the token request"

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-22 15:38:09 +02:00
Artem Navoiev
29e0850495
docs: fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-07-22 10:38:34 +02:00
Aliaksandr Valialkin
06f510899c
docs/CHANGELOG.md: document keep_original_host option at vmauth
This option has been added in the commit add2db12b2
2024-07-20 11:48:40 +02:00
Aliaksandr Valialkin
dad3eefd74
app/vmauth: test how User-Agent header is set in requests to backend 2024-07-20 11:43:24 +02:00
Aliaksandr Valialkin
e87b4d3768
app/vmauth: verify the correctness of X-Forwarded-For header processing at TestRequestHandler() 2024-07-20 11:28:14 +02:00
Aliaksandr Valialkin
cb76ff5c56
app/vmauth: add missing tests for requestHandler() 2024-07-20 11:22:36 +02:00
Aliaksandr Valialkin
78b1571eb8
app/vmauth: add more tests for requestHandler() 2024-07-20 10:19:45 +02:00
Aliaksandr Valialkin
0a8c9c5ee7
docs/vmauth.md: document the case with default url_prefix additionally to url_map 2024-07-20 09:46:01 +02:00
Aliaksandr Valialkin
9e0c37be2d
app/vmauth: properly proxy requests to backend paths ending with /
Previously the traling / was incorrectly removed when proxying requests from http://vmauth/

While at it, add more tests for requestHandler()
2024-07-19 17:29:04 +02:00
Aliaksandr Valialkin
add2db12b2
app/vmauth: properly proxy HTTP requests without body
The Request.Body for requests without body can be nil. This could break readTrackingBody.Read() logic,
which could incorrectly return "cannot read data after closing the reader" error in this case.
Fix this by initializing the readTrackingBody.r with zeroReader.

While at it, properly set Host header if it is specified in 'headers' section.
It must be set net/http.Request.Host instead of net/http.Request.Header.Set(),
since the net/http.Client overwrites the Host header with the value from req.Host
before sending the request.

While at it, add tests for requestHandler(). Additional tests for various requestHandler() cases
will be added in future commits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525
2024-07-19 16:24:12 +02:00
Aliaksandr Valialkin
b2e646280e
docs/goals.md: clarify that complicated CI/CD tasks make debugging harder 2024-07-19 08:50:58 +02:00
Aliaksandr Valialkin
6c8601681a
docs/vmauth.md: use /some_path/.* instead of /some_path/.+ in examples, so it matches /some_path/
The /some_path/.+ regexp matches /some_path/ followed by at least a single char.
This is unexpected by most users, since they expect it should match /some_path/.
Substitute .+ with .*, so this regexp matches /some_path/ .
2024-07-19 08:47:49 +02:00
Yury Molodov
efd70b2c52
vmui/logs: switched requests to sequential execution (#6624)
### Describe Your Changes

This PR changes `/select/logsql/query` and `/select/logsql/hits` to
execute sequentially
Fixed
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6558#issuecomment-2219298984

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-18 11:55:42 +02:00
Aliaksandr Valialkin
cda6a31ae8
docs/Single-server-VictoriaMetrics.md: recommend periodic running of make spellcheck command
This is a follow-up for fabf0b928e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6562
2024-07-18 11:51:53 +02:00
Aliaksandr Valialkin
c6e7fb688b
docs/goals.md: make clear that complication of CI/CD pipeline is bad
CI/CD pipeline is usually hard to debug. So it is good to keep it as simple as possible.
2024-07-18 11:51:52 +02:00
Andrii Chubatiuk
ef01df10bd
Code tooltip updates (#6657)
### Describe Your Changes

recently added custom `promtextmetric` and `influxtextmetric` prismjs
plugins to vmdocs to convert this
<img width="1081" alt="Screenshot 2024-07-17 at 12 13 29"
src="https://github.com/user-attachments/assets/66d4ea18-10fe-45ef-80b4-989d0eb3bd92">
to this
<img width="1087" alt="Screenshot 2024-07-17 at 12 24 34"
src="https://github.com/user-attachments/assets/60d4ab44-79e5-4c63-b966-54b989ead1aa">




### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-18 02:50:49 -07:00
Aliaksandr Valialkin
798256a655
docs/Release-Guide.md: shorten release guide for Ansible roles and RPM packages
There is no need in enumerating all the files, which must be updated, since these files
may be moved to other locations. It is enough to mention that versions for all the VictoriaMetrics
components must be updated.
2024-07-17 23:36:21 +02:00
Aliaksandr Valialkin
c8bc2f0ee5
app/vmselect/vmui: run make vmui-update after 959a4383c5 2024-07-17 23:09:18 +02:00
Aliaksandr Valialkin
9c0871786f
docs/vmbackup.md: an attempt to fix formatting issue at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables 2024-07-17 23:01:23 +02:00
Aliaksandr Valialkin
94ad22d140
docs/stream-aggregation.md: an attempt to fix yaml formatting at https://docs.victoriametrics.com/stream-aggregation/#stream-aggregation-config 2024-07-17 22:43:34 +02:00
Aliaksandr Valialkin
45794de0f9
deployment: update VictoriaMetrics Docker image from v1.101.0 to v1.102.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0
2024-07-17 22:34:48 +02:00
Aliaksandr Valialkin
65bb429b81
docs/CHANGELOG.md: cut v1.102.0 release 2024-07-17 20:49:51 +02:00
Aliaksandr Valialkin
66a34acc3a
vendor: run make vendor-update 2024-07-17 20:47:20 +02:00
Aliaksandr Valialkin
ea5c3571e9
docs/CHANGELOG.md: consistently use new url format for the MetricsQL docs
Use https://docs.victoriametrics.com/metricsql/ instead of https://docs.victoriametrics.com/MetricsQL.html .
This removes unnecessary redirect from https://docs.victoriametrics.com/MetricsQL.html to https://docs.victoriametrics.com/metricsql/

This is a follow-up for 6a4bd5049b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6547
2024-07-17 20:31:17 +02:00
Aliaksandr Valialkin
bcbfea56f9
docs/CHANGELOG.md: document v1.97.6 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.6
2024-07-17 20:27:44 +02:00
Aliaksandr Valialkin
6a12d2f18b
docs/CHANGELOG.md: document v1.93.16 LTS release
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.16
2024-07-17 19:47:55 +02:00
Aliaksandr Valialkin
1e8987df29
docs/CHANGELOG.md: order the changes at tip, so they are easier to read 2024-07-17 18:55:46 +02:00
Fred Navruzov
e5beeb18d3
docs/vmanomaly - v1.13.3 patch notes (#6659)
### Describe Your Changes

Add patch note v1.13.3 to CHANGELOG doc page for `vmanomaly`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-17 19:52:05 +03:00
Aliaksandr Valialkin
e78a51bf39
docs/VictoriaLogs/querying/README.md: add link to Grafana datasource for VictoriaLogs to the top of the page 2024-07-17 18:34:41 +02:00
Aliaksandr Valialkin
aefc4538bd
docs/Single-server-VictoriaMetrics.md: explain why images must have the same filepath as the corresponding docs at docs/ folder
This is a follow-up for f2c1d30011
2024-07-17 18:26:16 +02:00
Aliaksandr Valialkin
1f96882367
docs: fix spellcheck errors found by make spellcheck
Thanks to the spellcheck Makefile rule added by @arkid15r at fabf0b928e
2024-07-17 18:14:20 +02:00
Aliaksandr Valialkin
65ce4e30ab
lib/backup/azremote: follow-up for 5fd3aef549
- Mention that credentials can be configured via env variables at both vmbackup and vmrestore docs.

- Make clear that the AZURE_STORAGE_DOMAIN env var is optional at https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables

- Use string literals as is for env variable names instead of indirecting them via string constants.
  This makes easier to read and understand the code. These environment variable names aren't going to change
  in the future, so there is no sense in hiding them under string constants with some other names.

- Refer to https://docs.victoriametrics.com/vmbackup/#providing-credentials-via-env-variables in error messages
  when auth creds are improperly configured. This should simplify figuring out how to fix the error.

- Simplify the code a bit at FS.newClient(), so it is easier to follow it now.
  While at it, remove the check when superflouos environment variables are set, since it is too fragile
  and it looks like it doesn't help properly configuring vmbackup / vmrestore.

- Remove envLookuper indirection - just use 'func(name string) (string, bool)' type inline.
  This simplifies code reading and understanding.

- Split TestFSInit() into TestFSInit_Failure() and TestFSInit_Success(). This simplifies the test code,
  so it should be easier to maintain in the future.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6518
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984
2024-07-17 17:55:06 +02:00
Aliaksandr Valialkin
eaed0465d2
all: substitute double "the the" with "the"
This is a follow-up for 8786a08d27

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6600
2024-07-17 14:28:12 +02:00
Aliaksandr Valialkin
9c4b0334f2
all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter
The %q formatter may result in incorrectly formatted JSON string if the original string
contains special chars such as \x1b . They must be encoded as \u001b , otherwise the resulting JSON string
cannot be parsed by JSON parsers.

This is a follow-up for c0caa69939

See https://github.com/VictoriaMetrics/victorialogs-datasource/issues/24
2024-07-17 13:52:13 +02:00
Aliaksandr Valialkin
8ff051b287
lib/protoparser/graphite: use Regex.ReplaceAllLiteralString instead of Regex.ReplaceAllString for the case when the replacement cannot contain placeholders for capturing groups
This is a follow-up for 74affa3aec
2024-07-17 13:01:05 +02:00
Aliaksandr Valialkin
74affa3aec
lib/protoparser/graphite: follow-up for 476faf5578
- Clarify the description of -graphite.sanitizeMetricName command-line flag at README.md
- Do not sanitize tag values - only metric names and tag names must be sanitized,
  since they are treated specially by Grafana. Grafana doesn't apply any restrictions on tag values.
- Properly replace more than two consecutive dots with a single dot.
- Disallow unicode letters in metric names and tag names, since neither Prometheus nor Grafana
  do not support them.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6489
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6077
2024-07-17 12:41:55 +02:00
Aliaksandr Valialkin
58a757cd01
lib: consistently use regexp.Regexp.ReplaceAllLiteralString instead of regexp.Regexp.ReplaceAllString in places where the replacement cannot contain matching group placeholders 2024-07-17 12:41:54 +02:00
rtm0
bdc0e688e8
Fix inconsistent error handling in Storage.AddRows() (#6583)
### Describe Your Changes

`Storage.AddRows()` returns an error only in one case: when
`Storage.updatePerDateData()` fails to unmarshal a `metricNameRaw`. But
the same error is treated as a warning when it happens inside
`Storage.add()` or returned by `Storage.prefillNextIndexDB()`.

This commit fixes this inconsistency by treating the error returned by
`Storage.updatePerDateData()` as a warning as well. As a result
`Storage.add()` does not need a return value anymore and so doesn't
`Storage.AddRows()`.

Additionally, this commit adds a unit test that checks all cases that
result in a row not being added to the storage.



---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-07-17 12:07:14 +02:00
Aliaksandr Valialkin
6f9f861f57
docs/CONTRIBUTING.md: mention that docs/CHANGELOG.md shouldn't contain technical details for the changes
The purpose of docs/CHANGELOG.md is to provide VictoriaMetrics users clear and concise information
on what's changed at VictoriaMetrics components. Technical details of the change are unclear
to most of VictoriaMetrics users, who are not familiar with VictoriaMetrics source code.
These details complicate reading the docs/CHANGELOG.md by ordinary users, so do not clutter
the changelog with technical details. If the user wants technical details, he can click
the link to the related GitHub issue and/or pull request and dive into all the details he wants.
2024-07-17 11:46:24 +02:00
Aliaksandr Valialkin
7ed719b46a
app/vmauth: properly handle the case when zero backend hosts are resolved at SRV DNS
When zero backend hosts are resolved, then vmauth must return 'no backend hosts' error instead of crashing with panic

This is a follow-up for 590aeccd7d and 3a45bbb4e0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6401
2024-07-17 11:31:05 +02:00
Aliaksandr Valialkin
7ee5797493
app/vmauth: pool readTrackingBody structs in order to reduce pressure on Go GC
- use pool for readTrackingBody structs in order to reduce pressure on Go GC
- allow re-reading partially read request body
- add missing tests for various cases of readTrackingBody usage

This is a follow-up for ad6af95183 and 4d66e042e3.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6446
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533
2024-07-17 11:06:18 +02:00
Aliaksandr Valialkin
277aad18d8
app/vmauth: use more clear names for the field and function added at e666d64f1d
- Rename overrideHostHeader() function to hasEmptyHostHeader()
- Rename overrideHostHeader field at UserInfo to useBackendHostHeader

This should simplify the future maintenance of the code

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6525
2024-07-16 19:08:38 +02:00
Aliaksandr Valialkin
ad6af95183
Revert "app/vmauth: reader pool to reduce gc & mem alloc (#6533)"
This reverts commit 4d66e042e3.

Reasons for revert:

- The commit makes unrelated invalid changes to docs/CHANGELOG.md
- The changes at app/vmauth/main.go are too complex. It is better splitting them into two parts:
  - pooling readTrackingBody struct for reducing pressure on GC
  - avoiding to use readTrackingBody when -maxRequestBodySizeToRetry command-line flag is set to 0

Let's make this in the follow-up commits!

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6445
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6533
2024-07-16 18:59:16 +02:00
Aliaksandr Valialkin
9cf187c83b
docs: remove the remaining mentioning about snap packages after the commit 0a42c8fd8b
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6543
2024-07-16 18:35:38 +02:00
Aliaksandr Valialkin
c1e32f4517
lib/promrelabel: add test for IfExpression.String() function
While at it, simplify this function a bit after the commit 861852f262

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6462
2024-07-16 18:31:05 +02:00
Zakhar Bessarab
63dfeec0de
app/vmagent/kafka: fix non-unique metric naming (#774)
* app/vmagent/kafka: fix non-unique metric naming

Fix panic when using multiple topics with the same name.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6636
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/changelog: document bugfix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/vmagent: more examples for Kafka ingestion with multiple brokers/topics groups

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-07-16 18:03:24 +02:00
Aliaksandr Valialkin
4304950391
lib/promscrape/discovery/yandexcloud: follow-up for 070abe5c71
- Obtain IAM token via GCE-like API instead of Amazon EC2 IMDSv2 API,
  since it looks like IMDBSv2 API isn't supported by Yandex Cloud
  according to https://yandex.cloud/en/docs/security/standard/authentication#aws-token :

  > So far, Yandex Cloud does not support version 2, so it is strongly recommended
  > to technically disable getting a service account token via the Amazon EC2 metadata service.

- Try obtaining IAM token via GCE-like API at first and then fall back to the deprecated Amazon EC2 IMDBSv1.
  This should prevent from auth errors for instances with disabled GCE-like auth API.
  This addresses @ITD27M01 concern at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513#issuecomment-1867794884

- Make more clear the description of the change at docs/CHANGELOG.md , add reference to the related issue.

P.S. This change wasn't tested in prod because I have no access to Yandex Cloud.
It is recommended to test this change by @ITD27M01 and @vmazgo , who filed
the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5513

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6524
2024-07-16 17:58:40 +02:00