Commit graph

117 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8b76a40715
lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag()
This is a follow-up for 61dce6f2a1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329
2024-07-16 01:08:41 +02:00
justinrush
e65e55e2dd
lib/backup: add support for Azure Managed Identity (#6518)
### Describe Your Changes

These changes support using Azure Managed Identity for the `vmbackup`
utility. It adds two new environment variables:

* `AZURE_USE_DEFAULT_CREDENTIAL`: Instructs the `vmbackup` utility to
build a connection using the [Azure Default
Credential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity@v1.5.2#NewDefaultAzureCredential)
mode. This causes the Azure SDK to check for a variety of environment
variables to try and make a connection. By default, it tries to use
managed identity if that is set up.

This will close
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5984

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

### Testing

However you normally test the `vmbackup` utility using Azure Blob should
continue to work without any changes. The set up for that is environment
specific and not listed out here.

Once regression testing has been done you can set up [Azure Managed
Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)
so your resource (AKS, VM, etc), can use that credential method. Once it
is set up, update your environment variables according to the updated
documentation.

I added unit tests to the `FS.Init` function, then made my changes, then
updated the unit tests to capture the new branches.

I tested this in our environment, but with SAS token auth and managed
identity and it works as expected.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Justin Rush <jarush@epic.com>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 5fd3aef549)
2024-07-10 12:26:21 +02:00
Arkadii Yakovets
a6655322b1
docs: fix docs/ and README.md spelling errors (#6362)
Fixes `docs/` and `README.md` typos and errors.

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>

(cherry picked from commit c740a8042e)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:53:33 +02:00
Roman Khavronenko
f3e893f699
lib/backup: add -s3TLSInsecureSkipVerify command-line flag (#6318)
* The new flag can be used for for skipping TLS certificates
verification when connecting to S3 endpoint. Affects vmbackup,
vmrestore, vmbackupmanager.

* replace deprecated `EndpointResolver` with `BaseEndpoint`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1056

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit ac836bcf6c)
2024-05-22 16:40:06 +02:00
hagen1778
d4e901e212
Revert "docs: removed code-style highlighting for commanad-line flags of VM components (#6147)"
This reverts commit 9bedbcfa2f.

(cherry picked from commit 6aaf1768f4)
2024-04-25 13:07:58 +02:00
Denys Holius
cad2cb2b56
docs: removed code-style highlighting for commanad-line flags of VM components (#6147)
Using `sh` or `console` formatting doesn't do word-breaking on render. This makes flags description
harder to read, as users need to scroll the web page horizontally.
Removing the formatting renders the description with normal word-breaking.

(cherry picked from commit 9bedbcfa2f)
2024-04-22 15:04:49 +02:00
Aliaksandr Valialkin
b933001d2d
all: replace old https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/single-server-victoriametrics/ 2024-04-18 03:11:49 +02:00
Aliaksandr Valialkin
a21d1fcf57
all: replace old https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/cluster-victoriametrics/ 2024-04-18 02:56:28 +02:00
Aliaksandr Valialkin
a501e237d2
all: replace old https://docs.victoriametrics.com/vmbackupmanager.html url with the new one - https://docs.victoriametrics.com/vmbackupmanager/ 2024-04-18 02:04:39 +02:00
Aliaksandr Valialkin
a85602bb2d
all: replace old https://docs.victoriametrics.com/vmrestore.html url with the new one - https://docs.victoriametrics.com/vmrestore/ 2024-04-18 02:01:18 +02:00
Aliaksandr Valialkin
6fe21520ce
all: replace old https://docs.victoriametrics.com/vmbackup.html url with the new one - https://docs.victoriametrics.com/vmbackup/ 2024-04-18 01:58:00 +02:00
Aliaksandr Valialkin
164032cd9b
lib/httpserver: add support for automatic issuing of TLS certificates via Lets Encrypt service
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949
2024-04-17 23:53:51 +02:00
Aliaksandr Valialkin
18e521dbff
docs: change old url from https://docs.victoriametrics.com/enterprise.html to new url https://docs.victoriametrics.com/enterprise/ 2024-04-04 15:22:38 +03:00
Aliaksandr Valialkin
abf347ec0f
docs: replace speed up with more clear accelerate wording 2024-03-12 03:03:15 +02:00
Artem Navoiev
1348a822f8
docs:vmbackup fix typo sped -> speed
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-03-12 02:55:08 +02:00
Aliaksandr Valialkin
7559b6eb97
docs: update -help output after recent changes in VictoriaMetrics components 2024-03-01 05:31:13 +02:00
Aliaksandr Valialkin
dc5b1e4dc1
lib/httpserver: return back the default value for -http.connTimeout to 2 minutes
It has been appeared that there are VictoriaMetrics users, who rely on the fact that
VictoriaMetrics components were closing incoming connections to -httpListenAddr every 2 minutes
by default. So let's return back this value by default in order to fix the breaking change
made at d8c1db7953 .

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450 .
2024-02-24 00:20:11 +02:00
Aliaksandr Valialkin
8793817fbe
docs/vmbackup.md: remove the unneeded -storageDataPath command-line from the example for making server-side copy of the backup 2024-02-12 22:24:41 +02:00
Aliaksandr Valialkin
f5680a6857
all: upgrade Go builder from Go1.21.7 to Go1.22.0
See https://go.dev/doc/go1.22
2024-02-12 22:14:00 +02:00
Aliaksandr Valialkin
46de5d7eb1
docs: update docs after ae8a867924
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470
2024-02-09 04:20:09 +02:00
Aliaksandr Valialkin
fc8328a87f
docs: sync with enterprise branch 2024-02-08 17:25:26 +02:00
Aliaksandr Valialkin
428109a427
docs: update -help output after 83e55456e2 2024-02-08 17:22:42 +02:00
hagen1778
2aa86313bc
docs: sync flags description for vmbackup and vmbackupmanager
Sync description after changes in 83e55456e2

Remove extra text in flags description for vmbackupmanager as it breaks layout
when rendered at docs.victoriametrics.com

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-08 17:12:53 +02:00
Khushi Jain
a076cb4a93
app/vmbackup: support client-side TLS configuration for create/delete snapshot API (#5738)
(cherry picked from commit 83e55456e2)
2024-02-08 15:58:34 +01:00
Aliaksandr Valialkin
eff4dc3685
docs: sync -help output after recent changes 2024-02-01 15:06:19 +02:00
hagen1778
216f268c1a
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 17:06:26 +01:00
Aliaksandr Valialkin
885ee160c2
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-22 01:23:23 +02:00
Aliaksandr Valialkin
dc5066368a
docs/{vmbackup,vmbackupmanager}.md: clarify why storing backups to S3 Glacier can be time-consuming and expensive
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5614
This is a follow-up for e14e3d9c8c
2024-01-17 01:06:51 +02:00
Roman Khavronenko
c69d5041bb
docs: mention requirements for latest backups (#5614)
Add docs to explain that `latest` backup folder can be accessed
more frequently than others. Hence, it is recommended to keep it
on storages with low access price.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-16 21:28:27 +02:00
Aliaksandr Valialkin
0a99c819bf
all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page
This may be needed for systems, which require this metadata such as Google Cloud Managed Prometheus.
See https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
2023-12-19 03:26:02 +02:00
Aliaksandr Valialkin
9540d29154
lib/pushmetrics: add -pushmetrics.header and -pushmetrics.disableCompression command-line flags 2023-12-17 19:58:14 +02:00
Aliaksandr Valialkin
de1b394af8
docs: sync -help output for VictoriaMetrics components after recent changes 2023-12-10 01:13:51 +02:00
Aliaksandr Valialkin
68c0038a5d
docs/vmbackup.md: fix links to https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages
This is a follow-up for 2fc7e9f47e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
2023-11-15 23:27:00 +01:00
Dima Lazerka
ed8fc04898
lib/httpserver: add flags to specify HSTS / Frame-Options / CSP headers for httpserver (#5111)
support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options`
HTTP headers in all VictoriaMetrics components.
The values for headers can be specified by users via the following flags:
`-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`.

Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit ad839aa492)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 11:41:38 +01:00
Aliaksandr Valialkin
94e061087f
docs: use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest instead of https://github.com/VictoriaMetrics/VictoriaMetrics/releases link where needed
The https://github.com/VictoriaMetrics/VictoriaMetrics/releases link may show non-latest
releases at the top, such as LTS releases or VictoriaLogs releases.
So it is better to use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest link,
which always redirect to the latest available release of VictoriaMetrics.
2023-10-26 09:23:17 +02:00
Aliaksandr Valialkin
d6f585ec0f
docs/vmbackup.md: clarify documentation about -deleteAllObjectVersions command-line flag
Updates 2fc7e9f47e
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
2023-10-16 14:02:37 +02:00
hagen1778
709a2bad66
docs: remove extra / in the end of the link
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-16 13:24:30 +02:00
Aliaksandr Valialkin
dfc67aedcb
lib/license: cleanups and prettifications for log messages and docs related to licensing
- Make more clear the docs at docs/enterprise.md, so readers could figure out faster
  on how to obtain enterprise key and how to pass it to VictoriaMetrics Enterprise components.

- Fix examples at docs/enterprise.md, which were referring to non-existing `-license-file` command-line flag.
  The `-licenseFile` command-line flag must be used instead.

- Improve the description of `-license*` command-line flags, so users could understand
  faster how to use them.

- Improve the warning message, which is emitted when the deprecated -eula command-line flag is passed,
  so the user could figure out how to switch faster to -license* command-line flags.

- Disallow running VictoriaMetrics components with both -license and -licenseFile command-line flags.

- Disallow running VictoriaMetrics components when -licensFile points to an empty file.

- Consistently use the phrase "This flag is available only in Enterprise binaries" across
  all the enterprise-specific command-line flags.

- Remove unneeded level of indirection for `noLicenseMessage` and `expiredMessage` string contants
  in order to improve code readability and maintainability.

- Remove unneded `return` statements after `logger.Fatalf()` calls, since these calls exit the app and never return.

- Make sure that the info log message about successful license verification is emitted
  when the license is verified successfully. Previously the error message could be logged
  when the license payload is invalid or if it misses some required features.
2023-10-16 12:51:37 +02:00
Zakhar Bessarab
8b42a1733c
lib/backup: add -deleteAllObjectVersions command-line flag (#5147)
New flag enforces removal of all versions of the object in remote object storage.

See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5121
- https://docs.victoriametrics.com/vmbackup.html#permanent-deletion-of-objects-in-s3-compatible-storages

(cherry picked from commit 2fc7e9f47e)
2023-10-10 14:14:21 +02:00
Artem Navoiev
65b2a0ce60
docs: update the license flags description
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

(cherry picked from commit b3cc22b159)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-10 12:45:06 +02:00
Aliaksandr Valialkin
e275be05b0
docs/vmbackup.md: mention that -filestream.disableFadvise command-line flag can be used for reducing CPU usage on systems with big number of CPU cores
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120

This is a follow-up for 75dd7b30ba
2023-10-04 16:38:53 +02:00
Aliaksandr Valialkin
4e5d3d38f8
lib/filestream: add -filestream.disableFadvise syscall for unconditional disabling of fadvise syscall
This may be needed in rare cases when performing backups on systems with big number of CPU cores
and big value passed to -concurrency command-line flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120
2023-10-04 16:30:19 +02:00
Zakhar Bessarab
86eaf6906b
docs/vmbackup: update docs for different authentication options, add examples (#5046)
Updates: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5023

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-09-22 11:42:34 +02:00
Zakhar Bessarab
b842a0cd25
docs: sync description for license flags (#4977)
- update eula flag to add deprecation notice
- add new license flags description

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-09-08 23:26:25 +02:00
Aliaksandr Valialkin
0bc0d2610c
go.mod: increase the minimum supported Go version from Go1.19 to Go1.20 2023-09-07 12:18:12 +02:00
Artem Navoiev
696f143eb5
use correct abbriviation for ESA legal doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-09-05 16:59:47 +02:00
Artem Navoiev
40c795a1e3
change link to the enterprise legal doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-09-05 16:58:51 +02:00
Aliaksandr Valialkin
1361239393
app/vmbackup: add ability to make server-side copying of existing backups 2023-08-13 17:26:26 -07:00
Aliaksandr Valialkin
1b7d97787a
docs: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:40:06 -07:00
Roman Khavronenko
80768d53dd
docs: follow-up after aec4b5db81 (#4638)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-19 14:48:17 -07:00