Alexander Marshalov
7e9a74c865
fixed error with double slash in vmbackupmanager ( #557 )
...
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-11 13:39:06 -07:00
Aliaksandr Valialkin
146b3bd088
lib/backup/azremote: fix after upgrading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from v0.6.1 to v1.0.0
2023-02-08 09:19:10 -08:00
Zakhar Bessarab
decf46d72b
app/vmbackupmanager: add metrics for better observability ( #488 )
...
* app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response
* app/vmbackupmanager: drop old metrics before creating new ones
* app/vmbackupmanager: use `_total` postfix for counter metrics
* app/vmbackupmanager: remove `_total` postfix for gauge-like metrics
* app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention
* app/vmbackupmanager: address review feedback
* app/vmbackupmanager: fix metric name
* app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage
* app/vmbackupmanager: improve performance for backup size calculation
* app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic
* {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package
* app/vmbackupmanager: fix creating new metrics instead of reusing existing ones
* lit/formatutil: add comment to make linter happy
* app/vmbackupmanager: address review feedback
2022-12-20 14:18:43 -08:00
Zakhar Bessarab
1e58eabde6
lib/backup/azremote: fix copying for parts larger than 256M by using async copy ( #3479 )
...
* lib/backup/azremote: fix copying for parts larger than 256M by using async copy
* lib/backup/azremote: add description of an error for log message
2022-12-13 09:36:36 -08:00
Aliaksandr Valialkin
dc890ab80c
lib/backup: remove logging duplicate path values in a single error message
2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin
450a32970a
lib/envtemplate: allow referring env vars from other env vars via %{ENV_VAR} syntax
...
This is a follow-up for 02096e06d0
2022-10-26 14:51:02 +03:00
Nikolay
07140e0877
lib/backup: set s3 default region to us-west-2 ( #3224 )
...
* lib/backup: set s3 default region to us-west-2
it should fix an error with region detection for bucket, if AWS_REGION env var is not set
* Update lib/backup/s3remote/s3.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-13 12:06:24 +03:00
Aliaksandr Valialkin
f926db1de4
lib/backup/azremote: typo fixes after 03872025b747fcc4ee98710ad10fc98764328511
2022-10-07 01:04:37 +03:00
Zakhar Bessarab
a5861407cc
app/vmbackup: fix compatibility with latest azure sdk ( #461 )
2022-10-07 01:04:37 +03:00
Zakhar Bessarab
db791a254b
lib/backup/s3remote: fix error checking for alternative S3 providers ( #3191 )
2022-10-06 13:37:23 +03:00
Aliaksandr Valialkin
cc0d70c3d6
lib/backup/azremote: remove unused methods after the 262ce77e2d
2022-10-06 13:30:47 +03:00
Zakhar Bessarab
6a6dcc059b
lib/backup: add support of Azure Blob Storage ( #460 )
...
* lib/backup: add support of Azure Blob Storage
* lib/backup: add enterprise support of Azure Blob Storage
2022-10-06 00:36:19 +03:00
Zakhar Bessarab
5b7e8d1309
vmbackup: update AWS SDK to v2 ( #3174 )
...
* lib/backup/s3remote: update AWS SDK to v2
* Update lib/backup/s3remote/s3.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
* lib/backup/s3remote: refactor error handling
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-10-01 17:13:04 +03:00
Zakhar Bessarab
6c65ee18d9
vmbackup: configure retries for GCS remote FS ( #3156 )
2022-09-26 16:32:53 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
ba3ca5d1cd
lib/backup/actions: rename removeLockFile -> removeRestoreLock to have consistent naming with createRestoreLock function
2022-08-04 17:43:24 +03:00
Nikolay
6cdc934c3d
adds restore.lock ( #1988 )
...
* adds restore.lock
it must prevent from running storage after incomplete restore process
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958
* return back flock file deletion
* Apply suggestions from code review
* wip
* docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-22 13:10:56 +02:00
Nikolay
cf1d2f289b
removes FileSize from backup part key ( #1872 )
...
* removes FileSize from backup part key
it should fix download restoration for backups
* Update lib/backup/common/part.go
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2021-12-01 12:30:03 +02:00
Lan
6662714c6c
Add flag of S3ForcePathStyle ( #1802 )
2021-11-17 01:10:22 +02:00
Aliaksandr Valialkin
8dcf814c48
app/{vmbackup,vmrestore}: switch from gcs://...
to gs://...
urls for backups to GCS
...
The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls,
while leaving support for `gcs://` urls for backwards compatibility.
2021-09-29 12:12:37 +03:00
Nikolay
6d47e750be
adds check for region with custom s3 endpoint ( #1465 )
2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin
81789da731
lib/backup: increase backup chunk size from 128MB to 1GB
...
This should reduce costs for object storage API calls by 8x. See https://cloud.google.com/storage/pricing#operations-pricing
2021-01-13 12:16:39 +02:00
Aliaksandr Valialkin
de1c07b937
lib/backup: add MustStop() method for all remote filesystems
2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin
bf6d523bef
lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter
2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin
ad0950f630
lib/backup/actions: improve logging to be more clear to humans
2020-10-08 14:23:00 +03:00
Nikolay Khramchikhin
fb356c434b
Changed s3 configProfile flag default, ( #749 )
...
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin
8cc2e01386
lib/backup: allow using ~/.aws/config
without region
...
Use us-west-2 for determining bucket region.
2020-08-04 13:08:05 +03:00
Aliaksandr Valialkin
fe77d661b3
all: use errors.As
instead of type assertion for detecting net.Error
2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
cf91a94daf
lib/backup: properly create missing parent directories in fs.CreateFile
2020-06-05 19:28:25 +03:00
Aliaksandr Valialkin
8b0d9df51d
lib/backup/fsremote: create all the parent directories before creating file in CreateFile
2020-06-05 10:25:28 +03:00
Aliaksandr Valialkin
0f3d46810b
lib/backup: remove misleading -dst
mention in error message
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482
2020-05-15 17:13:27 +03:00
Aliaksandr Valialkin
e72518e8c6
lib/backup: donload only the remaining parts for partially downloaded files after vmrestore
restart
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487
2020-05-15 17:03:25 +03:00
Aliaksandr Valialkin
7545784a49
all: fix golangci-lint
issues
2020-03-10 19:40:03 +02:00
Aliaksandr Valialkin
7cde594696
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
ea53a21b02
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
1029b6ab34
lib/backup/s3remote: check whether the file exists before deleting it
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284
2020-01-09 23:20:51 +02:00
Aliaksandr Valialkin
705af61587
app/{vmbackup,vmrestore}: add backup complete
file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin
7e734433a3
lib/backup: cosmetic fixes after #243
2019-11-29 18:07:41 +02:00
glebsam
4a192cb832
Add option to provide custom endpoint for S3, add option to specify S3 config profile ( #243 )
...
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile
* make fmt
2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin
409c939621
lib/backup: remove flock.lock
file in empty dirs
...
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:
1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.
`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin
17eca31989
lib/backup: retrieve only the required metadata when reading GCS objects
2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin
216a260ced
app/{vmbackup,vmrestore}: add -maxBytesPerSecond
command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin
9d1ee1e2ae
lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore
2019-11-19 18:35:59 +02:00
Oleg Kovalov
74ba42d111
fix misspelled words ( #229 )
2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin
7d7fbf890e
app/{vmbackup,vmrestore}: add vmbackup
and vmrestore
tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-07 21:26:43 +02:00