VictoriaMetrics/docs/operator/backups.MD

sort
5

Backups

vmbackupmanager

You can check vmbackupmanager documentation. It contains a description of the service and its features. This documentation covers vmbackumanager integration in vmoperator

vmbackupmanager is a part of VictoriaMetrics Enterprise offer

Before using it, you must have signed contract and accept EULA https://victoriametrics.com/assets/VM_EULA.pdf

Usage examples

VMSingle and VMCluster has built-in backup configuration, it uses vmbackupmanager - proprietary tool for backups. It supports incremental backups (hourly, daily, weekly, monthly) with popular object storages (aws s3, google cloud storage).

The configuration example is the following:

---
apiVersion: v1
kind: Secret
metadata:
  name: remote-storage-keys
type: Opaque
stringData:
  credentials: |-
    [default]
    aws_access_key_id = your_access_key_id
    aws_secret_access_key = your_secret_access_key    
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example-vmsingle
spec:
  # Add fields here
  retentionPeriod: "1"
  vmBackup:
    # This is Enterprise Package feature you need to have signed contract to use it
    # and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
    acceptEULA: true
    destination: "s3://your_bucket/folder"
    credentialsSecret:
      name: remote-storage-keys
      key: credentials
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: example-vmcluster-persistent
spec:
  retentionPeriod: "4"
  replicationFactor: 2
  vmstorage:
    replicaCount: 2
    vmBackup:
      # This is Enterprise Package feature you need to have signed contract to use it
      # and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
      acceptEULA: true
      destination: "s3://your_bucket/folder"
      credentialsSecret:
        name: remote-storage-keys
        key: credentials

NOTE: for cluster version operator adds suffix for destination: "s3://your_bucket/folder", it becomes "s3://your_bucket/folder/$(POD_NAME)". It's needed to make consistent backups for each storage node.

You can read more about backup configuration options and mechanics here

Possible configuration options for backup crd can be found at link

Restoring backups

There are several ways to restore with vmrestore or vmbackupmanager.

Manually mounting disk

You have to stop VMSingle by scaling it replicas to zero and manually restore data to the database directory.

Steps:

1. edit VMSingle CRD, set replicaCount: 0
2. wait until database stops
3. ssh to some server, where you can mount VMSingle disk and mount it manually
4. restore files with vmrestore
5. umount disk
6. edit VMSingle CRD, set replicaCount: 1
7. wait database start

Using VMRestore init container

1. add init container with vmrestore command to VMSingle CRD, example:

   apiVersion: operator.victoriametrics.com/v1beta1
   kind: VMSingle
   metadata:
    name: vmsingle-restored
    namespace: monitoring-system
   spec:
    initContainers:
    - name: vmrestore
      image: victoriametrics/vmrestore:latest
      volumeMounts:
      - mountPath: /victoria-metrics-data
        name: data
      - mountPath: /etc/vm/creds
        name: secret-remote-storage-keys
        readOnly: true
      args:
      - -storageDataPath=/victoria-metrics-data
      - -src=s3://your_bucket/folder/latest
      - -credsFilePath=/etc/vm/creds/credentials
    vmBackup:
      # This is Enterprise Package feature you need to have signed contract to use it
      # and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
     acceptEULA: true
     destination: "s3://your_bucket/folder"
     extraArgs:
         runOnStart: "true"
     image:
         repository: victoriametrics/vmbackupmanager
         tag: v1.83.0-enterprise
     credentialsSecret:
      name: remote-storage-keys
      key: credentials
   

2. apply it, and db will be restored from s3

3. remove initContainers and apply crd.

Note that using VMRestore will require adjusting src for each pod because restore will be handled per-pod.

Using VMBackupmanager init container

Using VMBackupmanager restore in Kubernetes environment is described here.

Advantages of using VMBackupmanager include:

• automatic adjustment of src for each pod when backup is requested
• graceful handling of case when no restore is required - VMBackupmanager will exit with successful status code and won't prevent pod from starting