LabHub

Blog

Kubernetes Velero Backup and Disaster Recovery Strategy

2026-03-08 · 21 min read · #kubernetes#velero#backup#disaster-recovery#migration#2026-03

한국어English日本語

Kubernetes Velero Backup

Introduction

When you run a Kubernetes cluster in production, sooner or later you face the question "what happens if this whole cluster disappears?". etcd failures, cloud region outages, a bad Helm release rollback, an accidental namespace deletion - there are more scenarios that cause data loss than you might expect.

Velero (formerly Heptio Ark) is an open source project led by VMware (now Broadcom) and the de facto standard tool for backing up and restoring Kubernetes resources and persistent volumes (PVs). With Velero you can implement namespace-level backups, CSI volume snapshots, cross-cluster migration and schedule-based automatic backups.

This article covers Velero from its architecture through installation, scheduled backup configuration, CSI snapshot integration, multi-region DR strategy, cross-cluster migration, and recovery procedures for real failure cases, at a level you can apply directly in operations.

Velero Architecture and How It Works

Core Components

Velero consists of server-side components and a CLI client.

Backup Methods Compared: CSI Snapshots vs File System Backup

Velero offers two ways of backing up persistent volume data.

CSI snapshot method: uses the Kubernetes CSI VolumeSnapshot API to create snapshots at the storage provider level. Because these are block-level snapshots they are fast and highly consistent. The CSI driver does have to support snapshots.

File system backup (FSB) method: uses Kopia (or Restic) to upload the files inside the PV directly to object storage. It works regardless of the storage provider, but because it is a file-level copy it takes a long time on large volumes.

AspectCSI snapshotFile system backup (Kopia)
SpeedFast (block level)Slow (file level)
Storage dependencyCSI driver requiredUniversal (every PV supported)
ConsistencyCrash consistency guaranteedFile-level consistency
Cross-region restoreLimited (tied to the snapshot region)Easy (object storage based)
Resource loadLowHigh (CPU/memory)
Incremental backupStorage dependentBuilt into Kopia

The Backup Flow in Detail

Velero performs a backup in the following order.

  1. The user creates a Backup CR and the Velero controller detects it.
  2. It reads the target resources (Deployment, Service, ConfigMap, Secret, CRD and so on) through the Kubernetes API and serialises them to JSON.
  3. It compresses the serialised resources into a tarball and uploads it to the BSL (object storage).
  4. If PVs are included, it backs up the volume data with the selected method (CSI snapshot or FSB).
  5. It stores the backup metadata and logs in the BSL as well.

Installation and Initial Setup

Prerequisites

# Install the Velero CLI (latest stable version)
curl -fsSL -o velero-v1.15.0-linux-amd64.tar.gz \
  https://github.com/vmware-tanzu/velero/releases/download/v1.15.0/velero-v1.15.0-linux-amd64.tar.gz
tar -xvf velero-v1.15.0-linux-amd64.tar.gz
sudo mv velero-v1.15.0-linux-amd64/velero /usr/local/bin/

# Check the version
velero version --client-only

# If you use CSI snapshots, confirm the CRDs are installed
kubectl get crd | grep volumesnapshot
# volumesnapshotclasses.snapshot.storage.k8s.io
# volumesnapshotcontents.snapshot.storage.k8s.io
# volumesnapshots.snapshot.storage.k8s.io

Installing on AWS S3

An installation example targeting AWS S3, the most commonly used backend.

# Create the S3 bucket
aws s3api create-bucket \
  --bucket velero-backup-prod \
  --region ap-northeast-2 \
  --create-bucket-configuration LocationConstraint=ap-northeast-2

# Create the IAM credentials file
cat > credentials-velero <<EOF
[default]
aws_access_key_id=<YOUR_ACCESS_KEY>
aws_secret_access_key=<YOUR_SECRET_KEY>
EOF

# Install Velero (CSI snapshots + Node Agent enabled)
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.11.0 \
  --bucket velero-backup-prod \
  --backup-location-config region=ap-northeast-2 \
  --snapshot-location-config region=ap-northeast-2 \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --features EnableCSI \
  --wait

# Verify the installation
kubectl get pods -n velero
# NAME                      READY   STATUS    RESTARTS   AGE
# node-agent-xxxxx          1/1     Running   0          30s
# node-agent-yyyyy          1/1     Running   0          30s
# velero-xxxxxxxxx-zzzzz    1/1     Running   0          30s

Installing with the Helm Chart

In production it is recommended to manage Velero declaratively with the Helm chart.

# values-velero.yaml
configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero-backup-prod
      config:
        region: ap-northeast-2
  volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: ap-northeast-2
  features: EnableCSI
  defaultVolumesToFsBackup: false

credentials:
  useSecret: true
  secretContents:
    cloud: |
      [default]
      aws_access_key_id=<YOUR_ACCESS_KEY>
      aws_secret_access_key=<YOUR_SECRET_KEY>

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.11.0
    volumeMounts:
      - mountPath: /target
        name: plugins
  - name: velero-plugin-for-csi
    image: velero/velero-plugin-for-csi:v0.8.0
    volumeMounts:
      - mountPath: /target
        name: plugins

deployNodeAgent: true

nodeAgent:
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: '2'
      memory: 2Gi

resources:
  requests:
    cpu: 500m
    memory: 256Mi
  limits:
    cpu: '1'
    memory: 1Gi

schedules:
  daily-production:
    disabled: false
    schedule: '0 2 * * *'
    useOwnerReferencesInBackup: false
    template:
      ttl: '168h'
      includedNamespaces:
        - production
        - staging
      snapshotVolumes: true
      storageLocation: default
      volumeSnapshotLocations:
        - default

# Install with Helm
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
helm install velero vmware-tanzu/velero \
  --namespace velero \
  --create-namespace \
  -f values-velero.yaml

Scheduled Backup Strategy and RTO/RPO Design

RTO and RPO Explained

A disaster recovery strategy hinges on two metrics.

The Velero backup interval determines the RPO, and the speed of the restore procedure determines the RTO.

Backup Strategy by Workload Tier

Workload tierRPO targetBackup intervalRetentionBackup method
Tier 1 (mission critical)1 hourEvery hour72 hoursCSI snapshot + FSB
Tier 2 (business critical)4 hoursEvery 6 hours7 daysCSI snapshot
Tier 3 (general)24 hoursOnce a day30 daysFSB
Tier 4 (dev/test)1 weekOnce a week14 daysFSB

Scheduled Backup YAML Examples

# Tier 1: mission critical workloads - hourly backup
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: tier1-hourly-backup
  namespace: velero
spec:
  schedule: '0 * * * *'
  useOwnerReferencesInBackup: false
  template:
    ttl: 72h0m0s
    includedNamespaces:
      - payment
      - order-service
    snapshotVolumes: true
    storageLocation: default
    volumeSnapshotLocations:
      - default
    defaultVolumesToFsBackup: false
    metadata:
      labels:
        backup-tier: 'tier1'
        environment: 'production'
---
# Tier 3: general workloads - daily backup
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: tier3-daily-backup
  namespace: velero
spec:
  schedule: '0 2 * * *'
  useOwnerReferencesInBackup: false
  template:
    ttl: 720h0m0s
    includedNamespaces:
      - monitoring
      - logging
      - internal-tools
    snapshotVolumes: false
    defaultVolumesToFsBackup: true
    storageLocation: default
    metadata:
      labels:
        backup-tier: 'tier3'
        environment: 'production'

Intelligent Retention Policy (GFS Pattern)

Implementing the Grandfather-Father-Son pattern with Velero schedules keeps fine-grained recovery points in the short term while cutting storage costs in the long term.

# Hourly backup (24-hour retention)
velero schedule create hourly-backup \
  --schedule="0 * * * *" \
  --ttl 24h0m0s \
  --include-namespaces production \
  --snapshot-volumes

# Daily backup (7-day retention)
velero schedule create daily-backup \
  --schedule="0 3 * * *" \
  --ttl 168h0m0s \
  --include-namespaces production,staging \
  --snapshot-volumes

# Weekly backup (30-day retention)
velero schedule create weekly-backup \
  --schedule="0 4 * * 0" \
  --ttl 720h0m0s \
  --snapshot-volumes

# Monthly backup (365-day retention)
velero schedule create monthly-backup \
  --schedule="0 5 1 * *" \
  --ttl 8760h0m0s \
  --snapshot-volumes

CSI Snapshot Integration in Depth

Installing the CSI Snapshot Controller

To use CSI snapshots, the snapshot controller and its CRDs must be installed in the cluster.

# Install the CSI snapshot CRDs
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

# Install the snapshot controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.2.0/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Configuring VolumeSnapshotClass

For Velero to use CSI snapshots, the VolumeSnapshotClass needs a particular label.

# VolumeSnapshotClass for the AWS EBS CSI driver
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-csi-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: 'true'
driver: ebs.csi.aws.com
deletionPolicy: Retain
parameters:
  tagSpecification_1: 'velero-backup=true'

Velero automatically detects a VolumeSnapshotClass carrying the velero.io/csi-volumesnapshot-class: "true" label and uses it to create CSI snapshots. Setting deletionPolicy to Retain preserves the actual snapshot data even when the VolumeSnapshot resource is deleted.

Moving CSI Snapshot Data (Data Mover)

CSI snapshots are stored locally inside the storage provider by default. For cross-region or cross-cloud DR the snapshot data has to be moved to object storage. From Velero 1.12 the built-in Data Mover provides this.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: production-with-datamover
  namespace: velero
spec:
  includedNamespaces:
    - production
  snapshotMoveData: true
  storageLocation: default
  datamover: velero
  snapshotVolumes: true

Specifying snapshotMoveData: true copies the data to the object storage of the BSL once the CSI snapshot has been created. That resolves the problem of the original snapshot being tied to one particular region.

Cross-Cluster Migration

Migration Architecture

The key to cross-cluster migration with Velero is that the source cluster and the target cluster share the same BSL (object storage). The target cluster can reach a backup created on the source cluster and restore from it.

Migration flow:

  1. Create a backup of the namespaces to migrate on the source cluster
  2. Install Velero on the target cluster (with the same BSL configuration)
  3. Sync the BSL on the target cluster and check the backup list
  4. Restore from the chosen backup
  5. Verify the restored resources and switch DNS/Ingress

Running the Migration

# [Source cluster] create the migration backup
velero backup create migration-app-v2 \
  --include-namespaces app-v2 \
  --snapshot-volumes \
  --snapshot-move-data \
  --wait

# Check the backup status
velero backup describe migration-app-v2 --details

# [Target cluster] sync the BSL (Velero has to be pointing at the same S3)
# Sync runs every minute by default; force an immediate sync with the command below
kubectl -n velero patch backupstoragelocation default \
  --type merge \
  --patch '{"spec":{"accessMode":"ReadOnly"}}'

# Switch back to ReadWrite after a moment
kubectl -n velero patch backupstoragelocation default \
  --type merge \
  --patch '{"spec":{"accessMode":"ReadWrite"}}'

# Check the backup list
velero backup get

# [Target cluster] run the restore
velero restore create migration-restore \
  --from-backup migration-app-v2 \
  --namespace-mappings app-v2:app-production \
  --wait

# Check the restore result
velero restore describe migration-restore --details
kubectl get pods -n app-production

Namespace Mapping and Resource Filtering

A migration often needs a namespace rename or the exclusion of particular resources.

# Restore with a namespace rename
velero restore create --from-backup my-backup \
  --namespace-mappings old-namespace:new-namespace

# Restore only particular resource types
velero restore create --from-backup my-backup \
  --include-resources deployments,services,configmaps,secrets

# Restore excluding particular resource types
velero restore create --from-backup my-backup \
  --exclude-resources storageclasses,persistentvolumes

# Restore only the resources matching a label selector
velero restore create --from-backup my-backup \
  --selector app=frontend

Multi-Region DR Strategy

Configuring Multiple BSLs

In production you should not depend on a single object storage bucket; configure several BSLs to obtain geographic redundancy.

# Primary BSL - Seoul region
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: primary-seoul
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: velero-backup-ap-northeast-2
    prefix: cluster-prod
  config:
    region: ap-northeast-2
  accessMode: ReadWrite
  default: true
---
# Secondary BSL - Tokyo region (for DR)
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: secondary-tokyo
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: velero-backup-ap-northeast-1
    prefix: cluster-prod-dr
  config:
    region: ap-northeast-1
  accessMode: ReadWrite
  credential:
    name: velero-dr-credentials
    key: cloud

# Create the backup in both BSLs
velero backup create dr-backup-$(date +%Y%m%d) \
  --include-namespaces production \
  --snapshot-volumes \
  --snapshot-move-data \
  --storage-location primary-seoul

# Replicate the backup to the DR site as well
velero backup create dr-backup-$(date +%Y%m%d)-replica \
  --include-namespaces production \
  --snapshot-move-data \
  --storage-location secondary-tokyo

Using S3 Cross-Region Replication

Instead of duplicating the BSL, S3 Cross-Region Replication (CRR) replicates backup data to another region automatically, with no separate backup command. This approach has less operational overhead, but the S3 CRR cost is added on top. On the DR cluster, point a BSL at the replicated bucket and access it in ReadOnly mode.

Recovery Procedures by DR Scenario

Scenario 1: recovering from a single deleted namespace

# Check the accidentally deleted namespace
kubectl get ns production
# Error from server (NotFound): namespaces "production" not found

# Check the latest backup
velero backup get --selector backup-tier=tier1 | head -5

# Restore the namespace
velero restore create ns-recovery-$(date +%s) \
  --from-backup tier1-hourly-backup-20260308020000 \
  --include-namespaces production \
  --wait

# Check the restore status
velero restore describe ns-recovery-$(date +%s) --details

Scenario 2: whole-cluster DR (region outage)

  1. Provision a new Kubernetes cluster in the DR region
  2. Install Velero (pointing at the DR BSL)
  3. Restore everything from the most recent backup
  4. Switch DNS and verify the services

Scenario 3: rolling back a specific resource (recovering from a bad deployment)

# Restore only the bad Deployment to its previous state
velero restore create deployment-rollback \
  --from-backup daily-backup-20260307 \
  --include-namespaces production \
  --include-resources deployments \
  --selector app=api-server \
  --existing-resource-policy update \
  --wait

The --existing-resource-policy update option overwrites resources that already exist with their state at backup time. The default is none, which skips a resource when it already exists.

Backup Tools Compared: Velero vs Kasten K10 vs etcd Backup

When choosing a Kubernetes backup solution it is important to understand the characteristics of these three approaches.

AspectVeleroKasten K10Native etcd backup
LicenceOpen source (Apache 2.0)Commercial (free up to 5 nodes)Open source
Backup scopeNamespace/resource levelApplication levelWhole cluster (etcd data)
UICLI centricBuilt-in web dashboardCLI (etcdctl)
PV backupCSI snapshot + FSBCSI snapshot + KanisterNot supported (needs separate work)
Multi-clusterManual, via a shared BSLUnified through a central consoleIndependent per cluster
EncryptionSingle key basedEnvelope encryption (master key + DEK)Not supported
Restore granularityNamespace/resource/labelApplication/componentFull restore only
DB-consistent backupImplemented with hooksBuilt-in blueprints providedConsistency only for etcd
Suitable environmentSmall to mid size, cost sensitiveLarge enterpriseControl plane DR
Learning curveMediumLow (UI provided)High

Choosing between them:

Using Pre/Post Backup Hooks

Workloads such as databases, where file system consistency is not guaranteed while they are running, need a quiesce step before the backup. Velero's Pre/Post Hooks let you automate that.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: db-consistent-backup
  namespace: velero
spec:
  includedNamespaces:
    - database
  snapshotVolumes: true
  hooks:
    resources:
      - name: postgresql-hook
        includedNamespaces:
          - database
        labelSelector:
          matchLabels:
            app: postgresql
        pre:
          - exec:
              container: postgresql
              command:
                - /bin/bash
                - -c
                - 'pg_dump -U postgres -d myapp > /var/lib/postgresql/backup/pre_backup.sql && sync'
              onError: Fail
              timeout: 120s
        post:
          - exec:
              container: postgresql
              command:
                - /bin/bash
                - -c
                - 'rm -f /var/lib/postgresql/backup/pre_backup.sql'
              onError: Continue
              timeout: 30s

Setting onError to Fail in the Pre Hook aborts the backup itself when the hook fails. That prevents a backup with no consistency guarantee from being created.

Monitoring and Alerting

Collecting Prometheus Metrics

Velero exposes Prometheus metrics out of the box. Configure a ServiceMonitor to monitor backup and restore status.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: velero
  namespace: velero
  labels:
    app.kubernetes.io/name: velero
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: velero
  namespaceSelector:
    matchNames:
      - velero
  endpoints:
    - port: monitoring
      interval: 30s

Key Monitoring Metrics

Alertmanager Rule Example

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: velero-alerts
  namespace: velero
spec:
  groups:
    - name: velero.rules
      rules:
        - alert: VeleroBackupFailure
          expr: increase(velero_backup_failure_total[1h]) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: 'Velero backup failure detected'
            description: 'A Velero backup failed within the last hour. It needs immediate attention.'
        - alert: VeleroBackupNotRunning
          expr: time() - velero_backup_last_successful_timestamp{schedule!=""} > 86400
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: 'A Velero scheduled backup has not run for more than 24 hours'
            description: 'The last successful backup for schedule {{ $labels.schedule }} is more than 24 hours old.'
        - alert: VeleroBackupPartialFailure
          expr: increase(velero_backup_partial_failure_total[6h]) > 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: 'Velero backup partial failure detected'
            description: 'Some resources were not included in the backup. Check the volume snapshot or hook execution status.'

Troubleshooting Guide

Failure Case 1: Backup Status PartiallyFailed

Symptom: the backup completes but its status is shown as PartiallyFailed.

# Check the detailed cause
velero backup describe my-backup --details

# Look for the specific error in the logs
velero backup logs my-backup | grep -i "error\|warning"

Main causes and fixes:

Failure Case 2: PVC Pending After a Restore

Symptom: the restore completes but the PVC does not move on from the Pending state.

Cause: this happens when the StorageClass does not exist on the target cluster or has a different name.

# Check the status of the restored PVC
kubectl get pvc -n restored-namespace

# Check the StorageClasses
kubectl get sc

# Map the StorageClass with a ConfigMap
kubectl -n velero create configmap change-storage-class-config \
  --from-literal=old-storage-class=new-storage-class

# Restore with the mapping applied
velero restore create --from-backup my-backup \
  --include-namespaces production

When a ConfigMap named change-storage-class-config exists in the velero namespace, Velero maps the StorageClass name automatically.

Failure Case 3: BSL Not Reachable

Symptom: Velero cannot reach the object storage, so no backup is created.

# Check the BSL status
velero backup-location get
# NAME      PROVIDER   BUCKET/PREFIX              PHASE         LAST VALIDATED
# default   aws        velero-backup-prod/        Unavailable   2026-03-08 00:00:00

# Check the credentials
kubectl -n velero get secret cloud-credentials -o jsonpath='{.data.cloud}' | base64 -d

# Test BSL access manually
kubectl -n velero exec deploy/velero -- \
  aws s3 ls s3://velero-backup-prod/ --region ap-northeast-2

Fix: expired IAM credentials, a changed bucket policy, or outbound traffic blocked by a network policy can all be the cause. Using IRSA (IAM Roles for Service Accounts) removes the credential expiry problem at the root.

Failure Case 4: Version Mismatch Error

Symptom: the Velero CLI and server versions do not match, so commands fail.

# Check the versions
velero version
# Client:
#   Version: v1.15.0
# Server:
#   Version: v1.14.1

# Reinstall the CLI at the same version as the server, or upgrade the server.

The major and minor versions of the client and the server must match. Patch version differences are compatible in most cases, but the official recommendation is to use identical versions.

Operational Cautions and Best Practices

Principles to Always Follow

  1. Regular restore testing: a backup existing and a restore working are two different things. Perform a real restore into a test namespace at least once a month to verify that the backups are valid.

  2. Enable BSL encryption: backup data contains sensitive information such as Secrets and ConfigMaps. Always enable server-side encryption (SSE) on the object storage.

  3. Apply least privilege RBAC: do not grant excessive permissions to the Velero ServiceAccount. Grant only read permission on the namespaces being backed up and write permission on the velero namespace.

  4. Backup immutability: use S3 Object Lock or an equivalent feature to prevent tampering with or deletion of backup data. It is essential as a defence against ransomware attacks.

  5. Node Agent resource allocation: the default resource settings can lead to an OOM kill when backing up large volumes. In production, set the Node Agent memory limit to at least 2Gi.

Mistakes to Avoid

DR Recovery Drill Checklist

Regular DR recovery drills are essential if you do not want to be caught out during a real incident. Use the checklist below to run a quarterly drill.

Preparation stage:

Restore stage:

Review stage:

Conclusion

Backing up and recovering a Kubernetes cluster is not optional. The belief that "our cluster will be fine" collapses the moment a real failure happens. Despite being open source, Velero provides enterprise-grade capabilities: CSI snapshots, file system backup, cross-cluster migration and schedule-based automatic backups.

The key points covered in this article can be summarised as follows. Split workloads into tiers by importance and design an RPO/RTO for each tier. CSI snapshots together with the Data Mover make cross-region DR possible. Do not relax simply because a backup exists - run restore tests regularly to verify that the backups are valid. Configure monitoring and alerting so that a backup failure is detected and handled immediately.

Velero alone cannot cover every DR scenario. The most robust strategy is to protect the control plane with native etcd backups, protect applications and data with Velero, and run an enterprise solution such as Kasten K10 alongside them when needed.

References

  1. Velero Official Docs - How Velero Works
  2. Velero Official Docs - Cluster Migration
  3. Velero Official Docs - CSI Support
  4. Velero Official Docs - File System Backup
  5. Velero Official Docs - Troubleshooting
  6. Velero GitHub Repository
  7. Kubernetes Official Docs - CSI Volume Snapshots
  8. Broadcom Knowledge Base - Velero Backup Failed or PartiallyFailed
  9. Veeam Kasten for Kubernetes
  10. Kubernetes Official Docs - Operating etcd clusters

Comments

No comments yet.

Sign in to leave a comment