Backup and Restore#

How to protect your Mailu data with backups and recover from failures.

Problem#

You need to protect email data, user accounts, configuration, and DKIM keys from hardware failures, human errors, or disasters. You also need a tested procedure to restore service after data loss.

Solution#

Implement a comprehensive backup strategy covering PostgreSQL database, PersistentVolumes (mailboxes), and Kubernetes configuration. Test restore procedures regularly to ensure backups are viable.

What to Backup#

Critical Data (Must Backup)#

Component	Data	Backup Method	Frequency	Priority
PostgreSQL	User accounts, domains, aliases	pg_dump	Daily	Critical
Dovecot PVC	User mailboxes (email content)	Volume snapshots	Daily	Critical
Admin PVC	DKIM keys, config	Volume snapshots	Weekly	High
Rspamd PVC	Spam learning data	Volume snapshots	Weekly	Medium

Optional Data (Nice to Backup)#

Component	Data	Priority
Postfix PVC	Mail queue (temporary)	Low (self-healing)
ClamAV PVC	Virus signatures (re-downloadable)	Low
Webdav PVC	Calendars, contacts	Medium

Configuration (Version Control)#

Component	Backup Method
CDK8S source code	Git repository
Generated manifests	Git repository
Kubernetes secrets	Encrypted backup or external secret manager

Backup PostgreSQL Database#

Manual Database Backup#

Create a one-time database backup:

# Dump entire database
kubectl exec -n postgres postgres-1 -- \
  pg_dump -U postgres -d mailu --clean --if-exists \
  > mailu-db-backup-$(date +%Y%m%d-%H%M%S).sql

# Compress for storage
gzip mailu-db-backup-*.sql

Backup size: Typically 1-10MB for small deployments, scales with user count.

Automated Database Backups#

Create a CronJob for daily database backups:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mailu-db-backup
  namespace: mailu
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: mailu-backup
          containers:
          - name: backup
            image: postgres:15
            command:
            - /bin/bash
            - -c
            - |
              set -e
              BACKUP_FILE="/backups/mailu-db-$(date +%Y%m%d-%H%M%S).sql"
              pg_dump -h postgres-rw -U mailu -d mailu --clean --if-exists > "$BACKUP_FILE"
              gzip "$BACKUP_FILE"
              echo "Backup completed: ${BACKUP_FILE}.gz"

              # Cleanup backups older than 30 days
              find /backups -name "mailu-db-*.sql.gz" -mtime +30 -delete
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-app
                  key: password
            volumeMounts:
            - name: backups
              mountPath: /backups
          volumes:
          - name: backups
            persistentVolumeClaim:
              claimName: mailu-db-backups
          restartPolicy: OnFailure

Prerequisites: Create PVC for backup storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mailu-db-backups
  namespace: mailu
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi  # Adjust based on retention period
  storageClassName: longhorn

CloudNativePG Automated Backups#

If using CloudNativePG (CNPG), configure S3 backups in the PostgreSQL cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres
  namespace: postgres
spec:
  instances: 3
  storage:
    size: 20Gi

  backup:
    barmanObjectStore:
      destinationPath: s3://backup-bucket/postgres-mailu/
      s3Credentials:
        accessKeyId:
          name: s3-credentials
          key: access-key
        secretAccessKey:
          name: s3-credentials
          key: secret-key
      wal:
        compression: gzip
        maxParallel: 2
    retentionPolicy: "30d"

  # Daily full backup at 2 AM
  scheduledBackup:
  - name: daily-backup
    schedule: "0 2 * * *"
    backupOwnerReference: self
    target: primary

Advantages: Point-in-time recovery (PITR), automatic WAL archiving, off-site storage.

Backup PersistentVolumes#

Longhorn Volume Snapshots#

If using Longhorn storage, create volume snapshots:

Manual Snapshot#

# Snapshot Dovecot mailboxes
kubectl create -f - <<EOF
apiVersion: longhorn.io/v1beta2
kind: VolumeSnapshot
metadata:
  name: dovecot-snapshot-$(date +%Y%m%d)
  namespace: longhorn-system
spec:
  volume: mailu-dovecot-pvc-xxx  # Get from: kubectl get pv
EOF

# List snapshots
kubectl get volumesnapshot -n longhorn-system

Automated Recurring Snapshots#

Configure Longhorn RecurringJob for automatic snapshots:

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: mailu-daily-snapshot
  namespace: longhorn-system
spec:
  cron: "0 3 * * *"  # Daily at 3 AM
  task: "snapshot"
  retain: 7  # Keep 7 daily snapshots
  concurrency: 2
  labels:
    app: mailu

Apply to Dovecot PVC:

# Label PVC for automatic snapshots
kubectl label pvc -n mailu mailu-dovecot-pvc recurring-job.longhorn.io/mailu-daily-snapshot=enabled

Longhorn Backup to S3#

Configure Longhorn backup target for off-site backups:

# Set S3 backup target (via Longhorn UI or kubectl)
kubectl patch settings.longhorn.io -n longhorn-system backup-target --type=merge -p '{"value":"s3://backup-bucket@region/"}'

# Create backup from snapshot
kubectl create -f - <<EOF
apiVersion: longhorn.io/v1beta2
kind: Backup
metadata:
  name: dovecot-backup-$(date +%Y%m%d)
  namespace: longhorn-system
spec:
  snapshotName: dovecot-snapshot-YYYYMMDD
  labels:
    app: mailu
    component: dovecot
EOF

Velero Cluster Backups#

For full cluster-level backups, use Velero:

# Install Velero with S3 backend
velero install \
  --provider aws \
  --bucket mailu-backups \
  --secret-file ./credentials-velero \
  --backup-location-config region=us-east-1

# Backup Mailu namespace
velero backup create mailu-backup-$(date +%Y%m%d) \
  --include-namespaces mailu \
  --default-volumes-to-fs-backup

# Schedule daily backups
velero schedule create mailu-daily \
  --schedule="0 2 * * *" \
  --include-namespaces mailu \
  --default-volumes-to-fs-backup \
  --ttl 720h  # 30 days retention

Backup Kubernetes Secrets#

Warning: Secrets contain sensitive data. Encrypt backups and store securely.

Manual Secret Backup#

# Export all Mailu secrets
kubectl get secrets -n mailu -o yaml > mailu-secrets-backup-$(date +%Y%m%d).yaml

# Encrypt with GPG
gpg --symmetric --cipher-algo AES256 mailu-secrets-backup-*.yaml

# Store encrypted file securely (off-site)

Sealed Secrets#

Use Sealed Secrets for safe secret storage in git:

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

# Create sealed secret
kubectl create secret generic mailu-secrets \
  --from-literal=secret-key="..." \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-secret-mailu.yaml

# Commit sealed secret to git (safe - encrypted)
git add sealed-secret-mailu.yaml
git commit -m "Add Mailu sealed secret"

Restore Procedures#

Restore PostgreSQL Database#

Full Database Restore#

# Stop Mailu pods to prevent write conflicts
kubectl scale deployment -n mailu mailu-admin --replicas=0
kubectl scale deployment -n mailu mailu-front --replicas=0

# Drop and recreate database (CAUTION: deletes current data)
kubectl exec -n postgres postgres-1 -- psql -U postgres -c "DROP DATABASE mailu;"
kubectl exec -n postgres postgres-1 -- psql -U postgres -c "CREATE DATABASE mailu OWNER mailu;"

# Restore from backup
gunzip < mailu-db-backup-20250110.sql.gz | \
  kubectl exec -i -n postgres postgres-1 -- psql -U postgres -d mailu

# Restart Mailu pods
kubectl scale deployment -n mailu mailu-admin --replicas=1
kubectl scale deployment -n mailu mailu-front --replicas=1

CloudNativePG Point-in-Time Recovery#

Restore CNPG cluster to specific timestamp:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-restored
  namespace: postgres
spec:
  instances: 3

  bootstrap:
    recovery:
      source: postgres-backup
      recoveryTarget:
        targetTime: "2025-01-10 14:00:00"  # Restore to this timestamp

  externalClusters:
  - name: postgres-backup
    barmanObjectStore:
      destinationPath: s3://backup-bucket/postgres-mailu/
      s3Credentials:
        accessKeyId:
          name: s3-credentials
          key: access-key
        secretAccessKey:
          name: s3-credentials
          key: secret-key

Restore PersistentVolumes#

Longhorn Volume Restore#

Restore from Longhorn snapshot:

# Method 1: Restore from snapshot (creates new PVC)
kubectl create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mailu-dovecot-restored
  namespace: mailu
spec:
  dataSource:
    name: dovecot-snapshot-20250110
    kind: VolumeSnapshot
    apiGroup: longhorn.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: longhorn
EOF

# Method 2: Restore from backup (Longhorn UI)
# 1. Go to Longhorn UI → Backup
# 2. Find backup: dovecot-backup-20250110
# 3. Click "Restore" → creates new volume
# 4. Create PVC pointing to restored volume

Update deployment to use restored PVC:

# Edit deployment to reference new PVC
kubectl edit deployment -n mailu mailu-dovecot

# Change:
#   volumes:
#   - name: mail
#     persistentVolumeClaim:
#       claimName: mailu-dovecot-restored  # Updated

Velero Namespace Restore#

Restore entire Mailu namespace from Velero backup:

# List available backups
velero backup get

# Restore from specific backup
velero restore create mailu-restore-$(date +%Y%m%d) \
  --from-backup mailu-backup-20250110

# Monitor restore progress
velero restore describe mailu-restore-YYYYMMDD
kubectl get pods -n mailu -w

Restore Kubernetes Secrets#

# Decrypt secret backup
gpg --decrypt mailu-secrets-backup-20250110.yaml.gpg > secrets.yaml

# Apply secrets
kubectl apply -f secrets.yaml

# Clean up decrypted file
rm secrets.yaml

Disaster Recovery Scenarios#

Scenario 1: Accidental Email Deletion#

Symptoms: User reports missing emails.

Recovery:

Restore Dovecot PVC from most recent snapshot (before deletion)
Mount restored volume to temporary pod
Extract deleted user’s mailbox
Copy mailbox to production Dovecot pod

# Create recovery pod with restored PVC
kubectl run -n mailu dovecot-recovery \
  --image=alpine --command -- sleep infinity
kubectl set volumes pod/dovecot-recovery -n mailu \
  --add --name=mail-restored \
  --claim-name=mailu-dovecot-restored

# Copy mailbox
kubectl exec -n mailu dovecot-recovery -- \
  tar czf /tmp/user-mailbox.tar.gz /mail/user@example.com

kubectl cp -n mailu dovecot-recovery:/tmp/user-mailbox.tar.gz ./user-mailbox.tar.gz

# Extract to production pod
kubectl cp -n mailu ./user-mailbox.tar.gz mailu-dovecot-xxx:/tmp/
kubectl exec -n mailu mailu-dovecot-xxx -- \
  tar xzf /tmp/user-mailbox.tar.gz -C /

Scenario 2: Database Corruption#

Symptoms: Admin UI errors, authentication failures.