Disaster Recovery with Proxmox

Disaster Recovery with Proxmox

Photo by K on Pexels

Backups protect your data, but disaster recovery (DR) is about getting your services back online quickly when everything goes wrong. A hard drive fails. A power surge damages your server. A ransomware attack encrypts everything. In this lesson, we'll build a disaster recovery plan using Proxmox's built-in tools and some free community tools.

The Disaster Recovery Mindset

Disaster recovery starts with asking two questions:

• RTO (Recovery Time Objective): How long can your business survive without this service? A customer-facing website might need to be back in 15 minutes. An internal wiki might tolerate 4 hours.
• RPO (Recovery Point Objective): How much data can you afford to lose? If you backup hourly, your worst-case RPO is 1 hour of data. For a financial system, you might need near-zero RPO.

Every service should have an RTO and RPO defined. Write them down. They determine your backup frequency, replication strategy, and failover procedures.

Storage Replication with ZFS

If you're using ZFS for your VM storage, Proxmox has built-in storage replication that's perfect for minimizing RPO and RTO:

1. Set up a Proxmox cluster with at least two nodes (we'll cover clustering in the next section).
2. In the Proxmox interface, select a VM that uses ZFS storage.
3. Go to the VM's "Replication" tab and click "Add."
4. Select the target node and schedule (e.g., every 15 minutes).
5. Proxmox will replicate the VM's disk to the target node using ZFS send/receive. Only changed blocks are transferred.

If the primary node fails, you can immediately start the replicated VM on the second node. Your RPO is the time since the last replication (15 minutes in this example) and your RTO is the time to start the VM (under a minute).

# Manual replication from command line
pvesr create 100 --target node2 --schedule "*/15"  # Replicate VM 100 every 15 min
pvesr list                                        # Show all replication jobs
pvesr disable                             # Temporarily disable a job

VM High Availability

For services that can't tolerate downtime, Proxmox supports High Availability (HA). With HA, Proxmox monitors your VMs and automatically restarts them on another node if the current node fails:

1. Set up a Proxmox cluster (minimum 3 nodes for quorum).
2. Configure shared storage (NFS, iSCSI, or Ceph) that all nodes can access.
3. Go to Datacenter then HA then Groups and create an HA group with your nodes.
4. Go to Datacenter then HA then Add and select the VM you want to make highly available.
5. Set the "Max Restart" and "Max Relocate" values — these control how many times Proxmox will try to restart the VM before giving up.

Important: HA requires shared storage. If you're using local storage, use replication instead (which gives you a similar result but with manual failover).

Creating a Disaster Recovery Plan

Here's a practical DR plan template for a small Proxmox deployment:

1. Inventory Your Services

List every VM and container, its purpose, and its criticality:

• Critical: Customer-facing web server (RTO: 15 min, RPO: 1 hour)
• Important: Internal wiki (RTO: 4 hours, RPO: 24 hours)
• Nice to have: Test environment (RTO: 24 hours, RPO: 7 days)

2. Backup Strategy

• Critical VMs: Replicate every 15 minutes + PBS backup nightly
• Important VMs: PBS backup daily with 7-day retention
• Nice-to-have: PBS backup weekly with 4-week retention

3. Offsite Backup

Copy your PBS datastore to an external drive weekly and take it offsite, or sync to a cloud storage provider:

# On PBS server, sync datastore to external drive
rsync -avz /mnt/datastore/main-backups/ /mnt/external-drive/main-backups/

# Or use rclone to sync to cloud storage (S3, Backblaze B2, etc.)
rclone sync /mnt/datastore/main-backups remote:backups/main-backups --progress

4. Recovery Procedures

Document the exact steps for each failure scenario:

• Disk failure: Replace disk, rebuild ZFS pool from redundancy, no VM downtime expected.
• Node failure: Start replicated/HA VMs on another node. Restore non-replicated VMs from PBS.
• Total site failure: Set up new Proxmox server, connect to PBS, restore all VMs from backup.
• Ransomware: Restore from PBS backups taken before the attack. Ensure the new system is patched before restoring data.

5. Testing Your DR Plan

At least quarterly, simulate a failure:

• Power off a VM and see if HA restarts it
• Restore a VM from PBS to a test environment and verify it works
• Simulate a node failure by shutting down a Proxmox server (in a maintenance window)
• Time how long each recovery takes and compare to your RTO targets

Free Tools for Disaster Recovery

• Proxmox Backup Server: Deduplicated, encrypted backups with incremental support
• ZFS Replication: Built-in to Proxmox with ZFS storage
• rclone: Free cloud sync tool for offsite backup copies
• BorgBackup: Alternative deduplicating backup tool for file-level backups
• Uptime Kuma: Free monitoring tool that alerts you when services are down (install in an LXC container)

Key Takeaways

• Define RTO and RPO for every service — this drives your entire DR strategy
• ZFS replication gives near-zero RPO with minimal infrastructure
• HA provides automatic failover for critical services but requires shared storage and a cluster
• Always keep an offsite backup copy — a backup on the same server that fails is useless
• Test your recovery procedures quarterly — an untested plan is just a theory