Clustering and High Availability

Clustering and High Availability

Photo by panumas nikhomkhai on Pexels

As your infrastructure grows, a single Proxmox server becomes a single point of failure. Clustering solves this by combining multiple Proxmox servers into a unified pool of resources that can share VMs, storage, and management — and when combined with High Availability (HA), services automatically survive hardware failures. In this lesson, we'll set up a cluster and configure HA from scratch.

What is a Proxmox Cluster?

A Proxmox cluster is a group of Proxmox servers (nodes) that are managed as a single entity. In a cluster, you can:

• Manage all nodes from a single web interface (access any node's UI and see all nodes)
• Migrate running VMs between nodes with no downtime (live migration)
• Share storage across nodes
• Configure High Availability for automatic failover
• Monitor all nodes' health from one dashboard

Cluster Requirements

• Minimum nodes: 3 for production (provides quorum — the ability to make decisions even if one node fails). You can create a 2-node cluster for testing, but it's not recommended for production because losing one node means losing quorum.
• Network: A dedicated network for cluster communication is recommended. A simple Gigabit switch works — you don't need 10GbE for small clusters.
• Shared storage: For live migration and HA, you need storage accessible from all nodes. Options: NFS share, iSCSI target, Ceph (distributed storage built into Proxmox), or ZFS replication (for non-shared-storage setups).
• Same Proxmox version: All nodes must run the same major Proxmox version.
• Same CPU architecture: All nodes should have compatible CPUs for migration (Intel/AMD mixing works if you use the x86-64 CPU type).

Step 1: Create the Cluster

On your first Proxmox node:

1. Go to Datacenter then Cluster.
2. Click "Cluster Join Information" then click "Create Cluster."
3. Enter a cluster name (e.g., beawit-cluster).
4. Select the network interface for cluster communication (use a dedicated NIC if available).
5. Click "Create."
6. Once created, click "Cluster Join Information" again and copy the provided command/token.

Step 2: Join Additional Nodes

On each additional node:

1. Log into the new node's web interface.
2. Go to Datacenter then Cluster.
3. Click "Join Cluster."
4. Paste the join information from the first node.
5. Enter the root password of the first node.
6. Click "Join Cluster."
7. Wait for the node to restart and join the cluster. This typically takes 1-2 minutes.

Repeat for each additional node. You can add nodes at any time — the cluster doesn't need to be built all at once.

Step 3: Configure Shared Storage

For live migration and HA, all nodes need access to the same storage:

Option A: NFS (Simplest):

1. Set up an NFS server (can be a VM or a NAS device like TrueNAS).
2. In Proxmox, go to Datacenter then Storage then Add then NFS.
3. Enter the NFS server IP, export path, and select content types (Disk image, Container template, ISO image).

Option B: Ceph (Built-in, distributed):

Ceph is a distributed storage system that runs on your Proxmox nodes themselves. It uses the local disks on each node to create a shared, redundant storage pool:

1. Go to Datacenter then Ceph then Install and click "Install" on each node.
2. After installation, go to each node then Disks then Ceph then Create OSD for each disk you want to contribute to the Ceph pool.
3. Go to Datacenter then Ceph then Pools then Create and set up a pool (replicated with size 3 for redundancy).
4. The Ceph RBD storage appears in Proxmox and can be used for VM disks.

Ceph requires at least 3 nodes with dedicated OSD disks (preferably SSD/NVMe). It's powerful but adds complexity — use NFS if you're just getting started.

Step 4: Enable High Availability

Once your cluster and shared storage are configured, setting up HA is simple:

1. Go to Datacenter then HA then Groups and create a group (e.g., "all-nodes" containing all your nodes).
2. Go to Datacenter then HA then Add.
3. Select the VM or container you want to protect.
4. Set:
   • Group: Select your HA group.
   • Max Restart: 1 (number of restart attempts on the same node before trying another).
   • Max Relocate: 1 (number of nodes to try before giving up).
5. Click "Create."

Now, if the node running this VM fails, Proxmox will automatically start the VM on another node in the cluster. The process typically takes 1-3 minutes from failure to service restoration.

Step 5: Live Migration

Even without HA, you can manually migrate VMs between nodes for maintenance:

1. Right-click a VM and select "Migrate."
2. Select the target node.
3. Choose whether to migrate with local storage (requires the storage to be replicated or shared).
4. Click "Migrate."

Live migration copies the VM's memory state to the target node while the VM keeps running. There's a brief pause (usually under 1 second) during the final cutover. Users typically don't notice.

Quorum and Corosync

Proxmox clusters use Corosync for cluster communication and quorum. Quorum means a majority of nodes agree on cluster state. With 3 nodes, you need 2 to agree. If 2 nodes fail, the 1 remaining node loses quorum and can't make changes (to prevent split-brain scenarios). Proxmox automatically handles this — but understand that odd numbers of nodes (3, 5, 7) are recommended for exactly this reason.

Key Takeaways

• Clustering combines multiple Proxmox servers into one managed entity
• 3 nodes minimum for production — provides quorum and HA capability
• Shared storage (NFS or Ceph) enables live migration and HA
• HA automatically restarts VMs on another node if the current node fails
• Live migration moves running VMs between nodes with sub-second downtime