Proxmox: Remove a Node From a Cluster Without Breaking Quorum

Migrate the guests off, power the node down permanently, then run pvecm delnode nodename from a remaining node. The order is not optional — removing a node that is still running and reachable can corrupt the cluster configuration, and a removed node must never be powered back on with its old cluster config.

Proxmox clustering is built on Corosync, which maintains a shared configuration filesystem across nodes and requires a quorum to allow changes. Removing a node touches both, and the failure modes are unusually unforgiving — a cluster that loses quorum goes read-only, and a node that rejoins with stale configuration can cause genuine damage.

The procedure is short. Reading it in full before starting is the sensible order.

Table of contents

• Before you remove anything
• Power the node off first
• Running the removal
• Cleaning up what delnode leaves behind
• Rejoining a node, and why it needs a reinstall
• How this fits the rest of the stack
• FAQ

Before you remove anything

Move the guests. Removing a node does not migrate its VMs and containers — they simply become inaccessible.

# on the node being removed
qm list
pct list

# migrate each one to a remaining node
qm migrate 101 node2 --online
pct migrate 201 node2

--online performs a live migration where shared storage allows it. Without shared storage the guest must be stopped, and the disk copied, which takes as long as the disk is large.

Check quorum arithmetic. A cluster needs more than half its votes to function. Going from 3 nodes to 2 leaves you needing both remaining nodes for quorum — lose either and the cluster goes read-only.

pvecm status
# Expected votes:   3
# Highest expected: 3
# Total votes:      3
# Quorum:           2

Two-node clusters are a genuinely awkward configuration. If that is where you are heading, plan for a QDevice — a small external vote provider that lets a two-node cluster survive one failure:

apt install corosync-qdevice        # on both nodes
pvecm qdevice setup <qdevice-ip>

Note the node’s details. Take a copy of /etc/pve/corosync.conf and record the node’s name and ID. If something goes wrong, you will want them.

Power the node off first

This is the step that is most often done in the wrong order, and the documentation is emphatic about it.

The node being removed must be powered off and must not come back on the cluster network with its existing configuration. If it does, it still believes it is a cluster member, still runs Corosync, and can rejoin the ring in a state the remaining nodes have already removed. That produces a genuinely broken cluster that is difficult to repair.

# on the node being removed, once guests are migrated
shutdown -h now

Confirm from a remaining node that it is offline before proceeding:

pvecm nodes
# Membership information
#     Nodeid  Votes Name
#          1      1 node1 (local)
#          2      1 node2
#          3      1 node3    <- should now be absent or offline

If you intend to reuse the hardware, plan on a clean reinstall of Proxmox rather than trying to unpick the cluster configuration. That is the officially recommended path and it is faster than the alternative.

Running the removal

# from any REMAINING node, not the one being removed
pvecm delnode node3

You may see this, and it is not a failure:

could not kill node (error = cs_err_not_exist)

That is Corosync reporting it could not signal a node that is already off. The removal itself proceeds. The Proxmox wiki calls this out explicitly as safe to ignore.

Verify afterwards:

pvecm status
pvecm nodes
cat /etc/pve/corosync.conf | grep -A3 "node {"

The expected vote count should have dropped, and the removed node should be absent from all three.

If the cluster has lost quorum and is read-only, you can temporarily lower the expected votes to regain write access — but only when you are certain the missing nodes are genuinely gone:

pvecm expected 2

This is a recovery tool, not a configuration. Setting expected votes below the real membership while nodes are merely unreachable is how split-brain happens.

Cleaning up what delnode leaves behind

pvecm delnode removes cluster membership but leaves some per-node state in the shared filesystem. Tidy it up on a remaining node:

ls /etc/pve/nodes/
# node1  node2  node3    <- node3 directory still present

rm -rf /etc/pve/nodes/node3

Check that directory is genuinely empty of guest configs before removing it. If it still contains files under qemu-server/ or lxc/, a guest was never migrated and you are about to delete its configuration.

ls -R /etc/pve/nodes/node3/

Also worth reviewing after any node removal:

• HA rules that reference the removed node — they persist and will fail. Check under Datacenter → HA.
• Backup jobs targeting it, under Datacenter → Backup.
• Storage definitions restricted to that node, in /etc/pve/storage.cfg.
• Ceph, if in use — a Ceph node needs its OSDs and monitors removed through Ceph’s own procedure before the Proxmox node removal, and that is a separate job with its own data-safety implications.
• SSH known_hosts entries, which otherwise produce warnings later.

Rejoining a node, and why it needs a reinstall

The clean answer is: reinstall Proxmox on the machine, then join it fresh. This is what the documentation recommends and it is genuinely the fastest route.

# on the freshly installed node
pvecm add <ip-of-existing-cluster-node>

The reason a reinstall is preferred: a previously-clustered node retains Corosync configuration, cluster certificates, and its old node ID in /etc/pve. Joining with that residue produces conflicts that surface later as authentication failures or filesystem errors, and diagnosing them is worse than a reinstall.

If a reinstall is genuinely impossible, the manual cleanup is roughly:

systemctl stop pve-cluster corosync
pmxcfs -l                                    # start the filesystem in local mode
rm /etc/pve/corosync.conf
rm -rf /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster

Treat that as a last resort with a full backup taken first. It is documented, and it is also the kind of procedure where a single missed file causes problems weeks later.

Reusing the same node name is fine after a clean reinstall, provided the removal on the cluster side completed and the /etc/pve/nodes/ directory was cleaned.

How this fits the rest of the stack

Cluster membership changes are one of the few routine operations where the order of steps genuinely determines whether you have a working system afterwards. Quorum arithmetic, powered-off nodes and leftover configuration are all things you have to hold in your head, and the cluster will not stop you getting them wrong.

That is the ongoing cost of running your own virtualisation layer, and it is a reasonable cost when you need the control. When you do not, the same workload as managed services removes the layer entirely — RunxBuild deploys applications from a GitHub repository onto a runtime with autoscaling between plans you choose, so scaling up or down is a plan change rather than a node join. The RunxBuild hosting calculator shows what a service, database and storage come to together if you want to compare against the hardware.

Useful related references:

• Cron in Node.js: node-cron, node-schedule, and System Cron
• ioredis vs node-redis: Which to Ship With
• Deploy a Node.js API for Free on RunxBuild
• Services on RunxBuild

FAQ

How do I remove a node from a Proxmox cluster?

Migrate all guests off it, power the node down permanently, then run pvecm delnode nodename from a remaining node. Afterwards remove the leftover /etc/pve/nodes/nodename directory. The node must be off before you run delnode.

What does could not kill node cs_err_not_exist mean?

Corosync could not signal a node that is already powered off. It is not an error in the removal — the node is still removed. The Proxmox documentation flags this message explicitly as safe to ignore.

Can I rejoin a removed node to the cluster?

Only after reinstalling Proxmox on it. A previously clustered node keeps Corosync config, certificates and its old node ID, and joining with that residue causes conflicts that surface later. Reinstalling is both the recommended path and the faster one.

What happens to quorum when I remove a node?

Expected votes drop, and a cluster needs more than half to allow changes. Going from three nodes to two means both remaining nodes are required — losing either makes the cluster read-only. Consider a QDevice to give a two-node cluster an external tie-breaking vote.

Why must the node be powered off before removal?

If it stays on the cluster network with its old configuration, it still believes it is a member and can rejoin a ring the other nodes have already removed it from. That produces an inconsistent cluster state which is considerably harder to repair than the removal was.