A large e-commerce company uses Vault to manage database credentials for microservices. They have a Vault cluster of 5 nodes using Integrated Storage (Raft). To increase capacity, they add a sixth node to the cluster. Shortly after, they notice intermittent 'no leader' errors in the Vault logs, and some clients experience failures when reading secrets. The cluster was functioning correctly before the addition. What is the most likely cause and the recommended action?
Trap 1: The cluster's Raft protocol requires all nodes to be voters, but…
New nodes are voters by default. Even if the new node were a non-voter, the existing 5 voters would still have an odd number; the problem is even-numbered voters.
Trap 2: The new node has a different storage performance, causing it to lag…
While performance can cause issues, the sudden 'no leader' errors after adding a node are more characteristic of even-number quorum problems than a slow node.
Trap 3: The new node was not properly joined to the cluster and is running…
If the node were standalone, it wouldn't affect the existing cluster's leader election. The entire cluster is experiencing 'no leader' errors.
- A
The cluster now has an even number of nodes, which can cause split-brain scenarios and leader election issues. Recommended action: remove the new node to revert to 5 nodes, or add a seventh node to make it odd.
Raft performs best with an odd number of nodes to avoid split-brain. With 6 nodes, quorum is 4, and a partition could lead to no leader.
- B
The cluster's Raft protocol requires all nodes to be voters, but the new node is a non-voter. Recommended action: promote the new node to voter.
Why wrong: New nodes are voters by default. Even if the new node were a non-voter, the existing 5 voters would still have an odd number; the problem is even-numbered voters.
- C
The new node has a different storage performance, causing it to lag behind and not participate in consensus. Recommended action: replace the new node with a faster instance.
Why wrong: While performance can cause issues, the sudden 'no leader' errors after adding a node are more characteristic of even-number quorum problems than a slow node.
- D
The new node was not properly joined to the cluster and is running as a standalone server. Recommended action: re-run the join command and ensure the node is part of the cluster.
Why wrong: If the node were standalone, it wouldn't affect the existing cluster's leader election. The entire cluster is experiencing 'no leader' errors.