When nodes join a cluster, I have some logic that tries to choose between using couchbase-cli rebalance and couchbase-cli server-add to join the cluster. If my X node is the last one, it will add itself via the rebalance call as all nodes have now joined. However sometimes this node may actually join earlier than expected while waiting / pinging the cluster to come online etc. When this happens the nodes that go after join the cluster but don’t rebalance.
So my question is around if I added all nodes via the rebalance call, are they queued, or does any subsequent one stop the current rebalance and start again to include the new node. Just want to get an idea of how it works.
When you add a node to the cluster the node is added, but put in pending add state. This means if the cluster were to crash after you added a node it would still be there when you restarted the cluster. When you rebalance the cluster then it will move some of the data from the existing nodes in the cluster onto the new node.
What I would recommend is always adding servers with the server-add command. Once you are ready to rebalance data onto the new nodes run the rebalance command.
As a side note, we will remove the --server-add flag from the rebalance command in the future and you will need to explicitly add nodes using the server-add command.
Thanks for the information @mikew. I’ll remove my rebalance call if it’s the last node in favour of running it manually. It was a bit flaky anyway, race conditions etc.