How to stop the cluster safely

#1

Hi,

We got a 4.5 cluster of about 50 nodes, it is quite full with data. We’re going to replace some core switches in our local network, so connectivity between all nodes will be disrupted for some minutes or hours. We’re afraid the cluster can become unusable, so we’d like to gracefully stop it before we break the network.

Is there a way to stop all nodes “at where they are” so they don’t try to elect new master, failover all others, rebalance etc?

Thanks,
Kirill

#2

Hi Kirill

Best way is to:

  1. Disable auto-failover to prevent a node getting accidentally failed over while you’re shutting down or restarting the nodes in the cluster. (Can be done in the UI or via the CLI https://docs.couchbase.com/server/4.5/rest-api/rest-cluster-autofailover-intro.html)

  2. Shutdown each node. No problem doing this in parallel. https://docs.couchbase.com/server/4.5/install/startup-shutdown.html.

-dave

1 Like
#3

Thanks Dave,

We did like you suggested and it generally worked, the cluster has booted up after shutdown and all data seems to be in place.

The only issue we got is some nodes did not terminate for about 20 minutes, they were rendered as “pend” in Web UI and “service couchbase-server status” was showing the service was still running. As our maintenance window was running out we had to force kill such nodes (kill -9 respective processes).

How quick does a node typically shut down (given there’s no requests coming in to it)?

Thanks,
Kirill