How many nodes could be removed from the cluster at once


#1

Hello guys.

I have made a cluster for a product, using 10 nodes.
However, inflowing traffic to the cluster was not so much as I thought, actually.
So I decided to shrink the cluster from 10 to 5 nodes.

Here is, I found a way, what I’m supposed to do is to remove a node using the function ‘Remove’ in the couchbase console provided.

But I wonder

  1. how many nodes I could remove from the cluster at once.
  2. is it related to the value of replicas? (in my case, replicas is 2)
    (In details, I mean, is it available to remove nodes over the value of replicas?)
  3. what point for performance should I take care of if several nodes would be removed. (Let’s suppose to remove 3~4 nodes at once)

Thank you in advance.


#2

I think you should remove nodes one by one by rebalancing.

Replica of 2 means you need to 3 data services in the cluster.

Secondly math should be done before scaling down. So that one node failure should not invoke cascading nodes failure.

Lets say every node is having 80% node and you have 5 nodes, so if one fails others will have 100% load. In this case now if one more node fails then every node will start failing like by bicycles in stands.


#3

I understand you are considering cascading failures, right?

But what if I’m sure the cascading failures would not going to be happened?
In addition, what if the math you’re worried about has no problem due to small traffic and using resources.
When assuming above is right, is it no problem to be removed several nodes functionally?

Plus, According to a link below,

It seems to be that the ‘Remove’ function is guaranteed by somehow internal process of couchbase. If it is, is it ok to be removed several nodes at once? (removing 3~5 nodes at once although replicas is 2) I mean, does removing multiple nodes has no any connection whatever with the value of replicas?

So then I ultimately wonder whether I could remove multiple nodes over than the value of replicas when the performance issue has no problem