Problem with adding and removing nodes to cluster


#1

Hello experts.
I’m using couchbase 3 cluster.
I had 4 aws r3.large nodes in a cluster.
They used the wrong data path so I wanted to fix it.
I followed this documentation and raise a new server, used the CLI to init with new data path and then added it to the cluster, removed one of the existing nodes and pressed rebalance.
The servers stayed in the pending to rebalance and there was a warning:

Fail Over Warning: Rebalance recommended, some data does not have the desired replicas configuration!

I tried to roll back and remove the new node but I can’t do it.
I’m removing it and press rebalance. I get the message rebalance failed, see logs for more details.
I’m not sure which log should I look into and didn’t get much data from the ones I looked in (debug.log).

I thought, by browsing for help, that couchbase has a problem with odd number of nodes so I’ve added another one.
The same error happen.

I now have a cluster of 6 nodes that in “up” (green) status but two (the new ones) of them don’t have any items or replicas in them so they don’t really function.
There are no pending to rebalance servers but the warning of “rebalance recommended” still appear.

Your help will be appreciated.


#2

I found the problem.
The old nodes had a directory path privileges problem.
sudo chown -R couchbase:couchbase on the data path folder fixed the problem.

Thanks


#3

Gluz,
Thanks for posting the answer. This is one of the common things to look at when data won’t replicate, new nodes don’t rebalance, etc.

Austin