Index initial build times taking forever since cluster rebalance


#1

I have a small bucket with a little over 6000 documents in it. Index creation used to take no more than 5-10 seconds.

This afternoon, I removed each node from the cluster (2 total), one at a time. Doing a rebalance each time. Ive done this many times before.

However this time, once both nodes were connected again, any index creation query takes ages to complete. Several minutes. I tried restarting the nodes, but nothing has fixed the issue. I tried creating indexes with each node isolated from each other and still same issue. I compacted the bucket, in case that would solve it, but still, same problem.

Im running 4.5.1 CE. Any ideas? Nothing in the logs is hinting to an issue.


#2

Just an update on this. I had to reinstall couchbase on both nodes for the slowness issues to go away. Any insight on the matter would be useful, as this seems to be a major detriment to removing and re-adding nodes to a cluster. In a production environment, it would be unacceptable.


#3

@kev, without looking at the logs it is difficult to say what went wrong. Usually you would see logs complaining about port connectivity issues or lack of file handles.

I assume the indexes in question are GSI indexes. Rebalance is done by production customers all the time without any downtime or slowness issue you mentioned.


#4

I agree with you. I can only imagine something went wrong that normally does not happen. All I know is that a full re-install solved the issue right away.