vBucket replicas not promoted to master on node autofailover

We are running Couchbase server community 2.2 with 6 nodes and Java client 1.4. Last night we had one of the nodes go down and auto-failover triggered. We started seeing the following error (with many different keys and vbuckets) until the re-balance completed.

WARN com.couchbase.client.vbucket.VBucketNodeLocator: The key 0b4366a7-35f5-4eb8-98e2-2ce531469915 pointed to vbucket 452, for which no server is responsible in the cluster map.This can be an indication that either no replica is defined for a failed server or more nodes have been failed over than replicas defined.

Trying to figure out if this is because Couchbase failed to promote replicas to master or if the client failed to update it’s cluster map to point to the new masters. Any suggestions on what could cause this or how I can investigate this further?

Thanks.

Hi, how many replicas the bucket have? And has it occur that some nodes down and hadn’t do rebalance before you run your test?

@hubo3085632, I have faced the same issue during maintenance activity.

Cluster Details:
3 node cluster with 1 replica.

Scenario:
First, Node 1 went down and failover triggered manually from GUI. Node 1 came back and delta recovery triggered but Rebalance didn’t trigger. The same procedure happens with Node 2 but this time Rebalance has been triggered and we start getting the same error until the rebalance completed.

Error:
The key ****** pointed to vbucket 209, for which no server is responsible in the cluster map. This can be an indication that either no replica is defined for a failed server or more nodes have been failed over than replicas defined.

Any help would be appreciated that what could cause this issue.