Views access time out during rebalance

Hi,
we have a very serious problem with our CB cluster. we’re using CB 3.0.1 (CE) and java client 1.2.3.
It’s a 3 node CB cluster with 4 buckets (all of type: couchbase).
One of the buckets has 4 views in 2 design documents (2 views each). This is a rapidly growing bucket (a lot of inserts and a lot of updates).
The workset of our largest bucket (the one with the views) is ~11M documents. Disk usage: 15GB, RAM usage: 3.5GB.

The problem is that when a rebalance takes place, we get a time-out every time we try to access a view. inserting/reading documents by key works fine.
we saw that the nodes use high level of CPU, so we quadrupled the amount of cores. this doesn’t help.

we also see that everytime a rebalance takes place CB performs indexing and compacting - does that make the indexes unavailable?
how can we monitor and fix the problem?

Thanks,
Asaf

I see the same issue, couchbase server 3.1 EE. When adding a new node to our 2 node cluster (so going from 2 nodes to 3 nodes) the views are constantly re-indexing. This makes our online portal unusable during rebalance because it relies on views.

I thought that couchbase rebalance was supposed to be able to be done live, is there some other way we should be doing it aside from adding a node and clicking rebalance? We are using production views only and have about 5 design documents.

hi @alexegli, what is the error you are getting from views during rebalance?

Hi Cihangirb,
For a 3 node couchbase cluster having 1 replica configured for index. In a scenario when one of the node failover, all the replica copies of index is available for view query.
But, what we are observing is all indexes are rebuilding when node gets fail-over. Views queries with stale=false are getting timed out.
This is causing application outage.

Request your help.

Regards
Pratheej