After restarting a server in Couchbase Cluster Getting Exception : Message: Could not dispatch request, cancelling instead of retrying

java
sdk

#1

Dear Team,
I am getting the below Exception
Scenario :

  1. Couchbase cluster is up with 4 nodes.
  2. All the application and up and running with no errors.
  3. Remove one node from the coucbase cluster for maintenance work and add it to cluster after the work.
  4. All the application during the maintenance work and after throwing the below exception:
    class com.couchbase.client.core.RequestCancelledException;
    Message: Could not dispatch request, cancelling instead of retrying. - {
    com.couchbase.client.core.retry.RetryHelper.retryOrCancel(RetryHelper.java: 51),
    com.couchbase.client.core.service.AbstractPoolingService.dispatch(AbstractPoolingService.java: 60),
    com.couchbase.client.core.service.AbstractDynamicService.send(AbstractDynamicService.java: 129),
    com.couchbase.client.core.node.CouchbaseNode.send(CouchbaseNode.java: 197),
    com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java: 210),
    com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java: 76),
    com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java: 128),
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1142),
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 617),
    com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java: 137),
    java.lang.Thread.run(Thread.java: 745)
    }

Couchbase SDK Version : 2.2.5
Couchbase Server Version :
Version: 4.6.1-3652 Enterprise Edition (build-3652)
Cluster State ID: 059-040-426

My Fetch Code for Async:

/** Fetching the Documents for the given Id's in the List **/
                List<JsonDocument> responseJsonDocuments = Observable.from(couchBaseInputObject.getIds())
                    .flatMap(id -> bucket.async().get(id).onErrorResumeNext(onGetError(bucket, id)))
                    .toList().toBlocking().single();

I am using the default CouchbaseEnvironment

My Queries :

  1. Why we need a application restart when we remove and add a node from the cluster?
  2. How can we fix this in order to avoid the exceptions and restarts when a node from the couchbase is down/removed?