Signal disconnected


#1

I have installed - Couchbase server - couchbase-server-community_4.5.1-ubuntu14.04_amd64 and using Couchbase-Java-Client-2.2.8 to connect.

I have created a cluster with two nodes and both are up and running. I am getting connection timed out. PFB the code used and the stack-trace:

Code

CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder().connectTimeout(50000).build();
		Cluster cluster = CouchbaseCluster.create(env, "IP address of 1st node", "IP address of 2nd node");
		Bucket bucket = cluster.openBucket("bucket_name","*****");
		JsonObject person = JsonObject.create();
		person.put("firstname", "Sweta");
		person.put("lastname", "Kedia");		
		JsonDocument document = JsonDocument.create("sweta", person);		
		bucket.upsert(document);
		System.out.println(bucket.get("sweta").content());

Console output:

Sep 26, 2017 1:51:08 PM com.couchbase.client.core.CouchbaseCore <init>
INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=50000, disconnectTimeout=25000, dnsSrvEnabled=false}
Sep 26, 2017 1:51:10 PM com.couchbase.client.core.node.CouchbaseNode signalConnected
INFO: Connected to Node ec2-13-56-229-101.us-west-1.compute.amazonaws.com
Sep 26, 2017 1:51:10 PM com.couchbase.client.core.node.CouchbaseNode signalConnected
INFO: Connected to Node ec2-54-183-207-37.us-west-1.compute.amazonaws.com
Sep 26, 2017 1:51:16 PM com.couchbase.client.core.node.CouchbaseNode <init>
WARNING: DNS Reverse Lookup of 172.31.25.65 is slow, took 4500ms
Sep 26, 2017 1:51:20 PM com.couchbase.client.core.node.CouchbaseNode <init>
WARNING: DNS Reverse Lookup of 172.31.27.0 is slow, took 4500ms
Sep 26, 2017 1:51:20 PM com.couchbase.client.core.config.DefaultConfigurationProvider$8 call
INFO: Opened bucket StellarA
Sep 26, 2017 1:51:20 PM com.couchbase.client.core.node.CouchbaseNode signalDisconnected
INFO: Disconnected from Node ec2-54-183-207-37.us-west-1.compute.amazonaws.com
Sep 26, 2017 1:51:20 PM com.couchbase.client.core.node.CouchbaseNode signalDisconnected
INFO: Disconnected from Node ec2-13-56-229-101.us-west-1.compute.amazonaws.com
Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
Sep 26, 2017 1:51:21 PM com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise notifyListener0
WARNING: An exception was thrown by com.couchbase.client.core.endpoint.AbstractEndpoint$2.operationComplete()
rx.exceptions.OnErrorNotImplementedException: connection timed out: /172.31.25.65:11210
	at rx.Observable$26.onError(Observable.java:7955)
	at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
	at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
	at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
	at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
	at rx.internal.operators.OperatorSingle$ParentSubscriber.onError(OperatorSingle.java:139)
	at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onError(OperatorTakeLastOne.java:164)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
	at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:810)
	at rx.internal.operators.OperatorSingle$ParentSubscriber.onError(OperatorSingle.java:139)
	at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onError(OperatorTakeLastOne.java:164)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
	at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:810)
	at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
	at rx.internal.operators.OperatorSingle$ParentSubscriber.onError(OperatorSingle.java:139)
	at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onError(OperatorTakeLastOne.java:164)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
	at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:810)
	at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
	at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
	at rx.internal.operators.OperatorSingle$ParentSubscriber.onError(OperatorSingle.java:139)
	at rx.internal.operators.OperatorTakeLastOne$ParentSubscriber.onError(OperatorTakeLastOne.java:164)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
	at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:810)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:227)
	at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)
	at com.couchbase.client.core.endpoint.AbstractEndpoint$2.operationComplete(AbstractEndpoint.java:311)
	at com.couchbase.client.core.endpoint.AbstractEndpoint$2.operationComplete(AbstractEndpoint.java:277)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:604)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:564)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:416)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
	at com.couchbase.client.core.endpoint.kv.KeyValueFeatureHandler$1.operationComplete(KeyValueFeatureHandler.java:140)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:568)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:416)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
	at com.couchbase.client.core.endpoint.kv.KeyValueAuthHandler$3.operationComplete(KeyValueAuthHandler.java:316)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:604)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:564)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:425)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:222)
	at com.couchbase.client.deps.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at com.couchbase.client.deps.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:374)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.couchbase.client.deps.io.netty.channel.ConnectTimeoutException: connection timed out: /172.31.25.65:11210
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:220)
	... 7 more

Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][QueryEndpoint]: Socket connect took longer than specified timeout.
Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][ViewEndpoint]: Socket connect took longer than specified timeout.
Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][KeyValueEndpoint]: Socket connect took longer than specified timeout.
Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][QueryEndpoint]: Socket connect took longer than specified timeout.
Sep 26, 2017 1:51:21 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
WARNING: [null][ViewEndpoint]: Socket connect took longer than specified timeout.
java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
	at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:359)
	at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:354)
	at com.src.couchbase.Sample.main(Sample.java:31)
Caused by: java.util.concurrent.TimeoutException
	... 4 more

Note:

I have allowed connections on following ports in both the nodes:
4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11209 to 11211,
11214, 11215, 18091 to 18093, and from 21100 to 21299.

The same code is working fine if I am using a single node (installed in another server).


#2

I resolved the timeout issue using the socketConnectTimeout param in env by modifying following in my code:

CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder().connectTimeout(50000).socketConnectTimeout(50000).build();

Now I am getting following error:

Sep 26, 2017 2:22:44 PM com.couchbase.client.core.CouchbaseCore
INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile=‘null’, sslKeystorePassword=‘null’, queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=50000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=50000, disconnectTimeout=25000, dnsSrvEnabled=false}
Sep 26, 2017 2:22:47 PM com.couchbase.client.core.node.CouchbaseNode signalConnected
INFO: Connected to Node ec2-13-56-229-101.us-west-1.compute.amazonaws.com
Sep 26, 2017 2:22:47 PM com.couchbase.client.core.node.CouchbaseNode signalConnected
INFO: Connected to Node ec2-54-183-207-37.us-west-1.compute.amazonaws.com
Sep 26, 2017 2:22:52 PM com.couchbase.client.core.node.CouchbaseNode
WARNING: DNS Reverse Lookup of 172.31.25.65 is slow, took 4726ms
Sep 26, 2017 2:22:58 PM com.couchbase.client.core.node.CouchbaseNode
WARNING: DNS Reverse Lookup of 172.31.27.0 is slow, took 5937ms
Sep 26, 2017 2:22:58 PM com.couchbase.client.core.config.DefaultConfigurationProvider$8 call
INFO: Opened bucket StellarA
Sep 26, 2017 2:22:58 PM com.couchbase.client.core.node.CouchbaseNode signalDisconnected
INFO: Disconnected from Node ec2-54-183-207-37.us-west-1.compute.amazonaws.com
Sep 26, 2017 2:22:58 PM com.couchbase.client.core.node.CouchbaseNode signalDisconnected
INFO: Disconnected from Node ec2-13-56-229-101.us-west-1.compute.amazonaws.com
java.lang.RuntimeException: java.util.concurrent.TimeoutException
** at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)**
** at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:359)**
** at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:354)**
** at com.src.couchbase.Sample.main(Sample.java:32)**
Caused by: java.util.concurrent.TimeoutException
** … 4 more**