CouchbaseCluster.create() could not connect to endpoint which is inside Amazon VPC

#1

My current setting is 3 nodes in same Amazon VPC/Couchbase Cluster, and everything works well such as my app server in same VPC can connect well, web interface has never had problem, and ssh to any node never has issue either.
However, whenever I run my app server that is not in same VPC, the server cannot connect to any node and keeps retrying until TimeoutException is thrown, and no logs on nodes’ side about this connecting attempt at all.
I also had tried a load balancer but still in vain.Any hint or suggestions are very appreciated!

#2

Can you please share the logs and your connect code? Otherwise it’s hard to help - thanks!

#3

@daschl Thanks!!

Jan 14, 2015 12:21:22 PM com.couchbase.client.core.CouchbaseCore <init>
    INFO: CoreEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=8, computationPoolSize=8, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, packageNameAndVersion=couchbase-java-client/2.0.1 (git: 2.0.1)}
    Jan 14, 2015 12:21:23 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Connected to Node ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:23 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Connected to Node ec2-54-187-7-64.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:23 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Connected to Node ec2-50-112-162-12.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:23 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 2ms: 
    java.net.ConnectException: Connection refused: /170.x.x.x:11210
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208)
    	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    	at java.lang.Thread.run(Thread.java:745)
    
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call
    INFO: Opened bucket default
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Disconnected from Node ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Disconnected from Node ec2-54-187-7-64.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.node.CouchbaseNode$5 call
    INFO: Disconnected from Node ec2-50-112-162-12.us-west-2.compute.amazonaws.com
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call
    INFO: Opened bucket default
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call
    INFO: Opened bucket default
    Jan 14, 2015 12:21:24 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][ViewEndpoint]: Could not connect to endpoint, retrying with delay 2ms: 
    java.net.ConnectException: Connection refused: /170.x.x.x:8092
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208)
    	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    	at java.lang.Thread.run(Thread.java:745)
    
    Jan 14, 2015 12:21:25 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][ConfigEndpoint]: Could not connect to endpoint, retrying with delay 2ms: 
    java.net.ConnectException: Connection refused: /170.x.x.x:8091
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208)
    	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    	at java.lang.Thread.run(Thread.java:745)
    
    Jan 14, 2015 12:21:26 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][ConfigEndpoint]: Could not connect to endpoint, retrying with delay 4ms: 
    java.net.ConnectException: Connection refused: /170.x.x.x:8091
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208)
    	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    	at java.lang.Thread.run(Thread.java:745)
    
    Jan 14, 2015 12:21:27 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 4ms: 
    java.net.ConnectException: Connection refused: /170.x.x.x:11210
    	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    	at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208)
    	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    	at java.lang.Thread.run(Thread.java:745)
       .
       .(repeated logs eliminated due to limit on number of characters for post here)
       .
       .
 
   
    Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException
    	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:482)
    	at rx.observables.BlockingObservable.single(BlockingObservable.java:349)
    	at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:579)
    	at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:564)
    	at couchdoc_bulkdel_tool.CouchTool.main(CouchTool.java:53)
    Caused by: java.util.concurrent.TimeoutException
    	at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:169)
    	at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:42)
    	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:45)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    	at java.lang.Thread.run(Thread.java:745)
#4

More info: I encountered this problem when I wrote some simple code in Java to bulk delete documents, and I ended up using curl in bash script with the official REST API to successfully delete my docs.

#5

My connection code(I tried many different ways to use CouchbaseCluster.create(), such as with/without environment, single ip, list of ip and default/my own environment, but same result):

public static void main(String[] args) {
		// TODO Auto-generated method stub
		//DefaultCouchbaseEnvironment.Builder couchBaseEnv = new DefaultCouchbaseEnvironment.Builder();
		//couchBaseEnv.queryEnabled(true);
		//couchBaseEnv.ioPool(new NioEventLoopGroup());
		//couchBaseEnv.scheduler(Schedulers.immediate());
		//couchBaseEnv.build();		
		ArrayList<String> nodes = new ArrayList<String>();
		
		nodes.add("pub.lic.ip.ofmyinstance");
		nodes.add("pub.lic.ip.ofmyinstance");
		nodes.add("pub.lic.ip.ofmyinstance");
		CouchbaseCluster cluster=CouchbaseCluster.create(nodes);
#6

I compared the log I just pasted here with the log from my working server within the VPC, and I see that [null] value should be the ip or DNS of the Couchbase node to be connected:

Bad log:

com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
    WARNING: [null][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 2ms:

Good:

2015-01-14 21:41:46,625 [cb-io-1-1] DEBUG com.couchbase.client.core.endpoint.Endpoint- [ip-x-x-x-xx.us-west-2.compute.internal/x.x.x.x:11210][KeyValueEndpoint]:
#7

@nikkki from the tag I see that you’re not using 2.0.3 - can you retry with that? We had some fixes around connecting, so just to make sure its not already something that we fixed.

Connection refused points to me that there might be some network issues related - can you 1) check that you can ping all nodes from those addrs 2) telnet to 11210 to see if this port is also open? Also make sure that the nodes on how the cluster is configured maps 1:1 with the one you provide (so there is no accidental mix-up of internal and external ip addrs).

#8

@daschl I just tried telnet to 11210 on all my nodes and was able to connect without problem. And I’ll post the result of 2.0.3 and checking the cluster-node mapping later.