RequestCancelledException after connection idle


#1

Have a service that accesses Couchbase Community 4.0 using the 2.2.6 java client. When the service is left unused overnight and service is accessed (couchbase queried), the below exception occurs a few times before it works normally. I’m trying to figure out if this is expected and I need to catch and handle these exceptions or if this is some sort of firewall or general configuration issue.

com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:478)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:441)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:416)
at com.couchbase.client.deps.io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:144)
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:223)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:950)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)

#2

The Java client is up to release 2.3.2, so I’m not sure what version you mean with 2.6.

That looks like the operation was canceled at some other level, but wouldn’t be logged by this thread handling IO. I am sure @daschl can confirm, but I believe the cancellation might be from any other thread, including your own. Typically this happens if a connection needs to be rebuilt owing to some kind of network error and we can’t automatically determine the status of this particular request.

That network error may have it’s own log messages though.

I suspect you’re looking at effect in this log message, not cause.


#3

Typo, client is 2.2.6. Thanks for suggestion I’ll look into it.


#4

A firewall was determined to be the root cause of this error. I would have thought the Socket Keepalive Interval which defaults to 30 seconds would prevent firewall from killing these stale connections but apparently not.


#5

Thanks for letting us (and the community) know. This has come up very rarely over the years. Usually it’s a firewall that will purge “old” connections or consciously ignores the keepalive.