Netty IllegalReferenceCountException in Java client 2.7.9

Java SDK
1

We were on 2.7.4, but we upgraded to 2.7.9 to get a fix for: https://issues.couchbase.com/browse/JCBC-1333

We started running into…

com.couchbase.client.deps.io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
	at com.couchbase.client.deps.io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:90)
	at com.couchbase.client.deps.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:74)
	at com.couchbase.client.deps.io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:59)
	at com.couchbase.client.deps.io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:84)
	at com.couchbase.client.deps.io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:292)
	at com.couchbase.client.deps.io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:617)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.closeOutboundBufferForShutdown(AbstractChannel.java:627)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:620)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:893)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:313)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:847)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1266)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:770)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:762)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:743)
	at com.couchbase.client.deps.io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:770)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:762)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:743)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:533)
	at com.couchbase.client.deps.io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:358)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:770)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:762)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:743)
	at com.couchbase.client.deps.io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:770)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:762)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:35)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext$16.run(AbstractChannelHandlerContext.java:750)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

We were using 2.7.4 previously and didn’t seem to have this issue. I found a similar issue on StackOverflow: https://stackoverflow.com/questions/58018771/how-to-fix-these-warning-messages-com-couchbase-client-deps-io-netty-util-ill

2

@unhuman we need a little more help to track this down properly. The leak detector should provide more info around it - are you able to reproduce this outside of production as well? When does it happen? (i.e. regular operations, failover, rebalance, timeouts,…?)

From the stack trace it looks like a write operation failed and then it frees buffers which are already freed. Do you see anything else in the logs what could’ve gone wrong in the first place?

3

We’ve not been able to duplicate it locally so far, but it’s happening on multiple applications deployed.

One of the applications is a “consumer” which only does Get operations, yet still is having this issue. Nothing else in the logs we have indicates otherwise.

We’re continuing to work this. Thanks for the reply.