java.lang.RuntimeException: Could not decode snappy-compressed value

jda · May 6, 2019, 10:05am

I have an application on a web server where I use the Java SDK to connect to Couchbase. I had a few issues on the Couchbase server that I solved by removing one node from the cluster (2 nodes in total) and delete/re-create all indexes.

Now when I try to open my application I get this error:

Caused by: com.couchbase.client.core.CouchbaseException: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:369)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:811)
Caused by: java.lang.RuntimeException: Could not decode snappy-compressed value.
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:397)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:949)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:132)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:338)
	... 33 more
Caused by: com.couchbase.client.deps.org.iq80.snappy.CorruptionException: Invalid copy offset for opcode starting at 1
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.decompressAllTags(SnappyDecompressor.java:165)
	at com.couchbase.client.deps.org.iq80.snappy.SnappyDecompressor.uncompress(SnappyDecompressor.java:47)
	at com.couchbase.client.deps.org.iq80.snappy.Snappy.uncompress(Snappy.java:85)
	at com.couchbase.client.core.endpoint.kv.KeyValueHandler.handleSnappyDecompression(KeyValueHandler.java:395)
	... 36 more

I can query the data directly using the dashboard in the browser.

I have not set anything (on purpose) for “snappy”. The application worked prior to removing and readding the node. This could of course have been something “waiting to be triggered/activated”. I have restarted the database nodes (and the application server) - and rebalanced the cluster afterwards.

Any ideas as to how I can troubleshoot this?

Couchbase: Community Edition 6.0.0 build 1693
Java SDK: 2.7.4

jda · May 6, 2019, 9:46pm

Ok, not sure how I “broke” the database… It seems that it was somehow broken.

It was in my test environment - so I decided to flush the bucket and restore a new backup copy from the production environment. Seems that sorted the problem

unhuman · June 17, 2019, 5:50pm

This isn’t much of a solution, but we just encountered a similar issue. We got the same error and noticed metadata flags on the document to be: 33554438

When we recreated the document, it had flags: 33554432

I couldn’t identify any Couchbase documentation with a full list of flags.

It’s important to note that the original document appeared in the UI as valid JSON, but obviously the application/client couldn’t retrieve it.

As a result of this (or something else) we wound up in a state that was causing timeouts. And, then we see:

WARN  [2019-06-17 16:02:57,047] com.couchbase.client.core.endpoint.AbstractGenericHandler: [/172.24.171.204:11210][KeyValueEndpoint]: Got error while consuming KeepAliveResponse.
! java.util.concurrent.TimeoutException: null
! at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166)
! at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191)
! at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
! at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
! at java.util.concurrent.FutureTask.run(FutureTask.java)
! at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
! at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
! at java.lang.Thread.run(Thread.java:748)

@daschl Any idea?

Edit: we found this with another document, which has flags: 33554432

on this cluster, we performed an in-place upgrade from 5.5.3 -> 6.0.1. I’m trying to isolate this problem to a simple test.

unhuman · June 18, 2019, 3:01pm

Another item of interest here. Snappy encryption got added (supposed to be disabled by default) in 2.6.0, but that doesn’t appear to be the case. Works in 2.5.8, breaks in all versions >= 2.6.0.

Adding:

System.setProperty("com.couchbase.snappyEnabled", "False");

before initializing Couchbase seems to mitigate the issue.

sonalkesarwani · July 2, 2019, 7:01am

this worked for me too. Thanks. Not able to understand the cause though.

daschl · July 2, 2019, 7:30am

@sonalkesarwani you’ve effectively disabled the snappy compression with that which brings it back to the old state. That said, I think we still need to figure out what’s going on here.

@unhuman were you able to come up with a unit test to reproduce or so? I know many users are 2.6.0+ now and I haven’t seen these kinds of issues reported…

ingenthr · October 7, 2019, 11:38pm

Current thought from another thread with @jda…

Somehow the CE release is sending responses indicating the items are compressed. It’s not clear how that could happen, since the client should request compression, the CE release should indicate it is not supported, then the client should not receive items with snappy compression from the cluster.

Further tracking in MB-36299.