Continuously get CouchbaseException: Unknown query error

Couchbase Server
1

Hi, our servers continuously get CouchbaseException: Unknown query error (mostly)
and sometimes get UnambiguousTimeoutException: QueryRequest, Reason: TIMEOUT.

We often got this when try to got 100 records from DB while some other IFs are trying to update part of them (maybe) . We guess it’s the root cause and try to reproduce it in other ENV by using gatling.
But can only reproduce UnambiguousTimeoutException: QueryRequest, Reason: TIMEOUT when come to a high QPS. Does anyone have idea about this?

FYI
stack_trace
com.couchbase.client.core.error.CouchbaseException: Unknown query error
at com.couchbase.client.core.io.netty.query.QueryChunkResponseParser.errorsToThrowable(QueryChunkResponseParser.java:151)
at com.couchbase.client.core.io.netty.query.QueryChunkResponseParser.lambda$new$8(QueryChunkResponseParser.java:99)
at com.couchbase.client.core.json.stream.JsonStreamParser.emitCapturedValue(JsonStreamParser.java:213)
at com.couchbase.client.core.json.stream.StructureNavigator.finishValue(StructureNavigator.java:285)
at com.couchbase.client.core.json.stream.StructureNavigator.access$800(StructureNavigator.java:32)
at com.couchbase.client.core.json.stream.StructureNavigator$FastForwarding.accept(StructureNavigator.java:276)
at com.couchbase.client.core.json.stream.StructureNavigator$FastForwarding.accept(StructureNavigator.java:256)
at com.couchbase.client.core.json.stream.StructureNavigator.accept(StructureNavigator.java:90)
at com.couchbase.client.core.json.stream.JsonStreamParser.processTokens(JsonStreamParser.java:172)
at com.couchbase.client.core.json.stream.JsonStreamParser.feed(JsonStreamParser.java:123)
at com.couchbase.client.core.io.netty.chunk.BaseChunkResponseParser.feed(BaseChunkResponseParser.java:153)
at com.couchbase.client.core.io.netty.chunk.ChunkedMessageHandler.handleHttpContent(ChunkedMessageHandler.java:243)
at com.couchbase.client.core.io.netty.chunk.ChunkedMessageHandler.channelRead(ChunkedMessageHandler.java:196)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.jav…

2nd one

com.couchbase.client.core.error.UnambiguousTimeoutException: QueryRequest, Reason: TIMEOUT
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:167)
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)

2

Hi Seyashizuo,
Could you check if you see any errors on the cluster dashboard under the “Logs” menu?

Thank you!

3

@chedim
Hi, thanks for you reply. There is no related errors in the “Logs” menu. I try to reproduce after that.
And I think I might find root cause. (Not 100% sure though)

The errors happens when calling 2 big indexes in our DB.
One is about 600,000 MB and the other is nearly 500,000 MB size.
Roughly 600,000,000 records for each index.
And our queries is look like this.

select xxxxxx
where meta().id like "xxxx%" and xxxx order by xxxxx asc (5000 + records a time)offset $offset limit $limit`. ( limit = 100)

I guess the index is too big so the query takes a long time and return Unknown query error sometimes.
So maybe index eat too much memory was the root cause? (resident on memory is 50% - 60% those 2 indexes)

Do you have any idea about it?

Thank you!