Error Connection to couchbase and can not reconnect

Hi All,

I’m using Couchbase 6.5.1 and Java SDK 3.1.0
I’m building REST API that retrieve data from couchbase.
I have issue that suddenly the connection is closed and can not reconnect anymore.
Here’s the first error message before that :

2021-05-15 03:03:10,034 ERROR [rea.cor.pub.Operators] (cb-io-kv-5-2) Operator called default onErrorDropped: com.couchbase.client.core.error.RequestCanceledException: CarrierGlobalConfigRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) {“cancelled”:true,“completed”:true,“coreId”:“0xfe397c1700000220”,“idempotent”:true,“lastChannelId”:“FE397C1700000220/00000000C9C70D6E”,“lastDispatchedFrom”:“172.18.0.5:53364”,“lastDispatchedTo”:“xxxxx:11210”,“reason”:“NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)”,“requestId”:134041,“requestType”:“CarrierGlobalConfigRequest”,“retried”:0,“service”:{“opaque”:“0x279a8”,“target”:“xxxxx”,“type”:“kv”},“timeoutMs”:2500}
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:169)
at com.couchbase.client.core.retry.RetryOrchestrator.lambda$maybeRetry$0(RetryOrchestrator.java:79)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
at com.couchbase.client.core.retry.RetryOrchestrator.maybeRetry(RetryOrchestrator.java:62)
at com.couchbase.client.core.io.netty.kv.KeyValueMessageHandler.channelInactive(KeyValueMessageHandler.java:260)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)

How to resolve this?
THanks

@Han_Chris1 this is is a result of the fact that the socket got closed while an operation (here an internal one to update the configuration of the cluster) was in-flight. The question we need to resolve here is: why did the socket close in the first place? To answer this, more logs around that error (especially before) can help, as well as looking at the server logs if the server closed the connection on the client.

I wouldn’t worry so much about this specific exception, it’s just a side-effect.

Hi @daschl , Thank you for your response.

I set my app log level to warn and only see these logs before that happened:

2021-05-15 01:39:46,223 WARN [com.cou.tracing] (cb-events) [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“QueryRequest”,“last_local_address”:“172.18.0.5:46078”,“last_remote_address”:“xxxxx:8093”,“last_dispatch_us”:1480155,“last_operation_id”:“7e6461e7-41b4-4111-bf1e-fa473658637e”,“total_us”:1482576}],“service”:“query”,“count”:1}]
2021-05-15 01:52:47,294 WARN [com.cou.tracing] (cb-events) [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“QueryRequest”,“last_local_address”:“172.18.0.5:52510”,“last_remote_address”:“xxxxx:8093”,“last_dispatch_us”:1479073,“last_operation_id”:“041a61bb-d35f-441f-86e8-c8a440e43dd3”,“total_us”:1501880}],“service”:“query”,“count”:1}]
2021-05-15 02:04:38,064 ERROR [com.xxx.api.ser.MyService] (executor-thread-14) e: com.couchbase.client.core.error.DecodingFailureException: Deserialization of content into target class com.xxxxxx.api.model.CountData failed; encoded = {“address1”:“xxx”}
2021-05-15 02:23:51,406 ERROR [com.xxx.api.ser.MyService] (executor-thread-16) e: com.couchbase.client.core.error.DecodingFailureException: Deserialization of content into target class com.xxxxxx.api.model.CountData failed; encoded = {“totalClient”:281}
2021-05-15 02:47:11,728 WARN [com.cou.tracing] (cb-events) [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“QueryRequest”,“last_local_address”:“172.18.0.5:34986”,“last_remote_address”:“xxxxx:8093”,“last_dispatch_us”:2126183,“last_operation_id”:“5638e5ab-0f9e-451c-8ddd-320935251b33”,“total_us”:2126464}],“service”:“query”,“count”:1}]

I know it’s the side effect, but is there any reconnect mechanism ? bcs I need to restart my service to make it reconnect to couchbase

@Han_Chris1 yes the client transparently reconnects - but in your other message it says you shutdown the client, which is why it didn’t reconnect?

Hi @daschl ,

That error message comes after this error, so after no more retry, client suddenly shutdown the connection

com.couchbase.client.core.error.RequestCanceledException: QueryRequest, Reason: SHUTDOWN {“cancelled”:true,“completed”:true,“coreId”:“0xfe397c1700000261”,“idempotent”:false,“reason”:“SHUTDOWN”,“requestId”:149839,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“bbe70bd7-9583-4c54-9f0b-5a8afe833ce8”,

and did not reconnect anymore

The client does never shut itself down - it has to be initiated by the user (you), calling cluster.disconnect

I saw from my code there’s idle conn check,
Let me try to comment the disconnect part and see the result

Hi…A Couchbase association string is a comma-delimited rundown of IP addresses and additionally hostnames, alternatively followed by a rundown of boundaries.

The boundary list is very much like the inquiry part of a URI; name-esteem sets have an equivalents sign (=) isolating the name and worth, with an ampersand (and) between each pair. Similarly as in a URI, the main boundary is prefixed by a question mark