We are using Couchbase Java SDK 2.4.7. Very intermittently in production, we get the below error, which leads to wildfly server going out of memory eventually, Basically, the couch client keeps retrying to connect to a node which does not send a keep alive response perhaps (may be a temporary network delay?). We have 3 data, 2 query and 2 index nodes. Also at this point, Couchbase server (4.6.1) is absolutely healthy and other applications connect to the same bucket without any issues. The only thing is when Wildfly encounters this error, it cannot recover from it and throws Out of Memory exception. The only way to solve this is restart wildfly in production. We had this issue with the earlier jar version too and were asked to upgrade to 2.4.7. Has anyone else experienced the same issue? We will try upgrading to 2.7.1, just not sure if this issue would go away or if it is due to something else.
There is no pattern to the behavior and it can happen even with a select query to get just one record which is indexed.
[com.couchbase.client.core.endpoint.AbstractGenericHandler] (RxComputationScheduler-3) [XXXXX.matrixhealth.net/X.X.X.X:xxxxx][KeyValueEndpoint]: Got error while consuming KeepAliveResponse.: java.util.concurrent.TimeoutException at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:177) at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:41) at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189) at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)