We are using Reactive Connection factory to connect coucbase 5.1, though use traditional couchbase repository i.e CouchbaseRepository to perform database operations. After some time the threads which execute the couchbase queries are stuck and stay in WAITING state in all the subsequent thread dumps they stay in same state . They neither timeout nor return response. When access the couchbase bucket from web client are able to get the response of the same query, but the stuck threads do not return any response
Code
Perform following operation on bean couchbaseDatabaseOperations
public String getById( String id) {
couchbaseDatabaseOperations.findById(id)
}
or
public void save(String object){
couchbaseDatabaseOperations.save(object);
}
where the couchbaseDatabaseOperations is a bean defined as
@Repository
@N1qlPrimaryIndexed
public interface CouchbaseDatabaseOperations extends CouchbaseRepository<Model, String> {
}
Connection Factory Details
@Configuration
@EnableCouchbaseRepositories(basePackages = “com.test”)
@Slf4j
public class CouchbaseConfig extends AbstractCouchbaseConfiguration {
/**
- bootstrapHosts variable.
*/
@Value("#{’{test:{couchbaseserver.hostNames:}}’.split(’,’)}")
private List bootstrapHosts;
@Value("{test.{couchbaseserver.bucketName:}}")
private String bucketName;
Thread dump( One of the nio thread that is stuck, cb-io thread are in Runnable state, while cb-computation threads are also in WAITING state(stuck)
“http-nio-9590-exec-28” #551 daemon prio=5 os_prio=0 tid=0x00007f30c4015000 nid=0x298 waiting on condition [0x00007f30365a2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068a597be0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at rx.internal.util.BlockingUtils.awaitForComplete(BlockingUtils.java:47)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:460)
at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:368)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:363)
at org.springframework.data.couchbase.core.CouchbaseTemplate$11.doInBucket(CouchbaseTemplate.java:615)
at org.springframework.data.couchbase.core.CouchbaseTemplate$11.doInBucket(CouchbaseTemplate.java:598)
at org.springframework.data.couchbase.core.CouchbaseTemplate.execute(CouchbaseTemplate.java:548)
at org.springframework.data.couchbase.core.CouchbaseTemplate.doPersist(CouchbaseTemplate.java:598)
at org.springframework.data.couchbase.core.CouchbaseTemplate.save(CouchbaseTemplate.java:245)
at org.springframework.data.couchbase.core.CouchbaseTemplate.save(CouchbaseTemplate.java:240)
at org.springframework.data.couchbase.repository.support.SimpleCouchbaseRepository.save(SimpleCouchbaseRepository.java:87)
at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:359)
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:644)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:608)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.lambda$invoke$3(RepositoryFactorySupport.java:595)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor$$Lambda$1507/1083290460.get(Unknown Source)
at org.springframework.data.repository.util.QueryExecutionConverters$$Lambda$1506/64378994.apply(Unknown Source)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:595)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.couchbase.repository.support.ViewPostProcessor$ViewInterceptor.invoke(ViewPostProcessor.java:87)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy178.save(Unknown Source)
at sun.reflect.GeneratedMethodAccessor217.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy178.save(Unknown Source)
.save
CB IO threads are in Runnable state
“cb-io-1-4” #66 daemon prio=5 os_prio=0 tid=0x00007f310c016000 nid=0xab runnable [0x00007f313cde8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000684500320> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000684500310> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000684500338> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
CB Computation Threads are also in waiting state
“cb-computations-4” #76 daemon prio=5 os_prio=0 tid=0x00007f326d7d1800 nid=0xa1 waiting on condition [0x00007f313f8f3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000068953f050> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at rx.internal.util.BlockingUtils.awaitForComplete(BlockingUtils.java:47)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:460)
at rx.observables.BlockingObservable.singleOrDefault(BlockingObservable.java:372)
at com.pearson.glp.gms.data.repository.CacheRepository.getObjectAsRawJson(CacheRepository.java:119)
at com.pearson.glp.gms.data.repository.CacheRepository.getDocumentById(CacheRepository.java:160)