Scheduled events from SDK fails

We currently have a Couchbase cluster with 3 nodes.
After a node failure with a primary Index on i. The node came up a bit later and we had to recreate the primary index.
Suddenly we are getting some errors on our sdk

2023-01-24 16:04:24.625 WARN 8 — [ cb-events] com.couchbase.tracing : [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“QueryRequest”,“last_local_address”:“.”,“last_remote_address”:“…”,“last_dispatch_us”:29532433,“last_operation_id”:“…”,“total_us”:29544277}],“service”:“query”,“count”:1}]
2023-01-24 16:05:22.954 ERROR 8 — [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task

com.couchbase.client.core.error.IndexFailureException: The server reported an issue with the underlying index

Suppressed: java.lang.Exception: #block terminated with an error

Hi @kaizerman.

Can you share the complete stack trace, please?

Thanks,
David

Thanks for the response. Here is the full stack

2023-01-25 14:22:32.967 ERROR 8 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

com.couchbase.client.core.error.IndexFailureException: The server reported an issue with the underlying index
        at com.couchbase.client.core.io.netty.query.QueryChunkResponseParser.errorsToThrowable(QueryChunkResponseParser.java:148)
        at com.couchbase.client.core.io.netty.query.QueryChunkResponseParser.lambda$new$8(QueryChunkResponseParser.java:99)
        at com.couchbase.client.core.json.stream.JsonStreamParser.emitCapturedValue(JsonStreamParser.java:213)
        at com.couchbase.client.core.json.stream.StructureNavigator.finishValue(StructureNavigator.java:285)
        at com.couchbase.client.core.json.stream.StructureNavigator.access$800(StructureNavigator.java:32)
        at com.couchbase.client.core.json.stream.StructureNavigator$FastForwarding.accept(StructureNavigator.java:276)
        at com.couchbase.client.core.json.stream.StructureNavigator$FastForwarding.accept(StructureNavigator.java:256)
        at com.couchbase.client.core.json.stream.StructureNavigator.accept(StructureNavigator.java:90)
        at com.couchbase.client.core.json.stream.JsonStreamParser.processTokens(JsonStreamParser.java:172)
        at com.couchbase.client.core.json.stream.JsonStreamParser.feed(JsonStreamParser.java:123)
        at com.couchbase.client.core.io.netty.chunk.BaseChunkResponseParser.feed(BaseChunkResponseParser.java:153)
        at com.couchbase.client.core.io.netty.chunk.ChunkedMessageHandler.handleHttpContent(ChunkedMessageHandler.java:234)
        at com.couchbase.client.core.io.netty.chunk.ChunkedMessageHandler.channelRead(ChunkedMessageHandler.java:187)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at com.couchbase.client.core.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
                at reactor.core.publisher.Mono.block(Mono.java:1703)
                at org.springframework.data.couchbase.core.ExecutableFindByQueryOperationSupport$ExecutableFindByQuerySupport.all(ExecutableFindByQueryOperationSupport.java:70)
                at org.springframework.data.couchbase.repository.query.N1qlRepositoryQueryExecutor.execute(N1qlRepositoryQueryExecutor.java:71)
                at org.springframework.data.couchbase.repository.query.CouchbaseRepositoryQuery.execute(CouchbaseRepositoryQuery.java:42)
                at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
                at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
                at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
                at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
                at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
                at org.springframework.data.couchbase.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
                at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
                at com.sun.proxy.$Proxy157.findAllByStatus(Unknown Source)
                at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
                at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
                at com.sun.proxy.$Proxy157.findAllByStatus(Unknown Source)
                at ...our packages...
                at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
                at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                ... 1 common frames omitted

It looks like you’re using an older version of the SDK that doesn’t include error context information in the stack trace. I would upgrade to at least 3.2.1 (or the latest version, if you can!) so the stack trace tells you more about the error.

Thanks,
David

I have finally got the error, from the driver.

The server reported an issue with the underlying index {"completed":true,"coreId":"0x96a0455700000001","errors":[{"additional":{"retry":true},"code":12008,"message":"Error performing bulk get operation  - cause: {1 errors, starting with bulkget exceeded MaxBulkRetries for default(vbid:714,keys:<ud>[030b60b9-08b3-4008-8cbb-58e30013115b]</ud>)}"}],"idempotent":false,"lastDispatchedFrom":"10.0.131.100:60228","lastDispatchedTo":"couch-0000.couch.default.svc:8093","requestId":194,"requestType":"QueryRequest","retried":0,"service":{"operationId":"152d1cd4-6ff8-4f2f-bed7-fa29fb97de43","statement":"SELECT META(`default`).id AS __id, META(`default`).cas AS __cas, `filename`, `status`, `uploadDate`, `nodeselectors`, `step2DatasetFileId`, `step4DatasetFileId`, `parameters` FROM `default` WHERE `_class` = \"DataFile\" AND `status` = $1","type":"query"},"timeoutMs":75000,"timings":{"dispatchMicros":29995481,"totalDispatchMicros":29995481,"totalMicros":30132393}}
1 Like

Hi @kaizerman

Looks like it’s not an issue with the SDK. This other post talks about a similar error message: Code 12008 :Error performing bulk get operation - starting with bulkget exceeded MaxBulkRetries - #2 by vsr1

If the suggestions there don’t solve the issue, you might want to start a new topic in the N1QL category.

Thanks,
David

1 Like