Severe error while querying view using Java SDK


#1

While querying a view again CBL 3.0.0 using the Java Client (2.0.1-SNAPSHOT), after a few iterations of the code, the Java SDK consistently errors out and dies with the following error:

Oct 16, 2014 10:34:35 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 2225 com.couchbase.client.core.RequestEvent@49250abd
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
	at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:744)

Exception in thread "cb-core-3-2" java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
	at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@38b07646
	at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	... 4 more
Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:481)
	at rx.observables.BlockingObservable.single(BlockingObservable.java:348)
	at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:591)
	at com.couchbase.client.java.CouchbaseBucket.query(CouchbaseBucket.java:564)
	at com.voltaireapp.changeslistener.ChangesListener.runChangesListener(ChangesListener.java:78)
	at com.voltaireapp.changeslistener.ChangesListener.main(ChangesListener.java:135)
Caused by: java.util.concurrent.TimeoutException
	at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:169)
	at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:42)
	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:43)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Continuing the discussion from 2.0 Client Exception, flags not correct:

Continuing the discussion from 2.0 Client Exception, flags not correct:


#2

Hey folks,

thanks all for reporting this, we’re working on fixing it for 2.0.1 - thanks!


#4

Thanks, @daschl! I tried to search JIRA/issues before I raised this here but couldn’t find anything, sorry if this was a dupe/already known issue, not much came up, maybe I didn’t search properly.

At any rate, is there an issue # I can go follow in JIRA/Issue tracker?


#5

FWIW - the following workaround seems to work (wrap any upsert/insert/replace) that will catch the issue, reset the client, and keep going.

Testing this out a bit more, and if it works, the workaround until the fix hits 2.0.1 will just be to build a wrapper that wraps upsert et al and does the same thing…

try {
    bucket.upsert(doc);
} catch (Exception e) {
    if (e.toString().contains("java.lang.IllegalStateException: Node not found for request")) {
        logger.error("Ugh, couchbase is being stupid and entering Microsoft emulation mode, resetting client and trying to recover...");
        cluster.disconnect();
        cluster = null;

        cluster = CouchbaseCluster.create(clusterString);
        bucket.upsert(doc);
    } else {
        e.printStackTrace();
        throw new RuntimeException(e);
    }
}

#6

Can verify it. Having the same trouble with “Node not found”.


#7

To answer my own question above, looks like this is perhaps covered in https://www.couchbase.com/issues/browse/JCBC-543?


#8

No, this was a different exception (node addr mismatch) that was fixed in 2.0, I need to mark it appropriately. I’ll get you the right one with more info.


#9

http://www.couchbase.com/issues/browse/JVMCBC-45

We’ll go a head and retry those config requests in the future to gracefully handle connect states. I’ll package a 2.0.1-pre1 very soon and get it to you upfront so you can try to see if it works.


#10

Can you try this 2.0.1-pre version and see if it works for you?

https://dl.dropboxusercontent.com/u/10007675/Couchbase-Java-Client-2.0.1-SNAPSHOT-b0921ce.zip


#11

I checked the pre-version and for me it works like a charm! Thank you. Just to be sure, i also replaced it once more with the 2.0.0 version and it almost immediately complains.


#12

Perfect, please make sure to upgrade to 2.0.1 once we’ve released it.


#13

@daschl, thanks for the quick response and patch, sorry for my late reply, just had time to try this and it seems to work. Will update to 2.0.1 when it hits maven. Just as a point of interest - with what frequency (if any) is the -SNAPSHOT updated in maven? I know 2.0.1-SNAPSHOT still exhibits this issue, so I am assuming the code you posted in the zip is off some commit post-snapshot (b9021ce?)? No big deal as it seems release of 2.0.1 is imminent anyway…


#14

Well, we are not publishing SNAPSHOTS as of now through or Jenkins, but we will do in the future. For now you have to build both the core-io and java-client from github directly through gradle, but it’s not that hard either.


#15

Hi I updated my pom and added 2.0.1 to my project. and for some reason i am still getting the error. can anyone confirm that the issue was fixed in the new release of the Java SDK.


#16

@chaddy012 can you please post the full log somewhere (including when it starts)?
Thanks


#17

I included the last line of log for the last operation i ran so you get a sense of the time.

INFO 2014-11-07 00:46:57,905 [_api].connector.http.mule.default.receiver.06] org.mule.api.processor.LoggerMessageProcessor: Create collection object
Nov 07, 2014 12:50:31 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 3 com.couchbase.client.core.RequestEvent@5af345c5
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

Exception in thread “cb-core-3-2” java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@479d308f
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
… 4 more
Nov 07, 2014 12:50:31 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 103 com.couchbase.client.core.RequestEvent@5d0a66c5
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

Exception in thread “cb-core-6-2” java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@53cd92cc
at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
… 4 more


#18

@chaddy012 I was explicitly requesting the first lines because on bootstrap we output configuration params and all that related stuff, so if you can include these parts as well it would be great.


#20

Hi i just realize after looking at the bootstrap portion that for some reason my maven was still using the 2.0.0 version.

Thank you.


#21

See, that’s why I wanted to see it and the reason we put it right in there! :wink: