2.0 Client Exception, flags not correct

As far as I can tell, the document looks like correct json.
I put a breakpoint at JsonTranscoder:60.
The “response” object looks like this:

GetResponse{bucket='load_test_bucket', status=SUCCESS, cas=16295220261941950, flags=9371651, request=com.couchbase.client.core.message.kv.GetRequest@6f0f604e, content={"userId":"2","userPromos":{}}}

I’m running against CB 2.5.1 (soon to be upgraded to 3.0)
Running on Java 1.8.0_20
java client 2.0.0

Here’s a code snippet that I’m using:

def createOrGetUser(userId: String): UserPromo = {
    Try({
      bucket.get(userKey(userId), 1, SECONDS) match {
        case null =>
         createEmptyUserPromo(userId)
        case doc: JsonDocument =>
          JsonParser(doc.content.toString).convertTo[UserPromo]
      }
    }) recover {
      case e =>
        null
    } get
  }

  def storeUser(userId: String, newPromo: UserPromo, prevPromo: UserPromo): Boolean = {
    val j = JsonDocument.create(userKey(userId), jsonTranscoder.stringToJsonObject(newPromo.toJson.prettyPrint))
    bucket.upsert(j, PersistTo.MASTER, 1, SECONDS) != null
  }

Exception:

18:26:52.618UTC ERROR[PromotionActorSystem-akka.actor.default-dispatcher-9] OneForOneStrategy - Flags (0x35d0003) indicate non-JSON document for id u_5, could not decode.
    com.couchbase.client.java.error.TranscodingException: Flags (0x35d0003) indicate non-JSON document for id u_5, could not decode.
    	at com.couchbase.client.java.transcoder.JsonTranscoder.doDecode(JsonTranscoder.java:60) ~[java-client-2.0.0.jar:2.0.0-beta-14-gbe5dc12-dirty]
    	at com.couchbase.client.java.transcoder.JsonTranscoder.doDecode(JsonTranscoder.java:40) ~[java-client-2.0.0.jar:2.0.0-beta-14-gbe5dc12-dirty]
    	at com.couchbase.client.java.transcoder.AbstractTranscoder.decode(AbstractTranscoder.java:41) ~[java-client-2.0.0.jar:2.0.0-beta-14-gbe5dc12-dirty]
    	at com.couchbase.client.java.CouchbaseAsyncBucket$1.call(CouchbaseAsyncBucket.java:134) ~[java-client-2.0.0.jar:2.0.0-beta-14-gbe5dc12-dirty]
    	at com.couchbase.client.java.CouchbaseAsyncBucket$1.call(CouchbaseAsyncBucket.java:130) ~[java-client-2.0.0.jar:2.0.0-beta-14-gbe5dc12-dirty]
    	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at rx.internal.operators.OperatorFilter$1.onNext(OperatorFilter.java:54) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at rx.internal.operators.OperatorSubscribeOn$1$1$1.onNext(OperatorSubscribeOn.java:76) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:224) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:96) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at com.couchbase.client.core.ResponseHandler.onEvent(ResponseHandler.java:98) ~[core-io-1.0.0.jar:1.0.0]
    	at com.couchbase.client.core.ResponseHandler.onEvent(ResponseHandler.java:43) ~[core-io-1.0.0.jar:1.0.0]
    	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) ~[core-io-1.0.0.jar:1.0.0]
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_20]
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_20]
    	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[core-io-1.0.0.jar:1.0.0]
    	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_20]
    Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: GetResponse.class
    	at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:98) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:56) ~[rxjava-1.0.0-rc.3.jar:1.0.0-rc.3]
    	... 11 common frames omitted

Hi, this is a known issue and is fixed on the current master branch (2.0.1 upcoming).

If you want, I can link you a pre-release to try - or do you want to try the master branches?

Thanks daschl

I’m now running the master branch off the git repo.
Also our CB test cluster was upgraded to 3.0 last night.

I’m seeing a new error now:

Oct 15, 2014 9:30:29 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 661 com.couchbase.client.core.RequestEvent@1fe07403
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@5eb9df36
	at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

Exception in thread "cb-core-3-2" java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@5eb9df36
	at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.kv.GetBucketConfigRequest@5eb9df36
	at com.couchbase.client.core.node.locate.KeyValueLocator.locate(KeyValueLocator.java:63)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
	at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
	at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	... 4 more

This generally leads to timeouts and bad connections from then on until I restart the service.

I am having the same (Node not found) issue (not the original issue), along with the same “restart server, things get OK for a few minutes” symptom. Also on CB 3.0, with 2.0.1-SNAPSHOT out of maven.

Thanks!

@lapetus999 - see the response about this from @daschl in separate thread about this being known issue for 2.0.1 fix. In the mean time, this workaround seems to work for me:

FWIW - the following workaround seems to work (wrap any upsert/insert/replace) that will catch the issue, reset the client, and keep going.

Testing this out a bit more, and if it works, the workaround until the fix hits 2.0.1 will just be to build a wrapper that wraps upsert et al and does the same thing…

try {
    bucket.upsert(doc);
} catch (Exception e) {
    if (e.toString().contains("java.lang.IllegalStateException: Node not found for request")) {
        logger.error("Ugh, couchbase is being stupid and entering Microsoft emulation mode, resetting client and trying to recover...");
        cluster.disconnect();
        cluster = null;

        cluster = CouchbaseCluster.create(clusterString);
        bucket.upsert(doc);
    } else {
        e.printStackTrace();
        throw new RuntimeException(e);
    }
}

Can you try this 2.0.1-pre version and see if it works for you?

https://dl.dropboxusercontent.com/u/10007675/Couchbase-Java-Client-2.0.1-SNAPSHOT-b0921ce.zip

I built master and changed the version to 2.0.1-SNAPSHOT but I still get an exception on every single Get operation and on most insert operation. The insert was throwing a Timeout exception and I worked around that by disconnecting my network connections. I have not been able to successfully get an object out of Couchbase since I started seeing this early today. I only started with couch recently to evaluate for my company and this is disheartening. Is the Java client library expected to be able to connect to a local instance of Couchbase?

Caused by:

rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: GetResponse.class
        at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:98)
        at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:56)
        at rx.internal.operators.OperatorFilter$1.onNext(OperatorFilter.java:54)
        at rx.internal.operators.OperatorSubscribeOn$1$1$1.onNext(OperatorSubscribeOn.java:76)

Sorry you’re running into troubles. It’s definitely intended to be used with a local instance and obviously many of us do that day in and day out. I’m sure we’ll be able to help find the issue.

Recently, a bug was found with the flags field in the 3.0.0 release. It won’t affect all operations, but it may have affected this one. See the release note about MB-12328.

Is it possible that you did a rebalance and the flags are now off? One easy way to find out is check with the “cbc cat” command and compare a working doc to a non-working doc. That’s in the cbc tools which are part of libcouchbase.

@daschl: what’s a good way to see this in the client? A higher debug level? Or a different command?

Hi,

sorry it’s not working for you right from the start. You can also try the 1.4.5 version if you want to go to the “tried and true” which nearly all of our users still run in production.

That said, of course it should work. Can you please send us the full logs (INFO) level should be fine, but anything with more detail (including FINEST) would be much appreciated.
Can you also please put the info into a ticket here: http://www.couchbase.com/issues/browse/JCBC

Thanks for your help. I found out the issue was the format of my documents. Today I learned that when inserted as a JsonStringDocument, it must be gotten as a JsonStringDocument.

For my use-case, I decided to make it a JsonDocument on the way in. Previously, I was converting from a map to a String so I could let Jackson do the work.

To achieve this, I create a JsonObject and set the top-level values from the Map. I was wondering about one design choice on the part of the SDK developers: How come I can put a Map as a value of a field in a JsonObject successfully but the SDK intentionally prevents creating a JSONObject from a Map?

Hi,

so when 2.0.1 ships (or you try the preview) linked in this thread, you can use the RawJsonDocument to give it a already marshalled JSON string from jackon.

like RawJsonDocument.create(“id”, stringFromjackson)

I think we can add a factory method to create a JSONObject from a Map, the only thing we need to make sure is that it will raise an exception if some content of it is not applyable to json (so it needs to only contain the supported types as well).

I’ve added a ticket to track this: http://www.couchbase.com/issues/browse/JCBC-614

Thanks. I believe I saw 2.0.1 available, I’ll check Maven. My confusion was that I thought JsonStringDocument acted in the way you describe RawJsonDocument.

Since that is not the case, what is the difference in intended use between JsonStringDocument and BinaryDocument? I ask this because I noticed the Json documents I insert as content for a JsonStringDocument are described as Binary in the admin console and aren’t readable there nor are they searchable in ElasticSearch.

Hi,

I’m planning to extending the documentation in that area, but in the meantime here is a quick rundown:

  • JsonStringDocument is intended as a JSON compatible string that is stored. So for example if you store foobar it will stored it as quoted foorbar like “foobar” according to the JSON spec.
  • RawJsonDocument will store it inquoted (since you could be storing anything, arrays, objects, strings, numbers) AND we are setting the internal JSON flag so if you load it through a normal JsonDocument will we will do the parsing for you - so you can pick what suites your needs.
  • BinaryDocument - sets a non-json internal flag and just stores the raw bytes you are giving it. very low level and it’s up to you what to put in there. Note that it does not set a JSON flag because it can be anything.

So if you ES compat: use the JsonDocument if you don’t use your own JSON lib (like jackson) and use RawJsonDocument if you do.

Thanks for pointing that out. It’s quite clear now that it’s JsonStringDocument, not JsonStringDocument.

I discovered the “convert” feature in Jackson which can convert the in-memory Objects consumed in our Controller’s method to couch base’s native JsonObject. I’ll check out 2.0.2 for any advantages over the Jackson conversion.

@a_couch if you go down that route, I recommend you not to do object -> jackson -> jsonobject -> JsonDocument, but rather object -> jackson -> JSON String -> RawJsonDocument.

The RawJsonDocument gives you and returns stringified JSON, so you save object allocations, GC and time. It is designed to be used with third party JSON serializers. Basically you can do RawJsonDocument.create(“id”, “{…}”); and avoid going through JsonObjects.

You make a good point. It turns out Jackson won’t convert to JsonObject so easily, it expects each field to be present in the JsonObject class rather than using put.

I built 2.0.2 and JsonObject.create().from() works well. The key for me is to put a document asynchronously and then get the ID from the resulting object in the listener.

I suppose I could set the ID in the listener before the insert if I want to maintain state like this. I suppose I thought it would be simpler to insert and then access fields in the content from the result of the insert method. If I use RawJsonDocument, the content of the result of the insert needs to be converted before any of its fields can be used. So getting what I need from the source object before creating the document and treating the Document object’s content as a black box seems to be the optimal way to use the JDK reactively.

Thank you.

@a_couch I’m not sure I can follow quite. Would you mind providing some sample code (or some meta code) on what you’d like to achieve with Jackson and so forth? I’m sure then I can get you the best implementation we can currently provide.

Hi I am using v2.0.2 and get the:
com.couchbase.client.java.error.TranscodingException: Flags (0x2000000) indicate non-String document

Any clue?

@ohassidi how did you store the original document?