No support for actual JSON in couchbase lite Android?

I think I MUST be mistaken, but am unable to find a single reference for how to actually store a JSON string to couchbase lite on Android.

All I can find is a reference to creating strongly typed Document or MutableDocument objects and calling an endless morass of setters (setString, setInt, setX, etc). These are nearly completely useless, as I have documents that frequently contain tens or even HUNDREDS of properties, and the last thing I can afford to do is manually call tens or hundreds of setters, when I need exactly none of them, as I already have the JSON string that perfectly represents the document I want to store.

In other words, All I want to accomplish is creating a document from a JSON string. I expect this to be the very first method for constructing any document, but shockingly, I can find no evidence for its support.

Oh pretty please tell me I’ve missed something fundamental as this an absolute deal breaker for me.

If you have a JSON string, then I am guessing that you can deserialize it into a JSON object. If so you can construct a MutableDocument using that object and if you wish to edit the data directly you can do so and reset it using the setData method. It sounds like the constructor will be enough for the case you describe though.

There is no method to directly store a JSON string, nor would it save you anything if there were one because the storage format is not JSON in a Couchbase Lite database. Even if it were present, it would be first deserialized into an object before being reserialized in the storage format.

The assumption is that you use our API to create and examine documents, instead of doing the JSON serialization yourself. We store documents in an efficient binary format that’s more compact than JSON. In the typical case where you only need to read a subset of the document’s properties, we don’t waste time parsing the entire document, so it’s much faster than JSON.

If you do already have all the JSON serialization written, though, this can be a bit awkward. Ideally you can change that JSON I/O code to use our API instead of whatever JSON API you’re using. In the worst case you can hack together some functions to parse JSON and call our API to store it into the document (and the reverse for reading.)

In the next major release (3.0) we’re adding a document property that exposes the JSON, for cases like this. But that won’t be available for “a while” (we can’t talk schedules yet.)

Thanks for the responses.

I have to admit, I’m quite disappointed to hear this. I understand and appreciate the explanations provided here. What I’m having more difficulty with is this:

If CBL doesn’t support N1QL AND it doesn’t even support JSON, then why is the product named Couchbase Lite? I chose CBL because I falsely assumed it was Couchbase, but a Lite version. But now it appears to be a standalone NoSQL product with little relation to its namesake. I’m afraid others may make the same assumption.

Here’s my use case, which I expect is shared by other Android developers:

  • In Android, I use Gson exclusively.
  • JSON strings representing documents are sent between Android and Couchbase Node via socket connection.
  • Android receives JSON strings on the socket. I expect to store these strings directly to CBL with zero fuss/intervention.
  • I expect to fetch documents using N1QL exclusively.
  • I have zero use for “building” a Document with CBL. My documents are either already built or I will build them - always using JSON
  • If I must, I expect to create a new CBL Document by calling new Document(myJSONString) and that’s it.
  • I expect CBL to silently and efficiently parse my JSON string into its native format (fleece?) as needed. Efficiency of parsing JSON to and from fleece should be CBL’s concern, not mine.
  • Support for N1QL means we can craft query projections, which would eliminate ‘whole document parsing’

It appears that I have 2 equally painful choices: craft a JSON to CBL adapter by hand, or abandon CBL. I don’t like either of these at all.

I wouldn’t mind adding this API but I don’t feel like we’ve committed a grave crime by omitting it (disclaimer: opinion is my own). We read forum posts and we discuss them (especially around requests), I promise you, and striking a bombastic tone like your initial post doesn’t make us any more likely to do anything differently.

Given the tone of the paragraph I’m going to assume this question is rhetoric and skip answering it.

Maybe, but not enough have spoken up to make this a priority. Plenty of them seem perfectly happy, reasonably or not, using the current API. Ironically there has been more demand for the converse of this (i.e. people want to see their data saved and retrieved as their own custom defined class) but I forget the distribution of which platforms were asking for what. Most people will use Couchbase Lite as an offline first data store in which their edge devices are creating data to be synced with a Couchbase Server instance in the cloud, via Sync Gateway, and then sent to other edge devices. Your case is actually fairly unusual, to be honest. I won’t dismiss it, certainly, but if you are ever wondering “why don’t we have X” it’s probably because of low demand.

We are working on N1QL support for mobile, you can see the progress of it if you take a peek at the Couchbase Lite Core repo and look at the N1QLParserTests. It’s not exposed up to the platform level yet but that part just requires a few wiring methods.

We will consider adding it. We don’t add everything that gets requested in the forum but this request at least has the benefit that it is actually related to a separate issue that we are currently discussing.

I don’t know about Java but in .NET this is really not that hard, given the information that I have. You are already using Gson and I assume it knows how to serialize into class objects containing the data inside a JSON string. Once you have that you have what you need to create and save a document (give or take some massaging regarding Gson classes vs straight Java classes). You literally take that and pass it into the constructor (the one that takes a dictionary as one of the arguments), you don’t need to start from an empty document and set every single property. Perhaps the latter part of that sentence didn’t come across?

I also have a third option available! The entire community edition is open source and quite a few of the building blocks needed to have what you want are actually there. Anyway I hope that you find a solution that works.

Thank you for the detailed response and forgive the ‘bombastic’ tone. As I originally posted, I expect I’ve missed something critical to my needs in the documentation.

You mentioned:

If there’s already support for specifying a data class representing the document in a constructor, then indeed 50% of my problem is likely to have just evaporated!

Can you provide some details regarding that constructor? I can’t seem to find it documented anywhere. Specifically, I don’t understand the Dictionary argument. Why a dictionary instead of an object - i.e. what’s the dictionary key for?

Having attempted to do this:

data class MyThing(
    var myProp:Int? = 0
)

val map: Map<String, MyThing> = mapOf("whatisthisfor" to MyThing(1))
val doc = MutableDocument(map)
db.save(doc)

throws an Exception: Non-string or null key in data to be deserialized.

Does this mean that my data class must be composed entirely of non-nullable string properties?

Fingers crossed…

It sounds like you want to send Couchbase Server documents to your Android app as JSON and insert them into Couchbase Lite. If so, you are trying to duplicate what the replicator already does for you. So you may be laboring under a basic misconception about how to use Couchbase Lite. Or did I misread you?

(FYI, I just literally copied and pasted your initial post into our internal Jira issue covering adding a JSON accessor. So you are being heard!)

throws an Exception: Non-string or null key in data to be deserialized.

You cannot put MyThing object as a value in the Map object. The Map object needs to contain JSON based values plus Blob object. In general, you will need to convert MyThing object into a map instead.

I don’t know what “the replicator” is. Are you referring to Sync Gateway?
And thanks for the FYI. I greatly appreciate the responsiveness of the team on this forum. Top marks!

I don’t follow. Can you provide a simple example?

Basically what you need is JSON in class form. JSON can only contain certain types like number, string, object, array. So the document data must consist exclusively of these classes (with the exception of the Blob class). Any custom types you have must be semi-serialized into such so if you had something like:

class MyThing {
     string name;
     int age;
}

Then you should pass in a dictionary containing the keys name and age with the corresponding values (as you would find them in the JSON string).

// Forgive my syntax, I am not a Java/Kotlin dev, so this is pseudo
val map  = {
     "name": "Fred",
     "age": 24
};

This goes back to what I said about people more often requesting ways to pass in their own data types directly to the library.

One common paradigm that people follow to comply with this is that their classes will all implement an interface that has a method to convert them to dictionaries (similar to the Apple platform NSSerialization class or .NET ISerializable) and to convert them to a dictionary they call the conversion method on their class, which would then recursively go through its data and add in the dictionaries representing its member object.

// Simple pseudo example
interface DocumentSerializable {
    Map<String, Object> toDocumentData();
}

class MyNestedThing implements DocumentSerializable {
    string id;

    MyNestedThing (Map<String, Object> documentData) {
        id= documentData.get("id");
    }

    Map<String, Object> toDocumentData() {
        return  {
            "id": id
        }
    }
}

class MyThing implements DocumentSerializable {
    String name;
    int age;
    MyNestedThing nestedThing;

    MyThing(Map<String, Object> documentData) {
        name = documentData.get("name");
        age = documentData.get("age");
        nestedThing = new MyNestedThing(documentData.get("nestedThing"));
     }

    Map<String, Object> toDocumentData() {
        return  {
            "name": name,
            "age": age,
            "nestedThing": nestedThing.toDocumentData()
        }
     }
}

Then if you have a MyThing instance you just need to make a new MutableDocument (or modify an existing one) with it:

  • new MutableDocument("my-chosen-id", myThing.toDocumentData())
  • existingMutableDoc.setData(myThing.toDocumentData())

This is coming from the other angle than I imagined based on the previous posts, which was you have a JSON string in which case you simply do this:

Map<String, Object> myData = deserialize(myJsonString);
MutableDocument myDoc = new MutableDocument("my-chosen-id", myData);

The second example is the one I am envisioning as not very complicated. The first, which is a separate problem, is more involved.

First, thanks again for taking the time to draft a thoughtful and thorough response.

Alas, this kind of overhead is just not at all suitable for my Android project.
Without support for N1QL or JSON, CBL “feels” nothing like Couchbase (to me!), hence my tiny rage-fest above.

I would really like to see CBL function this simply (in Kotlin):

//writing:
data class MyThing( 
    var prop1:String, 
    var prop2:String,
    var type:String? = "mything"
)
val myThing = MyThing("cool", "interesting")
//let me construct documents from JSON
val json = Gson().toJson(myThing)
val doc = Document("mymetaid", json)
doc.save()

//reading
val n1ql = "SELECT b.* FROM bucket b WHERE b.type='mything'"
val results = cluster.query(n1ql)

I understand that N1QL support is somewhere on the horizon and that’s great. Support for JSON would seal the deal.

Again, thanks for your eyes and ears.

To me it seems like what you are actually after is data object modeling, which would skip these two lines:

val json = Gson().toJson(myThing)
val doc = Document("mymetaid", json)

and let you skip straight to this:

val myThing = MyThing("cool", "interesting")
db.save(myThing); // In .NET this would be a generic class, unsure of the Java syntax

Otherwise what is the point of serializing all the way to JSON first? It would be way better to go direct from class to Fleece . This is the thing I mentioned above that is more requested than JSON support, but still oddly not as requested as other features which will take priority over this. In the end there is a chance that JSON support will come as a side effect of a different feature we are working on for 3.0 though. I’m guessing data object modeling is more inline with what your needs are, no? If so I can get you marked as a +1 for that feature request.

Well, yes, if CBL could seamlessly handle accepting ANY type in its document constructor, then that would be great too.

But there are typically a whole host of conditions the custom type must satisfy, e.g.:

  • must be a data class
  • must have a no-argument constructor
  • must have properties whose types are primitives
  • if properties reference custom types, those types must adhere to these rules too
  • must be mutable or immutable
  • must have a default value
  • etc

Pre-converting to JSON would automatically enforce such conditions without forcing your developers to learn yet another API. That’s the function of serializing to JSON first. :slight_smile:

Interesting. I hadn’t considered that angle. Those requirements are not present for serializing to/from JSON? Anyway though I understand the angle now and I’ll keep it in mind when we discuss this feature. Also I will CC @priya.rajagopal to make sure this request is visible.

YES, JSON libraries like GSON absolutely do enforce such requirements, and that’s sort of my point - that popular libraries are probably already doing the kind of type of enforcement that CBL needs to do internally in order to accept custom types. I’d like to see CBL “outsource” this type enforcement, so developers don’t have to learn multiple APIs that essentially perform identical duties. As JSON is much narrower than Java or Kotlin types, I would imagine it would be much simpler for the CBL team to parse and validate JSON than parsing and validating custom types - and this seems like a win/win. I hope this is clear. It’s a bit challenging to express cleanly.

Anyway, thanks again for taking the time to connect. It is much appreciated.