Execute a MapReduce query takes too long

Mobile
1

Hi
I’m using couchbase-nativescript plugin (with couchbase server 5.1.1 and Sync Gateway 2.0)

Following the readme, I did the following code to create and query a MapReduce view, where viewName is the name of the view that’s going to be created, options has the the executeQuery options and fn is a conditional function to filter some results.

public query(viewName, options, fn){
	this.getDatabase().createView(viewName, "1", function(document, emitter) {
	    if(document.type == viewName){
	        emitter.emit(document._id, document);
	    }
	});
    let list = this.getDatabase().executeQuery(viewName, options);
    let queryList = [];
    if(fn){
        for(var i = 0;  i < list.length; i++){
            if(fn(list[i])){
                queryList.push(list[i]);
            }
        }
        return queryList;
    }else{
        return list;
    }
}

I have documents of the following type in my couchbase server

{
  "data": {
    "abo_aer": true,
    "abo_flu": false,
    "abo_ter": true,
    "cer_com": "3",
    "cod_com": "GTUI-VG-01",
    "com_des": false,
    "id_sec": null,
    "id_sub_are": "c28f226f-8a4d-4a84-b235-2e45d6398df5",
    "met_est": "",
    "niv_end": "3",
    "nom_com": "asdfg",
    "nom_equ": [
      "hjklf",
      "dfgh",
      "cvbn"
    ],
    "ubi_com": {
      "coordinates": [
        2.73593333333333,
        -64.1070833333333
      ],
      "type": "Point"
    }
  },
  "type": "communities"
}

Then I call the following function:

let communityList = this.couchbaseService.query("communities", {}, null);

And it does work, but takes too long to return the data (around 12 seconds only in the executeQuery function for just 350 records). Is there a way to improve that time? I don’t think 350 records should take that long. Note that creating the view may take around 2 to 50 ms and I could move it to another function, but the real problem is during the executeQuery function.

Thank you.

2

I’m not familiar with NativeScript, or a CBL plug-in for it; I assume it’s an unofficial 3rd party project? 12 seconds is orders of magnitude slower than I’d expect, so I’m guessing there’s a lot of inefficiency inside the plugin implementation.

Your code also has some problems. You only need to create one view, which would emit the ‘type’ property as the key and null as the value. Then you can query it for docs of any type by setting the query’s ‘key’ property to that type string. This will save a lot of overhead since indexing only needs to be done once.

3

It just occurred to me: is this NativeScript plugin a server-side thing? Since you haven’t mentioned Couchbase Lite at all. If so, you should be asking in a sever SDK forum instead.

4

This is the plugin I’m using. Indeed, it uses Couchbase Lite, but I’ve already pulled the data to the device.

What I’m doing is to create a view for each type of document, why would I create only one if I always want to filter by the type? I thought that would make the executeQuery take longer as it would have even more records.

Reading again your answer, I wonder if I misunderstood the use of the emit function. The examples always uses emit(doc._id, doc) and I thought it was the way to store the record with their id. Anyway, after changing it to emit(document.type, null) it seems like it takes the same time…

5

It looks like you’re performing the query right after creating the view. Typically you’d set up your views at start up. Otherwise you’re waiting while it builds the whole view. The time does still seem long, even in that case though.

You definitely don’t want to emit the whole document. Jens’ suggestion is a typical approach, but it depends on problem details. You can do a key/value retrieval of the whole document very quickly. If you store the doc in the index, it becomes bloated without adding much value.

Are you sure your data is all synced to the device when you run your test? (I know you said that, but just checking details. E.g. maybe you have sync running but it wasn’t actually finished.)

6

Don’t have to say it again, I will move the createView function (but as I said, that’s not the problem here).

I tried what Jens said, but I got the same results. The only difference I got is that the executeQuery function now gives me some more records (like 415), but still takes like 15 seconds. Maybe there is something else I need to do. Can you explain me about storing the doc in the index?

I know my data is all synced cause I have some logs that gives me that information.

7

@jens yes, it’s a Couchbaselabs project.

8

Hmm. Did you up the version number when you changed the code (the “1” parameter, I believe)? Otherwise it won’t recreate the view from scratch. I’m still not sure that explains it, but worth a try.

How large are you documents?

It’s strange you got more records after refactoring. How many are you expecting? (Maybe check through Couchbase Server console.)

So, about storing the doc in the index. By emitting the whole document, you’re duplicating it into the index. I don’t know all the details, but you can imagine it causing performance problems for several reasons. If you have large docs, during a query, you’ll get CPU cache misses, possibly force moving data in and out of main memory, and so on. You’re basically doubling the footprint of your database.

9

@amlibtest: You should be aware that a view is indexed lazily, the first time it’s queried. So creating a view is fast; it’s the first call that takes time. (And thereafter, the next call after docs are updated will re-index those docs, so it will take extra time proportional to the number of docs that have changed.) Have you benchmarked how long the second call takes?

Attn @nraboy … any idea whether there are performance problems with views in NativeScript?

10

@hod.greeley I haven’t seen a function on the plugin that allows me to do that…
My largest type of documents are like the first one posted at the start of this thread.
It is not strange to get more records after this change: my original view was getting only the documents with a specific name for type (I’m using type as a way to differentiate between records of each kind, like a table name in SQL) so before making the change I was getting only “communities” records and after that I’m getting all records on couchbase.

@jens Actually, I did a second call of executeQuery in the function and it took the same amount of time

11

If the second call was also slow, there must be some performance problem in the NativeScript binding … sorry, I can’t help with that as it’s not an official project. Maybe @nraboy, its author, can help.