Tip for faster walkthrough when using prebuilt database or filter example on push


#1

Hi All,

If you are using 1.4.1 or any version of couch-base lite which got functionality of push filter replication, if you have greater number of documents and if you are using prebuilt database, you must have noticed that it takes a lot of time in doing walkthrough i.e. _revs_diff calls.

How to reduce it or How to filter push replication?

Simple whenever the app saves any document keep a field in the document which keeps the device id. This ID should be unique. Then one can apply filter on push replication by creating a filter as follows :

private void pushReplicationFilter(Replication replication)
{
pushReplication.getLocalDatabase().setFilter(“byDevice”, new ReplicationFilter() {
@Override
public boolean filter(SavedRevision revision, Map<String, Object> params) {
String nameParam = (String) params.get(“DeviceID”);
return nameParam != null && nameParam.equals(revision.getProperty(“DeviceID”));
}
});
}

Then this can be called when starting the push replication as follows :

private void setPushReplicationFilter(Replication replication) {
pushReplicationFilter(replication);
replication.setFilter(“byDevice”);
Map<String, Object> params = new HashMap<String, Object>(1);
params.put(“DeviceID”,deviceID(Application.getContext()));
replication.setFilterParams(params);
}

and get the device id by as follows

public static String deviceID(Context context) {
try {
return Settings.Secure.getString(context.getApplicationContext().getContentResolver(), Settings.Secure.ANDROID_ID);
} catch (Exception ex) {
return “”;
}
}

by this method documents which have changed by the app in the a particular device will only be pushed to the sync gateway by that device.

By this way you can easily apply the filter on push replication. I have reduced number of calls from 1200 to 34 while working with push filter.

Like the solution if you liked it.

CC :- @househippo

Regards
Pankaj Sharma


#2

Could you explain this issue in more detail? Are you talking about the first replication after the prebuilt database is installed? This shouldn’t be especially slow — if it is, that’s a sign that the saved checkpoint isn’t being used. That was a bug in earlier versions of CBL but it shouldn’t apply to 1.4, so I’m concerned to hear this.


#3

We have a service written in java which use couchbase lite and keep itself synced with sync gateway. And every 30 minutes this couchbase lite directory is zipped and put on CDN. If a new app installs the app ask for the latest zip and gets the pre built db.

Now app unzips the database and usase replace database and start using it. There are 50000 to 100,000 documents which are there in the local DB. So for the first time the App will do walkthrough with sync gateway.

This walkthrough used to take time because of _revs_diff calls.


#4

How exactly do you install the new database? Using the CBL API call, or is it a direct copy?


#5

We first copy the cbl database directory to a path. And then use
mManager.replaceDatabase(dbName, sourcePath);

which was suggested by you. Before that we had an issue where every CBL instance had same private and public ID and all very messing up the check points.

After the replace database the _revs_diff call will happen only once, but the number of calls can be number of documents/50


#6

Sorry, I don’t understand this. Is _revs_diff called once, or multiple times? It sounds like you’re saying both.

If the database has been copied properly using replaceDatabase, the first replication should be as fast as a normal incremental replication, only scanning the changes that were made since the canned database was created. So you should only see _revs_diff called on the new/updated docs in the local database, not on every single document.


#7

So to clarify in other words. When one use the replace database and start pull and push replication on the canned database if there is no filter on push replication then for the first time a walk through is done. In that walk through lot of _revs_diff calls are made. These can be number of documents /50 for android.

Once all these _revs_diff calls are done check point on sync gateway updated , after that only changed document’s _rev_diff is done.


#8

And as I’ve been saying, that must be a bug. Could you file an issue, please?