Couchbase Lite Bulk Insert Peformance on Android

Mobile
1

What are some tips for achieving maximum insert performance with Cb Lite on Android?

At current, I’m trying to load about 50K documents and can only load about 10-20 documents per second using a Galaxy Tab 4.

I’m also creating my own id. Is is more performant to do that or let cblite create the id and use a view instead?

I’ve tried using transactions, and that works well using the .NET driver. But I"m getting out of memory exception on Android with this many documents.

Is it possible and supported to bulk load directly to the sqlite file maybe?

Update: I tried bulk insert with letting cblite create the id and am getting speeds of 39 docs/second. That seems to be the better route so far.

Update2: I should clarify that not using my own id and wrapping the insert process in a transaction was providing a rate of 39 docs/sec. But it crashed after about 15K documents. Without a transaction I’m getting aobut 23 docs/sec. Perhaps, wrapping batches of documents in a transaction will get a little more peformance.

2

Hi @Krcourville,
I assume you are using CBL .NET. I am not .NET developer, but I work for CBL Android/Java. Following is my tips for Android.

  • You already uses. Transaction makes database operation faster.
  • Find out best batch size for each Transaction. By using Transaction, data is stored in memory till commit. In case app uses a lot of memory, Android system tries to release releasable memory, it makes app/system really slow.
  • Same reason with previous one, release unused object/memory as soon as possible.

Thanks!

3

That seems awfully slow. (For comparison, an iPhone can insert several hundred documents per second.)

  • It shouldn’t make any difference whether you provide a specific ID or not.
  • Hideki is correct that using a smaller batch size (multiple commits) can help.
  • How large are your documents? If they’re very large, the JSON marshaling can become a significant time factor.
4

Stats on this dataset:

  • Document count: 42K
  • Max doc size: 28K
  • Min doc size: 283 characters
  • Avg doc size: 2.7K

Yeah. That’s pretty much what I’m figuring. Need to find a magic number for the batch size and wrap each in a transaction.

Even still, the next challenge is indexing… This devices takes close to 5 minutes to apply an index against 42K documents. And the index is doing nothing more than emitting two properties if they exist (not null). Not sure what more I can do to improve that.

5

Hi @Krcourville,
Are you using CBL .NET or CBL Android? Which version of CBL are you using?
Thanks,

6

These tests are with Android CBL release 1.0.3.1.

We’re looking to update to the latest release. But doing so pushes us into
the 64k method limit. So we’re working to get multi dex going now.

7

We improved View.updateIndex() performance with current master branch.
If you are using Android Studio, Please try muster branch version of CBL Android by specifying version number “0.0.0-511” in build.gradle.
http://files.couchbase.com/maven2/com/couchbase/lite/couchbase-lite-android/0.0.0-511/

8

Hi @priya.rajagopal . Is there a way to insert a BULK Docs in android mobile. Like inserting 100 documents at a time Couchbase lite 2.0.

9

Refer to the inBatch API. Also assuming you are on a version that is much more recent than 2.0

public void inBatch(@NonNull [Runnable]  runnable) throws [CouchbaseLiteException])

Runs a group of database operations in a batch. Use this when performing bulk write operations like multiple inserts/updates; it saves the overhead of multiple database commits, greatly improving performance.

Parameters:

`runnable` - the action which is implementation of Runnable interface

Throws:

`CouchbaseLiteException` - Throws an exception if any error occurs during the operation.