I’ve written a poc to see what is the capability of couchbaselite in android. My aim was that I would change the database layer from sqlite to couchbaselite if everything would gone well, however it seemed not like what I dreamt about so far and I’ve decided to ask something here to understand better.
First of all I need to mention that my db is a little bit different than a standard mobile db. There are at least 1.5 million records in a sqlite table so what I expect from couchbase lite is to run with the same performance or better than sqlite. I simply planned that I would use sync gateway to get all the records during the first opening of the application and then server could feed me later when a change happened. Shortly I wanted to use sync feature.
here is my questions:
While working within up to 1000 records everything seemed well. However as I said that I have to work within a table which contains around 1.5 million records. Does it make sense to be running with 1.5 million records in couchbase lite? (If yes, my queries are very slow, sometimes I also get out of memory, do I make wrong something?)
Since there is no support for N1QL in couchbaselite android (afaik) is there any other way to get (for instance) row count rather than map/reduce?
How long it can take to pull 1.5 million records from server to mobile phone? (wifi quality is quite enough) (I’m getting 15000 records in around 20 minutes)
You should be able to work with 1.5 million records. If you’re running into problems, we very much want to know. Please give details here, or open an issue on github.
But do you really need each instance of your application to have access to all 1.5 million records? This seems unusual. Have you looked at separating data using channels in Sync Gateway?
Map/reduce should be quite efficient. It takes time during the initial load of the documents, but once that has happened it should be ok.
Maybe it will help you to reconsider the structure of your data? You talk about # or rows. It could help if you post some details of what you have done to restructure your data to fit use in a document database.
What are you emitting as the key/value results for your map? That can have a large impact.
15000 records in 20 minutes sounds slow. @adamf can give a better answer.
Have you considered preloading the database? You do this by including some or all of the initial data you need as part of the original application.
But do you really need each instance of your application to have access to all 1.5 million records? This seems unusual. Have you looked at separating data using channels in Sync Gateway?
Yes, I really need each instance of my application to have access to all those records. 1.5 million records are our product records and they contain barcode information, stock code information, descriptions, etc. While our store supervisors are making the daily basis (such as checking the packages, accepting the goods which are coming from the manufactory, checking the products, counting the products for verification, etc) with hand terminals (which are android phones) sometimes there may no internet connection where they work so we think that once they get those hand terminals to sync with server, they can work with devices as offline without any problem.
As I said my previous comment, I’ve made the synchronization with the server but I got lots of out of memory exception while received item count reached about 150.000.
Maybe it will help you to reconsider the structure of your data? You talk about # or rows. It could help if you post some details of what you have done to restructure your data to fit use in a document database. What are you emitting as the key/value results for your map? That can have a large impact.
I just create simple document with 4 or 5 properties (for testing purposes) as you’ll find in the poc codes above
15000 records in 20 minutes sounds slow. @adamf can give a better answer.
I can create 1000 documents in 10 seconds locally, that was the duration while getting the documents using gateway.
I’ll be very happy if you can emphasise where am I doing wrong by above code.
I was making a huge mistake, instead of getting the documents by their id I was querying them over views with the keys. That’s why my queries were running very slow. I’ve fixed it and runs perfect however still I’m getting “out of memory” while using sync gateway whenever count size reaches to 200.000.
Do you have any idea why its happening after 200.000 records?
Ps: other problems which I mentioned above are not valid anymore.
This time I didn’t get out of memory exception. I’m not sure if this is the reason or not but I’ve changed db as forestdb. Now I started to get java.lang.RuntimeException: Can’t create handler inside thread that has not called Looper.prepare()
11-30 12:57:39.531 328-551/? A/fusion_persist: Run db version matches for option 4 == 4
11-30 12:57:39.570 328-10136/? E/WifiHW: [LIB_LEGACY]-[wifi_unload_driver]-[driver was started and must not be unloaded…return success 11-30 12:57:39.718 3461-3497/com.couchbase.helloworld E/Sync: Exception calling changeListener.changed java.lang.RuntimeException: Can’t create handler inside thread that has not called Looper.prepare()
at android.os.Handler.(Handler.java:121)
at android.widget.Toast$TN.(Toast.java:322)
at android.widget.Toast.(Toast.java:91)
at android.widget.Toast.makeText(Toast.java:238)
at com.couchbase.helloworld.MainActivity$1.changed(MainActivity.java:108)
at com.couchbase.lite.replicator.Replication.changed(Replication.java:425)
at com.couchbase.lite.replicator.ReplicationInternal$13.run(ReplicationInternal.java:1262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
11-30 12:57:39.789 328-551/? A/fusion_persist: Run db version matches for option 4 == 4
11-30 12:57:39.796 328-328/? E/WifiService: Param is :UserType
11-30 12:57:39.804 328-328/? A/fusion_persist: Run db version matches for option 4 == 4 11-30 12:57:39.867 3461-3497/com.couchbase.helloworld E/Sync: Exception calling changeListener.changed java.lang.RuntimeException: Can’t create handler inside thread that has not called Looper.prepare()
at android.os.Handler.(Handler.java:121)
at android.widget.Toast$TN.(Toast.java:322)
at android.widget.Toast.(Toast.java:91)
at android.widget.Toast.makeText(Toast.java:238)
at com.couchbase.helloworld.MainActivity$1.changed(MainActivity.java:108)
at com.couchbase.lite.replicator.Replication.changed(Replication.java:425)
at com.couchbase.lite.replicator.ReplicationInternal$13.run(ReplicationInternal.java:1262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
11-30 12:57:39.882 3461-3497/com.couchbase.helloworld E/Sync: Exception calling changeListener.changed
java.lang.RuntimeException: Can’t create handler inside thread that has not called Looper.prepare()
at android.os.Handler.(Handler.java:121)
at android.widget.Toast$TN.(Toast.java:322)
at android.widget.Toast.(Toast.java:91)
at android.widget.Toast.makeText(Toast.java:238)
at com.couchbase.helloworld.MainActivity$1.changed(MainActivity.java:108)
at com.couchbase.lite.replicator.Replication.changed(Replication.java:425)
at com.couchbase.lite.replicator.ReplicationInternal$13.run(ReplicationInternal.java:1262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
11-30 12:57:40.023 10143-10143/? A/libc: Fatal signal 13 (SIGPIPE) at 0x0000279f (code=0), thread 10143 (sh)
11-30 12:57:40.281 461-461/? E/wpa_supplicant: TI: Driver not initialized yet
11-30 12:57:40.281 461-461/? E/wpa_supplicant: TI: Driver not initialized yet
11-30 12:57:40.281 461-461/? E/wpa_supplicant: [SUPPLICANT]-[wpa_supplicant_scan]-[271]-[Driver in Idle state ignoring scans] 11-30 12:57:40.429 3461-3497/com.couchbase.helloworld E/Sync: Exception calling changeListener.changed
java.lang.RuntimeException: Can’t create handler inside thread that has not called Looper.prepare()
at android.os.Handler.(Handler.java:121)
at android.widget.Toast$TN.(Toast.java:322)
at android.widget.Toast.(Toast.java:91)
at android.widget.Toast.makeText(Toast.java:238)
at com.couchbase.helloworld.MainActivity$1.changed(MainActivity.java:108)
at com.couchbase.lite.replicator.Replication.changed(Replication.java:425)
at com.couchbase.lite.replicator.ReplicationInternal$13.run(ReplicationInternal.java:1262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
11-30 12:57:40.671 10155-10155/? A/libc: Fatal signal 13 (SIGPIPE) at 0x000027ab (code=0), thread 10155 (sh)
11-30 12:57:41.335 10166-10166/? A/libc: Fatal signal 13 (SIGPIPE) at 0x000027b6 (code=0), thread 10166 (sh)
11-30 12:57:42.585 328-389/? E/WifiStateMachine: Error! unhandled message{ what=131156 when=-1ms }
11-30 12:57:44.734 10217-10217/? A/libc: Fatal signal 13 (SIGPIPE) at 0x000027e9 (code=0), thread 10217 (sh)
11-30 12:57:46.476 10244-10244/? A/libc: Fatal signal 13 (SIGPIPE) at 0x00002804 (code=0), thread 10244 (sh)
11-30 12:57:47.132 10252-10252/? A/libc: Fatal signal 13 (SIGPIPE) at 0x0000280c (code=0), thread 10252 (sh)
That error you’re getting is related to Android. It’s a common mistake, with an obscure error. It’s because you’re trying to put up a Toast from a worker thread, not the UI thread.