Ok finally got around to testing the scenarios you requested. For both test cases, I was able to open the database and query the database size. I didn’t test any further behavior but this is as far as I have ever gotten when trying to preload the database.
You had mentioned the database being tied to an ID, possibly hardware related. Am I correct to assume that opening the database as is like we’ve been doing in our tests only works because we are operating on the same phone?
Hey @dshyu,
Thanks to your help, I think we will have a fix for this, shortly, We have discovered one definite problem in our Database.copy routine that would certainly affect large databases. Your tests all but confirm that that is where the problem lies.
Will keep you posted.
The ID that I mentioned is simply a random UUID, meant to distinguish one instance of a database from all others, when they are synchronized through a Sync Gateway. Multiple databases with the same ID will function normallly. The only issue is that if the SGW gets inconsistent updates from multiple database instances that appear to it to be a single instance, it will not be able to optimize synchronization and will be much less efficient.
Checking in to see if there was any update on this.
I also had a couple quick questions, was hoping I could just ping you here instead of making another topic.
We have a couple queries for document counts. One is querying the sync database and one is querying the regular database. Both should have roughly the same document counts but for some reason the sync database query is taking a very long time (roughly a minute and a half). Additionally, we call the query from an Android Worker so it should be a background thread but it seems like the query is still causing the UI thread to hang. Is there any reason the sync database query takes so long? If the call to perform the query is done on a background thread, will the work be done on the same background thread or do I need to specify for couchbase to perform work on a background thread?
Thanks for all your help Blake.
The fix for the Database copy will be in the soon-to-be-released CBL v 3.0.2.
By “sync database” do you mean a database that has a replicator running?
All operations on a database must be threadsafe: they all must seize a single lock before they execute. A query operation, of course, seizes the lock. If you are running other operation, on the UI thread, that also attempt to access the database, they will, also, attempt to seize the database, and will hang the UI. The solution to this is to perform ALL database operations from some worker thread.
Ok sounds good. Can’t wait for the release.
Yea, for our app we called the database that interfaces with the couchbase server as the sync database. So in our application, if we have a worker that calls a function on a background thread and then that function calls query.execute, will couchbase execute the query on the calling worker thread or does it run the query on the UI thread?
In a more general sense, are all queries run on the UI thread by default?
In a more general sense, are all queries run on the UI thread by default?
No. Queries are run on the thread on which their execute method is called. Remember, though, that a query blocks all other use of the database. If something is trying to use the database on a different thread, it will be blocked until the query completes.
In general, it is a pretty good idea to treat the database as essentially single threaded. Trying multi-thread access to it will not speed things up, noticeably (because most uses block other uses, anyway) and actually may cause context-switching delays and even deadlocks.
I understand this was fixed in CBL 3.0.2 but our most recent efforts seem to be running into the same database corruption issue at large database sizes.
We’re running on the same hardware (S10 512GB) and we’ve updated our build.gradle file to use “implementation ‘com.couchbase.lite:couchbase-lite-android:3.0.2’”. Once again it is our sync database that is having issues. The 3 file sizes are WAL- 5 MB, SQLITE3 File 13.4 GB, and the SQLITE3-SHM file is 3.56 MB. After the Database.copy command finished, only the SQLITE3 File is copied and is only 1.42GB. The copy command seems to finish with no errors or warnings but when trying to create the database after the copy call, the create fails saying the data is corrupted.
Is there anything we’re doing wrong? Thanks!
@dshyu Ugg. I am quite confident that we fixed a but that caused a problem exactly like this, in 3.0.2 We reproduced the issue, found code that was clearly to blame, changed it, and could no longer reproduce the problem.
Sorry to ask, but can you, once again, describe the steps you follow to create the issue?
Steps 2-5 should all happen on a non-UI thread. I have try catch blocks around steps 4 and 5 and only 5 is throwing any exception (that the data is corrupt).
If there’s any other info I can provide, please let me know. Preemptively, I’ll try generating a sanitized database that I can share with you guys.
I’m filing a ticket to investigate this ( CBL-4572 ). I don’t know that we do, or can, support that by-hand db copy.
Do you mean step 1? Is there a recommended way to extract the cblite2 folders?
Our non sync database I’ve seemingly been able to import just fine (querying for count lined up) so I didn’t think there was an issue with manually copying them but if that is a potential risk or difference from recommended procedures, I’ll gladly try something else.
@dshyu We can’t support this.
Even Android doesn’t, in general, support using the command line to copy things out of an application sandbox and into external storage. In trying to reproduce this issue, I found that my results depended to a very large extent on the specific device and specific version of Android on the device.
If you have to do something like this, I suggest that your application close (important!!) and zip the database directory, and then export it via the network. We have a fair amount of evidence that this strategy will work.
Forgive my ignorance. Can you verify if my understanding of the steps is correct? I think I’m getting a little confused primarily on how to copy the database files.
First step is to change in code, where we would store the couchbase files. Currently, in our code, it is the default location which I believe is app specific so if I close the app, I can’t access those files anymore. I would need to change the location for the couchbase files so that it persists after the app is closed?
Zip the database files.
Load them in code and use Couchbase to preload database.
@dshyu : of course!!!
I’m not sure I remember exactly what you were trying to accomplish, in the long run. If I recall, one requirement was the ability to preserve the database of an un-installed app.
You may not need to change the location of the database that the app uses. More about that in a minute.
To use a zipped database you should:
To save the database, do the opposite except for the copy:
Putting the database somewhere other than the default location may accomplish two things:
I also sent you a link to the onedrive with the corrupted database as well as the logs that I pulled from that run.
So sorry for the delay. Yea I think my team has some next steps to try. I’ll mark your previous reply as a solution but I’m not sure I see where to mark the conversation as closed. Please feel free to close the conversation if you can do it from your end.
Thanks for all your time and help.