Prebuilt Database copy not completing

dshyu · March 22, 2022, 1:56pm

I’m building a phone app and we’re trying to have the option of loading a prebuilt database instead of syncing. The regular database seems to copy fine but the sync database seems like it is not. The database is about 4 GB but after the Database.copy call completes, when I look in ADB, the sqlite3 file is only 2 GB.

I can’t for the life of me figure out why it won’t copy the whole sqlite3 file. Any advice would be great.

jianminzhao · March 22, 2022, 5:30pm

Hi @dshyu
Thank you for the question. To be clear,

by “sync database,” do you mean the database generated by pulling from SG?
do you actually lose any documents? Could you please compare the doc count?
Thanks,
-Jianmin

dshyu · March 22, 2022, 6:50pm

Hi @jianminzhao

Yea so the database in question should be the one generated from pulling from the sync gateway. We technically have two databases [app-name].cblite2 and [app-name]-sync.cblite2. The one that does not seem to finish is the sync one.
So if I try to access the database via say, a query on the count of a type of document, I see error messages about the database disk image is malformed, specifically sqlite error code 11. Also, since the sqlite3 file is missing almost 2 GB, I’m assuming something is not getting copied over. I don’t see any errors with the Database.copy command itself, even with logging turned up.

I had initially been running this on the UI thread but I have since moved it to a background thread and I still don’t see the 4 GB file fully copied.

jianminzhao · March 22, 2022, 11:15pm

What version of CBL are you using?

jianminzhao · March 22, 2022, 11:16pm

Could you share the logs?

dshyu · March 23, 2022, 3:18am

Using couchbase lite 2.7.1.

The relevant logs are as follows:

2022-03-22 23:12:28.370 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: {N8litecore8DataFile6SharedE#1}==> N8litecore8DataFile6SharedE 0xdf4c68a8 @0xdf4c68a8
2022-03-22 23:12:28.370 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: {N8litecore8DataFile6SharedE#1} instantiated on /data/data/com.[app name]/files/CouchbaseLiteTemp/MVhkbg/db.sqlite3
2022-03-22 23:12:28.370 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: {DB#2}==> N8litecore14SQLiteDataFileE /data/user/0/com.[app name]/files/CouchbaseLiteTemp/MVhkbg/db.sqlite3 @0xdf5e2820
2022-03-22 23:12:28.370 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: {DB#2} Opening database
2022-03-22 23:12:28.393 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} begin transaction
2022-03-22 23:12:28.393 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} KeyStore(info) set previousPrivateUUID
2022-03-22 23:12:28.393 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} KeyStore(info) set publicUUID
2022-03-22 23:12:28.393 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} KeyStore(info) set privateUUID
2022-03-22 23:12:28.393 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} commit transaction
2022-03-22 23:12:28.394 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} Pre-close housekeeping: 0 of 1011647 pages free (0%)
2022-03-22 23:12:28.653 20471-10523/com.[app name] V/CouchbaseLite/DATABASE: {DB#2} Closed SQLite database
2022-03-22 23:12:28.654 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: {DB#2} Closing database
2022-03-22 23:12:28.654 20471-10523/com.[app name] I/CouchbaseLite/DATABASE: Moving source DB to destination DB…
2022-03-22 23:12:28.657 20471-10523/com.[app name] E/Auditing: [importDatabase] sync copied files

The last line is my own print statement which prints after Database.copy call. The only thing I see is 0 pages free but that seems to be preclose so after the copying? Are there any other logs you need to see?

dshyu · March 23, 2022, 12:07pm

Also, following with the example code here:

github.com

couchbaselabs/mobile-travel-sample/blob/753e5d408c49e48e58741e26ece872c0c6efac3c/android/TravelSample/app/src/main/java/com/couchbase/travelsample/util/DatabaseManager.java#L70

      
        
                DatabaseConfiguration config = new DatabaseConfiguration();
            
            
    config.setDirectory(String.format("%s/guest", appContext.getFilesDir()));
            
            
    try {
                    database = new Database("guest", config);
                } catch (CouchbaseLiteException e) {
                    e.printStackTrace();
                }
            }
            public void OpenDatabaseForUser(String username) {
                File dbFile = new File(appContext.getFilesDir()+"/"+ username, "travel-sample.cblite2");
                DatabaseConfiguration config = new DatabaseConfiguration();
                config.setDirectory(String.format("%s/%s", appContext.getFilesDir(),username));
                currentUser = username;
            
            
    if (!dbFile.exists()) {
                    AssetManager assetManager = appContext.getAssets();
                    try {
                        File path = new File(appContext.getFilesDir()+"");
                        unzip(assetManager.open("travel-sample.cblite2.zip"),path);

When I try create the new database after copying, I get the disk image malformed errors:

2022-03-23 07:58:05.369 22894-24000/com.[app name] I/CouchbaseLite/DATABASE: Opening Database{d4c8d62, name=’[app name]-sync’} at path /data/user/0/com.[app name]/files/[app name]-sync.cblite2
2022-03-23 07:58:05.369 22894-24000/com.[app name] I/CouchbaseLite/DATABASE: {N8litecore8DataFile6SharedE#3}==> N8litecore8DataFile6SharedE 0xdf4c68a8 0xdf4c68a8
2022-03-23 07:58:05.369 22894-24000/com.[app name] I/CouchbaseLite/DATABASE: {N8litecore8DataFile6SharedE#3} instantiated on /data/data/com.[app name]/files/[app name]-sync.cblite2/db.sqlite3
2022-03-23 07:58:05.370 22894-24000/com.[app name] I/CouchbaseLite/DATABASE: {DB#4}==> N8litecore14SQLiteDataFileE /data/user/0/com.[app name]/files/[app name]-sync.cblite2/db.sqlite3 0xdf5e5020
2022-03-23 07:58:05.370 22894-24000/com.[app name] I/CouchbaseLite/DATABASE: {DB#4} Opening database
2022-03-23 07:58:05.371 22894-24000/com.[app name] E/CouchbaseLite/DATABASE: SQLite error (code 11): database corruption at line 66365 of [884b4b7e50]
2022-03-23 07:58:05.371 22894-24000/com.[app name] E/CouchbaseLite/DATABASE: SQLite error (code 11): statement aborts at 1: [PRAGMA user_version] database disk image is malformed
2022-03-23 07:58:05.371 22894-24000/com.[app name] E/CouchbaseLite/DATABASE: database disk image is malformed (11/11)

dshyu · March 24, 2022, 6:53pm

I’ve also used the cblite tool and it does read the database as follows:

C:\Users\800061\AppData\Local\Android\Sdk\platform-tools>cblite file [app name]-sync.cblite2
Database: .[app name].cblite2
Total size: 4GB
Documents: 39791, last sequence 39791
Indexes: xType_index
UUIDs: public f0f66f8c928e45e5ab7df445872cf6ec, private c0bbd24be6ea44eaa472e7c9c9d3511a

That is the correct number of documents that I would expect and the total size lines up as well. I’ll try using the tool to run some queries just to make sure that nothing is corrupted but as it stands, I still feel like something is stopping the Database.copy call from finishing since it only grabs 2GB and I think the missing 2GB is what is leading to the corruption.

jianminzhao · March 25, 2022, 4:22pm

After the database file is copied to a temporary folder, we open and close it before moving it to the destination. In that process, apply SQLite3’s optimization pragma, “optimize” and “vaccum”. It appears that this process corrupted file, “image malformed.”
We have moved to 3.0 now. Do you have plan to move to 3.0?

dshyu · March 25, 2022, 7:31pm

Jian,

This corruption issue is fixed in 3.0? My team may upgrade to 3.0 pending customer input. Also, is the issue dependent on the size of the database?

Thanks!

jianminzhao · March 25, 2022, 8:41pm

I can be certain about it. But if you could find a db containing no sensitive data that can cause the problem, I will be happy to try it.

dshyu · March 28, 2022, 7:46pm

Hey @jianminzhao,

I’ll try and see if we can scrub a database for you guys to test but in the meantime, I was able to update CBL to 3.0 in a test setting. When I run database.copy, it does seem like it now copies all of the sqlite3 file as I can see the 3.9 GB. Additionally, when I use the cblitetool, the document counts are correct. However, I still see database image corrupt errors.

2022-03-28 15:43:17.292 24323-31082/com.[app name] E/CouchbaseLite/DATABASE: database disk image is malformed (11/11)
2022-03-28 15:43:17.292 24323-31082/com.[app name] E/CouchbaseLite/DATABASE: database disk image is malformed (11/11)
2022-03-28 15:43:17.292 24323-31082/com.[app name] I/CouchbaseLite/DATABASE: RELEASE SAVEPOINT roTransaction
2022-03-28 15:43:17.295 24323-31082/com.[app name] E/Auditing: [count] couchbase lite exception is 15: data is corrupted

jianminzhao · March 28, 2022, 10:15pm

Definitely better in 3.0, but, unfortunately, there’s still problem of corrupt data. It would be greatly helpful to us if you could scrub a file for us to repro on our side. Once I receive the file, I will create a ticket for it.

dshyu · March 31, 2022, 6:20pm

@jianminzhao,

Generating a dummy database as soon as I can. Do I just attach it to a comment here?

jianminzhao · April 6, 2022, 12:21am

Do you use google drive? If so, you could put there and send me a one-time sharing URL?

jianminzhao · April 13, 2022, 5:06pm

Hi @dshyu,

Have you got chance to generate a dummy database to help us repro the probem?

Thanks,
Jianmin

dshyu · April 15, 2022, 12:35am

@jianminzhao I’m trying to share a one drive link with specifically you. Is there an email you can provide?

jianminzhao · April 18, 2022, 7:42pm

jianmin.zhao@couchbase.com

blake.meike · April 27, 2022, 4:39pm

@dshyu : Here’s where we are:

We may have seen this issue before. We think it happens on Android (we have seen it in Xamarin Android).
We are unable to reproduce it. @jianminzhao was unable to repro it with the DB you provided and I have been unable to repro it with the DBs provided by others.

Our conclusion is that this, most definitely, is an issue. We are, however unable to get a handle on it. Anything you can do to get us closer to reproducing it would be very, very appreciated.

What kind of device are you using? What version of Android are you running? Can you pull the busted DB off the device and get it to us?

I’m going to try again at a repro as soon as I get a spare moment. We are in the middle of a release, so I may not get back to you immediately. I’ve opened this ticket to track the issue: https://issues.couchbase.com/browse/CBL-3052

dshyu · April 27, 2022, 4:57pm

Hey @blake.meike,

We’re on a Galaxy S10 running Android Version 10. I can’t provide the original production database that saw the issue but I was able to replicate it with a sanitized database which is the one I gave to Jianmin. That particular database has some 300k documents and the sync.sqlite3 database ends up being a little over 8 GB. Our team was considering trying smaller database sizes so are in the process of generating sanitized databases of 200k and 100k. If that’ll help, I can send them to you when they are generated.

Thanks for taking a look at this. Let me know if I can provide anything else.