Cbbackupmgr restore is not recognised by sync. manager

I’m in the process of migrating a cluster from one data centre to another. The sites cannot see each other so I have to backup/restore using cbbackupmgr.

The solution consists of a web based app that reads/writes to the database using the Java SDK and a mobile app using Xamarin (C#) updating through the Sync.Gateway. Any changes made in the web app are discovered by the mobile app instantly :slight_smile:

When I restore an incremental backup I see the changes in the web app so data is correctly restored. However, my mobile app doesn’t seem to “discover” the updates… Nothing is synced to the app.

I have tried with sync. manager running or stopped when doing the restore - same result. I also tried to take the database offline via Postman (http://sg1.domaino.dk:4985/data/_offline), then do a “resync” (http://sg1.domaino.dk:4985/data/_resync) and take it back online again (http://sg1.domaino.dk:4985/data/_online). But it still doesn’t sync. the new docs to the app… If I make any changes to the database (via a web app using the Java SDK to write directly to the DB) they show up on the mobile app at once. This is expected from these settings:

"enable_shared_bucket_access": true,
"import_docs": true,

in the sync_gateway.json

The installation is all Community Edition. Couchbase server: 7.1.1 and sync. gateway 2.8.3

Any ideas why Sync.gateway doesn’t see these changes from the restore?

Someone more familiar with sync_gateway configuration may need to explain if this is a sync_gateway configuration thing … but one thing about cbbackupmgr restores are that, in general, the restored documents will have the same CAS values (part of document metadata that’s also used as a timestamp) as when the documents were backed up. So, it’s possible that the sync is not recognizing these documents as “new” updates or “new” documents that need to be synced. Again, someone more familiar with sync_gateway configuration and setup would need to explain if this is possible. If that is the case, however, one workaround might be to do something like “UPDATE collectionName set Field1=Field1” that will update every document without actually updating any of the values (but would update CAS and revId, etc.). For example, the below would update the CAS of all documents in travel-sample bucket’s _default._default collection.

update `travel-sample`._default._default set id=id

Additional notes:
If you did a cbbackupmgr restore to a bucket that already has data, then cbbackupmgr will use conflict resolution to not restore “older” documents in the backup if the bucket already has “newer” document with the same doc id (key). To over-ride that, you can use the --force-updates option with cbbackupmgr restore. In 7.1, cbbackupmgr restore with --force-updates will restore the “older” document in the backup and update the CAS to current.

Thanks for your response.

The scenario I have is that I’m in the process of moving a cluster from one data centre to another. I found inspiration to the procedure from this article about migrating from a CB Server to CB Capella: How To Migrate From Couchbase Server to Couchbase Capella

Everything worked fine on the restore of the first “large” restore to the new servers. Now, I have had to do some testing and adjustments on the cluster as well as the web application being moved at the same time - so basically once I’m ready (which is almost now) I need to stop access to the current data centre, take an incremental backup and restore the changes since last backup to the new server.

If I have to create a new backup repo and backup everything again then it just seems a waste of time and extra downtime for the users - hence the idea of taking the incremental backup and restore only that. Following your explanations above I tried to re-restore using --force-updates but unfortunately that didn’t make the changes from the latest backups sync. to the app.

I’m not really keen on forcing an update of ALL documents as there are 171,000+ on the mobile app at present - and that sort of kills the idea of having e.g. a built-in database that we use for first time setup.

Perhaps I should try and clear all documents in the database and do a full restore of first and incremental backups…? I have those backups on the new server already so it would only add the restore time to the entire process.

I tried to update all documents created after a certain time (updates are a little more complicated - so wanted to test creation first):

UPDATE data set type = type
WHERE type IS VALUED
    AND revisioninfo.created >= '2022-07-11'

This updated a couple of hundred documents - but made no difference on the mobile app… :frowning:

So now I tried to flush the entire database in the new data centre (while having stopped sync.gateway) and restore everything (full and incremental backups). It obviously takes quite some more time!

After I restarted the sync.gateway it took a looooong time to sync. But the number of documents seems to be right.

So that seems to be a way around it… though I still quite understand why it doesn’t work with an incremental restore :smiley: i