Can I do a "post only once and forget" type of sync?

I have a requirement like this:

  1. I Create a specific document on the mobile of type say “Feedback”
  2. This type is not sync’ed to the mobile (using an import filter)
  3. When sync’ed I want the document to be sent to the server - and removed from mobile…

How should my import filter and sync. formula look for this to happen?

  1. Save the Document on mobile app CBL and start (or be running) a push replicator (you don’t need a pull unless you have data you don’t want to “forget”).
  2. The Gateway Sync function should not assign the Document to any channel that is synced (back) to the mobile client.
  3. Use the CBL 2.6 Document Replication events to get notified in your app that the Document has been pushed, and Purge it from the CBL.
1 Like

Ah, yes… that should work.

Then I just need to find out how to identify that document in the replication event - and then the purge should not be replicated… - or it is all c0mplicated…

The new Document replication events introduced in Couchbase Lite 2.6 allow you to identify which Documents got pushed or pulled, and it includes the Document ID.

The difference between Delete and Purge is that a Delete replicates to the rest of your system whereas a Purge means “remove locally only and don’t tell anyone”. That is why you need to make sure that the Sync function on the Gateway does not assign a syncable channel to the Document, or it WILL be replicated back to the client in a pull replication.

1 Like

Good point. I use an import filter to keep it away from the mobile.

What should I have in my sync. formula for it to be sent? Just checking that it is a new document, current user is the same and type as expected?

I really find it difficult to understand the “direction” of the doc. exchange inside of the sync. formula. When is it a server -> mobile and when is it mobile -> server sync…?

Couchbase Mobile’s focus is on read optimisation. The channels are used to achieve this. The Sync function is ONLY called when pushing changes from client -> server (write), not server -> client (read). Therefore you have to think ahead in your design to make sure that Documents get assigned to appropriate channels, as changes after the fact are difficult and costly (manual maintenance operation). You assign Documents to channels to make sure that the right client gets the Documents they are supposed to. Clients can filter what channels they want by requesting it in their replicator. Using a client-side filter is inefficient because ALL Documents for eligible channels are first synced to the client, and only then are they discarded if the client doesn’t want them.

What verifications you put in your Sync function depends on your system design. If all your users can access all data there is very little you need to check. Otherwise you will likely have some access rights checks and data validation checks. For your Sync function to be able to receive the Document you don’t need to do any checks, just set the Channel on it. Any checks on Document type or user etc. is up to your system requirements.

I would recommend you read the Couchbase Mobile online documentation about Channels and Replication.

Hmmmm… I’m not sure that it is entirely right that the sync. function is ONLY called when sending from client to server… That is where I set the channels. I understand that channels could be set and saved on each document - but I use the formulas to calculate the channels in one place. And those documents _get sent _ to the mobile…

Here is a snippet:

                // Allow anyone to create a Feedback on the server
                if (oldDoc == null && doc.type == 'Feedback') {
                    _log('Created feedback: ' + (doc._id || 'no id!') + ', key: ' + doc.key + ', user: ' + doc.userkey);
                    return;
                }
            
                // All public docs are available in the app
                if (doc.ispublic) {
                    _log('public, id: ' + (doc._id || 'no id!'));
                    channel('!');
                }
                // All non-specific user info is available (for stats)
                if (doc.type == 'User') {
                    _log('User doc, id: ' + (doc._id || 'no id!'));
                    channel('!');
                }
            
                // Only non-public docs "owned" by user can be replicated
                var userkey = _getUserKey(doc);
                if (userkey != null) {
                    if (oldDoc != null) {
                        // Update
                        if (oldDoc.type != doc.type) {
                            throw ({ forbidden: "Can't change doc type" });
                        }
                        if (oldDoc.key != doc.key) {
                            throw ({ forbidden: "Can't change doc key" });
                        }
                        if (oldDoc.userkey && oldDoc.userkey != doc.userkey) {
                            throw ({ forbidden: "Can't change user key" });
                        }
                    }
                    _log('User owned, id: ' + (doc._id || 'no id!') + ', type: ' + doc.type + ', user: ' + doc.userkey);
                    if(doc.ispublic){
                        requireAdmin();
                    }
                    requireUser(userkey);
                    channel('channel.' + userkey);
                    access(userkey, 'channel.' + userkey);
                }

And I have read the documentation about channels and replication. Several times. But I find it rather confusing… there was a lot of talk about differences between version 1.x and 2.x which is irrelevant to me. So the base concepts of the whole “journey” of a document back and forth somewhat dissappears in the details - and when (at least I find that) the overall concept is a little unclear then I suppose some of the finer details could be missed.

Just a note: I’m not unfamiliar with replication as a concept - I have been working with another database that replicates for 20+ years… So it’s more a matter of understanding the current direction of the document when it hits a block of code in my sync. function… - and it’s not that easy to debug!

I haven’t verified this but my guess is that Documents with no channel set always get replicated to all clients. Because you set no channel on the Feedback type it always gets synced back to your mobile client. I would try setting a channel on it (“feedback”?), which the user does not have access to.

Nope, Feedback docs are not sent to the mobile…

Here is a snippet more from my sync_gateway.json where you can see that guest is disabled - and I set an admin channel that contains all “public” documents:

			"import_docs": true,
            "users": { "GUEST": { "disabled": true, "admin_channels": ["!"] } },
            "import_filter": `
            function(doc) {
               // Some document types not allowed on mobile
                if (doc.type == 'EnvLake' || doc.type == 'EnvMeasurement' || doc.type == 'ActivityLog' || doc.type == 'Image') {
                    return false;
                }
                if ((doc.type == 'FishingTrip' || doc.type == 'Catch') && typeof doc.clubonlykey !== 'undefined' && doc.clubonlykey != '') {
                    return false;
                }
                return true;
            }

The import filter (which I didn’t implement until recently as it makes much more sense for restricting docs. going to mobile) does not mention “Feedback”…

I thought you didn’t want Feedback to sync to the mobile client?

I don’t - and they don’t… So something in my sync. function works Ok to handle that it should not be sent out. But this is what I find difficult to understand (and what I mean with “direction”)… But you’re right as to I probably should add the “Feedback” doc. type to the import filter :slight_smile:

So the way I understand it is:

I have to add a document either to the “public” channel ("!") or to a user specific channel:

requireUser(userkey);
channel('channel.' + userkey);
access(userkey, 'channel.' + userkey);

to make the document go TO the mobile device.

But then I’m uncertain as to how to know when I’m dealing with a document coming FROM the mobile - as I have some rules that needs to be followed (e.g. that a user can only change his/her “own” documents).

I suppose the AddDocumentReplicationListener() event may be able to report back to me what happened with a particular document… So that may be a way to test what is going on.

Coming FROM the mobile through replication as opposed to changes made to the Document on the Couchbase Server itself through some other software?

The Document is going TO the mobile because in the Sync function you assigned it to a channel that the user has access to when the Document was updated coming FROM the mobile (or … ?). When the mobile client pulls changes from the Gateway it doesn’t use the Sync function, it only looks at the channels saved in the Document vs the channels the user has access to (in your case Public and channel.-userkey-).

(doc.type == ‘EnvLake’ || doc.type == ‘EnvMeasurement’ || doc.type == ‘ActivityLog’ || doc.type == ‘Image’)
These document types should have a channel the user on the mobile does not have access to, so they don’t get pulled down, and you don’t need to filter them.

– I say “or … ?” because early on you were not allowed to change Documents directly on Couchbase Server when using Gateway. You had to always use the Gateway to make sure the metadata got updated correctly. This was changed so that even Documents changed through the Server APIs are updated by Gateway to work for replication properly. I don’t actually know if those pass the Sync function, I would assume so.

Yes, I have quite a few documents that could be changed on the server and need to get sync.'ed to mobile. The user can even edit his/her documents on the server (and these changes need to get sync.'ed to the mobile.

I don’t save a “channel” on each document… I understand that was a way to do it in earlier versions. I just checked and there is a “channel” object in the meta data of the documents. So it must be the sync. gateway itself that uses the sync. function to calculate what channels to add to a particular document. That also explains why you have to take the database offline and do a _resync whenever you have made changes to the sync. function that impacts channels.

Those documents that have no channels in their meta data do not sync to the mobile. I guess that could also be handled by the sync. gateway looking at the import filters (that I have been advised to use by @priya.rajagopal) when doing a resync.

I think that the requirement to update docs via the sync.gateway was a thing of the past (pre-version 2.x where I started). I just use the Java SDK to access the data directly in the cluster. There even isn’t yet a sync.gateway in my production environment as the app using the CBLites is under development… :slight_smile:

So it sounds like the concept was easier to grasp earlier.

Basically, I don’t understand when the document ends up in e.g. my code above when there is a userkey (can be on several types of documents)… Does this not happen both for documents being sync’ed TO the mobile (I’m setting the channel for ‘User’ and public documents to “!” - but for the ones belonging only to this user I set the channel for user and gives access)? When an update (or document) is received FROM mobile I guess that the requireUser(,,,) is what allows it to update?

I had another look at the documentation and the only hint that the doc object is FROM the mobile is the “This matches the JSON that was saved by the Couchbase Lite and replicated to Sync Gateway”.

So I guess if I just “accept” that the doc is always the mobile version of a document and the olddoc is always the server version of the document - then I can build my validation correctly - … although calculating the channels in the same function kind of reverses the perspective as we look at doc to find out what channels to assign to it…

Thank you very much for your patience - and I’ll play a little around with the new repl. event! :+1:

From https://docs.couchbase.com/sync-gateway/current/sync-function-api.html

The sync function is called every time a new revision/update is made to a document

When the Document travels from Server -> mobile it is not being updated, just being read. Only when you modify a document on a client, push it to the Gateway, does it pass through the Sync function (or when you make changes to a Document on CB Server when running Gateway).

Determining access when reading a Document is what you would want, it is the easy dynamic scenario, easy to comprehend, this is very much a up-side-down way of thinking. But as the documentation for a certain competing product that works in that way says:

Query-based Sync Permissions
It is worth noting that this added complexity also comes with reduced performance.

Thanks. Yes, I guess that in my mind an “update” is also happening when I write to the database from the serverbased web app - and obviously that confuses the rest of the description slightly :slight_smile:

I must say that I have really spent some time on reading and looking at examples/tutorials when I set all this up. And I don’t find it easy to understand the concept - nor what is “a good practice”…