Yep - I restored a fresh copy from my production environment - and flushed the bucket first. That should make a “clean” environment I guess?
That clean up should suffice.
Ok, so by “brute force” testing I have a solution that seems to work - but I still don’t understand why the problematic docs are writes nor why the types put in the filter still replicates to mobile.
As you noted in one of your earlier posts, you were able to verify that documents that you filtered out in the import filter function don’t get synced.
Only thing I can think of is if you may have inadvertently synced the document through some other means like through the mobile app (from an earlier version of app that had it locally persisted) . Also, can you confirm that you did not create these documents via the Sync Gateway REST API.
Because I can guarantee you that if you have your import filters filtering docs of specific type that are created directly on couchbase server, it will not be available for replication (and you have verified that as well)
I would suggest reviewing and debugging using the sync gateway logs to see exactly what’s going on.
I guess one difference between my “A1” and “A2” tests were that I set up the filter prior to creating those docs. in the database (directly via the Couchbase web console). For my real data the docs have already been created (via the Java SDK) prior to setting the filter and starting the sync.
I removed the local app (and thus the data) prior to starting the app with the sync.
So if I do a pull-only replication only the import filter should come into play, right? There is an error right now where it crashes if I set up pull-only so right now I have to start a pull-push even for an anonymous user (until that has been fixed in the CBLite version). But for a “blank” database where I don’t change/create any documents there should not be any writes should there?
I just tried to add this line as the first in my sync function:
throw ({ forbidden: "Don't do any updates... id=" + doc._id });
Then I stopped and started sync.gateway service, took database offline and ran resync. This results in a bunch of entries in sg_warn.log:
2019-10-13T12:33:47.477+02:00 [WRN] Error calling sync() on doc "Catch:14714977730634900544237": 403 Don't do any updates... id=Catch:14714977730634900544237 -- db.(*Database).UpdateAllDocChannels.func1.1() at database.go:1049
2019-10-13T12:33:47.479+02:00 [WRN] Error calling sync() on doc "User:Private:F3E33BA419523FD0C1258371005A2AA9": 403 Don't do any updates... id=User:Private:F3E33BA419523FD0C1258371005A2AA9 -- db.(*Database).UpdateAllDocChannels.func1.1() at database.go:1049
2019-10-13T12:33:47.480+02:00 [WRN] Error calling sync() on doc "User:EE9E6F4761B8B67AC125823C007D7E6C": 403 Don't do any updates... id=User:EE9E6F4761B8B67AC125823C007D7E6C -- db.(*Database).UpdateAllDocChannels.func1.1() at database.go:1049
and sg_info.log:
2019-10-13T12:33:13.001+02:00 [INF] CRUD: Doc "User:6D4732DBE37F8F65C1257F8B00715C53" / "1-e6e713feb8d085d12a4d8fca0697decf" in channels "{}"
2019-10-13T12:33:13.001+02:00 [INF] Access: Doc "User:6D4732DBE37F8F65C1257F8B00715C53" grants channel access: map[]
2019-10-13T12:33:13.001+02:00 [INF] Access: Saving updated channels and access grants of "User:6D4732DBE37F8F65C1257F8B00715C53"
2019-10-13T12:33:13.012+02:00 [INF] c:#003 Sync fn rejected doc "User:Private:647A5DDBD951AD33C125804000765A49" / "" --> 403 Don't do any updates... id=User:Private:647A5DDBD951AD33C125804000765A49
2019-10-13T12:33:13.012+02:00 [WRN] Error calling sync() on doc "User:Private:647A5DDBD951AD33C125804000765A49": 403 Don't do any updates... id=User:Private:647A5DDBD951AD33C125804000765A49 -- db.(*Database).UpdateAllDocChannels.func1.1() at database.go:1049
2019-10-13T12:33:13.012+02:00 [INF] CRUD: Doc "User:Private:647A5DDBD951AD33C125804000765A49" / "1-1c4396ab6b47817a3e89874cb3ccb4c9" in channels "{}"
2019-10-13T12:33:13.012+02:00 [INF] Access: Doc "User:Private:647A5DDBD951AD33C125804000765A49" grants channel access: map[]
2019-10-13T12:33:13.012+02:00 [INF] Access: Saving updated channels and access grants of "User:Private:647A5DDBD951AD33C125804000765A49"
2019-10-13T12:33:13.014+02:00 [INF] c:#003 Sync fn rejected doc "User:E04C91A83EB57FDEC125825D0047E942" / "" --> 403 Don't do any updates... id=User:E04C91A83EB57FDEC125825D0047E942
2019-10-13T12:33:13.014+02:00 [WRN] Error calling sync() on doc "User:E04C91A83EB57FDEC125825D0047E942": 403 Don't do any updates... id=User:E04C91A83EB57FDEC125825D0047E942 -- db.(*Database).UpdateAllDocChannels.func1.1() at database.go:1049
So, should I not run resync when having changed the sync. function? - or how should my sync. function allow the resync to do it’s job? By requiring Admin, or…?
Well, apparently that is not the way to do it… Following the above changes I started the app again (without a database). And it started replicating (I saw doc ids being sent to the mobile) - but no data was sent (or rather the database contains 0 documents…).
This is from thesg_info.log:
2019-10-13T13:13:23.716+02:00 [INF] HTTP: #031: GET /data/_blipsync (as anonymous)
2019-10-13T13:13:23.716+02:00 [INF] HTTP+: #031: --> 101 [bca696f] Upgraded to BLIP+WebSocket protocol (as anonymous) (0.0 ms)
2019-10-13T13:13:23.716+02:00 [INF] WS: c:[bca696f] Start BLIP/Websocket handler
2019-10-13T13:13:23.737+02:00 [INF] SyncMsg: c:[bca696f] #1: Type:getCheckpoint Client:cp-RqNn1eSH+yes9fjmP2JkXxcVOCc=
2019-10-13T13:13:23.770+02:00 [INF] SyncMsg: c:[bca696f] #2: Type:subChanges Since:127588 Continuous:true Filter:sync_gateway/bychannel Channels:!
2019-10-13T13:13:23.771+02:00 [INF] Sync: c:[bca696f] Sending changes since 127588
2019-10-13T13:13:23.771+02:00 [INF] Changes: c:[bca696f] MultiChangesFeed(channels: {!}, options: {Since:127588 Limit:0 Conflicts:false IncludeDocs:false Wait:true Continuous:true Terminator:0xc003f48420 HeartbeatMs:0 TimeoutMs:0 ActiveOnly:false Ctx:context.Background.WithValue(base.LogContextKey{}, base.LogContext{CorrelationID:"#031"}).WithValue(base.LogContextKey{}, base.LogContext{CorrelationID:"[bca696f]"})}) ... (to anonymous)
2019-10-13T13:13:23.771+02:00 [INF] Sync: c:[bca696f] Sent all changes to client
2019-10-13T13:13:23.774+02:00 [INF] SyncMsg: c:[bca696f] #3: Type:proposeChanges #Changes: 0
2019-10-13T13:17:52.105+02:00 [INF] HTTP: #032: GET /
2019-10-13T13:17:52.105+02:00 [INF] HTTP+: #032: --> 200 (0.1 ms)
Especially IncludeDocs:false I find interesting…???
Ok, so I changed my sync. function and added this at the beginning:
if (doc.ispublic) {
channel('!');
}
requireAdmin();
return;
… and now the docs. are added to the “!” channel. And I suppose the requireAdmin() is required for the REST call (resync) to be able to update the docs in the database?
So I guess what I find confusing is that the sync. function does have an impact on reads as well as writes. And it all very easily gets mixed up.
I’ld like to get a good “pattern” for building the sync. function. I would expect something like this:
- Deletions - requireUser if doc has userkey else requireAdmin. Then stop processing (–> return)
- Validation of required fields being present (e.g. key and type). Throw error if not Ok
- Check documents to see if they are public and add to “!” channel - and requireAdmin (is this right??)
- Allow creation of docs (no oldDoc) without userkey (restrict to certain types). Then stop processing (–> return)
- If a doc has a userkey then check userkey is not changed (if not a new doc), add to user-channel, give access to the channel for the user and requireUser.
- If there is no userkey then throw an exception that doc. cannot be created/updated.
I have attached my current sync. function to reflect the above.
Is the above pattern a “good” one? I don’t get the exact same number of docs. on mobile as I would expect when querying the database directly on the server. I’m investigating why - but I guess it has to do with something not being right in the sync. function.
Currently, I have two else’s on item 5-6 to say that if there is no userkey. The first one checks if doc. ispublic and then requireAdmin - but I guess that is already covered earlier in the code where I add public docs to “!” channel and requireAdmin so that this one has no effect?
What happens if a doc is public and has a userkey for the user? This is a valid combination as a user can select to make a doc. public (so others can see it). Then I would have called requireAdmin - and requireUser() - but how will you interpret that? As per my earlier tests I seem to need requireAdmin for the resync to work… Argh… really confusing that I cannot make it work out… 
Any comments on my understanding about the above is much appreciated… 
function (doc, oldDoc) {
function _log(t) {
// Write to sg_info.log
console.log('SG: ' + t);
}
function _getUserKey(d) {
var key = null;
if (d) {
if (d.type == 'User') {
key = d.key;
} else {
key = d.userkey;
}
}
return key;
}
if (doc && doc._deleted) {
// Doc. deleted -> if public then require
if(oldDoc){
var userkey = _getUserKey(oldDoc);
_log('delete doc id: ' + doc._id + ', userkey=' + userkey);
if (userkey != null) {
requireUser(userkey);
} else {
requireAdmin();
}
}
_log('doc deleted, id: ' + (doc._id || 'no id!') + ', ' + (oldDoc ? ('old key=' + oldDoc.key + ', userkey=' + userkey) : 'no oldDoc'));
return;
}
_log('doc id: ' + (doc._id || 'no id!') + ', ispublic: ' + doc.ispublic + ', userkey=' + userkey + ', ' + (oldDoc ? (oldDoc._deleted ? 'oldDoc is deleted' : ('old key=' + oldDoc.key + ', oldDoc.userkey=' + oldDoc.userkey + ' update')) : ' creation'));
// Document type is mandatory
if (typeof doc.type === 'undefined') {
_log('Document type missing: ' + JSON.stringify(doc));
throw ({ forbidden: "Document type is required. id=" + doc._id });
}
// Document key is mandatory
if (typeof doc.key === 'undefined') {
_log('Document key missing: ' + JSON.stringify(doc));
throw ({ forbidden: "Document key is required. id=" + doc._id });
}
// Update: Cannot allow change of type or key
if (oldDoc != null && !oldDoc._deleted) {
// Update
if (oldDoc.type != doc.type) {
throw ({ forbidden: "Can't change doc type" });
}
if (oldDoc.key != doc.key) {
throw ({ forbidden: "Can't change doc key" });
}
}
// Document sync is disabled (used for type Image)
if (doc.issyncdisabled) {
throw ({ forbidden: "Sync. disabled for id=" + doc._id });
}
// All public docs are available in the app
if (doc.ispublic) {
_log('public, id: ' + (doc._id || 'no id!'));
channel('!');
requireAdmin();
}
// All non-club fishing trips and catches are available (for stats)
if ((doc.type == 'FishingTrip' || doc.type == 'Catch') && doc.clubonlykey === 'undefined') {
_log('non-club trip/catch, id: ' + (doc._id || 'no id!'));
channel('!');
requireAdmin();
}
// All users are available (for stats)
if (doc.type == 'User') {
_log('User doc, id: ' + (doc._id || 'no id!'));
channel('!');
requireAdmin();
}
// Allow anyone to create a Feedback or Observation on the server
if (oldDoc == null && doc.userkey == null && (doc.type == 'Feedback' || doc.type == 'Observation')) {
_log('Created ' + doc.type + ': ' + (doc._id || 'no id!') + ', key: ' + doc.key + ' as anonymous user ');
return;
}
// Only non-public docs "owned" by user can be created/updated (and replicated)
var userkey = _getUserKey(doc);
if (userkey != null) {
if (oldDoc != null && ! oldDoc._deleted) {
// Update
if (oldDoc.userkey && oldDoc.userkey != doc.userkey) {
throw ({ forbidden: "Can't change user key" });
}
}
_log('User owned, id: ' + (doc._id || 'no id!') + ', type: ' + doc.type + ', user: ' + userkey);
channel('channel.' + userkey);
access(userkey, 'channel.' + userkey);
requireUser(userkey);
} else if(doc.ispublic){
requireAdmin();
} else {
// Creation/update without user
_log('Document type cannot be created without user key: ' + (doc.type === 'Image' ? doc._id : JSON.stringify(doc)));
throw ({ forbidden: "This document type cannot be created without user key. id=" + doc._id });
}
}