If sync gateway is running (with a sync function, in my case), saving a document directly to Couchbase via the web dashboard or any SDK will lead to OnUpdate() firing twice for all eventing functions.
Documents saved through sync gateway results in OnUpdate() firing once, which is the expected behavior.
When I kill sync gateway, saving documents to Couchbase results in OnUpdate() firing once, which is the expected behavior.
This is 100% reproducible. These double-events are rather impacful to our system. I am using Server 5.5 Beta, so I am not sure if this will be fixed in 6.0?
This is a known issue, when Sync Gateway & Eventing are leveraging same bucket. MB-29360 captures additional details.
Basically, Sync gateway would update metadata of the document within the bucket - which in turn generates an event for Eventing to process. Eventing can’t differentiate between events from Sync Gateway vs other events(doc updates via SDK, N1QL UPSERT and others).
Presently, there is no workaround to suppress OnUpdate() firing twice. We are actively in discussion with DCP team to allow differentiation between doc vs metadata updates in upcoming releases.
This is a known issue being tracked using this ticket.
A possible workaround could be to add an attribute(say ‘source’) in the json and track from where it got updated - i.e, via mobile or other applications. But, I do understand this might not be the most elegant or a feasible solution.
I don’t think I understand your workaround. If I add source: application, won’t I still get two events w/ that same JSON? I suppose I could use my own revision counter, but that seems like overkill.
The source attribute will be updated based on the application from which it is updated. the OnUpdate handler will still be triggered twice, but you can use an if clause at the start of the code which checks if the source equals the application from which you want to handle the mutation. This workaround will not work if you want to handle mutations from across the applications.
I do not assume your OnUpdate handler code to be idempotent - but do you?
Ok, understood - I don’t think that will work for us as we have many apps / services that can mutate documents. We actually have several scenarios where our OnUpdate handler code is assumed to be idempotent, as we’re using eventing to publish to a message broker that has many subscribers. Duplicate events leads to duplicate publish messages leads to a ripple of unnecessary actions. I’m wondering if we need to rethink our dependence on eventing.
The event function on the bucket w/ sync gateway inserts into a second bucket that does not have sync gateway, using a rev counter as part of the ID. The first insert will succeed, all subsequent inserts will fail as the ID already exists. I then trigger the event function in the second isolated bucket. Small TTL in the second bucket as I’m only using these docs as triggers.
Really messy and I’d definitely want the real fix before going to production, but would this be a reasonable approach?
This assumption may not work. Sync Gateway updates Extended attributes for a document in the bucket, which also increments revision ID.
As a workaround, you could generate digest of relevant document fields and store it as part of the document itself. If the digest hasn’t changed, which would be the case when Sync Gateway updates a document’s metadata - you could return from OnUpdate/OnDelete or any other Javascript function immediately. Otherwise, if digest has changed - then you run your usual application logic.
function OnUpdate(doc, meta) {
// digest verification logic
if (doc.digest) {
docDigest = digest(doc.field1, doc.field2, doc.field3);
if docDigest === doc.digest {
log("Skipping and returning because of unchanged digest, probably because of metadata update from SG")
return;
}
} else {
// digest being created for the very first time
doc["digest"] = digest(doc.field1, doc.field2, doc.field3)
}
// application logic
}
function digest() {
var i, sum = 0;
for (i = 0; i < arguments.length; i++) {
sum += arguments[i];
}
return sum;
}
Actual fix for this issue would be to differentiate b/w metadata vs document update within our key-value engine. Will drop a note to KV team about it - MB-30901
Thanks @asingh, yeah I should have clarified “a rev counter” would be my own digest. I tried this moving from one bucket to another, but I still saw two events fired… perhaps the second event triggered before the document was saved?
One question based on your comment above. Does eventing now support updating the source document directly? I know this was not supported in 5.5 (we had to update a target bucket) but this would definitely be a great addition.
Given SDK is doing updates to the document(whose metadata gets modified by SG) - the app leveraging SDK would need to stamp digest to the document. That would allow Javascript logic within Eventing to figure difference b/w actual doc vs metadata update.
We’re shipping it in next release(after 6.0). Would be interested to provide you an early developer build in few weeks timeframe, if you would like.
We have plans for implementation that will automatically detect and suppress these. But that will likely not make it to upcoming release. In the meanwhile, we’d like to provide an easy way to generate a digest, so you can do logic like:
function onUpdate(doc, meta) {
var crc = crc64(doc);
if (lookup_bucket[meta.id] == crc) return; // already seen
lookup_bucket[meta.id] = crc;
}
We are planning to add the crc64() macro in the upcoming release to enable logic such as above.
Isn’t this very critical for automatic conflict resolution? The document with higher revision number would be taken priority which makes Mobile App data rarely get synced when there are conflicts.