Anyway to do XDCR excluding Delete Ops?

Couchbase Server
1

Just wondering … if it is possible to define an XDCR from one cluster to another excluding DELETE op. Requirement is like the following two XDCRs one from A --> B and another from B --> A:

  1. Cluster A needs to be able to perform Writes, as well as client at Cluster B also can perform some Writes
  2. Cluster A can obtain XDCRed data from Cluster B
  3. Cluster A is somewhat constrained in infrastructure, so we need to keep only last 3 months worth of data. So any doc older than 3 months can be deleted here;
  4. Cluster A data should be XDCRed into Cluster B, however B should keep all the data that ever created - so we do NOT want the DELETE ops happened at Cluster A to propagate to Cluster B.

Possible?

~AP

2

Hi @appajee1,

I don’t think there is anything like this currently built-in to XDCR as far as I know.

However, this might be a good use case for the new Eventing service. I don’t think OnDelete would work, but you could write an OnUpdate Function that contains logic to copy certain documents over to a separate “backup” bucket. Any thoughts, @venkat?

3

Do the writes from A have to propagate to B almost immediately? What are the expected SLAs?

4

Not quite real time, but the usual XDCR speeds (which by the way is amazingly fast, thanks) with the acceptable network latency to get the eventually catch up is just fine (say few minutes to even an hour or two may be reasonable --basically proportional to the Writes volume on either end).

Another simplification I can offer in my requirement is that – same docs won’t be dirtied on both clusters during same time windows, so conflict resolution is something we don’t need to solve for.

-AP

5

What I can think of using Eventing is:

Let the XDCR between the clusters for the given bucket(Call it as Foo) remain as-is.
Create a Master Bucket(Call it Bar) in Cluster B.
Write a Function in Cluster B that has only the OnUpdate handler (no OnDelete) and listens to Foo(source bucket).
For every update that happens in Foo@Cluster B, write that document to Bar@Cluster B. This way, no document will be lost in Cluster B and Bar@Cluster B will have all documents.

1 Like
6

Ok I will play with this idea. Thanks

7

Following this up - did it work? :slight_smile:

1 Like
8

Hmm, not sure what exactly the deal is … but it did not work. This is what I setup:

Cluster 1
Bucket A

Cluster 2
BucketADelta
Bucket AMaster

XDCRs between
Cluster1/Bucket A --> Cluster2/Bucket ADelta
Cluster2/Bucket ADelta --> Cluster1/BucketA

3a
XDCR between Cluster2/BucketADelta --> Cluster2BucketAMaster
(Or, instead of XDCR … do a function just like in 3b to achiveve the same onUpdate)

3b.
Function on Cluster2/BucketAMaster
OnUpdate
create the same id item into BucketADelta
OnDelete
– empty body

If I just do 1,2, 3a - things appear to work. Buckets A, ADelta keep in sync bi-directionally. ADelta’s changes flow into AMaster. Including delete’s between A & ADelta. AMaster keeps getting sync’ed except delete;s. All seemed good. But as soon as I introduce 3b (so that any changes on AMaster flow back to ADelta and A) – the deleted records re-appear (which obviously I do not want).

My simplified use case is just getting 3a & 3b working together, without 1/2 steps. But I explored little bit more.

thanks
AP

9

Because:

  • You do not need 3a.
  • In 3b, create Function on Cluster2/BucketADelta and have the OnUpdate clause to write into Cluster2/BucketAMaster. (the other way of what you are doing now)

On in other words, Cluster2/BucketAMaster gets data only via the Function defined(only OnUpdate) on Cluster2/BucketADelta.