We’re currently investigating a problem in our couchbase server clusters where a select set of documents seem to be ignored by replication - in one specific cluster.
No matter what we’ve tried, we can’t seem to make XDCR recognize that it should be replicating this document. We’ve tried:
- upserting the document with the same content in the source datacenter
- pausing and resuming the replication
- restarting the couchbase-server service on both source and destination servers
- deleting the problematic document from the source, waiting, and inserting the document back in
- creating the document specifically in the datacenter where it is missing, then deleting from the source datacenter (it doesn’t get deleted from the destination cluster)
- based on prior experience, we’ve created a different set of documents within the bucket and they all get replicated just fine (my script ensures that it inserts into every vBucket)
- we have multiple destination datacenters, and only one destination is missing the document
Is there anything we can look into to see why couchbase is ignoring replication for just this one document?
In case it helps, right around the time the document was being inserted in the source, we were rebalancing the destination cluster and a server being added was powered off unexpectedly. We were able to successfully rebalance the cluster once the server was back online.