Snapshot backup of high-availability couchbase clusters

Hi,

We are setting up a high-availability Couchbase cluster for our application and we are struggling with creating reliable snapshot backups. The documentation describes all kinds of possibilities and limitations, but is actually contradicting (or not working at all) at certain points.

In our high-availability MySQL cluster, in order to create a snapshot backup, a single node is detached from the cluster (replication is stopped), and the backup is created from that node. After backup has finished, the node will be attached again and modifications are replicated. If the node has been fully replicated again, the node will take traffic again.

We’ve tried this same approach for our Couchbase cluster, but this continuously fails:

sudo ./cbbackup -b <bucket> -u <username> -p <password> couchstore-files:///opt/couchbase/var/lib/couchbase/data /tmp/cb-20191022

On a online node: “error: cannot use a dead worker
On a offline node (gracefully failed-over): “error: missing vbuckets in source_bucket: <bucket>

How should one create a reliable snapshot backup of an online high-availability Couchbase cluster without affecting performance in any way?

Thanks in advance,
Raymond