Move bucket to a new cluster

Hi,
Is there a documented way to move a bucket to a new cluster without losing availability of the bucket?
We have a production cluster with 3 bucket and we wish to move one of the bucket to a separate cluster to optimise the bucket resource needs but since the system is in production we don’t want to loose data.

Thank you.

You’ll have to either modify your application or take some downtime when moving a bucket. The way I can think of that you may be able to do this is to use XDCR at the old and new cluster, as long as you have system resources. The app may need be coordinated in this change.

Using XDCR and taking a short outage would be simpler though. It would only be a few minutes outage.

Thanks.
The bucket I was talking about is a write only with eventually consistency demand.
So I figure I could do a full backup, change the application to work with the new cluster, restore the full back, use differential incremental backup (after the bucket has no ops and the disk write queue zeroed), and restore this incremental backup.

Theoretically is should work, no?

@gluz you can of course do full backups and restore it on the other cluster, that said it will also include downtime. In general I’d also favour @ingenthr’s approach since it will allow you to warm up the other cluster with your data while the old is still serving traffic. Of course you need to have the system resources in place, so backup & restore will work too.

Why is the backup restore operation include downtime?

Well, to avoid data loss there needs to be a time when you stop writing to Cluster A, then you do the backup, you import it on Cluster B and then you start directing traffic to Cluster B. If you keep writing to Cluster A you’ll probably loose some data along the way.

Also, while you’re restoring a backup the bucket is not available (doing a backup is an online operation).

Here is another strategy that you might be able to use without downtime:

  • You modify your application to write data in both clusters (new data). You still keep reading from the old bucket.
  • In the background, you run some code that transfers old data from one bucket to the new one.
  • Once the old data has caught up and you ran some sanity checks to make sure the data is consistent, you switch over to reading from the new bucket (still writing to the old one so you can always switch back)
  • Then once you check all is good, you stop writing to the old bucket and shut it down.

You are ignoring my message:

The bucket I was talking about is a write only with eventually consistency demand.
So I figure I could do a full backup, change the application to work with the new cluster, restore the full back, use differential incremental backup (after the bucket has no ops and the disk write queue zeroed), and restore this incremental backup.

Why won’t this method work without downtime?

Hm I just checked the docs, looks like you are right - it can work indeed as an online operation. Make sure to use the --add option on cbrestore so that existing docs are not overwritten.

Great thanks.
For future readers, this operation can work without downtime only because the bucket is a write only bucket. If we needed to read from it, it wouldn’t work.

1 Like

How can I move the data of a particular bucket to the other bucket in different environment like from production to stage ?

I have tried using below command:

cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data -b <> http://127.0.0.1:8091 --username=Administrator --password=<> --bucket-destination=<>

[transferring data from source bucket of server to another server in the destination bucket]

But, the below error is coming:
error: unable to access REST API: 127.0.0.1:8091/pools/default/buckets; please check source URL, server status, username (-u) and password (-p)