I’ve setup XDCR of production cluster to backup node and use cbbackup to dump all buckets on a daily basis. Couchbase data directory is around 55GB. When I run cbbackup, it goes crazy in terms of memory usage. It’s using up to 60GB of memory - which is beyond machine RAM (it has 16GB, out of that 7GB is used by couchbase itself) so it ends up using SWAP excessively. This all makes the backup process very very slow, around 20hours. Disk I/O (on the data partition) and CPU usage is pretty small, definitely memory usage (and because of swapping - IO on swap drive) being the bottleneck.
Is there any way which we could reduce memory usage by cbbackup?
Looking on other datastores I’ve been using before (*SQL, MongoDB) it was never a problem to backup 50GB of data…
I’m running on Amazon EC2 - the backup host is m3.xlarge instance, which provides two 40GB ephemeral drives. I’m using raid0 of them for swap volume.
Available on gist - output of dstat command. xvdb, xcdc are ephemeral local SSD drives used for swap, xvdf is used for couchbase data and writing backup outputs (with IO capacity of 1500IOPS):