Cbbackup is hanging

#1

Cbrestore is hanging while trying to backup a bucket.

The strange thing is I had same issue on another bucket during full backup but calling against the bucket alone works well.

What seems to be an issue is the estimated messages keeps growing during the backup while the bucket is not being touched by any other application.

I am running a three-node cluster 3.0.2 on centos 64bit

/opt/couchbase/bin//cbbackup http://RTECouchbase3:8091 -b SctLive /backups/backup-test/ -u Administrator -p XXX -v
2015-02-11 11:16:20,817: mt cbbackup…
2015-02-11 11:16:20,817: mt source : http://RTECouchbase3:8091
2015-02-11 11:16:20,817: mt sink : /backups/backup-test/
2015-02-11 11:16:20,817: mt opts : {‘username’: ‘’, ‘verbose’: 1, ‘dry_run’: False, ‘extra’: {‘max_retry’: 10.0, ‘rehash’: 0.0, ‘data_only’: 0.0, ‘uncompress’: 0.0, ‘nmv_retry’: 1.0, ‘conflict_resolve’: 1.0, ‘cbb_max_mb’: 100000.0, ‘mcd_compatible’: 1.0, ‘try_xwm’: 1.0, ‘backoff_cap’: 10.0, ‘batch_max_bytes’: 400000.0, ‘report_full’: 2000.0, ‘seqno’: 0.0, ‘batch_max_size’: 1000.0, ‘report’: 5.0, ‘design_doc_only’: 0.0, ‘recv_min_bytes’: 4096.0}, ‘single_node’: False, ‘vbucket_list’: None, ‘threads’: 4, ‘mode’: ‘full’, ‘key’: None, ‘password’: ‘’, ‘id’: None, ‘bucket_source’: ‘SctLive’}
2015-02-11 11:16:20,917: mt bucket: SctLive
[####################] 97.6% (10998/estimated 11274 msgs)

The percentage where isstops varies but is always in the 95->98 range.

Couchbase 3.0.1 backup hangs
Cbbackup never terminates - does partial backup
Cbbackup never terminates - does partial backup
#2

Hi David, From what I read yes its a bug we already have a bug opened MB-13386. Please take a look and it is the same issue it will be fixed in upcoming Couchbase Server 3.0.3 bug-fix release.

Thanks!

Anil Kumar

#3

To anyone who might be struggling with cbbackup hanging indefinitely, as I was, I may have a workaround for you.

DISCLAIMER: This workaround is provided as is, with no warranty whatsoever. Use at your own risk.

The workaround is intended for those that are running Couchbase Server 3.0.1 Community Edition. It combines a couple of suggestions in the comments from https://issues.couchbase.com/browse/MB-14833 with an important twist.

As mentioned in the issue’s comments, you should reverse the order of DCPStreamSource and TAPDumpSource in the SOURCES property of lib/python/pump_transfer.py. Also, you will have to acquire a newer version of couchbase-cli (https://github.com/couchbase/couchbase-cli) than is shipped with 3.0.1 CE so that backups can run sequentially. Using revision c1209a3 or later of couchbase-cli will make sequential the default behaviour (though it can be overridden by the --parallelism flag).

For your convenience, you can download a pre-packaged copy of the workaround from https://s3-us-west-1.amazonaws.com/kashoo-public/custom-couchbase-cli.zip. It contains a fully isolated and patched copy of Couchbase Server’s CLI utilities as described above.

Note that the workaround has only been tested on CentOS 6.5; your mileage may vary with other distros and OSes. Also, while the pre-packaged workaround mimics the partial directory structure of a Couchbase Server installation, the archive is completely self contained, so it is highly recommended that its contents be extracted to its own isolated directory rather than overwriting the corresponding files from the Couchbase Server directory.

1 Like
Cbbackup never terminates - does partial backup