I am experiencing the same issue with couchbase server 3.0.1. I tried running a backup last night twice, and both times it failed. I used the command:
cbbackup http://<hostname>:8091 db_backup -u <couchbase username> -p <couchbase
password>
We have three buckets and the two smaller ones are backed up fine, but when it hits the larger bucket (~2 million documents) it runs for a while, slowly using more and more CPU and RAM, until I get an error in the couchbase console that the cluster can no longer connect to the node I was running the backup on. I think at that point the couchbase process must restart because shortly afterward everything is working fine again. In the shell displaying the backup status information it never displays an error, it just seems to be stopped or frozen. The first time I ran it the backup failed after about an hour. I tried running a diff backup afterward and it took forever to start, then displayed an error that it couldn’t decode a JSON object:
Exception in thread w1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/opt/couchbase/lib/python/pump.py", line 279, in run_worker
curx)
File "/opt/couchbase/lib/python/pump_bfd2.py", line 20, in check_spec
getattr(opts, "mode", "diff"))
File "/opt/couchbase/lib/python/pump_bfd.py", line 255, in find_seqno
json_data = json.load(json_file)
File "/opt/couchbase/lib/python/simplejson/__init__.py", line 267, in load
parse_constant=parse_constant, **kw)
File "/opt/couchbase/lib/python/simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/opt/couchbase/lib/python/simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/couchbase/lib/python/simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
[####### ] 34.7% (659955/estimated 1904339 msgs)
It then ran until 34.7% and seems to have failed sometime after that (I was running it overnight since it slows down the server to the point that it can’t be used). Our nodes have 4 cores at 2.0GHz each, and 7GB of RAM each (with 4 GB allocated to each couchbase node for buckets). I’ve run the backup before on a Couchbase 2.5.1 server successfully, but it only had just over 1 million documents at the time. Is this an issue with couchbase 3.0.1? Is there a better way to run the backup than the general full cluster backup that might be more successful?