Cbbackup never terminates - does partial backup


#1

This happens with Couchbase Server Community Edition 3.0.1 running on CentOS Linux 7

The backup begins and writes 4000 msgs to disk, which represents 88% of the bucket’s data (a small bucket) and then hangs: no more progress and no termination.

Any ideas?


#2

What I am seeing here may be the issue documented here: https://issues.couchbase.com/browse/MB-13509


#3

Sorry to not have a better reply but I just wanted to chime in and say that I see this issue as well, and I have not yet found a solution.


#4

Thanks! It does help to know I’m not the only one. Maybe we should just make more noise about it?


#5

All I could find besides the issue you linked to was this: Cbbackup is hanging
I’m not really sure if the issue linked in that post is whats causing the problem, but it states that it should be fixed in 3.0.3, which is currently only available in the enterprise edition, to which we no longer have a license :frowning: which of course also limits me to asking this question on the forum.
Have you tried with this version? I tried with both 3.0.1 and 3.0.2 but that did not help.


#6

I have not tried 3.0.2 or 3.0.3 yet and I am currently also limited to using the Community Edition.

According to one answer given to the entry you linked, the issue then was a different one: https://issues.couchbase.com/browse/MB-13386

I would really like to take a backup soon. I am lucky that this is a small bucket and only my staging server. I might write my on backup routine in Ruby to work around this. After all, the data is Json, Strings and Numbers only, so this shouldn’t be hard.

Still, backup is a somewhat essential feature and would be really nice to have back


#7

Me too !

When I run cbbackup, it hangs at same point for some bucket.
Rebalance also doesn’t work (basically hangs at the same place)

I also read this and that

Is there a utility that cleans corrupted docs in the database ?
What if I cannot upgrade because I cannot rebalance to the upgraded server ?

@ingenthr, @cihangirb


#8

@ingenthr, @cihangirb,

Still hanging with a production cluster that I cannot scale nor backup :disappointed_relieved:

Please advise


#9

If you ever find a solution to this issue, please post in one of the numerous threads about this issue, I still havn’t been able to find a solution, except creating a new bucket, then using a bucketcopy program we made and copy over all the data (which works fine) and finally deleting the old bucket.

But since I have seen this happen to serveral buckets this is not really a solution.


#10

Thanks for the info @viktorhanssonand @itay. I’ve chatted with a couple of other folks on this and we should be able to update this thread soon with what we’ve found and what we expect as a way to move forward.

cc/ @cihangirb, @anil


#11

Hi folks, there were a number of issues recently on cbbackup but if you are experiencing the issue, pls see the workaround provided in the bug. Also pls report back if the workaround has not fixed the issue for you.
https://issues.couchbase.com/browse/MB-14833
The fix is in progress but is not in any of the editions yet.
thanks
-cihan


#12

To anyone who might be struggling with cbbackup hanging indefinitely, as I was, I may have a workaround for you. See my response to “Cbbackup is hanging”: Cbbackup is hanging


#13

@OldSneerJaw, @viktorhansson, @mkrogemann

Just an update.
I upgraded to 4.0.0 CE and if before cbbackup hangs in the middle, now it just quits without reaching 100%, can be ~70%, ~80%, etc.
This means that the backup is partial as thus can be useless.
I think that if cbbackup reads a corrupted entry, it quits instead of skip to the next one.

My question is: “Is there a house keeping utility that can indicate / remove corrupted docs so that cbbackup can be free to work” ?

Itay