HELP Please! Buckets all empty after cluster restart


#1

Hi, really hope someone can help me / us:

We have a very old Couchbase cluster that has an enormous amount of data in there. This morning, all of our nodes were rebooted on AWS and the cluster didn’t come back up - one of the nodes was ‘Down’ and the other 3 were ‘Pending’. We had to fail over the ‘Down’ one and then everything rebalanced… now, however, although our buckets are all there again they have no data. How can I retrieve the data that were in there?

Thanks in advance

John


#2

Which version of Couchbase? How many replicas did you have configured?

Before going further, you should shutdown and snapshot/backup the filesystems.


#3

We took over the database relatively recently so are still familiarising ourselves with it (our main goal is migrating it to a newer version etc.) but here goes with what we have been able to figure out:

CB 2.5.1 EE (very old), we had 4 nodes running that all restarted simultaneously. 3 came back up, 1 didn’t…


#4

I can connect to each of the nodes, but really can’t see how to get the data back into the buckets.

Have taken snapshots in AWS of each of the filesystems


#5

I also don’t see large enough files on any of the filesystems that would be size-wise in line with the database size. I’m really hoping that the data aren’t gone…