We are using couchbase 4.6.0-3573 Enterprise Edition (build-3573). We have three nodes containing 5 buckets. Last evening, we were running maintenance and turned off internet switch for a period of 10 minutes or so. This made the node lose contact to each other. The logs we got at that point are following
- Haven’t heard from a higher priority node or a master, so I’m taking over.
- (Nodes are not being able to see each other
- Enabled auto-failover with timeout 45)
- Could not auto failover node
- (Then the nodes saw other nodes coming back up)
- Shutting down bucket (all the 5 buckets) for deletion
Then the worst thing happened and 150k document got deleted from just one bucket. Other 4 were fine.
Has anyone seen anything like this happen before or lost their data from a bucket like that? Can this be a bug on the version we are using?
We have no idea why this happened and documents got deleted. I will really appreciate any help.