I have a three node cluster that I have been building a prototype running Couchbase 3 on Linux. Linux often have updates, and I would love to update the node OS’s more often, but it seems to take a whole day to update the nodes. Failovers take a long time, rebalances fail on empty vbuckets which then lead to writing 1024+ dummy records to each bucket, so that rebalances work, and then remove the dummy records…
Does anyone have a standard procedure for doing a rolling restart of a cluster?
Failovers shouldn’t take much time unless you’re including rebalance duration into it. We basically promote already existing replicas vbuckets to active as part of failover.
For your specific case, I would suggest employing swap rebalance strategy. Mark X nodes for removal that are going for OS upgrade and add another set of X servers, trigger rebalance after that. That would certainly lower down amount of rebalances that you have to perform.
Yes, we’re aware of this bug. We are planning to roll out a CE version containing fix for it too.
Here is the C# code I use to solve the vBucket problem that stops Rebalances and Failovers
public static void rebalanceFix(bool create = true, string suffix = "")
{
string[] bucks = { "buck1", "buck2", "buck3", "default" };
foreach (string buckname in bucks)
{
using (var bucket = CMDMSvc.CouchbaseManager.cluster.OpenBucket(buckname))
{
for (int i = 1; i < 3080; i++)
{
var key = "rebalance::" + i.ToString() + (!string.IsNullOrWhiteSpace(suffix) ? "::" + suffix : "");
if (create)
{
Couchbase.Document<string> doc = new Couchbase.Document<string>();
doc.Id = key;//"rebalance::" + i.ToString();
//doc.Expiry = 10 * 60 * 3 * 3 * 1000; // Note I don't use this because I could not figure the time to do a rebalance. So I call this routine again to remove the keys after rebalance
doc.Content = "test" + i.ToString();
var result = bucket.Upsert<string>(doc);
Console.WriteLine(doc.Id + result.Message + result.Success.ToString());
}
else
{
var resultdel = bucket.Remove(key);
Console.WriteLine(key + resultdel.Message + resultdel.Success.ToString());
}
}
}
}
}