I upgraded my Couchbase cluster from 7.0.x to 7.1 which all run on VM based on Ubuntu 20.04.4 LTS.
The way i did the upgrade was that I first thought about doing an online upgrade so i moved one node out of the cluster and did rebalance so all data was moved to remaining nodes. At that point i shut down the Couchbase service and then started the upgrade process
a) sudo apt update
b) sudo apt upgrade
at this point i saw the message that it will install couchbase 7.1 over 7.0.x and it took some time to complete. After that i went and rebooted the system and expected to access this node again. When
i tried to login to admin console i kept getting a connection refused. I checked if the port 8091 was avail and if there was no firewall issues after update.
i checked via sudo lsof -i -P -n | grep LISTEN what was running and found that only a couple couchbase ports where listening. When i manually started the service again i saw some more active ports which went away within a sec or so.
As i was not able to find any useful info in syslog there was no way to determine the cause for me.
So my question is what could have caused this upgrade failure ?
How can one determine the cause so it can be fixed.
In my case the fix was to uninstall Couchbase from server and do a clean install, after that the server started fine and was able to rejoin the cluster. My other 3 nodes i jut took offline and run the update as above and they all went fine without hitch. All those 4 VM’s a 100 % identical in setup and based on same template.