Issue Upgrading Couhbase to 6.5 using swap rebalance

Hi,

I have tried to upgrade my couchbase version from 6.0.1 to 6.5.1
I have 11 nodes ( 4 data, 4 index, 2 query, 1 eventing) and 1 data node has around 1TB data
I try using swap rebalance, but found it takes a very long time to transfer the data & indexes.
Is there any work around for this?

Hi @Han_Chris1,

There are some other options for upgrading: https://docs.couchbase.com/server/current/install/upgrade-strategies.html

A quote from the docs: “If the speed of an upgrade is a primary concern for your cluster, see Graceful Failover or Performing the Offline Upgrade.”

Just to add, the preferred option for upgrade of index service node is (assuming each index has a replica):

  1. Failover one index service node at a time
  2. Upgrade the software
  3. Recover the node using rebalance

This allows for index service node to resume indexing from the point of failover and is usually a very fast operation while still being completely online.

1 Like

Hi @matthew.groves & @deepkaran.salooja,

Thanks for your response.
My current condition is I’m using docker
Actually I’m still confused for the offline upgrade step, there’s no detail step by to upgrade, it redirect to install single node.
Last time I’ve tried to stop the node, change docker image directly inside docker-compose. (I assume this is the step for offline upgrade using docker), then during bring up data service, it can not be started, and almost all of my data is corrupt.
That’s why I choose swap rebalance, but it takes longer time.

@Han_Chris1,

Now I’m a little confused, because there’s a 9-step process for Offline Upgrade at the above link (Upgrade an Offline Cluster | Couchbase Docs). Is there a particular step that you’re having trouble with?

Hi @matthew.groves ,

For step number 8, didn’t have a detail step for upgrade, only redirect to https://docs.couchbase.com/server/current/install/install-intro.html

Okay, I will rephrase my question,
How to do offline upgrade using docker?

Thanks

@Han_Chris1,

I don’t think we’re looking at the same page, because the step 8 I’m looking at links to https://docs.couchbase.com/server/current/install/upgrade-individual-nodes.html

But nevertheless, I think the steps will be similar, as long as you’ve mapped Couchbase node-specific data to a local folder/volume of /opt/couchbase/var/ (as discussed here: https://hub.docker.com/_/couchbase/). I’ve not personally tried this, but after a backup, could you stop the old docker image, and start a new docker image using the same volume?

Hi @matthew.groves,

we’re lookong at the same page, in that link, move forward to step number 6, it will redirect to install instead of upgrade.

For docker part, I’ve tried that step previously. and the result is my data node can not be started, it’s like corrupt.
So I rename my var folder, then it can be started but my old data can not be recovered.

Thanks,

I think we ran into the same problem. Until now we just updated the version in Dockerfile (FROM couchbase/server:enterprise-6.0.4). We then have a docker-compose file which maps the volume /opt/couchbase/var to our local disk and we just redeploy the newly built image.

Until now the new container always ran fine and the data and indexes were available in the new version of couchbase. Now the indexer in the 6.5/6.6 version seems to crash after such upgrade.

Similarly to these:

We also run into this log message in babysitter.log https://github.com/couchbase/kv_engine/blob/master/engines/ep/src/warmup.cc#L1016

Our nodes also didn’t come up from 6.0.4 to 6.6.0. Using docker (stop/start with new version). Maybe the same issue. We have enterprise support and I uploaded our logs to CB support (Request #35959).

This is what I got back for my case from couchbase support. I also asked them to try to keep the upgrade process for docker users more simple than currently advised.


The issue here comes from the fact that from a Couchbase Server installation perspective, an upgrade wasn’t performed. Instead, a new instance running the upgraded version started up, and ran with the config files of the previous version. This will cause an issue when a change between versions requires the upgrade processing to run - in this case, this is due to a change in the data file format. As Couchbase Server 6.6 expects the files to have been migrated to the new format before it starts, it is unable to read the files in the older version.

We are looking into any changes we can make to our documentation in order to make the upgrade steps clearer when using Docker. In the meantime however, for any future upgrades you should approach these by adding a new node to the cluster on the upgraded version and then removing the older node.

In order to recover the one node cluster you currently have, you should be able to execute the following command within the container :

# /opt/couchbase/bin/cbupgrade -a yes --namespace_upgrade_only -c /opt/couchbase/var/lib/couchbase/config/ && pkill memcached

This will manually trigger the data file upgrade and then kill the Data Service, forcing it to restart and pick up the change. It should be noted that this is provided as a workaround only, and should not be used as a routine step for any future upgrades. Before running this command, you should back up the container’s mapped config directory .

1 Like

Thanks @Koen_De_Jaeger for sharing this workaround.
I agree that couchbase should enhance the way to upgrade the version.
Yes, the proper way is using swap rebalance (remove old node and add new node then rebalance), but if we’ve already had so many nodes and billions of data, it will take a bery long process to upgrade.
Since this couchbase often release newer version, hopefully in the future couchbase can make faster and simpler way to upgrade