Cluster stuck in compatibility mode after upgrade

Couchbase Server
1

After upgrading cluster from community-4.5.0 to community 5.0.0 (build 3519) we have it stuck in compatibility mode:

This cluster contains multiple server versions and is running in 4.5 compatibility mode.

Logs show:

Unsuccessful user storage upgrade

…which would suggest what we were hit by Loading..., which should be fixed in build 3336 (we are running 3519).

Please advise.

2

Exact error log message:

Unsuccessful user storage upgrade.
{error,{badmatch,{abort,{error,roles_validation,[ro_admin]}}},
   [{menelaus_users,do_upgrade_to_spock,2,
                    [{file,"src/menelaus_users.erl"},{line,594}]},
    {menelaus_users,upgrade_to_spock,2,
                    [{file,"src/menelaus_users.erl"},{line,547}]},
    {cluster_compat_mode,do_upgrades,5,
                         [{file,"src/cluster_compat_mode.erl"},
                          {line,212}]},
    {cluster_compat_mode,do_consider_switching_compat_mode,2,
                         [{file,"src/cluster_compat_mode.erl"},
                          {line,233}]},
    {ns_orchestrator,consider_switching_compat_mode_dont_exit,0,
                     [{file,"src/ns_orchestrator.erl"},{line,1039}]},
    {ns_orchestrator,consider_switching_compat_mode,0,
                     [{file,"src/ns_orchestrator.erl"},{line,1029}]},
    {ns_orchestrator,handle_info,3,
                     [{file,"src/ns_orchestrator.erl"},{line,511}]},
    {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,505}]}]}
3

I have exactly the same problem.

I have 2 instances in my cluster and I’m sure that the both upgrades went smoothly without error messages that would explain this status.

4

This seems to be casused by read-only user. Unfortunately, 5.0.0 does not allow to remove it.

@RogerP What did work was:

  1. adding a new 4.5.0 node (docker) to the cluster (no rebalance),
  2. logging in on to 4.5.0 node and removing read-only user,
  3. canceling node add on 5.0.0 node.

I did not re-add the user since I no longer need it, but re-adding it after should work just fine.

2 Likes
5

Koder, thanks a lot for this hint, it took me some time but it finally worked like you described!

1 Like
6

Had this same issue on 5.0.1 and was able to resolve with @koder 's above steps