Couchbase server is in constant rebalance with key length error even after flushing all data

Hi,

We have a couchbase server cluster. It has been constantly doing rebalance with error “key length exceeds 250”. Initially the suspicion was that some of the data is causing the problem. So we have flushed data in bucket. It is still trying rebalance with no success.

Rebalance exited with reason {mover_crashed,
{unexpected_exit,
{'EXIT',<0.14442.859>,
{{{function_clause,
[{mc_client_binary,
handle_docs_estimate_result,
[{memcached_error,einval,
<<"{\"error\":{\"context\":\"Key length exceeds 250\"}}">>},
{0,0,<<"does_not_exist">>}],
[{file,"src/mc_client_binary.erl"},
{line,728}]},
{ns_memcached,do_handle_call,3,
[{file,"src/ns_memcached.erl"},
{line,565}]},
{ns_memcached,worker_loop,3,
[{file,"src/ns_memcached.erl"},
{line,247}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]},
{gen_server,call,
['ns_memcached-order-management',
{get_dcp_docs_estimate,1022,

Additional details:

  • Using couchbase kubernetes operator
  • image: couchbase/server:6.6.0

Apparently it’s not a document key length issue, it looked like an issue with pod hostname. When we reduced the size of the cluster name/label etc and redeployed it. The error seems to go away (we may need to monitor for few days).

One thing I am not confident about is, since we did redeploy and hostname change together. I am not sure which one action exactly fixed the issue or the combination.

It would be great if someone can clarify that.

That is correct, this is a limitation on the cluster name itself. There should be a fix for this in 6.6.4

1 Like