Re-balance failing with error - Rebalance exited with reason {badmatch,failed}

server

#1

I am setting up a cluster. I tried to join 3 nodes but while rebalancing. I got above error.

=========================CRASH REPORT=========================
crasher:
initial call: service_agent:-spawn_connection_waiter/2-fun-0-/0
pid: <0.18486.7>
registered_name: []
exception exit: {no_connection,“index-service_api”}
in function service_agent:wait_for_connection_loop/3 (src/service_agent.erl, line 305)
ancestors: [‘service_agent-index’,service_agent_children_sup,
service_agent_sup,ns_server_sup,ns_server_nodes_sup,
<0.170.0>,ns_server_cluster_sup,<0.89.0>]
messages: []
links: [<0.18481.7>,<0.18490.7>]
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 27
reductions: 1195
neighbours:

[ns_server:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:service_agent-index<0.18481.7>:service_agent:terminate:264]Terminating abnormally
[ns_server:debug,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:<0.18487.7>:ns_pubsub:do_subscribe_link:145]Parent process of subscription {ns_config_events,<0.18481.7>} exited with reason {linked_process_died,
<0.18486.7>,
{no_connection,
“index-service_api”}}
[error_logger:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server ‘service_agent-index’ terminating
** Last message in was {‘EXIT’,<0.18486.7>,
{no_connection,“index-service_api”}}
** When Server state == {state,index,
{dict,6,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[[{uuid,<<“55a14ec6b06d72205b3cd956e6de60e7”>>}|
‘ns_1@xuodf7.firebrand.com’]],
[],
[[{uuid,<<“c5e67322a74826bef8edf27d51de3257”>>}|
‘ns_1@xuodf8.firebrand.com’]],
[],
[[{uuid,<<“3b55f7739e3fe85127dcf857a5819bdf”>>}|
‘ns_1@xuodf9.firebrand.com’]],
[],
[[{node,‘ns_1@xuodf7.firebrand.com’}|
<<“55a14ec6b06d72205b3cd956e6de60e7”>>],
[{node,‘ns_1@xuodf8.firebrand.com’}|
<<“c5e67322a74826bef8edf27d51de3257”>>],
[{node,‘ns_1@xuodf9.firebrand.com’}|
<<“3b55f7739e3fe85127dcf857a5819bdf”>>]],
[],[],[],[],[],[],[],[],[]}}},
undefined,undefined,<0.18626.7>,#Ref<0.0.5.56873>,
<0.18639.7>,
{[{<0.18646.7>,#Ref<0.0.5.56891>}],[]},
undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {linked_process_died,<0.18486.7>,{no_connection,“index-service_api”}}

[error_logger:error,2018-02-12T13:54:43.532-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: service_agent:init/1
pid: <0.18481.7>
registered_name: 'service_agent-index’
exception exit: {linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [service_agent_children_sup,service_agent_sup,ns_server_sup,
ns_server_nodes_sup,<0.170.0>,ns_server_cluster_sup,
<0.89.0>]
messages: [{‘EXIT’,<0.18639.7>,
{linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}}]
links: [<0.18487.7>,<0.4805.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 28690
stack_size: 27
reductions: 6334
neighbours:

[error_logger:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,service_agent_children_sup}
Context: child_terminated
Reason: {linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}
Offender: [{pid,<0.18481.7>},
{name,{service_agent,index}},
{mfargs,{service_agent,start_link,[index]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[ns_server:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_rebalancer:run_rebalance:80]Agent terminated during the rebalance: {‘DOWN’,#Ref<0.0.5.56860>,process,
<0.18481.7>,
{linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}}
[error_logger:info,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,service_agent_children_sup}
started: [{pid,<0.20369.7>},
{name,{service_agent,index}},
{mfargs,{service_agent,start_link,[index]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_agent-index<0.20369.7>:service_agent:handle_call:186]Got rebalance-only call {if_rebalance,<0.18626.7>,unset_rebalancer} that doesn’t match rebalancer pid undefined
[ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_agent:process_bad_results:815]Service call unset_rebalancer (service index) failed on some nodes:
[{‘ns_1@xuodf9.firebrand.com’,nack}]
[ns_server:warn,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_rebalancer:run_rebalance:89]Failed to unset rebalancer on some nodes:
{error,{bad_nodes,index,unset_rebalancer,
[{‘ns_1@xuodf9.firebrand.com’,nack}]}}
[error_logger:error,2018-02-12T13:54:43.535-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: service_rebalancer:-spawn_monitor/6-fun-0-/0
pid: <0.18626.7>
registered_name: 'service_rebalancer-index’
exception exit: {linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}
in function service_rebalancer:run_rebalance/7 (src/service_rebalancer.erl, line 92)
ancestors: [cleanup_process,ns_janitor_server,ns_orchestrator_child_sup,
ns_orchestrator_sup,mb_master_sup,mb_master,<0.4893.0>,
ns_server_sup,ns_server_nodes_sup,<0.170.0>,
ns_server_cluster_sup,<0.89.0>]
messages: [{‘EXIT’,<0.18640.7>,
{linked_process_died,<0.18486.7>,
{no_connection,“index-service_api”}}}]
links: []
dictionary: []
trap_exit: true
status: running
heap_size: 2586
stack_size: 27
reductions: 6359
neighbours:

[ns_server:error,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:cleanup_process<0.18625.7>:service_janitor:maybe_init_topology_aware_service:84]Initial rebalance for index failed: {error,
{initial_rebalance_failed,index,
{linked_process_died,<0.18486.7>,
{no_connection,
“index-service_api”}}}}
[ns_server:debug,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {“projector-cbauth”,<0.5099.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.538-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {“goxdcr-cbauth”,<0.479.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.539-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {“cbq-engine-cbauth”,<0.5124.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.540-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {“fts-cbauth”,<0.5129.0>} needs_update


#2

@umaris, the rebalance error is coming due to index service. You can check indexer.log to see if there are any errors and the process is able to bootstrap correctly.

Please make sure the communication ports are open as mentioned here: https://developer.couchbase.com/documentation/server/current/install/install-ports.html

projector_port 9999 being blocked can lead to this.