After adding a new node to cluster rebalance stuck and server is unable to start or stop


#1

After adding a node to a cluster the rebalance hanged and while trying to restart the server I got into a state that the sever cant be run or stopped

I run out of disk space while doing so deleting some files got the server up and running

It will be nice to get notified that this is a hard drive space problem

debug.log
{sasl_password,“"},
{repl_type,dcp},
{uuid,<<“c4d9b138cb36b031980b775e0b91c893”>>},
{replica_index,false},
{type,membase},
{num_vbuckets,1024},
{replication_topology,star},
{servers,[‘ns_1@172.31.28.114’]},
{map_opts_hash,133465355},
{flushseq,1}]},
{“sparkup”,
[{map,[{479,
[‘ns_1@172.31.28.114’,undefined],
[‘ns_1@172.31.16.242’,‘ns_1@172.31.28.114’]}]},
{fastForwardMap,[]},
{repl_type,dcp},
{uuid,<<“682128cefecb12b1863147d74feb9a41”>>},
{num_replicas,1},
{replica_index,false},
{ram_quota,104857600},
{auth_type,sasl},
{sasl_password,"
”},
{autocompaction,false},
{purge_interval,undefined},
{flush_enabled,false},
{num_threads,3},
{eviction_policy,value_only},
{type,membase},
{num_vbuckets,1024},
{replication_topology,star},
{servers,[‘ns_1@172.31.28.114’,‘ns_1@172.31.16.242’]},
{map_opts_hash,133465355}]}]}]
[ns_server:debug,2016-03-01T08:32:02.498Z,ns_1@172.31.28.114:capi_doc_replicator-Crawl<0.23155.2>:doc_replicator:loop:64]doing replicate_newnodes_docs
[ns_server:debug,2016-03-01T08:32:02.498Z,ns_1@172.31.28.114:capi_doc_replicator-default<0.23170.2>:doc_replicator:loop:64]doing replicate_newnodes_docs
[ns_server:debug,2016-03-01T08:32:02.498Z,ns_1@172.31.28.114:capi_doc_replicator-sparkup<0.23158.2>:doc_replicator:loop:64]doing replicate_newnodes_docs
[ns_server:debug,2016-03-01T08:32:02.514Z,ns_1@172.31.28.114:<0.23791.2>:ns_rebalance_observer:handle_vbucket_move_done:293]Noted vbucket move end (vbucket 479)
[ns_server:debug,2016-03-01T08:32:02.514Z,ns_1@172.31.28.114:<0.24559.3>:ns_vbucket_mover:spawn_workers:326]Got actions: []
[rebalance:info,2016-03-01T08:32:02.583Z,ns_1@172.31.28.114:<0.26186.3>:janitor_agent:bulk_set_vbucket_state:382]Doing bulk vbucket 475 state change
[]
[rebalance:info,2016-03-01T08:32:02.583Z,ns_1@172.31.28.114:<0.26188.3>:janitor_agent:bulk_set_vbucket_state:382]Doing bulk vbucket 475 state change
[{‘ns_1@172.31.28.114’,replica,undefined,‘ns_1@172.31.16.242’}]
[rebalance:info,2016-03-01T08:32:02.583Z,ns_1@172.31.28.114:<0.26190.3>:janitor_agent:set_vbucket_state:397]Doing vbucket 475 state change: {‘ns_1@172.31.28.114’,replica,undefined,
‘ns_1@172.31.16.242’}
[ns_server:debug,2016-03-01T08:32:02.664Z,ns_1@172.31.28.114:janitor_agent-sparkup<0.465.0>:janitor_agent:handle_cast:892]Got reply to call {update_vbucket_state,478,replica,undefined,
‘ns_1@172.31.16.242’} from apply_vbucket_states_worker: ok
[rebalance:info,2016-03-01T08:32:02.664Z,ns_1@172.31.28.114:<0.26127.3>:ns_single_vbucket_mover:on_move_done_body:630]Moving vbucket 478 done. Will delete it on: []
[ns_server:info,2016-03-01T08:32:02.665Z,ns_1@172.31.28.114:<0.586.0>:ns_memcached:do_handle_call:564]Changed vbucket 477 state to replica
[ns_server:debug,2016-03-01T08:32:02.665Z,ns_1@172.31.28.114:dcp_consumer_conn-sparkup-ns_1@172.31.16.242<0.24633.3>:dcp_commands:add_stream:66]Add stream for partition 477, opaque = 0x1DD, type = add
[ns_server:debug,2016-03-01T08:32:02.665Z,ns_1@172.31.28.114:dcp_consumer_conn-sparkup-ns_1@172.31.16.242<0.24633.3>:dcp_consumer_conn:handle_call:198]Setup DCP streams:


#2

We have an issue for running out of disk space filed at the link below. You can track this ticket to find out when a fix is available.

https://issues.couchbase.com/browse/MB-8067