Adding new node(s) into cluster and rebalancing hangs


#1

Added one more node into current cluster and perform the “rebalance” operation.
But the rebalance process is always 0%…
Stop and restart it doesn’t work …
Does anyone can tell me the reason or some tips to resolve this problem?
Thanks a lot !

Here is some information about the cluster:
6 VM nodes(not Amazon VM) in this cluster;
Server quota 2.4GB and bucket quota 512MB;
Total data in the 4.68GB (2 replicas)

And some error reports in the log file:
[ns_server:error,2013-12-24T23:05:03.872,ns_1@10.11.20.54:<0.8438.3>:ns_memcached:verify_report_long_call:294]call topkeys took too long: 705873 us

[stats:error,2013-12-24T23:10:08.523,ns_1@10.11.20.54:<0.8042.33>:stats_reader:log_bad_responses:170]Some nodes didn’t respond: [‘ns_1@10.11.20.59’]

[ns_server:error,2013-12-24T23:10:33.896,ns_1@10.11.20.54:<0.8439.3>:ns_memcached:verify_report_long_call:294]call topkeys took too long: 516901 us

[ns_server:error,2013-12-24T23:46:36.334,ns_1@10.11.20.54:<0.8167.33>:ns_single_vbucket_mover:spawn_and_wait:87]Got unexpected exit signal {‘EXIT’,<0.8149.33>,shutdown}

[error_logger:error,2013-12-24T23:46:36.392,ns_1@10.11.20.54:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.8175.33>
registered_name: []
exception exit: {unexpected_exit,{‘EXIT’,<0.8149.33>,shutdown}}
in function ns_single_vbucket_mover:spawn_and_wait/1
in call from ns_single_vbucket_mover:mover_inner/6
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.8149.33>,<0.8053.33>]
messages: [{‘EXIT’,<0.8149.33>,shutdown}]
links: [<0.8149.33>]
dictionary: [{cleanup_list,[<0.8284.33>,<0.8295.33>]}]
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 4618
neighbours:

10.11.20.54 is the pool master
10.11.20.59 is the added node.