Service_agent-cbas<0.947.4>:service_agent:handle_info:287] Rebalancer <0.1083.4> died unexpectedly: shutdown

Couchbase Server
1

Getting below error message in info.log file.

Couchbase version 6.5 EE

[ns_server:info,2020-02-25T21:27:26.396+05:30,ns_1@:<0.596.0>:ns_orchestrator:handle_info:523]Skipping janitor in state janitor_running
[ns_server:error,2020-02-25T21:27:26.396+05:30,ns_1@:cleanup_process<0.1082.4>:service_janitor:do_orchestrate_initial_rebalance:129]Initial rebalance of service cbas takes too long (timeout 120000)
[ns_server:error,2020-02-25T21:27:26.397+05:30,ns_1@:service_rebalancer-cbas<0.1083.4>:service_rebalancer:run_rebalance_worker:128]Got exit message from parent: {‘EXIT’,<0.1082.4>,shutdown}
[ns_server:error,2020-02-25T21:27:26.397+05:30,ns_1@:cleanup_process<0.1082.4>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for cbas failed: {error,{initial_rebalance_timeout,cbas}}
[ns_server:error,2020-02-25T21:27:26.398+05:30,ns_1@:service_agent-cbas<0.947.4>:service_agent:handle_info:287]Rebalancer <0.1083.4> died unexpectedly: shutdown
[ns_server:error,2020-02-25T21:27:26.502+05:30,ns_1@:service_agent-cbas<0.947.4>:service_agent:cleanup_service:560]Failed to cleanup some stale tasks:
[[{<<“rev”>>,<<“MzUy”>>},
{<<“id”>>,<<“rebalance/596a1c8f11a87cb29c2616ba8abde905”>>},
{<<“type”>>,<<“task-rebalance”>>},
{<<“status”>>,<<“task-failed”>>},
{<<“isCancelable”>>,true},
{<<“progress”>>,0},
{<<“errorMessage”>>,
<<“cancelled waiting for condition on http://:9111/analytics/cluster”>>},
{<<“extra”>>,
{[{<<“cc-ejecting”>>,false},
{<<“rebalanceId”>>,<<“596a1c8f11a87cb29c2616ba8abde905”>>}]}}]]
[ns_server:error,2020-02-25T21:27:26.503+05:30,ns_1@:service_agent-cbas<0.947.4>:service_agent:terminate:312]Terminating abnormally
[ns_server:error,2020-02-25T21:27:26.503+05:30,ns_1@:service_agent-cbas<0.947.4>:service_agent:terminate:317]Terminating json rpc connection for cbas: <0.975.4>

2

Please update for my requested error message and Let me know if need to share more details.

3

Hi @Gulasan_Mishra,
This error is indicating that the initial rebalance for the Analytics Service couldn’t be completed successfully. This could be caused by the Analytics java process failing to start due to incompatible/bad JVM or some ports conflict. You can take a look at the log file (analytics_info.log). It might give you a better idea for the exact cause of the failure. We will need a complete list of logs using cbcollect info in order to help further.

4

Hello Murtadha

Thank you for update and suggestion,

below message is from analytics_info.log. meanwhile looking from my side, if you can suggest something it would be grateful.

2020-02-28T21:30:23.016+05:30 INFO CBAS.cbas not updating balance state balanced-65 for fb5a677fe5b8fdad12939b98564ce36a (current state is unknown for 0934e816bf9927405f6c4c79e
b5bfc87)
2020-02-28T21:30:23.016+05:30 INFO CBAS.cbas not updating balance state balanced-65 for fed4def5a63ebf03ffbddb7b6f5e092d (current state is unknown for 0934e816bf9927405f6c4c79e
b5bfc87)
2020-02-28T21:30:23.016+05:30 INFO CBAS.cbas ignoring older topology 89d35dd61e33529f57788226ffea22dc/218 (was 0934e816bf9927405f6c4c79eb5bfc87/219)
2020-02-28T21:30:23.016+05:30 INFO CBAS.cbas not updating balance state balanced-65 for 89d35dd61e33529f57788226ffea22dc (current state is unknown for 0934e816bf9927405f6c4c79e
b5bfc87)
2020-02-28T21:30:23.016+05:30 INFO CBAS.cbas our node is within topology refresh 0934e816bf9927405f6c4c79eb5bfc87/219; ensuring driver is running…
2020-02-28T21:30:23.059+05:30 INFO CBAS.cbas nodeConfigMap: map[ccAnalyticsHttpListenPort:9111 ccConsoleListenPort:9114 ccHeartbeatPeriod:30000 dataListenPort:9116 replicationL
istenPort:9120 analyticsCcHttpPort:9111 analyticsHttpsListenPort:18095 ccClientListenPort:9113 clusterCompatibility:393221 initialRun:false ncservicePort:-1 sslEnabled:false cc
AnalyticsMetadataPartitionId:0 ccControllerId:0 clusterListenPort:9115 messagingListenPort:9118 replicationLogBufferPagesize:262144 resultListenPort:9117 cbasPort:9122 ccHeartb
eatMaxMisses:4 defaultDir:/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0 iodevices:[/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data
/@analytics/v_iodevice_0] metadataNode:75ca3e89cd66373fb6e19553beb81693 publicAddress:192.168.0.71 address:0.0.0.0 metadataCallbackPort:9119 replicationEnabled:true analyticsHt
tpListenAddress:0.0.0.0 analyticsHttpListenPort:8095 clusterAddress:192.168.0.71 nsServerAddress:127.0.0.1 replicationStrategy:metadata logDir:/disk1/app/couchbase/opt/couchbas
e/var/lib/couchbase/logs metadataListenPort:9121 additionalCcNodes: analyticsCbHome:/disk1/app/couchbase/opt/couchbase analyticsHttpsListenAddress:0.0.0.0 cbasAddress:127.0.0
.1 ccClusterListenPort:9112 ccPublicAddress:192.168.0.71 nsServerPort:8091 startingPartitionId:0 txnLogBufferPagesize:262144 nsServerSslPort:18091 replicationFactor:4096 analyt
icsHttpAdminListenPort:9110 analyticsNodeName:192.168.0.71:8091 analyticsSslEnabled:true ccAddress:0.0.0.0 ccSslEnabled:false clusterPort:9112]
2020-02-28T21:30:23.136+05:30 INFO CBAS.cbas my node is in topology but not registered in cbauth db; sleeping for 2s before trying again…
2020-02-28T21:30:25.140+05:30 INFO CBAS.cbas analytics driver exited; restarting
2020-02-28T21:30:25.174+05:30 INFO CBAS.cbas nodeConfigMap: map[replicationStrategy:metadata ccControllerId:0 ccSslEnabled:false messagingListenPort:9118 metadataNode:75ca3e89c
d66373fb6e19553beb81693 replicationLogBufferPagesize:262144 analyticsCbHome:/disk1/app/couchbase/opt/couchbase ccAddress:0.0.0.0 ccHeartbeatMaxMisses:4 replicationFactor:4096 r
eplicationListenPort:9120 ccClientListenPort:9113 clusterPort:9112 nsServerPort:8091 clusterListenPort:9115 iodevices:[/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data
/@analytics/v_iodevice_0] logDir:/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/logs resultListenPort:9117 sslEnabled:false analyticsHttpListenAddress:0.0.0.0 analyticsHt
tpListenPort:8095 clusterCompatibility:393221 startingPartitionId:0 clusterAddress:192.168.0.71 defaultDir:/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/
v_iodevice_0 metadataListenPort:9121 nsServerAddress:127.0.0.1 nsServerSslPort:18091 analyticsCcHttpPort:9111 analyticsHttpsListenPort:18095 ccAnalyticsMetadataPartitionId:0 tx
nLogBufferPagesize:262144 analyticsHttpAdminListenPort:9110 analyticsHttpsListenAddress:0.0.0.0 ccClusterListenPort:9112 publicAddress:192.168.0.71 address:0.0.0.0 ccPublicAddr
ess:192.168.0.71 ncservicePort:-1 cbasAddress:127.0.0.1 cbasPort:9122 ccAnalyticsHttpListenPort:9111 ccConsoleListenPort:9114 ccHeartbeatPeriod:30000 additionalCcNodes: analy
ticsNodeName:192.168.0.71:8091 analyticsSslEnabled:true replicationEnabled:true dataListenPort:9116 initialRun:false metadataCallbackPort:9119]
2020-02-28T21:30:25.249+05:30 INFO CBAS.cbas my node is in topology but not registered in cbauth db; sleeping for 2s before trying again…
2020-02-28T21:30:27.249+05:30 INFO CBAS.cbas analytics driver exited; restarting
2020-02-28T21:30:27.256+05:30 INFO CBAS.cbas nodeConfigMap: map[additionalCcNodes: iodevices:[/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_
0] ccAddress:0.0.0.0 ccControllerId:0 clusterAddress:192.168.0.71 clusterCompatibility:393221 metadataListenPort:9121 analyticsCbHome:/disk1/app/couchbase/opt/couchbase analyti
csHttpAdminListenPort:9110 analyticsHttpListenPort:8095 metadataNode:75ca3e89cd66373fb6e19553beb81693 ccClientListenPort:9113 ccConsoleListenPort:9114 logDir:/disk1/app/couchba
se/opt/couchbase/var/lib/couchbase/logs nsServerPort:8091 replicationListenPort:9120 analyticsHttpsListenAddress:0.0.0.0 analyticsNodeName:192.168.0.71:8091 ccAnalyticsHttpList
enPort:9111 replicationLogBufferPagesize:262144 resultListenPort:9117 startingPartitionId:0 replicationEnabled:true txnLogBufferPagesize:262144 analyticsHttpListenAddress:0.0.0
.0 analyticsHttpsListenPort:18095 ccAnalyticsMetadataPartitionId:0 dataListenPort:9116 initialRun:false nsServerAddress:127.0.0.1 ccClusterListenPort:9112 ccHeartbeatPeriod:300
00 ccSslEnabled:false replicationFactor:4096 replicationStrategy:metadata ccPublicAddress:192.168.0.71 clusterPort:9112 messagingListenPort:9118 defaultDir:/disk1/app/couchbase
/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0 analyticsSslEnabled:true cbasPort:9122 ccHeartbeatMaxMisses:4 clusterListenPort:9115 metadataCallbackPort:9119 ncs
ervicePort:-1 nsServerSslPort:18091 publicAddress:192.168.0.71 address:0.0.0.0 analyticsCcHttpPort:9111 cbasAddress:127.0.0.1 sslEnabled:false]
2020-02-28T21:30:27.264+05:30 INFO CBAS.cbas my node is in topology but not registered in cbauth db; sleeping for 2s before trying again…
2020-02-28T21:30:29.264+05:30 INFO CBAS.cbas analytics driver exited; restarting
2020-02-28T21:30:29.271+05:30 INFO CBAS.cbas nodeConfigMap: map[dataListenPort:9116 nsServerAddress:127.0.0.1 publicAddress:192.168.0.71 additionalCcNodes: analyticsHttpsList
enAddress:0.0.0.0 ccClusterListenPort:9112 clusterListenPort:9115 replicationListenPort:9120 analyticsHttpsListenPort:18095 ccSslEnabled:false initialRun:false iodevices:[/disk
1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0] metadataListenPort:9121 replicationStrategy:metadata sslEnabled:false analyticsCbHome:/disk1/app/c
ouchbase/opt/couchbase cbasPort:9122 ccAnalyticsMetadataPartitionId:0 logDir:/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/logs ncservicePort:-1 nsServerPort:8091 ccAddr
ess:0.0.0.0 ccHeartbeatMaxMisses:4 clusterPort:9112 defaultDir:/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0 ccHeartbeatPeriod:30000 ccPubli
cAddress:192.168.0.71 replicationFactor:4096 resultListenPort:9117 nsServerSslPort:18091 txnLogBufferPagesize:262144 analyticsHttpListenPort:8095 analyticsSslEnabled:true ccCon
trollerId:0 clusterAddress:192.168.0.71 ccClientListenPort:9113 messagingListenPort:9118 replicationLogBufferPagesize:262144 analyticsCcHttpPort:9111 analyticsHttpAdminListenPo
rt:9110 analyticsNodeName:192.168.0.71:8091 cbasAddress:127.0.0.1 clusterCompatibility:393221 metadataCallbackPort:9119 metadataNode:75ca3e89cd66373fb6e19553beb81693 replicatio
nEnabled:true address:0.0.0.0 analyticsHttpListenAddress:0.0.0.0 ccAnalyticsHttpListenPort:9111 ccConsoleListenPort:9114 startingPartitionId:0]
2020-02-28T21:30:29.302+05:30 INFO CBAS.cbas my node is in topology but not registered in cbauth db; sleeping for 2s before trying again…
2020-02-28T21:30:31.303+05:30 INFO CBAS.cbas analytics driver exited; restarting

5

It looks like there was a series of events that led to this node being in a bad state in the cluster. It could be due to a previous deployment of couchbase server in the same environment but I can’t tell exactly what happened without all the logs. If this is a single node cluster, you can backup your data, uninstall couchbase server by following the steps in (https://docs.couchbase.com/server/current/install/install-uninstalling.html), then install couchbase server again and restore your data. If it is a multi-node cluster, you can remove all the nodes with the Analytics service then Rebalance, and after that, add the nodes with the Analytics service back to the cluster.

6

Hello Murtadha,

Haven’t done previous deployment on this server, it is first time installed Couchbase 6.5 EE.

As earlier you said, it is might due to some port conflict.
some what same port issue seems below in analytics_cbas_debug.log file.
looking now how to resolve this port issue now.

2020-02-29T17:22:41.148+05:30 DEBU CBAS.cbas Setting /cbas/config/node/75ca3e89cd66373fb6e19553beb81693 to {“configVersion”:1,“config”:{“additionalCcNodes”:,“address”:“0.0.0.0”,“analyticsCbHome”:"/disk1/app/couchbase/opt/couchbase",“analyticsCcHttpPort”:“9111”,“analyticsHttpAdminListenPort”:“9110”,“analyticsHttpListenAddress”:“0.0.0.0”,“analyticsHttpListenPort”:“8095”,“analyticsHttpsListenAddress”:“0.0.0.0”,“analyticsHttpsListenPort”:“18095”,“analyticsNodeName”:“192.168.0.71:8091”,“analyticsSslEnabled”:true,“cbasAddress”:“127.0.0.1”,“cbasPort”:“9122”,“ccAddress”:“0.0.0.0”,“ccAnalyticsHttpListenPort”:“9111”,“ccAnalyticsMetadataPartitionId”:0,“ccClientListenPort”:“9113”,“ccClusterListenPort”:“9112”,“ccConsoleListenPort”:“9114”,“ccControllerId”:“0”,“ccHeartbeatMaxMisses”:“4”,“ccHeartbeatPeriod”:“30000”,“ccPublicAddress”:“192.168.0.71”,“ccSslEnabled”:false,“clusterAddress”:“192.168.0.71”,“clusterCompatibility”:393221,“clusterListenPort”:“9115”,“clusterPort”:“9112”,“dataListenPort”:“9116”,“defaultDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0",“initialRun”:false,“iodevices”:["/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0"],“logDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/logs",“messagingListenPort”:“9118”,“metadataCallbackPort”:“9119”,“metadataListenPort”:“9121”,“metadataNode”:“75ca3e89cd66373fb6e19553beb81693”,“ncservicePort”:"-1",“nsServerAddress”:“127.0.0.1”,“nsServerPort”:8091,“nsServerSslPort”:18091,“publicAddress”:“192.168.0.71”,“replicationEnabled”:true,“replicationFactor”:4096,“replicationListenPort”:9120,“replicationLogBufferPagesize”:262144,“replicationStrategy”:“metadata”,“resultListenPort”:“9117”,“sslEnabled”:false,“startingPartitionId”:“0”,“txnLogBufferPagesize”:262144},“nodeId”:“75ca3e89cd66373fb6e19553beb81693”} for 0xc4204813a0
2020-02-29T17:22:41.679+05:30 DEBU CBAS.cbas ignoring error from setRequestAuth(): Unable to find given hostport in cbauth database: 192.168.0.71:8095' 2020-02-29T17:22:41.679+05:30 DEBU CBAS.cbas Sending request to http://192.168.0.71:9111/analytics/cluster 2020-02-29T17:22:41.679+05:30 DEBU CBAS.cbas Got err: Get http://192.168.0.71:9111/analytics/cluster: dial tcp 192.168.0.71:9111: getsockopt: connection refused 2020-02-29T17:22:41.680+05:30 DEBU CBAS.cbas Retrying in 1s... 2020-02-29T17:22:42.688+05:30 DEBU CBAS.cbas ignoring error from setRequestAuth(): Unable to find given hostport in cbauth database: 192.168.0.71:8095’
2020-02-29T17:22:42.688+05:30 DEBU CBAS.cbas Sending request to http://192.168.0.71:9111/analytics/cluster
2020-02-29T17:22:42.689+05:30 DEBU CBAS.cbas Got err: Get http://192.168.0.71:9111/analytics/cluster: dial tcp 192.168.0.71:9111: getsockopt: connection refused
2020-02-29T17:22:42.689+05:30 DEBU CBAS.cbas Retrying in 1s…
2020-02-29T17:22:43.237+05:30 DEBU CBAS.cbas Setting /cbas/config/node/75ca3e89cd66373fb6e19553beb81693 to {“configVersion”:1,“config”:{“additionalCcNodes”:,“address”:“0.0.0.0”,“analyticsCbHome”:"/disk1/app/couchbase/opt/couchbase",“analyticsCcHttpPort”:“9111”,“analyticsHttpAdminListenPort”:“9110”,“analyticsHttpListenAddress”:“0.0.0.0”,“analyticsHttpListenPort”:“8095”,“analyticsHttpsListenAddress”:“0.0.0.0”,“analyticsHttpsListenPort”:“18095”,“analyticsNodeName”:“192.168.0.71:8091”,“analyticsSslEnabled”:true,“cbasAddress”:“127.0.0.1”,“cbasPort”:“9122”,“ccAddress”:“0.0.0.0”,“ccAnalyticsHttpListenPort”:“9111”,“ccAnalyticsMetadataPartitionId”:0,“ccClientListenPort”:“9113”,“ccClusterListenPort”:“9112”,“ccConsoleListenPort”:“9114”,“ccControllerId”:“0”,“ccHeartbeatMaxMisses”:“4”,“ccHeartbeatPeriod”:“30000”,“ccPublicAddress”:“192.168.0.71”,“ccSslEnabled”:false,“clusterAddress”:“192.168.0.71”,“clusterCompatibility”:393221,“clusterListenPort”:“9115”,“clusterPort”:“9112”,“dataListenPort”:“9116”,“defaultDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0",“initialRun”:false,“iodevices”:["/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0"],“logDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/logs",“messagingListenPort”:“9118”,“metadataCallbackPort”:“9119”,“metadataListenPort”:“9121”,“metadataNode”:“75ca3e89cd66373fb6e19553beb81693”,“ncservicePort”:"-1",“nsServerAddress”:“127.0.0.1”,“nsServerPort”:8091,“nsServerSslPort”:18091,“publicAddress”:“192.168.0.71”,“replicationEnabled”:true,“replicationFactor”:4096,“replicationListenPort”:9120,“replicationLogBufferPagesize”:262144,“replicationStrategy”:“metadata”,“resultListenPort”:“9117”,“sslEnabled”:false,“startingPartitionId”:“0”,“txnLogBufferPagesize”:262144},“nodeId”:“75ca3e89cd66373fb6e19553beb81693”} for 0xc420481810
2020-02-29T17:22:43.690+05:30 DEBU CBAS.cbas ignoring error from setRequestAuth(): Unable to find given hostport in cbauth database: 192.168.0.71:8095' 2020-02-29T17:22:43.690+05:30 DEBU CBAS.cbas Sending request to http://192.168.0.71:9111/analytics/cluster 2020-02-29T17:22:43.691+05:30 DEBU CBAS.cbas Got err: Get http://192.168.0.71:9111/analytics/cluster: dial tcp 192.168.0.71:9111: getsockopt: connection refused 2020-02-29T17:22:43.691+05:30 DEBU CBAS.cbas Retrying in 1s... 2020-02-29T17:22:44.692+05:30 DEBU CBAS.cbas ignoring error from setRequestAuth(): Unable to find given hostport in cbauth database: 192.168.0.71:8095’
2020-02-29T17:22:44.692+05:30 DEBU CBAS.cbas Sending request to http://192.168.0.71:9111/analytics/cluster
2020-02-29T17:22:44.693+05:30 DEBU CBAS.cbas Got err: Get http://192.168.0.71:9111/analytics/cluster: dial tcp 192.168.0.71:9111: getsockopt: connection refused
2020-02-29T17:22:44.693+05:30 DEBU CBAS.cbas Retrying in 1s…
2020-02-29T17:22:45.271+05:30 DEBU CBAS.cbas Setting /cbas/config/node/75ca3e89cd66373fb6e19553beb81693 to {“configVersion”:1,“config”:{“additionalCcNodes”:,“address”:“0.0.0.0”,“analyticsCbHome”:"/disk1/app/couchbase/opt/couchbase",“analyticsCcHttpPort”:“9111”,“analyticsHttpAdminListenPort”:“9110”,“analyticsHttpListenAddress”:“0.0.0.0”,“analyticsHttpListenPort”:“8095”,“analyticsHttpsListenAddress”:“0.0.0.0”,“analyticsHttpsListenPort”:“18095”,“analyticsNodeName”:“192.168.0.71:8091”,“analyticsSslEnabled”:true,“cbasAddress”:“127.0.0.1”,“cbasPort”:“9122”,“ccAddress”:“0.0.0.0”,“ccAnalyticsHttpListenPort”:“9111”,“ccAnalyticsMetadataPartitionId”:0,“ccClientListenPort”:“9113”,“ccClusterListenPort”:“9112”,“ccConsoleListenPort”:“9114”,“ccControllerId”:“0”,“ccHeartbeatMaxMisses”:“4”,“ccHeartbeatPeriod”:“30000”,“ccPublicAddress”:“192.168.0.71”,“ccSslEnabled”:false,“clusterAddress”:“192.168.0.71”,“clusterCompatibility”:393221,“clusterListenPort”:“9115”,“clusterPort”:“9112”,“dataListenPort”:“9116”,“defaultDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0",“initialRun”:false,“iodevices”:["/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0"],“logDir”:"/disk1/app/couchbase/opt/couchbase/var/lib/couchbase/logs",“messagingListenPort”:“9118”,“metadataCallbackPort”:“9119”,“metadataListenPort”:“9121”,“metadataNode”:“75ca3e89cd66373fb6e19553beb81693”,“ncservicePort”:"-1",“nsServerAddress”:“127.0.0.1”,“nsServerPort”:8091,“nsServerSslPort”:18091,“publicAddress”:“192.168.0.71”,“replicationEnabled”:true,“replicationFactor”:4096,“replicationListenPort”:9120,“replicationLogBufferPagesize”:262144,“replicationStrategy”:“metadata”,“resultListenPort”:“9117”,“sslEnabled”:false,“startingPartitionId”:“0”,“txnLogBufferPagesize”:262144},“nodeId”:“75ca3e89cd66373fb6e19553beb81693”} for 0xc420481cb0

7

@Gulasan_Mishra,
This connection refused error message is expected when the java process isn’t running. As per the previous log you posted, the java process isn’t running because the cluster manager believes that the node with IP 192.168.0.71 shouldn’t be part of the cluster, and that’s why I recommended starting from a clean state.