Index node hangs with error

Couchbase Server
1

Hi Team,

Index server hangs with below exception

Rebalance failed. See logs for detailed reason. You can try again.
in indexing log we see following exception

2019-10-01T18:21:10.415+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 179] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 177] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:10.415+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 179] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:10.419+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 179]
2019-10-01T18:21:10.419+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 179]
2019-10-01T18:21:11.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:11.415+08:00 [Info] Rebalancer::computeProgress 0
2019-10-01T18:21:11.416+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 180] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 178] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:11.416+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 180] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:11.418+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 180]
2019-10-01T18:21:11.418+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 180]
2019-10-01T18:21:12.324+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.50:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId]) partitionScheme:SINGLE whereExpression:((type = “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.328+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.50:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.328+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.50:9999
2019-10-01T18:21:12.328+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.51:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId]) partitionScheme:SINGLE whereExpression:((type = “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.331+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.51:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.331+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.51:9999
2019-10-01T18:21:12.331+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.52:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId]) partitionScheme:SINGLE whereExpression:((type = “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.335+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.52:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.335+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.52:9999
2019-10-01T18:21:12.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:12.415+08:00 [Info] Rebalancer::computeProgress 0
2019-10-01T18:21:12.415+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 181] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:12.415+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 181] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 179] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:12.417+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 181]
2019-10-01T18:21:12.417+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 181]
2019-10-01T18:21:13.310+08:00 [Info] Rebalancer::waitForIndexBuild Index optima:idx_processinstance_processInstDocId: State INDEX_STATE_INITIAL Pending 1.0146875e+07 EstTime 10
2019-10-01T18:21:13.335+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error from Projector dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:13.335+08:00 [Error] Indexer::startBucketStream Stream MAINT_STREAM Bucket optima Error from Projector dial tcp 10.126.112.46:9105: getsockopt: no route to host. Retrying 5.
2019-10-01T18:21:13.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:13.415+08:00 [Info] Rebalancer::computeProgress 0

2

It looks like the node 10.126.112.46 is not reachable. Can you please check if
a. You are able to ping the machine 10.126.112.46
b. The port 9105 is opened and not blocked by firewall. Please refer to Couchbase Server Ports | Couchbase Docs for the details on all the ports that need to be opened and available for proper functionality of couchbase-server

Thanks,
Varun