Hi Team,
Index server hangs with below exception
Rebalance failed. See logs for detailed reason. You can try again.
in indexing log we see following exception
2019-10-01T18:21:10.415+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 179] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 177] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:10.415+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 179] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:10.419+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 179]
2019-10-01T18:21:10.419+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 179]
2019-10-01T18:21:11.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:11.415+08:00 [Info] Rebalancer::computeProgress 0
2019-10-01T18:21:11.416+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 180] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 178] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:11.416+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 180] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:11.418+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 180]
2019-10-01T18:21:11.418+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 180]
2019-10-01T18:21:12.324+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.50:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId
]) partitionScheme:SINGLE whereExpression:((type
= “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.328+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.50:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.328+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.50:9999
2019-10-01T18:21:12.328+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.51:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId
]) partitionScheme:SINGLE whereExpression:((type
= “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.331+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.51:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.331+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.51:9999
2019-10-01T18:21:12.331+08:00 [Info] KVSender::sendMutationTopicRequest Projector 10.126.112.52:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Instances [indexInstance:<instId:4690826406376890930 state:IndexPending definition:<defnID:5013097842794997973 bucket:optima isPrimary:false name:idx_processinstance_processInstDocId using:0 exprType:N1QL secExpressions:([processInstDocId
]) partitionScheme:SINGLE whereExpression:((type
= “processinstance”)) > singlePartn:endpoints:“10.126.112.46:9105” > ]
2019-10-01T18:21:12.335+08:00 [Error] KVSender::sendMutationTopicRequest Projector 10.126.112.52:9999 Topic MAINT_STREAM_TOPIC_6a2bb21016fc39f162b1ba7b73c97a10 optima
Unexpected Error dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:12.335+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error Received dial tcp 10.126.112.46:9105: getsockopt: no route to host from 10.126.112.52:9999
2019-10-01T18:21:12.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:12.415+08:00 [Info] Rebalancer::computeProgress 0
2019-10-01T18:21:12.415+08:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 12 181] [6a2bb21016fc39f162b1ba7b73c97a10] true }
2019-10-01T18:21:12.415+08:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 12 181] [{[0 0 0 0 0 0 0 0] prepare/938e198be3420b75d1be3b33de8457db task-prepared task-running true 0 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]} {[0 0 0 0 0 0 12 179] rebalance/938e198be3420b75d1be3b33de8457db task-rebalance task-running true 0.1 map map[rebalanceId:938e198be3420b75d1be3b33de8457db]}]}
2019-10-01T18:21:12.417+08:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 12 181]
2019-10-01T18:21:12.417+08:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 12 181]
2019-10-01T18:21:13.310+08:00 [Info] Rebalancer::waitForIndexBuild Index optima:idx_processinstance_processInstDocId: State INDEX_STATE_INITIAL Pending 1.0146875e+07 EstTime 10
2019-10-01T18:21:13.335+08:00 [Error] KVSender::openMutationStream MAINT_STREAM optima Error from Projector dial tcp 10.126.112.46:9105: getsockopt: no route to host
2019-10-01T18:21:13.335+08:00 [Error] Indexer::startBucketStream Stream MAINT_STREAM Bucket optima Error from Projector dial tcp 10.126.112.46:9105: getsockopt: no route to host. Retrying 5.
2019-10-01T18:21:13.415+08:00 [Info] Rebalancer::getBuildProgressFromStatus 4690826406376890930 0 0
2019-10-01T18:21:13.415+08:00 [Info] Rebalancer::computeProgress 0