I’ve created a new EC2 server on AWS and I tried to add it to my cluster but when I hit rebalance it fails, this is what I see. in the logs:
Rebalance exited with reason {badmatch,
{leader_activities_error,
{default,{ns_janitor,“api”,apply_config}},
{no_quorum,
[{required_quorum,
{all,
{set,5,16,16,8,80,48,
{,,,,,,,,,,,,,
,,},
{{[‘ns_1@10.10.200.100’,
‘ns_1@10.10.100.50’],
,,
[‘ns_1@10.10.100.93’,
‘ns_1@10.10.100.133’],
,,
[‘ns_1@10.10.200.166’],
,,,,,,,,}}}}},
{leases,
[‘ns_1@10.10.100.50’,‘ns_1@10.10.200.100’,
‘ns_1@10.10.100.93’,
‘ns_1@10.10.100.133’]}]}}}
in addition there are weird errors in the logs when I connect via ssh, I see a lot:
Projector.log - [Error] PROJ[:9999] ##2 acquireFeed(): projector.topicMissing
indexer.log:
2020-09-21T16:01:44.872+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 10.100.1.145:55610. Error = EOF. Kill Pipe.
2020-09-21T16:01:44.872+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2020-09-21T16:01:44.872+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.100.1.145:55610)
2020-09-21T16:01:44.872+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 10.100.1.145:55611. Error = EOF. Kill Pipe.
2020-09-21T16:01:44.872+00:00 [Error] LeaderSyncProxy.updateAcceptEpochAfterQuorum(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2020-09-21T16:01:44.872+00:00 [Error] LeaderServer:startProxy(): Leader Fail to synchronization with follower (TCP conn = 10.100.1.145:55611)
error.log:
[ns_server:error,2020-09-21T15:09:07.177Z,ns_1@10.10.200.149:service_status_keeper_worker<0.450.0>:rest_utils:get_json_local:63]Request to (indexer) getIndexStatus failed: {error,
{econnrefused,
[{lhttpc_client,send_request,1,
[{file,
“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},
{line,220}]},
{lhttpc_client,execute,9,
[{file,
“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},
{line,169}]},
{lhttpc_client,request,9,
[{file,
“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},
{line,92}]}]}}
I’ve created and added new servers before, not sure what happened al of the sudden I’m using the same method as before, any help?
thank you very much