Rebalance after adding new node failed

I’m currently in Couchbase 2.1.1. I need to updgrade two of my nodes to 2.2.0 so I added one extra server for temporary swapspace and it happens to be 2.0.1. Once I have added the new server and I clicked rebalance, I got this error:

Rebalance exited with reason {unexpected_exit,
{‘EXIT’,<0.6995.1477>,
{badmatch,
[{‘EXIT’,
{{badmatch,{error,ehostunreach}},
{gen_server,call,
[<16025.11383.0>,had_backfill,30000]}}},
{‘EXIT’,
{shutdown,
{gen_server,call,
[<0.6992.1477>,had_backfill,30000]}}}]}}}

I’m guessing it has something to do with host unreachable which is strange consider I have opened up all the necessary ports and they all can ping each other. and just for the giggle, I turned off all the firewall and I’m still having the same problem.

I was thinking perhaps it’s because the new server I’m trying to add is couchbase 2.0.1. However after I upgraded the 2.0.1 to 2.1.1, I’m still seeing the same problem. Please advise?

Many thanks.

Can you share the ports you have open?
thanks
-cihan

Sure. Here you go straight from the firewall…

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8091 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8092 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 11209:11211 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4369 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 21100:21199 -j ACCEPT