Hi,
I’m on Ubuntu 14.04, Couchbase Server Community 4.0.
One server acts as reverse proxy (Nginx) to route incoming traffic to 3 Couchbase servers which are in 1 Couchbase cluster. I followed this guide and additionally opened the ports 4984 and 4985 for all 4 servers. Including the servers themselves as a last resort. The ufw status is the same in all 3 Couch servers.
ufw status
Status: active
To Action From
-- ------ ----
22 ALLOW Anywhere
8091 ALLOW Anywhere
8092 ALLOW Anywhere
8093 ALLOW Anywhere
9100:9105/tcp ALLOW Anywhere
9998 ALLOW Anywhere
9999 ALLOW Anywhere
11207 ALLOW Anywhere
18091 ALLOW Anywhere
18092 ALLOW Anywhere
4369 ALLOW Anywhere
21100:21299/tcp ALLOW Anywhere
4984 ALLOW Reverse Proxy IP
4985 ALLOW Reverse Proxy IP
4984 ALLOW IP of this server (Couch Server 1)
4985 ALLOW IP of this server (Couch Server 1)
4984 ALLOW IP of Couch Server 2
4985 ALLOW IP of Couch Server 2
4984 ALLOW IP of Couch Server 3
4985 ALLOW IP of Couch Server 3
22 (v6) ALLOW Anywhere (v6)
8091 (v6) ALLOW Anywhere (v6)
8092 (v6) ALLOW Anywhere (v6)
8093 (v6) ALLOW Anywhere (v6)
9100:9105/tcp (v6) ALLOW Anywhere (v6)
9998 (v6) ALLOW Anywhere (v6)
9999 (v6) ALLOW Anywhere (v6)
11207 (v6) ALLOW Anywhere (v6)
18091 (v6) ALLOW Anywhere (v6)
18092 (v6) ALLOW Anywhere (v6)
4369 (v6) ALLOW Anywhere (v6)
21100:21299/tcp (v6) ALLOW Anywhere (v6)
Rebalancing, or adding a new bucket will not work. The result is that it takes too long and the operation will be stopped. It works after disabling ufw and redoing the operation. Am I missing other ports?
Here is the Log output when adding a new server and then rebalancing:
Log 1:
Bucket "bucket_name" loaded on node 'ns_1@Couchbase-Server-3-IP(this server was added to cluster)' in 0 seconds. ns_memcached000 ns_1@Couchbase-Server-3-IP(this server was added to cluster) 11:03:41 - Thu Feb 25, 2016
Log 2:
Bucket "bucket_name" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@Couchbase-Server-1-IP(Master) 11:03:41 - Thu Feb 25, 2016
Log 3:
<0.22538.2> exited with {unexpected_exit,
{'EXIT',<0.22545.2>,
{bulk_set_vbucket_state_failed,
[{'ns_1@Couchbase-Server-3-IP(this server was added to cluster)',
{'EXIT',
{{{{case_clause,
{error,
{{{badmatch,
{error,
{{badmatch,{error,etimedout}},
[{mc_replication,connect,1,
[{file,"src/mc_replication.erl"},
{line,30}]},
{mc_replication,connect,1,
[{file,"src/mc_replication.erl"},
{line,49}]},
{dcp_proxy,connect,4,
[{file,"src/dcp_proxy.erl"},
{line,174}]},
{dcp_proxy,maybe_connect,1,
[{file,"src/dcp_proxy.erl"},
{line,161}]},
{dcp_producer_conn,init,2,
[{file,"src/dcp_producer_conn.erl"},
{line,30}]},
{dcp_proxy,init,1,
[{file,"src/dcp_proxy.erl"},
{line,46}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},
{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,239}]}]}}},
[{dcp_replicator,init,1,
[{file,"src/dcp_replicator.erl"},
{line,49}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{child,undefined,'ns_1@Couchbase-Server-1-IP(Master)',
{dcp_replicator,start_link,
['ns_1@Couchbase-Server-1-IP(Master)',
"bucket_name"]},
temporary,60000,worker,
[dcp_replicator]}}}},
[{dcp_sup,start_replicator,2,
[{file,"src/dcp_sup.erl"},{line,53}]},
{dcp_sup,
'-manage_replicators/2-lc$^2/1-2-',2,
[{file,"src/dcp_sup.erl"},{line,69}]},
{dcp_replication_manager,handle_call,3,
[{file,"src/dcp_replication_manager.erl"},
{line,87}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,585}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server,call,
['dcp_replication_manager-bucket_name',
{get_replicator_pid,1023},
infinity]}},
{gen_server,call,
[{'janitor_agent-bucket_name',
'ns_1@Couchbase-Server-3-IP(this server was added to cluster)'},
{if_rebalance,<0.22504.2>,
{update_vbucket_state,1022,replica,passive,
'ns_1@Couchbase-Server-2-IP(Already in cluster)'}},
infinity]}}}}]}}} ns_vbucket_mover000 ns_1@Couchbase-Server-1-IP(Master) 11:05:49 - Thu Feb 25, 2016
Log 4:
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.22545.2>,
{bulk_set_vbucket_state_failed,
[{'ns_1@Couchbase-Server-3-IP(this server was added to cluster)',
{'EXIT',
{{{{case_clause,
{error,
{{{badmatch,
{error,
{{badmatch,{error,etimedout}},
[{mc_replication,connect,1,
[{file,
"src/mc_replication.erl"},
{line,30}]},
{mc_replication,connect,1,
[{file,
"src/mc_replication.erl"},
{line,49}]},
{dcp_proxy,connect,4,
[{file,"src/dcp_proxy.erl"},
{line,174}]},
{dcp_proxy,maybe_connect,1,
[{file,"src/dcp_proxy.erl"},
{line,161}]},
{dcp_producer_conn,init,2,
[{file,
"src/dcp_producer_conn.erl"},
{line,30}]},
{dcp_proxy,init,1,
[{file,"src/dcp_proxy.erl"},
{line,46}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},
{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,239}]}]}}},
[{dcp_replicator,init,1,
[{file,"src/dcp_replicator.erl"},
{line,49}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},
{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,239}]}]},
{child,undefined,
'ns_1@Couchbase-Server-1-IP(Master)',
{dcp_replicator,start_link,
['ns_1@Couchbase-Server-1-IP(Master)',
"bucket_name"]},
temporary,60000,worker,
[dcp_replicator]}}}},
[{dcp_sup,start_replicator,2,
[{file,"src/dcp_sup.erl"},{line,53}]},
{dcp_sup,
'-manage_replicators/2-lc$^2/1-2-',2,
[{file,"src/dcp_sup.erl"},{line,69}]},
{dcp_replication_manager,handle_call,
3,
[{file,
"src/dcp_replication_manager.erl"},
{line,87}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,585}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server,call,
['dcp_replication_manager-bucket_name',
{get_replicator_pid,1023},
infinity]}},
{gen_server,call,
[{'janitor_agent-bucket_name',
'ns_1@Couchbase-Server-3-IP(this server was added to cluster)'},
{if_rebalance,<0.22504.2>,
{update_vbucket_state,1022,replica,
passive,'ns_1@Couchbase-Server-2-IP(Already in cluster)'}},
infinity]}}}}]}}}
ns_orchestrator002 ns_1@Couchbase-Server-1-IP(Master) 11:05:49 - Thu Feb 25, 2016
Regards, Ben