The Join Cluster / Start new Cluster for a cluster Setup does not navigate to the subsequent page and the “Loading” message continues to display.
4.0.0-4051 Enterprise Edition (build-4051)
Are you trying to create a new cluster, or join an existing one? Which OS are you on and is it one of the supported platforms?
Are you trying to create a new cluster, or join an existing one?
Create a new cluster, if a default cluster is not provided.
Which OS are you on and is it one of the supported platforms?
Ubuntu Server 14.04 LTS (HVM), SSD Volume Type on Amazon EC2 ami-d05e75b8. Yes, supported.
Couchbase Server is run in a Docker container. Docker image is
https://hub.docker.com/r/couchbase/server/
We’d probably have to look at some logs to debug it fully. Can you either log an issue at issues.couchbase.com and submit a cbcollect info or maybe to try to more easily diagnose it you can post the output of ps -u couchbase and look at the error.log for clues?
Snippets from info.log, debug.log and error.log are as follows; complete log messages available if required. Some messages could have been repeated in log files as multiple tries to start a cluster were made.
info.log
22T15:50:38.448Z,ns_1@127.0.0.1:ns_couchdb_port<0.181.0>:ns_port_server:log:210]ns_couchdb<0.181.0>: Apache CouchDB (LogLevel=info) is starting.
ns_couchdb<0.181.0>: Apache CouchDB has started. Time to relax.
[ns_server:info,2015-10-22T15:50:38.493Z,ns_1@127.0.0.1:ns_server_sup<0.199.0>:dir_size:start_link:39]Starting quick version of dir_size with program name: godu
[ns_server:warn,2015-10-22T15:50:38.499Z,ns_1@127.0.0.1:ns_log<0.204.0>:ns_log:read_logs:128]Couldn’t load logs from “/opt/couchbase/var/lib/couchbase/ns_log” (perhaps it’s first startup): {error,
enoent}
[ns_server:warn,2015-10-22T15:50:38.512Z,ns_1@127.0.0.1:ns_config_isasl_sync<0.206.0>:ns_memcached:connect:1283]Unable to connect: {error,{badmatch,{error,econnrefused}}}, retrying.
[ns_server:info,2015-10-22T15:50:38.676Z,ns_1@127.0.0.1:ns_couchdb_port<0.181.0>:ns_port_server:log:210]ns_couchdb<0.181.0>: 107: Booted. Waiting for shutdown request
ns_couchdb<0.181.0>: working as port
[user:info,2015-10-22T15:50:39.515Z,ns_1@127.0.0.1:ns_cookie_manager<0.141.0>:ns_cookie_manager:do_cookie_init:86]Initial otp cookie generated: ahkpnhrujmqupnvg
[user:info,2015-10-22T15:50:39.578Z,ns_1@127.0.0.1:mb_master<0.254.0>:mb_master:init:86]I’m the only node, so I’m the master.
[ns_server:info,2015-10-22T15:50:39.601Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_online_config_upgrader:upgrade_config_from_2_0_to_2_5:51]Performing online config upgrade to 2.5 version
[ns_server:info,2015-10-22T15:50:39.601Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_online_config_upgrader:upgrade_config_from_2_5_to_3_0:55]Performing online config upgrade to 3.0 version
[ns_server:info,2015-10-22T15:50:39.602Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_online_config_upgrader:upgrade_config_from_3_0_to_4_0:60]Performing online config upgrade to 4.0 version
[ns_server:info,2015-10-22T15:50:39.611Z,ns_1@127.0.0.1:ns_ssl_services_setup<0.158.0>:ns_ssl_services_setup:handle_info:376]Got certificate and pkey change
[ns_server:info,2015-10-22T15:50:39.613Z,ns_1@127.0.0.1:ns_ssl_services_setup<0.158.0>:ns_ssl_services_setup:maybe_generate_local_cert:444]Failed to read node certificate. Perhaps it wasn’t created yet. Error: {error,
{badmatch,
{error,
enoent}}}
[user:warn,2015-10-22T15:50:39.614Z,ns_1@127.0.0.1:<0.258.0>:ns_orchestrator:consider_switching_compat_mode:1130]Changed cluster compat mode from undefined to [4,0]
[user:info,2015-10-22T15:50:39.689Z,ns_1@127.0.0.1:ns_server_sup<0.199.0>:menelaus_sup:start_link:46]Couchbase Server has started on web port 8091 on node ‘ns_1@127.0.0.1’. Version: “4.0.0-4051-enterprise”.
[ns_server:warn,2015-10-22T15:50:39.712Z,ns_1@127.0.0.1:<0.319.0>:ns_memcached:connect:1283]Unable to connect: {error,{badmatch,{error,econnrefused}}}, retrying.
[ns_server:info,2015-10-22T15:50:39.732Z,ns_1@127.0.0.1:<0.323.0>:ns_memcached_log_rotator:init:28]Starting log rotator on “/opt/couchbase/var/lib/couchbase/logs”/“memcached.log”* with an initial period of 39003ms
[ns_server:info,2015-10-22T15:50:41.387Z,ns_1@127.0.0.1:ns_ssl_services_setup<0.158.0>:ns_ssl_services_setup:do_generate_local_cert:432]Saved local cert for node ‘ns_1@127.0.0.1’
[ns_server:info,2015-10-22T15:50:41.399Z,ns_1@127.0.0.1:ns_ssl_services_setup<0.158.0>:ns_ssl_services_setup:handle_info:379]Wrote new pem file
[stats:warn,2015-10-22T19:31:34.780Z,ns_1@127.0.0.1:<0.365.0>:base_stats_collector:latest_tick:69](Collector: global_stats_collector) Dropped 1 ticks
[stats:warn,2015-10-22T19:31:34.853Z,ns_1@127.0.0.1:system_stats_collector<0.347.0>:base_stats_collector:latest_tick:69](Collector: system_stats_collector) Dropped 1 ticks
[stats:warn,2015-10-22T19:33:37.200Z,ns_1@127.0.0.1:query_stats_collector<0.360.0>:base_stats_collector:latest_tick:69](Collector: query_stats_collector) Dropped 2 ticks
[user:info,2015-10-22T19:49:07.722Z,ns_1@127.0.0.1:<0.8469.2>:ns_log:crash_consumption_loop:70]Service ‘goxdcr’ exited with status 1. Restarting. Messages: MetadataService 2015-10-22T19:48:33.602Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=2
MetadataService 2015-10-22T19:48:33.602Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=3
MetadataService 2015-10-22T19:48:33.602Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=4
RemoteClusterService 2015-10-22T19:48:33.602Z [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
[goport] 2015/10/22 19:48:33 /opt/couchbase/bin/goxdcr terminated: exit status 1
[ns_server:error,2015-10-22T19:49:07.723Z,ns_1@127.0.0.1:ns_log<0.8468.2>:ns_log:handle_cast:209]unable to notify listeners because of badarg
[user:info,2015-10-22T19:49:07.725Z,ns_1@127.0.0.1:<0.8469.2>:ns_log:crash_consumption_loop:70]Service ‘goxdcr’ exited with status 1. Restarting. Messages: MetadataService 2015-10-22T19:48:41.527Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=2
MetadataService 2015-10-22T19:48:41.527Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=3
MetadataService 2015-10-22T19:48:41.527Z [ERROR] metakv.ListAllChildren failed. path=/remoteCluster/, err=Get http://127.0.0.1:8091/_metakv/remoteCluster/: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused, num_of_retry=4
RemoteClusterService 2015-10-22T19:48:41.527Z [ERROR] Failed to get all entries, err=metakv failed for max number of retries = 5
[goport] 2015/10/22 19:48:41 /opt/couchbase/bin/goxdcr terminated: exit status 1
[ns_server:error,2015-10-22T19:49:07.725Z,ns_1@127.0.0.1:ns_log<0.8468.2>:ns_log:handle_cast:209]unable to notify listeners because of badarg
22T19:53:21.897Z,ns_1@127.0.0.1:mb_master<0.8521.2>:mb_master:terminate:299]Synchronously shutting down child mb_master_sup
[ns_server:info,2015-10-22T19:53:22.144Z,ns_1@127.0.0.1:ns_couchdb_port<0.8288.2>:ns_port_server:log:210]ns_couchdb<0.8288.2>: 718: got shutdown request. Exiting
ns_couchdb<0.8288.2>: [os_mon] memory supervisor port (memsup): Erlang has closed
ns_couchdb<0.8288.2>: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[ns_server:info,2015-10-22T19:54:56.579Z,ns_1@127.0.0.1:ns_couchdb_port<0.9949.2>:ns_port_server:log:210]ns_couchdb<0.9949.2>: Apache CouchDB (LogLevel=info) is starting.
[ns_server:info,2015-10-22T19:55:02.425Z,ns_1@127.0.0.1:ns_couchdb_port<0.9949.2>:ns_port_server:log:210]ns_couchdb<0.9949.2>: Apache CouchDB has started. Time to relax.
[ns_server:info,2015-10-22T19:55:04.959Z,ns_1@127.0.0.1:ns_couchdb_port<0.9949.2>:ns_port_server:log:210]ns_couchdb<0.9949.2>: 837: Booted. Waiting for shutdown request
ns_couchdb<0.9949.2>: 837: got shutdown request. Exiting
ns_couchdb<0.9949.2>: [os_mon] memory supervisor port (memsup): Erlang has closed
ns_couchdb<0.9949.2>: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[ns_server:info,2015-10-22T19:55:05.040Z,ns_1@127.0.0.1:ns_couchdb_port<0.9949.2>:ns_port_server:log:210]ns_couchdb<0.9949.2>: working as port
[ns_server:error,2015-10-22T19:55:05.041Z,ns_1@127.0.0.1:<0.153.0>:restartable:loop:85]Failed to restart child {ns_server_nodes_sup,start_link,[]}: {error,
{shutdown,
{failed_to_start_child,
wait_for_couchdb_node,
timeout}}}
[ns_server:error,2015-10-22T19:55:05.041Z,ns_1@127.0.0.1:<0.8561.2>:menelaus_web:loop:184]Server error during processing: [“web request failed”,
{path,"/nodes/self/controller/settings"},
{method,‘POST’},
{type,error},
{what,
{badmatch,
{error,
{shutdown,
{failed_to_start_child,
wait_for_couchdb_node,timeout}}}}},
{trace,
[{menelaus_web,handle_node_settings_post,2,
[{file,“src/menelaus_web.erl”},
{line,2633}]},
{request_throttler,do_request,3,
[{file,“src/request_throttler.erl”},
{line,59}]},
{menelaus_web,loop,2,
[{file,“src/menelaus_web.erl”},
{line,162}]},
{mochiweb_http,headers,5,
[{file,
"/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
{line,94}]},
{proc_lib,init_p_do_apply,3,
[{file,“proc_lib.erl”},{line,239}]}]}]
debug.log
[ns_server:info,2015-10-22T15:50:36.097Z,nonode@nohost:<0.89.0>:ns_server:init_logging:144]Started & configured logging
[ns_server:info,2015-10-22T15:50:36.130Z,nonode@nohost:dist_manager<0.130.0>:dist_manager:read_address_config_from_path:86]Reading ip config from “/opt/couchbase/var/lib/couchbase/ip_start”
[ns_server:info,2015-10-22T15:50:36.130Z,nonode@nohost:dist_manager<0.130.0>:dist_manager:read_address_config_from_path:86]Reading ip config from “/opt/couchbase/var/lib/couchbase/ip”
[ns_server:info,2015-10-22T15:50:36.130Z,nonode@nohost:dist_manager<0.130.0>:dist_manager:init:163]ip config not found. Looks like we’re brand new node
[ns_server:info,2015-10-22T15:50:36.135Z,nonode@nohost:dist_manager<0.130.0>:dist_manager:bringup:214]Attempting to bring up net_kernel with name ‘ns_1@127.0.0.1’
[ns_server:debug,2015-10-22T15:50:36.149Z,ns_1@127.0.0.1:dist_manager<0.130.0>:dist_manager:bringup:228]Attempted to save node name to disk: ok
[ns_server:debug,2015-10-22T15:50:36.149Z,ns_1@127.0.0.1:dist_manager<0.130.0>:dist_manager:wait_for_node:235]Waiting for connection to node ‘babysitter_of_ns_1@127.0.0.1’ to be established
[ns_server:info,2015-10-22T15:50:36.193Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_config:load_config:995]Loading static config from “/opt/couchbase/etc/couchbase/config”
[ns_server:info,2015-10-22T15:50:36.194Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_config:load_config:1009]Loading dynamic config from “/opt/couchbase/var/lib/couchbase/config/config.dat”
[ns_server:info,2015-10-22T15:50:36.194Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_config:load_config:1014]No dynamic config file found. Assuming we’re brand new node
[ns_server:debug,2015-10-22T15:50:36.195Z,ns_1@127.0.0.1:ns_config<0.146.0>:ns_config:load_config:1017]Here’s full dynamic config we loaded:
[[]]
[ns_server:warn,2015-10-22T15:50:38.499Z,ns_1@127.0.0.1:ns_log<0.204.0>:ns_log:read_logs:128]Couldn’t load logs from “/opt/couchbase/var/lib/couchbase/ns_log” (perhaps it’s first startup): {error,
enoent}
=========================PROGRESS REPORT=========================
supervisor: {local,ns_server_sup}
started: [{pid,<0.205.0>},
{name,ns_crash_log_consumer},
{mfargs,{ns_log,start_link_crash_consumer,[]}},
{restart_type,{permanent,4}},
{shutdown,1000},
{child_type,worker}]
[ns_server:debug,2015-10-22T15:50:38.506Z,ns_1@127.0.0.1:ns_config_isasl_sync<0.206.0>:ns_config_isasl_sync:init:63]isasl_sync init: ["/opt/couchbase/var/lib/couchbase/isasl.pw","_admin",
“683f2873134f5cd7def60f77bcec3bf8”]
[ns_server:debug,2015-10-22T15:50:38.506Z,ns_1@127.0.0.1:ns_config_isasl_sync<0.206.0>:ns_config_isasl_sync:init:71]isasl_sync init buckets: []
[ns_server:debug,2015-10-22T15:50:38.507Z,ns_1@127.0.0.1:ns_config_isasl_sync<0.206.0>:ns_config_isasl_sync:writeSASLConf:143]Writing isasl passwd file: “/opt/couchbase/var/lib/couchbase/isasl.pw”
[ns_server:warn,2015-10-22T15:50:38.512Z,ns_1@127.0.0.1:ns_config_isasl_sync<0.206.0>:ns_memcached:connect:1283]Unable to connect: {error,{badmatch,{error,econnrefused}}}, retrying.
[ns_server:info,2015-10-22T15:50:38.676Z,ns_1@127.0.0.1:ns_couchdb_port<0.181.0>:ns_port_server:log:210]ns_couchdb<0.181.0>: 107: Booted. Waiting for shutdown request
ns_couchdb<0.181.0>: working as port
[ns_server:debug,2015-10-22T15:50:39.575Z,ns_1@127.0.0.1:<0.251.0>:mb_master:check_master_takeover_needed:141]Sending master node question to the following nodes: []
[ns_server:debug,2015-10-22T15:50:39.575Z,ns_1@127.0.0.1:<0.251.0>:mb_master:check_master_takeover_needed:143]Got replies: []
[ns_server:debug,2015-10-22T15:50:39.575Z,ns_1@127.0.0.1:<0.251.0>:mb_master:check_master_takeover_needed:149]Was unable to discover master, not going to force mastership takeover
[user:info,2015-10-22T15:50:39.578Z,ns_1@127.0.0.1:mb_master<0.254.0>:mb_master:init:86]I’m the only node, so I’m the master.
[ns_server:debug,2015-10-22T16:07:40.039Z,ns_1@127.0.0.1:compaction_new_daemon<0.379.0>:compaction_scheduler:schedule_next:60]Finished compaction too soon. Next run will be in 30s
[ns_server:debug,2015-10-22T19:48:43.563Z,ns_1@127.0.0.1:<0.8290.2>:ns_server_nodes_sup:do_wait_link_to_couchdb_node:133]ns_couchdb is not ready: false
error.log
[ns_server:error,2015-10-22T19:49:07.723Z,ns_1@127.0.0.1:ns_log<0.8468.2>:ns_log:handle_cast:209]unable to notify listeners because of badarg
[ns_server:error,2015-10-22T19:55:05.041Z,ns_1@127.0.0.1:<0.153.0>:restartable:loop:85]Failed to restart child {ns_server_nodes_sup,start_link,[]}: {error,
{shutdown,
{failed_to_start_child,
wait_for_couchdb_node,
timeout}}}
[ns_server:error,2015-10-22T21:30:32.428Z,ns_1@127.0.0.1:<0.5742.3>:menelaus_web:loop:184]Server error during processing: [“web request failed”,
{path,"/nodes/self/controller/settings"},
{method,‘POST’},
{type,error},
{what,
{badmatch,
{error,
{shutdown,
{failed_to_start_child,
wait_for_couchdb_node,timeout}}}}},
{trace,
[{menelaus_web,handle_node_settings_post,2,
[{file,“src/menelaus_web.erl”},
{line,2633}]},
{request_throttler,do_request,3,
[{file,“src/request_throttler.erl”},
{line,59}]},
{menelaus_web,loop,2,
[{file,“src/menelaus_web.erl”},
{line,162}]},
{mochiweb_http,headers,5,
[{file,
"/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
{line,94}]},
{proc_lib,init_p_do_apply,3,
[{file,“proc_lib.erl”},{line,239}]}]}]
[ns_server:error,2015-10-22T22:01:59.873Z,ns_1@127.0.0.1:<0.8474.3>:restartable:loop:85]Failed to restart child {ns_server_nodes_sup,start_link,[]}: {error,
{shutdown,
{failed_to_start_child,
wait_for_couchdb_node,
timeout}}}
It looks like it’s having some trouble starting some of the underlying services. Best thing is to create a cbcollect_info and file an issue.
Submitted a JIRA issue with collect.zip included.
As an update, the issue was fixed by modifying the IP Address in the /etc/hosts to the Public IP Address of the Amazon EC2 instance from 127.0.0.1.
1 Like