Couchbase server crash and not starting


#1

Couchbase server is not starting after crash… Below are the error in logs.

[ns_server:info,2018-06-28T14:44:15.243+05:30,nonode@nohost:<0.87.0>:ns_server:init_logging:138]Started & configured logging
[ns_server:info,2018-06-28T14:44:15.342+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]Static config terms:
[{error_logger_mf_dir,
“C:/Program Files/Couchbase/Server/var/lib/couchbase/logs”},
{path_config_bindir,“C:/Program Files/Couchbase/Server/bin”},
{path_config_etcdir,“C:/Program Files/Couchbase/Server/etc/couchbase”},
{path_config_libdir,“C:/Program Files/Couchbase/Server/lib”},
{path_config_datadir,“C:/Program Files/Couchbase/Server/var/lib/couchbase”},
{path_config_tmpdir,
“C:/Program Files/Couchbase/Server/var/lib/couchbase/tmp”},
{path_config_secdir,“C:/Program Files/Couchbase/Server/etc/security”},
{nodefile,
“C:/Program Files/Couchbase/Server/var/lib/couchbase/couchbase-server.node”},
{loglevel_default,debug},
{loglevel_couchdb,info},
{loglevel_ns_server,debug},
{loglevel_error_logger,debug},
{loglevel_user,debug},
{loglevel_menelaus,debug},
{loglevel_ns_doctor,debug},
{loglevel_stats,debug},
{loglevel_rebalance,debug},
{loglevel_cluster,debug},
{loglevel_views,debug},
{loglevel_mapreduce_errors,debug},
{loglevel_xdcr,debug},
{loglevel_xdcr_trace,error},
{loglevel_access,info},
{disk_sink_opts,
[{rotation,
[{compress,true},
{size,41943040},
{num_files,10},
{buffer_size_max,52428800}]}]},
{disk_sink_opts_json_rpc,
[{rotation,
[{compress,true},
{size,41943040},
{num_files,2},
{buffer_size_max,52428800}]}]},
{disk_sink_opts_xdcr_trace,
[{rotation,[{compress,false},{size,83886080},{num_files,5}]}]},
{net_kernel_verbosity,10}]
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter error_logger_mf_dir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_bindir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_etcdir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_libdir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_datadir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_tmpdir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter path_config_secdir, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter nodefile, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_default, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_couchdb, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_ns_server, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_error_logger, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_user, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_menelaus, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_ns_doctor, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_stats, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_rebalance, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_cluster, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_views, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_mapreduce_errors, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_xdcr, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_xdcr_trace, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter loglevel_access, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter disk_sink_opts, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter disk_sink_opts_json_rpc, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter disk_sink_opts_xdcr_trace, which is given from command line
[ns_server:warn,2018-06-28T14:44:15.343+05:30,nonode@nohost:<0.87.0>:ns_server:log_pending:32]not overriding parameter net_kernel_verbosity, which is given from command line
[error_logger:info,2018-06-28T14:44:15.448+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_server_cluster_sup}
started: [{pid,<0.142.0>},
{name,local_tasks},
{mfargs,{local_tasks,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

[ns_server:info,2018-06-28T14:44:15.521+05:30,nonode@nohost:ns_server_cluster_sup<0.141.0>:log_os_info:start_link:25]OS type: {win32,nt} Version: {6,2,9200}
Runtime info: [{otp_release,“R16B03-1”},
{erl_version,“5.10.4.0.0.1”},
{erl_version_long,
“Erlang R16B03-1 (erts-5.10.4.0.0.1) [source-53c0d14eb] [64-bit] [smp:4:4] [async-threads:16]\n”},
{system_arch_raw,“win32”},
{system_arch,“win64”},
{localtime,{{2018,6,28},{14,44,15}}},
{memory,
[{total,23214032},
{processes,9933296},
{processes_used,9933000},
{system,13280736},
{atom,339441},
{atom_used,326056},
{binary,93488},
{code,7703147},
{ets,2255568}]},
{loaded,
[ns_info,log_os_info,local_tasks,restartable,
ns_server_cluster_sup,calendar,ale_default_formatter,
‘ale_logger-metakv’,‘ale_logger-rebalance’,
‘ale_logger-xdcr_trace’,‘ale_logger-menelaus’,
‘ale_logger-stats’,‘ale_logger-json_rpc’,
‘ale_logger-access’,‘ale_logger-ns_server’,
‘ale_logger-user’,‘ale_logger-ns_doctor’,
‘ale_logger-cluster’,‘ale_logger-xdcr’,otp_internal,
ns_log_sink,ale_disk_sink,misc,io,couch_util,ns_server,
io_lib_fread,memsup,disksup,filelib,os_mon_sysinfo,os_mon,
release_handler,overload,alarm_handler,sasl,timer,
tftp_sup,httpd_sup,httpc_handler_sup,httpc_cookie,
inets_trace,httpc_manager,httpc,httpc_profile_sup,
httpc_sup,ftp_sup,inets_sup,inets_app,ssl,lhttpc_manager,
lhttpc_sup,lhttpc,tls_connection_sup,ssl_session_cache,
ssl_pkix_db,ssl_manager,ssl_sup,ssl_app,crypto_server,
crypto_sup,crypto_app,ale_error_logger_handler,
‘ale_logger-ale_logger’,‘ale_logger-error_logger’,
beam_opcodes,beam_dict,beam_asm,beam_validator,beam_z,
beam_flatten,beam_trim,beam_receive,beam_bsm,beam_peep,
beam_dead,beam_split,beam_type,beam_bool,beam_except,
beam_clean,beam_utils,beam_block,beam_jump,beam_a,
v3_codegen,v3_life,v3_kernel,sys_core_dsetel,erl_bifs,
sys_core_fold,cerl_trees,sys_core_inline,core_lib,cerl,
v3_core,erl_bits,erl_expand_records,sys_pre_expand,sofs,
erl_internal,sets,ordsets,erl_lint,compile,
dynamic_compile,ale_utils,io_lib_pretty,io_lib_format,
io_lib,ale_codegen,dict,ale,ale_dynamic_sup,ale_sup,
ale_app,epp,ns_bootstrap,child_erlang,file_io_server,
orddict,erl_eval,file,c,kernel_config,user_io,user_sup,
supervisor_bridge,standard_error,code_server,unicode,
hipe_unified_loader,gb_sets,filename,ets,binary,code,
file_server,net_kernel,global_group,erl_distribution,os,
inet_parse,inet,inet_udp,inet_config,inet_db,global,
gb_trees,rpc,supervisor,kernel,application_master,sys,
application,gen_server,erl_parse,proplists,erl_scan,lists,
application_controller,proc_lib,gen,gen_event,
error_logger,heart,error_handler,erts_internal,erlang,
erl_prim_loader,prim_zip,zlib,prim_file,prim_inet,
prim_eval,init,otp_ring0]},
{applications,
[{lhttpc,“Lightweight HTTP Client”,“1.3.0”},
{os_mon,“CPO CXC 138 46”,“2.2.14”},
{public_key,“Public key infrastructure”,“0.21”},
{asn1,“The Erlang ASN1 compiler version 2.0.4”,“2.0.4”},
{kernel,“ERTS CXC 138 10”,“2.16.4”},
{ale,“Another Logger for Erlang”,“5.1.1-5723-enterprise”},
{inets,“INETS CXC 138 49”,“5.9.8”},
{ns_server,“Couchbase server”,“5.1.1-5723-enterprise”},
{crypto,“CRYPTO version 2”,“3.2”},
{ssl,“Erlang/OTP SSL application”,“5.3.3”},
{sasl,“SASL CXC 138 11”,“2.3.4”},
{stdlib,“ERTS CXC 138 10”,“1.19.4”}]},
{pre_loaded,
[erts_internal,erlang,erl_prim_loader,prim_zip,zlib,
prim_file,prim_inet,prim_eval,init,otp_ring0]},
{process_count,108},
{node,nonode@nohost},
{nodes,[]},
{registered,
[ale_dynamic_sup,os_mon_sysinfo,‘sink-disk_xdcr_errors’,
code_server,tftp_sup,memsup,‘sink-disk_xdcr’,disksup,
application_controller,‘sink-disk_debug’,sasl_sup,
‘sink-disk_error’,crypto_server,‘sink-disk_default’,
httpc_sup,standard_error,crypto_sup,error_logger,
standard_error_sup,httpc_profile_sup,tls_connection_sup,
ssl_sup,httpc_manager,timer_server,httpc_handler_sup,
ssl_manager,ftp_sup,ns_server_cluster_sup,release_handler,
overload,init,inet_db,inets_sup,rex,user,alarm_handler,
kernel_sup,‘sink-ns_log’,lhttpc_sup,global_name_server,
file_server_2,lhttpc_manager,global_group,httpd_sup,
sasl_safe_sup,‘sink-disk_json_rpc’,ale_stats_events,
‘sink-disk_metakv’,local_tasks,ale,‘sink-disk_access_int’,
kernel_safe_sup,‘sink-disk_access’,erl_prim_loader,
ale_sup,‘sink-xdcr_trace’,‘sink-disk_reports’,os_mon_sup,
‘sink-disk_stats’]},
{cookie,nocookie},
{wordsize,8},
{wall_clock,8}]
[ns_server:info,2018-06-28T14:44:15.589+05:30,nonode@nohost:ns_server_cluster_sup<0.141.0>:log_os_info:start_link:27]Manifest:
[]

[error_logger:info,2018-06-28T14:44:15.630+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_server_cluster_sup}
started: [{pid,<0.143.0>},
{name,timeout_diag_logger},
{mfargs,{timeout_diag_logger,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[ns_server:info,2018-06-28T14:44:15.668+05:30,nonode@nohost:dist_manager<0.144.0>:dist_manager:read_address_config_from_path:86]Reading ip config from “c:/Program Files/Couchbase/Server/var/lib/couchbase/ip_start”
[ns_server:info,2018-06-28T14:44:15.668+05:30,nonode@nohost:dist_manager<0.144.0>:dist_manager:read_address_config_from_path:86]Reading ip config from “c:/Program Files/Couchbase/Server/var/lib/couchbase/ip”
[error_logger:info,2018-06-28T14:44:15.697+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,inet_gethost_native_sup}
started: [{pid,<0.146.0>},{mfa,{inet_gethost_native,init,[[]]}}]

[error_logger:info,2018-06-28T14:44:15.697+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,kernel_safe_sup}
started: [{pid,<0.145.0>},
{name,inet_gethost_native_sup},
{mfargs,{inet_gethost_native,start_link,[]}},
{restart_type,temporary},
{shutdown,1000},
{child_type,worker}]

[ns_server:info,2018-06-28T14:44:15.775+05:30,nonode@nohost:dist_manager<0.144.0>:dist_manager:bringup:214]Attempting to bring up net_kernel with name ‘ns_1@192.168.101.14’
[error_logger:info,2018-06-28T14:44:15.791+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,net_sup}
started: [{pid,<0.148.0>},
{name,erl_epmd},
{mfargs,{erl_epmd,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

[error_logger:info,2018-06-28T14:44:15.792+05:30,nonode@nohost:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,net_sup}
started: [{pid,<0.149.0>},
{name,auth},
{mfargs,{auth,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

[ns_server:debug,2018-06-28T14:44:15.796+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:configure_net_kernel:258]Set net_kernel vebosity to 10 -> 0
[error_logger:info,2018-06-28T14:44:15.796+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,net_sup}
started: [{pid,<0.150.0>},
{name,net_kernel},
{mfargs,
{net_kernel,start_link,
[[‘ns_1@192.168.101.14’,longnames]]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

[error_logger:info,2018-06-28T14:44:15.796+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,kernel_sup}
started: [{pid,<0.147.0>},
{name,net_sup_dynamic},
{mfargs,
{erl_distribution,start_link,
[[‘ns_1@192.168.101.14’,longnames]]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,supervisor}]

[ns_server:info,2018-06-28T14:44:15.830+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:save_node:147]saving node to “C:/Program Files/Couchbase/Server/var/lib/couchbase/couchbase-server.node”
[ns_server:debug,2018-06-28T14:44:15.901+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:bringup:228]Attempted to save node name to disk: ok
[ns_server:debug,2018-06-28T14:44:15.901+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:wait_for_node:235]Waiting for connection to node ‘babysitter_of_ns_1@127.0.0.1’ to be established
[error_logger:info,2018-06-28T14:44:15.901+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================INFO REPORT=========================
{net_kernel,{connect,normal,‘babysitter_of_ns_1@127.0.0.1’}}
[ns_server:debug,2018-06-28T14:44:15.915+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:wait_for_node:247]Observed node ‘babysitter_of_ns_1@127.0.0.1’ to come up
[ns_server:debug,2018-06-28T14:44:15.953+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:init:187]Found rename marker. Old Node = "ns_1@127.0.0.1"
[ns_server:debug,2018-06-28T14:44:15.990+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:notify_couchdb_node:328]Couchdb node is not started. Don’t need to notify
[ns_server:debug,2018-06-28T14:44:15.990+05:30,ns_1@192.168.101.14:dist_manager<0.144.0>:dist_manager:complete_rename:338]Renaming node from ‘ns_1@127.0.0.1’ to ‘ns_1@192.168.101.14’.
[error_logger:error,2018-06-28T14:44:16.032+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: dist_manager:init/1
pid: <0.144.0>
registered_name: []
exception exit: {noproc,
{gen_server,call,
[ns_config,
{update_with_changes,
#Fun<ns_config.5.50957994>}]}}
in function gen_server:init_it/6 (c:/cygwin64/home/vagrant/OTP_SR~1/lib/stdlib/src/gen_server.erl, line 328)
ancestors: [ns_server_cluster_sup,<0.87.0>]
messages: []
links: [<0.141.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 4185
stack_size: 27
reductions: 13922
neighbours:

[error_logger:error,2018-06-28T14:44:16.032+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_server_cluster_sup}
Context: start_error
Reason: {noproc,
{gen_server,call,
[ns_config,
{update_with_changes,#Fun<ns_config.5.50957994>}]}}
Offender: [{pid,undefined},
{name,dist_manager},
{mfargs,{dist_manager,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[error_logger:error,2018-06-28T14:44:16.033+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: application_master:init/4
pid: <0.86.0>
registered_name: []
exception exit: {{shutdown,
{failed_to_start_child,dist_manager,
{noproc,
{gen_server,call,
[ns_config,
{update_with_changes,
#Fun<ns_config.5.50957994>}]}}}},
{ns_server,start,[normal,[]]}}
in function application_master:init/4 (c:/cygwin64/home/vagrant/OTP_SR~1/lib/kernel/src/application_master.erl, line 133)
ancestors: [<0.85.0>]
messages: [{‘EXIT’,<0.87.0>,normal}]
links: [<0.85.0>,<0.7.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1598
stack_size: 27
reductions: 194
neighbours:

[error_logger:info,2018-06-28T14:44:16.033+05:30,ns_1@192.168.101.14:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================INFO REPORT=========================
application: ns_server
exited: {{shutdown,
{failed_to_start_child,dist_manager,
{noproc,
{gen_server,call,
[ns_config,
{update_with_changes,
#Fun<ns_config.5.50957994>}]}}}},
{ns_server,start,[normal,[]]}}
type: permanent