Erlang server crashed when running couchbase community server edition on centos 6.4 64 bit Virtual Machine


#1

installed: 2.2.0-837-rel couchbase server
[product]
title = Couchbase Server
version = 2.2.0-837-rel
license = community

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg1-lv_root
5.0G 891M 3.9G 19% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 1008M 110M 848M 12% /boot
/dev/mapper/vg1-lv_home
5.0G 4.0G 786M 84% /home
/dev/mapper/vg1-lv_opt
1008M 680M 278M 72% /opt
/dev/mapper/vg1-lv_tmp
2.0G 68M 1.9G 4% /tmp
/dev/mapper/vg1-lv_usr
9.9G 4.0G 5.4G 43% /usr
/dev/mapper/vg1-lv_var
5.0G 436M 4.3G 10% /var
/dev/mapper/vg1-lv_var_lib_mysql
20G 2.8G 16G 15% /var/lib/mysql
/dev/mapper/vg1-lv_var_log
5.0G 222M 4.5G 5% /var/log
/dev/mapper/vg1-lv_u01
60G 11G 46G 19% /u01

/opt/couchbase/bin $ ./couchdb
Apache CouchDB 1.2.0a-01dda76-git (LogLevel=info) is starting.

=CRASH REPORT==== 6-May-2014::11:59:29 ===
crasher:
initial call: couch_event_sup:init/1
pid: <0.99.0>
registered_name: []
exception exit: {error,
{file_error,
"/opt/couchbase/var/log/couchdb/couch.log",
eacces}}
in function gen_server:init_it/6
ancestors: [couch_primary_services,couch_server_sup,<0.31.0>]
messages: []
links: [<0.90.0>,<0.6.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 233
stack_size: 24
reductions: 122
neighbours:

=SUPERVISOR REPORT==== 6-May-2014::11:59:29 ===
Supervisor: {local,couch_primary_services}
Context: start_error
Reason: {error,{file_error,"/opt/couchbase/var/log/couchdb/couch.log",
eacces}}
Offender: [{pid,undefined},
{name,couch_log},
{mfargs,{couch_log,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=SUPERVISOR REPORT==== 6-May-2014::11:59:29 ===
Supervisor: {local,couch_server_sup}
Context: start_error
Reason: shutdown
Offender: [{pid,undefined},
{name,couch_primary_services},
{mfargs,{couch_primary_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]

=CRASH REPORT==== 6-May-2014::11:59:29 ===
crasher:
initial call: application_master:init/4
pid: <0.30.0>
registered_name: []
exception exit: {bad_return,
{{couch_app,start,
[normal,
["/opt/couchbase/etc/couchdb/default.ini",
"/opt/couchbase/etc/couchdb/local.ini"]]},
{‘EXIT’,
{{badmatch,{error,shutdown}},
[{couch_server_sup,start_server,1},
{application_master,start_it_old,4}]}}}}
in function application_master:init/4
ancestors: [<0.29.0>]
messages: [{‘EXIT’,<0.31.0>,normal}]
links: [<0.29.0>,<0.7.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 24
reductions: 154
neighbours:

=INFO REPORT==== 6-May-2014::11:59:29 ===
application: couch
exited: {bad_return,{{couch_app,start,
[normal,
["/opt/couchbase/etc/couchdb/default.ini",
"/opt/couchbase/etc/couchdb/local.ini"]]},
{‘EXIT’,{{badmatch,{error,shutdown}},
[{couch_server_sup,start_server,1},
{application_master,start_it_old,4}]}}}}
type: temporary

also some log snippets:

ls: cannot access erl_crash.dump.*: No such file or directory

[ns_server:info,2014-05-06T14:09:16.267,babysitter_of_ns_1@127.0.0.1:<0.22631.0>:ns_port_server:log:168]ns_server<0.22631.0>: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

[ns_server:info,2014-05-06T14:09:17.709,babysitter_of_ns_1@127.0.0.1:<0.22631.0>:ns_port_server:log:168]ns_server<0.22631.0>: Crap error:{badmatch,{error,{shutdown,{ale_app,start,[normal,[]]}}}}
ns_server<0.22631.0>: [{child_erlang,do_child_start,1},
ns_server<0.22631.0>: {child_erlang,child_start,1},
ns_server<0.22631.0>: {init,start_it,1},
ns_server<0.22631.0>: {init,start_em,1}]

[ns_server:info,2014-05-06T14:09:18.521,babysitter_of_ns_1@127.0.0.1:<0.22630.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for ns_server failed: {abnormal,3}

the crash is in a file: erl_crash.dump.1399320803.9379 ( too big to paste here )

=erl_crash_dump:0.1
Mon May 5 16:13:25 2014
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})
System version: Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:16] [kernel-poll:false]
Compiled: Wed Aug 21 19:17:37 2013

here is additional information:

./cbcollect_info info.zip
uname (uname -a) - OK
Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 2
Process list snapshot (export TERM=’’; top -Hb -n1 || top -H n1) - OK
Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK
Raw /proc/vmstat (cat /proc/vmstat) - OK
Raw /proc/mounts (cat /proc/mounts) - OK
Raw /proc/partitions (cat /proc/partitions) - OK
Raw /proc/diskstats (cat /proc/diskstats) - OK
Raw /proc/interrupts (cat /proc/interrupts) - OK
Swap configuration (free -t) - OK
Swap configuration (swapon -s) - OK
Kernel modules (lsmod) - OK
Distro version (cat /etc/redhat-release) - OK
Distro version (lsb_release -a) - OK
Installed software (rpm -qa) - OK
Installed software (COLUMNS=300 dpkg -l) - Exit code 127
Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - OK
Core dump settings (find /proc/sys/kernel -type f -name ‘core’ -print -exec cat ‘{}’ ‘;’) - OK
sysctl settings (sysctl -a) - OK
netstat -nap (netstat -nap) - OK
relevant lsof output (lsof -n | grep ‘moxi|memcached|vbucketmigrator|beam|couch_compact|godu|portsigar’) - Exit code 1
LVM info (lvdisplay) - OK
LVM info (vgdisplay) - OK
LVM info (pvdisplay) - OK
Network configuration (ifconfig -a) - OK
Taking sample 2 after 10.000000 seconds -
OK
Network configuration (echo link addr neigh rule route netns | xargs -n1 – sh -x -c ‘ip $1 list’ --) - Exit code 124
Network status (netstat -an) - OK
Network routing table (netstat -rn) - OK
Arp cache (arp -na) - OK
Filesystem (df -ha) - OK
System activity reporter (sar 1 10) - OK
System paging activity (vmstat 1 10) - OK
System uptime (uptime) - OK
couchbase user definition (getent passwd couchbase) - OK
couchbase user limits (su couchbase -c “ulimit -a”) - OK
membase user definition (getent passwd membase) - Exit code 2
couchbase user limits (su couchbase -c “ulimit -a”) - OK
membase user limits (su membase -c “ulimit -a”) - Exit code 125
Interrupt status (intrstat 1 10) - Exit code 127
Processor status (mpstat 1 10) - OK
System log (cat /var/adm/messages) - Exit code 1
System log (cat /var/log/syslog) - Exit code 1
System log (cat /var/log/messages) - OK
All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2
Relevant proc data ((pgrep moxi; pgrep beam.smp; pgrep memcached; pgrep couch_compact; pgrep portsigar ; pgrep godu) | xargs -n1 – sh -c ‘echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/sched; echo’ --) - OK
NUMA data (numactl --hardware) - OK
NUMA data (numactl --show) - OK
NUMA data (cat /sys/devices/system/node/node*/numastat) - OK
Kernel log buffer (dmesg) - OK
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - OK
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - OK
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs…
Failed to read initargs:
{error,enoent}
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs…
Failed to read initargs:
{error,enoent}
Checking for server guts in /root/Library/Application Support/Couchbase/var/lib/couchbase/initargs…
Failed to read initargs:
{error,enoent}