Crash on couchbase-server 2.2 start

I have installed couchbase-server-2.2 on rhel 6.4 system and I get the following error while starting couchbase-server as a deamon:

daemon “/bin/su -s /bin/sh -c "/opt/couchbase/bin/couchbase-server – -noinput -detached" couchbase”

[error_logger:error,2014-01-28T13:42:30.803,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_bucket_sup}
Context: shutdown_error
Reason: normal
Offender: [{pid,<0.387.0>},
{name,buckets_observing_subscription},
{mfargs,{ns_bucket_sup,subscribe_on_config_events,}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[error_logger:error,2014-01-28T13:42:30.907,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_server_cluster_sup}
Context: shutdown_error
Reason: killed
Offender: [{pid,<0.291.0>},
{name,vbucket_filter_changes_registry},
{mfargs,
{ns_process_registry,start_link,
[vbucket_filter_changes_registry]}},
{restart_type,permanent},
{shutdown,100},
{child_type,worker}]

[error_logger:error,2014-01-28T13:42:30.910,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.220.0> terminating
** Last message in was {‘EXIT’,<0.210.0>,killed}
** When Server state == {db,<0.220.0>,<0.221.0>,nil,<<“1390916546558891”>>,
<0.217.0>,<0.222.0>,
{db_header,11,1,
<<0,0,0,0,13,103,0,0,0,0,0,51,0,0,0,0,1,0,0,0,
0,0,0,0,0,0,13,69>>,
<<0,0,0,0,13,154,0,0,0,0,0,49,0,0,0,0,1>>,
nil,0,nil,nil},
1,
{btree,<0.217.0>,
{3431,
<<0,0,0,0,1,0,0,0,0,0,0,0,0,0,13,69>>,
51},
#Fun<couch_db_updater.7.108595492>,
#Fun<couch_db_updater.8.83286466>,
#Fun<couch_btree.1.72034400>,
#Fun<couch_db_updater.9.132659170>,1279,2558,
true},
{btree,<0.217.0>,
{3482,<<0,0,0,0,1>>,49},
#Fun<couch_db_updater.10.23517937>,
#Fun<couch_db_updater.11.59258139>,
#Fun<couch_db_updater.6.7170572>,
#Fun<couch_db_updater.12.109813745>,1279,2558,
true},
{btree,<0.217.0>,nil,identity,identity,
#Fun<couch_btree.1.72034400>,nil,1279,2558,
true},
1,<<“_users”>>,
“/opt/couchbase/var/lib/couchbase/data/_users.couch.1”,
,nil,
{user_ctx,null,,undefined},
nil,
[before_header,after_header,on_file_open],
}
** Reason for termination ==
** killed

The above error was logged in /opt/couchbase/var/lib/couchbase/logs/error.1
If you need more info or other logs, please let me know.
Thanks in advance!

I see you are starting it with this command “opt/couchbase/bin/couchbase-server – -noinput -detached” are you installing it an non-root?

I have installed it as root, but I start it as couchbase user.

did you try starting it as a service EX. # /etc/init.d/couchbase-server start

When I start couchbase as a service “/etc/init.d/couchbase-server start”, I get the following error. It seems that coucbase server starts for some seconds and then craches.

[error_logger:error,2014-01-30T9:28:01.325,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_bucket_sup}
Context: shutdown_error
Reason: normal
Offender: [{pid,<0.390.0>},
{name,buckets_observing_subscription},
{mfargs,{ns_bucket_sup,subscribe_on_config_events,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[error_logger:error,2014-01-30T9:28:01.428,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_server_cluster_sup}
Context: shutdown_error
Reason: killed
Offender: [{pid,<0.294.0>},
{name,vbucket_filter_changes_registry},
{mfargs,
{ns_process_registry,start_link,
[vbucket_filter_changes_registry]}},
{restart_type,permanent},
{shutdown,100},
{child_type,worker}]

[error_logger:error,2014-01-30T9:28:01.436,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: couch_file:spawn_writer/2
pid: <0.216.0>
registered_name: []
exception exit: {noproc,
{gen_server,call,
[couch_file_write_guard,
{remove,<0.216.0>},
infinity]}}
in function gen_server:call/3
in call from couch_file:writer_loop/4
ancestors: [<0.214.0>,couch_server,couch_primary_services,
couch_server_sup,cb_couch_sup,ns_server_cluster_sup,
<0.59.0>]
messages: []
links: []
dictionary: []
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 651
neighbours:

[error_logger:error,2014-01-30T9:28:01.436,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.223.0> terminating
** Last message in was {‘EXIT’,<0.213.0>,killed}
** When Server state == {db,<0.223.0>,<0.224.0>,nil,<<“1391074080161336”>>,
<0.220.0>,<0.225.0>,
{db_header,11,1,
<<0,0,0,0,13,103,0,0,0,0,0,51,0,0,0,0,1,0,0,0,
0,0,0,0,0,0,13,69>>,
<<0,0,0,0,13,154,0,0,0,0,0,49,0,0,0,0,1>>,
nil,0,nil,nil},
1,
{btree,<0.220.0>,
{3431,
<<0,0,0,0,1,0,0,0,0,0,0,0,0,0,13,69>>,
51},
#Fun<couch_db_updater.7.108595492>,
#Fun<couch_db_updater.8.83286466>,
#Fun<couch_btree.1.72034400>,
#Fun<couch_db_updater.9.132659170>,1279,2558,
true},
{btree,<0.220.0>,
{3482,<<0,0,0,0,1>>,49},
#Fun<couch_db_updater.10.23517937>,
#Fun<couch_db_updater.11.59258139>,
#Fun<couch_db_updater.6.7170572>,
#Fun<couch_db_updater.12.109813745>,1279,2558,
true},
{btree,<0.220.0>,nil,identity,identity,
#Fun<couch_btree.1.72034400>,nil,1279,2558,
true},
1,<<"_users">>,
"/opt/couchbase/var/lib/couchbase/data/_users.couch.1",
[],nil,
{user_ctx,null,[],undefined},
nil,
[before_header,after_header,on_file_open],
[]}
** Reason for termination ==
** killed

[error_logger:error,2014-01-30T9:28:01.436,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: couch_db:init/1
pid: <0.223.0>
registered_name: []
exception exit: killed
in function gen_server:terminate/6
ancestors: [couch_server,couch_primary_services,couch_server_sup,
cb_couch_sup,ns_server_cluster_sup,<0.59.0>]
messages: []
links: []
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 24
reductions: 214
neighbours:

Am also facing same issue. Version - 2.2.0 Community (837)
Can some one pls tell me … why this CB is Crashing again and again (Please see below logs)

erl_crash_dump:0.1
Wed Jul 9 14:30:57 2014
Slogan: eheap_alloc: Cannot allocate 2280657000 bytes of memory (of type “heap”).
System version: Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:16] [kernel-poll:true]
Compiled: Wed Aug 21 19:17:37 2013
Taints: couch_ejson_compare,couch_view_parser,ejson,mapreduce,snappy,crypto,mlockall
Atoms: 19408
=memory
total: 14599947408
processes: 14311880520
processes_used: 14298201928
system: 288066888
atom: 1370969
atom_used: 1344288
binary: 98316416
code: 14036470
ets: 162139496
=hash_table:atom_tab
size: 12853
used: 10052
objs: 19408
depth: 8
=index_table:atom_tab
size: 19456
limit: 1048576
entries: 19408
=hash_table:module_code
size: 397
used: 303
objs: 41

[mapreduce_errors:error,2014-07-09T0:35:52.746,ns_1@pute-1.amazonaws.:<0.20442.19>:couch_set_view_updater:load_doc:483]Bucket ap_uid_attr, main group _design/dev_error_data, skipping document with ID Ve8c8f130276c72a81b1865838ee3273938ebc987: too large body (1214405 bytes)

[ns_server:error,2014-07-09T10:10:34.328,ns_1@ute-1.amazonaws.814.0>:ns_memcached:verify_report_long_call:294]call {stats,<<“timings”>>} took too long: 627594 us
[ns_server:error,2014-07-09T10:12:33.157,ns_1@ec2-23-23-221-58.compute-1.amazonaws.com:ns_memcached-default<0.12736.0>:ns_memcached:handle_info:671]handle_info(ensure_bucket,…) took too long: 1260503 usns_server:error,2014-07-09T10:12:34.467,67,ns_1@ute-1.amazonaws.<0.12817.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 1015471 us

xdcr.2:[xdcr:debug,2014-07-10T14:52:39.586,ns_1@ompute-1.amazonaws.com:<0.10962.3>:xdc_vbucket_rep_xmem_worker:handle_call:139][xmem_worker 1 for vb 954]: cmd select_bucket no longer supported (bucket: “ap_uid_attr”, username: “ap_uid_attr”) at node “us-west-mazonaws.com”, error msg: <<“Not supported”>>

vbucket_rep_ckpt:update_checkpoint_status_to_parent:302]replicator (vb: 977, source: <<“ap_uid_attr”>>, dest: <<"/remoteClusters/4df1d158502da910246d6c710c5bd476/buckets/ap_uid_attr">>) reports checkpoint status: {succ: true, msg: “no error”} to parent: <0.9725.0>
xdcr.2: 977,true,“no error”}) to ckpt history of replication (src: <<“ap_uid_attr”>>, target: <<"/remoteClusters/4df1d158502da910246d6c710c5bd476/buckets/ap_uid_attr">>)

info.1:[ns_server:error,2014-07-10T14:08:58.119,ns_1@compute-1.amazonaws.com:<0.14850.0>:ns_memcached:verify_report_long_call:294]call {stats,<<>>} took too long: 501054 us

babysitter.1:[ns_server:info,2014-07-10T13:50:50.986,babysitter_of_ns_1@127.0.0.1:ns_crash_log<0.69.0>:ns_crash_log:init:43]crash_log.bin path: /opt/couchbase/var/lib/couchbase/logs/crash_log.bin
babysitter.1:[ns_server:warn,2014-07-10T13:50:50.986,babysitter_of_ns_1@127.0.0.1:ns_crash_log<0.69.0>:ns_crash_log:read_crash_log:135]Couldn’t load crash_log from /opt/couchbase/var/lib/couchbase/logs/crash_log.bin (perhaps it’s first startup): {error,
babysitter.1: {name,ns_crash_log},
babysitter.1: {mfargs,{ns_crash_log,start_link,[]}},
debug.1: {name,ns_crash_log_consumer},
debug.1: {mfa,{ns_log,start_link_crash_consumer,[]}},
debug.1:[error_logger:error,2014-07-10T14:11:17.844,ns_1@ec2-54-225-238-44.compute-1.amazonaws.com:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** gen_event handler {ns_pubsub,#Ref<0.0.0.24721>} crashed.
debug.1:=========================CRASH REPORT=========================
debug.1: crasher:
debug.1: exception exit: {handler_crashed,master_activity_events_ingress,
debug.1: Reason: {handler_crashed,master_activity_events_ingress,
debug.1:=========================CRASH REPORT=========================
debug.1: crasher:
debug.1:=========================CRASH REPORT=========================
debug.1: crasher:
error.1:[error_logger:error,2014-07-10T14:11:17.844,ns_1@ec2-54-225-238-44.compute-1.amazonaws.com:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** gen_event handler {ns_pubsub,#Ref<0.0.0.24721>} crashed.
error.1:=========================CRASH REPORT=========================
error.1: crasher:
error.1: exception exit: {handler_crashed,master_activity_events_ingress,
error.1: Reason: {handler_crashed,master_activity_events_ingress,
error.1:=========================CRASH REPORT=========================
error.1: crasher:
info.1: {name,ns_crash_log_consumer},
info.1: {mfa,{ns_log,start_link_crash_consumer,[]}},
info.1:[error_logger:error,2014-07-10T14:11:17.844,ns_1@ec2-54-225-238-44.compute-1.amazonaws.com:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** gen_event handler {ns_pubsub,#Ref<0.0.0.24721>} crashed.
info.1:=========================CRASH REPORT=========================
info.1: crasher:
info.1: exception exit: {handler_crashed,master_activity_events_ingress,
info.1: Reason: {handler_crashed,master_activity_events_ingress,
info.1:=========================CRASH REPORT=========================
info.1: crasher:
info.1:=========================CRASH REPORT=========================
info.1: crasher:
[root@ip-10-231-38-148 logs]#

error.1: <<“0x00007fcf59ae08f8 Return addr 0x0000000000872878 ()”>>]},
error.1: in function gen_server:terminate/6
error.1: Context: child_terminated
error.1: <<“0x00007fceef7c7f10 Return addr 0x0000000000872878 ()”>>]},
info.1: Context: child_terminated
info.1: in function gen_server:terminate/6
info.1: Context: child_terminated
info.1: Context: child_terminated

=========================CRASH REPORT=========================
crasher:
initial call: xdc_vbucket_rep:init/1
pid: <0.8937.5>
registered_name: []
exception exit: {badmatch,{error,all_nodes_failed,
<<“Failed to grab remote bucket ap_uid_attrfrom any of known nodes”>>}}
in function gen_server:terminate/6
ancestors: [<0.10271.0>,<0.10266.0>,xdc_replication_sup,ns_server_sup,
ns_server_cluster_sup,<0.59.0>]
messages: []
links: [<0.10271.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 6765
stack_size: 24
reductions: 13874
neighbours:

0.21013.5>:ns_orchestrator:rebalance_progress:183]Couldn’t talk to orchestrator: {exit,
{timeout,
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
rebalance_progress,2000]}}}

11097.0>:xdc_vbucket_rep:start_replication:781]Replication <<"98a2998ed085c8180fe9d636fc941b5f/ap_userkey_uid/ap_userkey_uid">> is using:
4 worker processes
a worker batch size of 5000
a worker batch size (KiB) 3048
20 HTTP connections
a connection timeout of 180 seconds
2 retries per request
socket options are: [{keepalive,true},{nodelay,false}]
source start sequence 217985

[xdcr:info,2014-07-10T9:37:52.579,2.compute-1.amazonaws.com:<0.11097.0>:xdc_vbucket_rep:start_replication:865]replicator of vb 722 for replication from src <<“ap_userkey_uid/722”>> to target <<1.a2/ap_userkey_uid%2f722%3bcc79d210a9e844a3082f7257e614cea5">> has been started (xmem remote (.com", port: 11210, bucket: “ap_userkey_uid”), xmem srv: <0.12408.0>).
[xdcr:debug,2014-07-10T9:37:52.587,azonaws.com:<0.10530.0>:xdc_replication:handle_info:278]get src_db_udpated from vb 527, but the vb replicator has not been initialized yet or has been deleted.
[xdcr:debug,2014-07-10T9:37:52.587,ns_1@ec:<0.17078.0>:xdc_replication:handle_info:278]get src_db_udpated from vb 527, but the vb replicator has not been initialized yet or has been deleted.

master<0.9592.0>:mb_master:master:412]Got master heartbeat from ‘ns_1@ec2-54-221-255-232.compute-1.amazonaws.com’ when I’m master
[error_logger:error,2014-07-10T15:52:50.643,ns_1@ec2-23-23-221-58.compute-1.amazonaws.com:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]Detected time forward jump (or too large erlang scheduling latency). Skipping 20 samples (or 16000 milliseconds) ({{1404987680477,
#Ref<0.0.0.24975>},
{repeat,
800,
<0.9655.0>},
{timer2,
send,
[<0.9655.0>,
{cascade,
minute,
hour,
4}]}})
[ns_server:error,2014-07-10T15:52:51.046,ns_1@eompute-1.amazonaws.com:ns_doctor<0.9582.0>:ns_doctor:update_status:235]The following buckets became not ready on node

[ns_server:error,2014-07-10T15:52:50.866,ns_1@ec2-23-23-221-58.compute-1.amazonaws.com:timeout_diag_logger<0.9501.0>:timeout_diag_logger:do_diag:107]
{<0.3702.2>,
[{registered_name,[]},
{status,waiting},
{initial_call,{proc_lib,init_p,5}},
{backtrace,[<<“Program counter: 0x00007f28240142c0 (gen_server:loop/6 + 256)”>>,
<<“0x00007f2687a85a08 Return addr 0x00007f282faef5e0 (proc_lib:init_p_do_apply/3 + 56)”>>,
<<“0x00007f2687a85a40 Return addr 0x0000000000872878 ()”>>]},
{error_handler,error_handler},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,512},
{minor_gcs,40}]},
{heap_size,196418},
{total_heap_size,710647},
{links,[<0.10524.0>]},
{memory,5686016},
{message_queue_len,0},
{reductions,652329},
{trap_exit,false}]}
[ns_server:info,2014-07-10T15:52:51.313,ns_1@ec2-23-23-221-58.compute-1.amazonaws.com:mb_master<0.9592.0>:mb_master:master:412]Got master heartbeat from

memcached<0.454.0>: Sat Jul 5 03:13:19.196473 IST 3: (ap_uid_attr) Failed to connect to: "127.0.0.1:11213"
memcached<0.454.0>: Sat Jul 5 03:13:19.196941 IST 3: (ap_userkey_uid) Failed to connect to: "127.0.0.1:11213"
memcached<0.454.0>: Sat Jul 5 03:13:19.196979 IST 3: (ap_userkey_uid) Failed to connect to: "127.0.0.1:11213"
memcached<0.454.0>: Sat Jul 5 03:13:19.201577 IST 3: (ap_uid_attr) Failed to connect to: "127.0.0.1:11213"
memcached<0.454.0>: Sat Jul 5 03:13:19.201593 IST 3: (ap_uid_attr) Failed to connect to: “127.0.0.1:11213”

ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:l
og_msg:76]** Generic server <0.466.0> terminating
** Last message in was {#Port<0.3320>,{exit_status,134}}
** When Server state == {state,#Port<0.3320>,ns_server,
{[“eheap_alloc: Cannot allocate 13593774640 bytes of memory (of type “heap”).”,
“Crash dump was written to: erl_crash.dump.1403604346.1257”,
[],“Erlang has closed”,
"/opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup: Erlang has closed. ",
“working as port”,
“Apache CouchDB has started. Time to relax.”,
“Apache CouchDB 1.2.0a-01dda76-git (LogLevel=info) is starting.”,
empty],
[empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty]},
undefined,[],0,true}

                           undefined,[],0,true}

** Reason for termination ==
** {abnormal,134}

[error_logger:error,2014-07-04T22:40:51.133,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.466.0>
registered_name: []
exception exit: {abnormal,134}
in function gen_server:terminate/6
ancestors: [<0.465.0>,child_ns_server_sup,ns_babysitter_sup,<0.55.0>]
messages: [{#Port<0.3320>,eof}]
links: [<0.465.0>,#Port<0.3320>]
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 3633
neighbours:

Some more logs
error_logger:error,2014-07-
04T21:56:51.717,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_child_ports_sup}
Context: child_terminated
Reason: {abnormal,71}
Offender: [{pid,<0.1004.0>},
{name,
{memcached,"/opt/couchbase/bin/memcached",
["-X",
"/opt/couchbase/lib/memcached/stdin_term_handler.so",
"-X",
"/opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/opt/couchbase/var/lib/couchbase/logs/memcached.log",
[error_logger:error,2014-07-04T21:57:38.237,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.1007.0> terminating
** Last message in was {#Port<0.3586>,{exit_status,71}}
** When Server state == {state,#Port<0.3586>,memcached,
{[“Sat Jul 5 03:26:51.750151 IST 3: failed to listen on TCP port 11210: Address already in use”,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,
empty,empty,empty,empty,empty,empty,empty,

[couchdb:info,2014-07-05T3:09:54.878,ns_1@mazonaws.com:couch_server<0.212.0>:couch_log:info:39]Deleting file /data/couchbase/ap_userkey_uid/510.couch.424
[couchdb:info,2014-07-05T3:09:54.924,ns_1@zonaws.com:couch_server<0.212.0>:couch_log:info:39]Deleting database ap_userkey_uid/511
[couchdb:info,2014-07-05T3:09:54.927,ns_1@naws.com:couch_server<0.212.0>:couch_log:info:39]Deleting file /data/couchbase/ap_userkey_uid/511.couch.426
[couchdb:info,2014-07-05T3:09:54.974,ns_1@.com:couch_server<0.212.0>:couch_log:info:39]Deleting database ap_userkey_uid/512

SYS LOGS on one node :

/var/log/messages:Jul 8 08:16:22 ip-10-231-46-163 epmd: epmd: got KILL_REQ - terminates normal
/var/log/messages:Jul 8 08:53:10 ip-10-231-46-163 epmd: epmd: got KILL_REQ - terminates normal
/var/log/messages:Jul 9 10:31:51 ip-10-231-46-163 epmd: epmd: got KILL_REQ - terminates normal
/var/log/messages:Jul 10 08:21:43 ip-10-231-46-163 epmd: epmd: got KILL_REQ - terminates normal
/var/log/messages-20140623:Jun 23 14:43:25 ip-10-231-46-163 init: serial (ttyS0) main process (1669) killed by TERM signal
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: [] ? oom_kill_process+0x82/0x2a0
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: Out of memory: Kill process 1888 (memcached) score 635 or sacrifice child
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: Killed process 1888, UID 497, (memcached) total-vm:40441420kB, anon-rss:40006768kB, file-rss:2068kB
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
/var/log/messages-20140706:Jul 4 20:32:46 ip-10-231-46-163 kernel: [] ?