I think the problem was that the the nodes were too tight in memory, with the RAM quota per node too high compared with the total amount of memory. I have added a third node to the cluster, giving more (total) memory to the buckets but reducing the RAM quota per node so there is more memory available to the OS.
Anyway, here you have a portion of the babysitter log:
[ns_server:info,2015-11-30T19:41:00.454Z,babysitter_of_ns_1@127.0.0.1:<0.391.0>:ns_port_server:log:210]memcached<0.391.0>: 2015-11-30T19:41:00.253097Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.267755Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.281142Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.293183Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.301124Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.309098Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
memcached<0.391.0>: 2015-11-30T19:41:00.318449Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database
[ns_server:info,2015-11-30T19:41:15.655Z,babysitter_of_ns_1@127.0.0.1:<0.390.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for memcached failed: {abnormal,137}
[ns_server:debug,2015-11-30T19:41:15.655Z,babysitter_of_ns_1@127.0.0.1:<0.393.0>:supervisor_cushion:init:39]starting ns_port_server with delay of 5000
[error_logger:error,2015-11-30T19:41:15.655Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.391.0> terminating
** Last message in was {#Port<0.3883>,{exit_status,137}}
** When Server state == {state,#Port<0.3883>,memcached,
{["2015-11-30T19:41:00.318449Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database",
"2015-11-30T19:41:00.309098Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database",
"2015-11-30T19:41:00.301124Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database",
"2015-11-30T19:41:00.293183Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database"],
["2015-11-30T19:41:00.281142Z WARNING (Bingo_Card) Engine warmup is complete, request to stop loading remaining database"]},
undefined,undefined,[],0}
** Reason for termination ==
** {abnormal,137}
[error_logger:error,2015-11-30T19:41:15.656Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.391.0>
registered_name: []
exception exit: {abnormal,137}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [<0.390.0>,<0.389.0>,ns_child_ports_sup,ns_babysitter_sup,
<0.56.0>]
messages: [{‘EXIT’,#Port<0.3883>,normal}]
links: [<0.390.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 17731
stack_size: 27
reductions: 3270550
neighbours:
[error_logger:error,2015-11-30T19:41:15.656Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.390.0> terminating
** Last message in was {die,{abnormal,137}}
** When Server state == {state,memcached,5000,
{1448,911394,878319},
undefined,infinity}
** Reason for termination ==
** {abnormal,137}
[error_logger:error,2015-11-30T19:41:15.656Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: supervisor_cushion:init/1
pid: <0.390.0>
registered_name: []
exception exit: {abnormal,137}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [<0.389.0>,ns_child_ports_sup,ns_babysitter_sup,<0.56.0>]
messages: []
links: [<0.389.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 1811
neighbours:
[error_logger:error,2015-11-30T19:41:15.656Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.389.0>
registered_name: []
exception exit: {abnormal,137}
in function restartable:loop/4 (src/restartable.erl, line 69)
ancestors: [ns_child_ports_sup,ns_babysitter_sup,<0.56.0>]
messages: []
links: [<0.69.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 376
stack_size: 27
reductions: 133
neighbours:
[error_logger:error,2015-11-30T19:41:15.656Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_child_ports_sup}
Context: child_terminated
Reason: {abnormal,137}
Offender: [{pid,<0.389.0>},
{name,
{memcached,"/opt/couchbase/bin/memcached",
["-C",
"/opt/couchbase/var/lib/couchbase/config/memcached.json"],
[{env,
[{“EVENT_NOSELECT”,“1”},
{“MEMCACHED_TOP_KEYS”,“5”},
{“ISASL_PWFILE”,
"/opt/couchbase/var/lib/couchbase/isasl.pw"}]},
use_stdio,stderr_to_stdout,exit_status,
port_server_dont_start,stream]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[memcached,5000,infinity,ns_port_server,
start_link,
[#Fun<ns_child_ports_sup.2.49698737>]]},
86400000]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,worker}]
[error_logger:info,2015-11-30T19:41:15.657Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_child_ports_sup}
started: [{pid,<0.392.0>},
{name,
{memcached,"/opt/couchbase/bin/memcached",
["-C",
"/opt/couchbase/var/lib/couchbase/config/memcached.json"],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","5"},
{"ISASL_PWFILE",
"/opt/couchbase/var/lib/couchbase/isasl.pw"}]},
use_stdio,stderr_to_stdout,exit_status,
port_server_dont_start,stream]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[memcached,5000,infinity,ns_port_server,
start_link,
[#Fun<ns_child_ports_sup.2.49698737>]]},
86400000]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,worker}]
[ns_server:info,2015-11-30T19:41:16.939Z,babysitter_of_ns_1@127.0.0.1:<0.394.0>:ns_port_server:log:210]memcached<0.394.0>: 2015-11-30T19:41:16.735024Z WARNING NUMA: Set memory allocation policy to 'interleave'.
[ns_server:info,2015-11-30T19:41:18.058Z,babysitter_of_ns_1@127.0.0.1:<0.130.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for indexer failed: {abnormal,1}
[error_logger:error,2015-11-30T19:41:18.058Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.131.0> terminating
** Last message in was {#Port<0.3586>,{exit_status,1}}
** When Server state == {state,#Port<0.3586>,indexer,
{["[goport] 2015/11/30 19:41:18 /opt/couchbase/bin/indexer terminated: signal: killed",
"2015-11-30T19:41:10.707+00:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM Bingo_Card SnapType NO_SNAP",
"2015-11-30T19:41:10.246+00:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM Bingo_Card SnapType NO_SNAP",
"2015-11-30T19:41:09.362+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 247060000"],
["2015-11-30T19:41:08.382+00:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6090 Milliseconds Vbucket 1022"]},
indexer,undefined,[],0}
** Reason for termination ==
** {abnormal,1}
[error_logger:error,2015-11-30T19:41:18.059Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.131.0>
registered_name: []
exception exit: {abnormal,1}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [<0.130.0>,<0.87.0>,ns_child_ports_sup,ns_babysitter_sup,
<0.56.0>]
messages: [{‘EXIT’,#Port<0.3586>,normal}]
links: [<0.130.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 196650
stack_size: 27
reductions: 160071734
neighbours:
[error_logger:error,2015-11-30T19:41:18.059Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.130.0> terminating
** Last message in was {die,{abnormal,1}}
** When Server state == {state,indexer,5000,
{1448,548818,1509},
undefined,infinity}
** Reason for termination ==
** {abnormal,1}
[error_logger:error,2015-11-30T19:41:18.059Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: supervisor_cushion:init/1
pid: <0.130.0>
registered_name: []
exception exit: {abnormal,1}
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [<0.87.0>,ns_child_ports_sup,ns_babysitter_sup,<0.56.0>]
messages: []
links: [<0.87.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 1824
neighbours:
[error_logger:error,2015-11-30T19:41:18.059Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.87.0>
registered_name: []
exception exit: {abnormal,1}
in function restartable:loop/4 (src/restartable.erl, line 69)
ancestors: [ns_child_ports_sup,ns_babysitter_sup,<0.56.0>]
messages: []
links: [<0.69.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 1989
neighbours:
[error_logger:error,2015-11-30T19:41:18.059Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_child_ports_sup}
Context: child_terminated
Reason: {abnormal,1}
Offender: [{pid,<0.87.0>},
{name,
{indexer,"/opt/couchbase/bin/goport",[],
[use_stdio,exit_status,stderr_to_stdout,stream,
{log,“indexer.log”},
{env,
[{“GOPORT_ARGS”,
"["/opt/couchbase/bin/indexer","-vbuckets=1024","-cluster=127.0.0.1:8091","-adminPort=9100","-scanPort=9101","-httpPort=9102","-streamInitPort=9103","-streamCatchupPort=9104","-streamMaintPort=9105","-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i"]"},
{“GOTRACEBACK”,[]},
{“CBAUTH_REVRPC_URL”,
“http://%40:9ab273789342b2778368ff0594fbed8c@127.0.0.1:8091/index”}]}]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[indexer,5000,infinity,ns_port_server,
start_link,
[#Fun<ns_child_ports_sup.2.49698737>]]},
86400000]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,worker}]
[ns_server:debug,2015-11-30T19:41:18.060Z,babysitter_of_ns_1@127.0.0.1:<0.396.0>:supervisor_cushion:init:39]starting ns_port_server with delay of 5000
[error_logger:info,2015-11-30T19:41:18.071Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,ale_dynamic_sup}
started: [{pid,<0.398.0>},
{name,'sink-indexer'},
{mfargs,
{ale_disk_sink,start_link,
['sink-indexer',
"/opt/couchbase/var/lib/couchbase/logs/indexer.log",
[{rotation,
[{compress,true},
{size,41943040},
{num_files,10},
{buffer_size_max,52428800}]}]]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
[error_logger:info,2015-11-30T19:41:18.132Z,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_child_ports_sup}
started: [{pid,<0.395.0>},
{name,
{indexer,"/opt/couchbase/bin/goport",[],
[use_stdio,exit_status,stderr_to_stdout,stream,
{log,"indexer.log"},
{env,
[{"GOPORT_ARGS",
"[\"/opt/couchbase/bin/indexer\",\"-vbuckets=1024\",\"-cluster=127.0.0.1:8091\",\"-adminPort=9100\",\"-scanPort=9101\",\"-httpPort=9102\",\"-streamInitPort=9103\",\"-streamCatchupPort=9104\",\"-streamMaintPort=9105\",\"-storageDir=/opt/couchbase/var/lib/couchbase/data/@2i\"]"},
{"GOTRACEBACK",[]},
{"CBAUTH_REVRPC_URL",
"http://%40:9ab273789342b2778368ff0594fbed8c@127.0.0.1:8091/index"}]}]}},
{mfargs,
{restartable,start_link,
[{supervisor_cushion,start_link,
[indexer,5000,infinity,ns_port_server,
start_link,
[#Fun<ns_child_ports_sup.2.49698737>]]},
86400000]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,worker}]
[ns_server:info,2015-11-30T19:41:19.458Z,babysitter_of_ns_1@127.0.0.1:<0.394.0>:ns_port_server:log:210]memcached<0.394.0>: 2015-11-30T19:41:19.257628Z WARNING (No Engine) Bucket Bingo_Card registered with low priority
memcached<0.394.0>: 2015-11-30T19:41:19.257702Z WARNING (No Engine) Spawning 4 readers, 4 writers, 1 auxIO, 1 nonIO threads
memcached<0.394.0>: 2015-11-30T19:41:19.258963Z WARNING 42: Slow CREATE_BUCKET operation on connection (127.0.0.1:50093 => 127.0.0.1:11209): 2139 ms
[ns_server:info,2015-11-30T19:41:24.176Z,babysitter_of_ns_1@127.0.0.1:<0.394.0>:ns_port_server:log:210]memcached<0.394.0>: 2015-11-30T19:41:23.975621Z WARNING (No Engine) Bucket Bingo_PlayerShout registered with low priority
memcached<0.394.0>: 2015-11-30T19:41:23.976977Z WARNING 43: Slow CREATE_BUCKET operation on connection (127.0.0.1:7119 => 127.0.0.1:11209): 6856 ms
memcached<0.394.0>: 2015-11-30T19:41:23.977100Z WARNING 42: Slow SELECT_BUCKET operation on connection (127.0.0.1:50093 => 127.0.0.1:11209): 4716 ms
memcached<0.394.0>: 2015-11-30T19:41:23.980434Z WARNING (Bingo_Card) Updated cluster configuration - first 100 bytes: '{"rev":2179,"name":"Bingo_Card","uri":"/pools/default/buckets/Bingo_Card?bucket_uuid=0e2bcc25447b444'...
memcached<0.394.0>: 2015-11-30T19:41:23.982121Z WARNING (Bingo_PlayerShout) Updated cluster configuration - first 100 bytes: '{"rev":2179,"name":"Bingo_PlayerShout","uri":"/pools/default/buckets/Bingo_PlayerShout?bucket_uuid=1'...
[ns_server:info,2015-11-30T19:41:31.135Z,babysitter_of_ns_1@127.0.0.1:<0.394.0>:ns_port_server:log:210]memcached<0.394.0>: 2015-11-30T19:41:30.934034Z WARNING (No Engine) Bucket Bingo_PlayerCards registered with low priority
memcached<0.394.0>: 2015-11-30T19:41:30.934235Z WARNING 45: Slow CREATE_BUCKET operation on connection (127.0.0.1:12406 => 127.0.0.1:11209): 6952 ms
memcached<0.394.0>: 2015-11-30T19:41:30.934322Z WARNING 52: Slow SELECT_BUCKET operation on connection (10.32.3.212:41077 => 10.32.3.212:11210): 6952 ms
[ns_server:info,2015-11-30T19:41:36.564Z,babysitter_of_ns_1@127.0.0.1:<0.394.0>:ns_port_server:log:210]memcached<0.394.0>: 2015-11-30T19:41:36.363491Z WARNING (No Engine) Bucket Bingo_Game registered with low priority
memcached<0.394.0>: 2015-11-30T19:41:36.363688Z WARNING 45: Slow SELECT_BUCKET operation on connection (127.0.0.1:12406 => 127.0.0.1:11209): 5429 ms
memcached<0.394.0>: 2015-11-30T19:41:36.363697Z WARNING 59: Slow SELECT_BUCKET operation on connection (127.0.0.1:10301 => 127.0.0.1:11209): 5428 ms
memcached<0.394.0>: 2015-11-30T19:41:36.363800Z WARNING 44: Slow CREATE_BUCKET operation on connection (127.0.0.1:54848 => 127.0.0.1:11209): 12382 ms
memcached<0.394.0>: 2015-11-30T19:41:36.365948Z WARNING (Bingo_PlayerCards) Updated cluster configuration - first 100 bytes: '{"rev":2179,"name":"Bingo_PlayerCards","uri":"/pools/default/buckets/Bingo_PlayerCards?bucket_uuid=2'...
memcached<0.394.0>: 2015-11-30T19:41:36.369875Z WARNING (Bingo_Game) Updated cluster configuration - first 100 bytes: '{"rev":2179,"name":"Bingo_Game","uri":"/pools/default/buckets/Bingo_Game?bucket_uuid=deebecc74740afb'...
And dmesg grep with oom-killer:
Nov 30 10:54:43 ip-10-32-3-212 kernel: [347215.081330] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 11:14:55 ip-10-32-3-212 kernel: [348426.578255] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 11:59:25 ip-10-32-3-212 kernel: [351097.227147] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 12:17:30 ip-10-32-3-212 kernel: [352182.739029] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 12:38:34 ip-10-32-3-212 kernel: [353446.186443] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 13:02:41 ip-10-32-3-212 kernel: [354892.651343] cbq-engine invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 13:20:42 ip-10-32-3-212 kernel: [355973.933736] cbq-engine invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 13:37:46 ip-10-32-3-212 kernel: [356997.990963] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 13:54:18 ip-10-32-3-212 kernel: [357990.577587] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 14:09:39 ip-10-32-3-212 kernel: [358911.151613] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 14:30:10 ip-10-32-3-212 kernel: [360142.359675] cbq-engine invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 14:47:47 ip-10-32-3-212 kernel: [361199.738149] ntpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 15:06:43 ip-10-32-3-212 kernel: [362334.959155] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 15:23:48 ip-10-32-3-212 kernel: [363359.707498] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 15:40:39 ip-10-32-3-212 kernel: [364370.820677] indexer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 15:57:50 ip-10-32-3-212 kernel: [365402.080630] mc:worker 3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 16:19:52 ip-10-32-3-212 kernel: [366724.338910] goxdcr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 16:37:07 ip-10-32-3-212 kernel: [367758.834485] indexer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 16:59:29 ip-10-32-3-212 kernel: [369101.652364] goxdcr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 17:17:05 ip-10-32-3-212 kernel: [370157.829432] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 17:37:40 ip-10-32-3-212 kernel: [371391.793876] mc:writer_5 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 17:53:53 ip-10-32-3-212 kernel: [372364.801814] indexer invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0
Nov 30 18:11:47 ip-10-32-3-212 kernel: [373439.655298] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 18:34:05 ip-10-32-3-212 kernel: [374776.625951] ntpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 18:51:19 ip-10-32-3-212 kernel: [375811.160988] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 19:07:15 ip-10-32-3-212 kernel: [376766.832195] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 19:07:15 ip-10-32-3-212 kernel: [376767.281656] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 19:23:14 ip-10-32-3-212 kernel: [377725.932775] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Nov 30 19:41:15 ip-10-32-3-212 kernel: [378806.800754] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 19:41:15 ip-10-32-3-212 kernel: [378807.145096] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 21:37:01 ip-10-32-3-212 kernel: [385753.526360] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Nov 30 22:35:21 ip-10-32-3-212 kernel: [389252.891067] goxdcr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 01:59:36 ip-10-32-3-212 kernel: [401507.487052] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 01:59:36 ip-10-32-3-212 kernel: [401507.831900] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 02:45:37 ip-10-32-3-212 kernel: [404268.434741] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 03:10:36 ip-10-32-3-212 kernel: [405767.458390] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 03:10:36 ip-10-32-3-212 kernel: [405767.807478] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 03:43:59 ip-10-32-3-212 kernel: [407771.576795] indexer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 04:13:14 ip-10-32-3-212 kernel: [409525.769219] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 04:33:12 ip-10-32-3-212 kernel: [410724.513248] moxi invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 04:56:03 ip-10-32-3-212 kernel: [412094.179134] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 05:17:48 ip-10-32-3-212 kernel: [413399.973475] projector invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 05:37:21 ip-10-32-3-212 kernel: [414573.129070] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 05:59:51 ip-10-32-3-212 kernel: [415923.652576] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 06:18:57 ip-10-32-3-212 kernel: [417069.003026] mc:reader_1 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 06:37:03 ip-10-32-3-212 kernel: [418154.844607] projector invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 06:37:03 ip-10-32-3-212 kernel: [418155.184220] projector invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 06:56:56 ip-10-32-3-212 kernel: [419348.740184] moxi invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 07:14:58 ip-10-32-3-212 kernel: [420430.152622] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 07:38:58 ip-10-32-3-212 kernel: [421869.660789] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Dec 1 08:06:50 ip-10-32-3-212 kernel: [423541.740733] beam.smp invoked oom-killer: gfp_mask=0x10200da, order=0, oom_score_adj=0
Dec 1 08:29:06 ip-10-32-3-212 kernel: [424878.169631] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 08:51:37 ip-10-32-3-212 kernel: [426229.338850] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 09:11:40 ip-10-32-3-212 kernel: [427432.588260] goxdcr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 09:30:59 ip-10-32-3-212 kernel: [428591.236604] indexer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 09:51:03 ip-10-32-3-212 kernel: [429794.776385] mc:nonio_9 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 09:51:03 ip-10-32-3-212 kernel: [429795.160876] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 12:01:07 ip-10-32-3-212 kernel: [437598.820621] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 12:48:26 ip-10-32-3-212 kernel: [440438.299320] df invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 13:15:38 ip-10-32-3-212 kernel: [442069.973793] indexer invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 13:45:45 ip-10-32-3-212 kernel: [443876.609278] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 1 14:24:33 ip-10-32-3-212 kernel: [446204.968305] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
BTW, I’ve been trying to do some N1QL queries now that the cluster seems to be stabilized and surprisingly it is complaining about indexes (that indeed exist) do not exist. Could this be because they are somehow corrupted after the indexer process being killed by the oom-killer?