Indexer always warming up

Hi,

I have a 3 node cluster, 1 data, 1 index query, 1 index query. The indexer on the 2nd node always warms up every now and then while querying. Specs for the 3 nodes are 8core cpu, 128gb ram, 512 ssd. Running on Linux and EE version 6

indexer logs

Service ‘indexer’ exited with status 137. Restarting. Messages:
2019-12-13T06:41:52.314+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore
2019-12-13T06:41:53.039+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 10.0.5.19:9999 Topic MAINT_STREAM_TOPIC_6fbb1a117da06c0f6cce41d1ec67e6d1. Err projector.topicMissing
2019-12-13T06:41:53.039+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 10.0.5.19:9999
2019-12-13T06:41:53.301+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 10.0.5.19:9999 Topic INIT_STREAM_TOPIC_6fbb1a117da06c0f6cce41d1ec67e6d1. Err projector.topicMissing
2019-12-13T06:41:53.301+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 10.0.5.19:9999
2019-12-13T06:48:07.405+00:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 10.0.5.21:44878. Error = EOF. Kill Pipe.
[goport(/opt/couchbase/bin/indexer)] 2019/12/13 07:22:35 child process exited with status 137

@phaigne,

These errors are expected during indexer bootstrap.

Service ‘indexer’ exited with status 137

This means that indexer is probably running out-of-memory. This looks like a sizing issue to start with.

Which storage mode are you using? What is the memory consumption on the node before the time of crash? How many documents are you indexing and what is the average size of the indexed value? How many indexes are you creating.

Can you share the output of indexer stats before the time of crash. You can get the index stats using the following curl call:

$ curl -X GET -u Administrator:password "http://<indexer_node_IP>:9102/api/v1/stats"

Thanks,
Varun