It looks like Linux has run out of RAM (OOM) and hence had to kill something to be able to continue operating.
This could simply be a sizing problem - how much RAM have your nodes got, and what have you set the Server Quoa to? Could you list post the previous lines of /var/log/messages - those listing the sizes of the various process when the OOM-killer was invoked.
Thanks @drigby for the information.
We have 4 nodes each with 8 GB RAM. Out of which 2 GB from each node is allocated as cluster Quota (8 GB total).
Following are the only lines for this process in /var/log/messages
Aug 4 07:07:52 ip-172-31-23-150 kernel: [3234847.720645] [29953] 220 29953 2121411 1143942 3857 808611 0 couch_view_grou
Aug 4 07:07:52 ip-172-31-23-150 kernel: [3234847.720656] Out of memory: Kill process 29953 (couch_view_grou) score 658 or sacrifice child
Aug 4 07:07:52 ip-172-31-23-150 kernel: [3234847.720668] Killed process 29953 (couch_view_grou) total-vm:8485644kB, anon-rss:4575768kB, file-rss:0kB
Aslo, if this is sizing problem, do we have sizing guidelines describing what should be the correct parameters/configurations for 4 node cluster. We use views very heavily. i would say at all time we fetch data through view and their size is few 100 times more than actual data size. What wiould be the c ideal RAM/Disk for such use case for couchbase to function smoothly?
thanks @martinesmann .
Could you please let me know which all log you will need ?
cbcollect_info generated a zip file with 140 MB, which am not able to upload.
Please let me know files you would need and i will only upload them.