Couchbase memcached critically overflows memory allocation


#1

We are experiencing a critical issue with the server ignoring the server and bucket RAM quotas. The Linux OOM kicks in and kills Couchbase due to this overflow.

Versions:
couchbase-server-community_2.2.0_x86_64.rpm
CentOS release 6.5 (Final)

Admin console stats for an offending node (all nodes are overflowing):
RAM:Couchbase Quota (4 GB) Total (6.69 GB)
In Use (2.66 GB) Other Data (3.76 GB) Free (276 MB)

However Top command shows:
Couchbase memcached process is taking 67.2% + beam 11% = 5.24 GB

Linux system log:
kernel: Out of memory: Kill process 50564 (memcached) score 787 or sacrifice child
/var/log/messages:May 8 16:27:46 sns-cb002 kernel: Killed process 50564, UID 498, (memcached) total-vm:5754632kB, anon-rss:5515928kB, file-rss:1984kB

Detailed Process Output:
5603 couchbas 20 0 2059m 742m 41m S 8.6 10.8 2835:37 beam.smp
51320 couchbas 20 0 4845m 4.5g 3104 S 3.3 67.2 21:05.82 memcached
6615 couchbas 20 0 6272 600 476 S 0.7 0.0 51:46.74 sigar_port
5553 couchbas 20 0 10828 564 416 S 0.3 0.0 0:03.39 epmd
5576 couchbas 20 0 1329m 32m 2052 S 0.0 0.5 0:35.70 beam.smp
5633 couchbas 20 0 103m 1084 884 S 0.0 0.0 0:02.88 sh
5634 couchbas 20 0 4056 556 476 S 0.0 0.0 0:11.81 memsup
5636 couchbas 20 0 4056 412 336 S 0.0 0.0 0:00.00 cpu_sup
6100 couchbas 20 0 10792 472 372 S 0.0 0.0 0:00.00 inet_gethost
6101 couchbas 20 0 10792 392 276 S 0.0 0.0 0:00.00 inet_gethost
8444 couchbas 20 0 39660 1904 1512 S 0.0 0.0 0:05.58 ssl_esock
8696 couchbas 20 0 10792 500 400 S 0.0 0.0 0:16.79 inet_gethost
8697 couchbas 20 0 10792 404 288 S 0.0 0.0 0:02.34 inet_gethost
8794 couchbas 20 0 10792 388 272 S 0.0 0.0 0:01.43 inet_gethost
11173 couchbas 20 0 413m 4704 880 S 0.0 0.1 1:42.76 moxi
11207 couchbas 20 0 775m 1840 436 S 0.0 0.0 2:52.30 i386-linux-godu
17596 couchbas 20 0 10792 384 268 S 0.0 0.0 0:00.60 inet_gethost
17610 couchbas 20 0 10792 388 272 S 0.0 0.0 0:02.74 inet_gethost


#2

We are also facing the same issue.

rpm -qa |grep couch

couchbase-server-2.2.0-837.x86_64

cat /etc/redhat-release

CentOS release 6.5 (Final)