We have observed that erl process leaks approximately 1GB in 60 hours even when the system is completely idle. We are observing this in Couchbase Server 4,5,0 Community Edition.
Is this the expected behaviour or known issue that is resolved?
@wellitssatish, you are probably hitting https://issues.couchbase.com/browse/MB-20521 (if you’re using the query service on this node). This has been fixed in all releases of Couchbase Server subsequent to 4.5.0.
In the meantime, there is a workaround which is to restart the cluster management process. This has the effect of emptying the stats cache, releasing the memory used. The downside of this approach is that for a very short period (while the process starts up again, usually a minute or so) there will be slight disruption:
- The web console on that particular node will be temporarily unavailable
- Connections on 8091 will be refused
- The node will be seen briefly as ‘pending’ by the other nodes in the cluster.
Please note that this will not affect data access as the SDKs and other components will connect directly to each other where necessary. However, due to the node being seen as ‘pending’ during this period it is important that before conducting the workaround that you disable automatic failover, this is to prevent erroneous failovers when actually the node is still available; it will be perfectly safe to re-enable automatic failover after the cluster is marked ‘active’ again.
The command to do this is as follows:
curl --data "erlang:halt()." -u username:password http://hostname:8091/diag/eval
Hope this helps!
Thanks Matt for the prompt response.
We have built the code for version 4.5.1 however, even now we observe the memory leak in erl process.
We are also experiencing this memory leak in 4.5.1-2844 community edition. We have to keep restarting couchbase everyday else it will run out of memory. Is there a fix for this? MB-20521 doesn’t seem its fixed.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5642 couchba+ 20 0 18.229g 0.017t 11444 S 0.0 55.5 308:17.05 cbq-engine
5617 couchba+ 20 0 4723904 2.190g 11708 S 67.6 7.0 85:37.97 indexer
5669 couchba+ 20 0 1624944 1.150g 9520 S 26.5 3.7 94:59.22 memcached
5504 couchba+ 20 0 2347620 893604 3568 S 26.5 2.7 133:03.93 beam.smp
5604 couchba+ 20 0 2300556 721844 8288 S 17.6 2.2 31:40.62 projector
5542 couchba+ 20 0 1818596 135728 13292 S 8.8 0.4 13:57.23 beam.smp