See this section from the TCMalloc allocator stats:
TCMalloc (the memory allocator used by Couchbase) doesn’t immediately return memory to the OS, in case it needs the memory again in the near future (and it would have been a waste of time to return it to have to then re-allocate it).
As you can see from that extract; there’s 19.7GB in TCMalloc’s freelists; should account for the difference you are seeing. That’s quite a large amount, TCMalloc doesn’t normally keep back such a quantity of memory. Has the node been up a long time?
In terms of resolving this you can always swap-rebalance out the node with a new one (and then re-insert the original node after restarting couchbase-server if you wish) - this will clear out the TCMalloc heap.
Another option is an (undocumented, unsupported) method to request TCMalloc to release it’s freelist memory which you could look into. It’s unsupported as it can have an impact on the performance of any future writes to the database as the memory allocator may not have any already free memory available). Additionally it’s not tested by Couchbase QA afiak.
If you decide to give it a try, you’ll need to run the mcctl command on the specific node(s):
/opt/couchbase/bin/mcctl -h localhost:11210 set release_free_memory
Note that this only requests TCMalloc to release memory, it doesn’t always necessarily release all free memory…