TCMalloc don't release memory


#1

Hi,

I’m running Couchase 3.0.3 for a while, and I have a problem with the memory usage, which never decrease. According to Couchbase graphs, memory is not consummed by Couchbase itself, but by TCMalloc, which prealloc a lot memory, as TCMalloc statistics says:

MALLOC:   119701952152 (114156.7 MiB) Bytes in use by application
MALLOC: +  18110562304 (17271.6 MiB) Bytes in page heap freelist
MALLOC: +  36195393472 (34518.6 MiB) Bytes in central cache freelist
MALLOC: +      3861344 (    3.7 MiB) Bytes in transfer cache freelist
MALLOC: +     28229192 (   26.9 MiB) Bytes in thread cache freelists
MALLOC: +    617197728 (  588.6 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: = 174657196192 (166566.1 MiB) Actual memory used (physical + swap)
MALLOC: +     36454400 (   34.8 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: = 174693650592 (166600.8 MiB) Virtual address space used

Between heap and central free list, I have about 50GB preallocated. I tried to call release_free_memory method via mcctl, but it realease only the heap free list, and don’t touch to the central free list, which is the bigger.

Here a graph of the evolution on one day. Red line is “Bytes in use by application”, and green line is "Actual memory used ", from TCMalloc statistics.

The bucket is setup in full eviction mode, but it was the same on value eviction. I took a look at the 3.1.0 Couchbase version, but TCMalloc version is the same. I’m not running any cbackup for the moment.

I already opened a topic on the issue few weeks ago, which gave me detailled informations on the memory usage, but I now stuck on this TCMalloc problem.

Any kind of help will be appreciated. Thanks in advance.


#2

Hi AlexisD,

This is an issue we are aware of, but with the current memory allocator (TCMalloc) there isn’t much more to be done - as you’ve seen release_free_memory asks TCMalloc to release what it can, but that isn’t always the bulk of the usage. As I mentioned in the previous thread you can swap-rebalance out the affected node(s) and then re-insert them - this will essentially restart the memcached process and hence reset the heap.

The good news is that we’ve made a number of changes in this area in the forthcoming 4.0 release - specifically moving to a different underlying memory allocator on Linux (jemalloc) and introducing a memory defragmentation task inside memcached to reduce the impact of fragmentation build-up.

I encourage you to try the 4.0-Beta on a test environment and see how that affects the memory usage with your workload.