TCMalloc don't release memory

Couchbase Server
1

Hi,

I’m running Couchase 3.0.3 for a while, and I have a problem with the memory usage, which never decrease. According to Couchbase graphs, memory is not consummed by Couchbase itself, but by TCMalloc, which prealloc a lot memory, as TCMalloc statistics says:

MALLOC:   119701952152 (114156.7 MiB) Bytes in use by application
MALLOC: +  18110562304 (17271.6 MiB) Bytes in page heap freelist
MALLOC: +  36195393472 (34518.6 MiB) Bytes in central cache freelist
MALLOC: +      3861344 (    3.7 MiB) Bytes in transfer cache freelist
MALLOC: +     28229192 (   26.9 MiB) Bytes in thread cache freelists
MALLOC: +    617197728 (  588.6 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: = 174657196192 (166566.1 MiB) Actual memory used (physical + swap)
MALLOC: +     36454400 (   34.8 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: = 174693650592 (166600.8 MiB) Virtual address space used

Between heap and central free list, I have about 50GB preallocated. I tried to call release_free_memory method via mcctl, but it realease only the heap free list, and don’t touch to the central free list, which is the bigger.

Here a graph of the evolution on one day. Red line is “Bytes in use by application”, and green line is "Actual memory used ", from TCMalloc statistics.

tcmalloc_usage.png1320×339 35.1 KB

The bucket is setup in full eviction mode, but it was the same on value eviction. I took a look at the 3.1.0 Couchbase version, but TCMalloc version is the same. I’m not running any cbackup for the moment.

I already opened a topic on the issue few weeks ago, which gave me detailled informations on the memory usage, but I now stuck on this TCMalloc problem.

Any kind of help will be appreciated. Thanks in advance.

2

Hi AlexisD,

This is an issue we are aware of, but with the current memory allocator (TCMalloc) there isn’t much more to be done - as you’ve seen release_free_memory asks TCMalloc to release what it can, but that isn’t always the bulk of the usage. As I mentioned in the previous thread you can swap-rebalance out the affected node(s) and then re-insert them - this will essentially restart the memcached process and hence reset the heap.

The good news is that we’ve made a number of changes in this area in the forthcoming 4.0 release - specifically moving to a different underlying memory allocator on Linux (jemalloc) and introducing a memory defragmentation task inside memcached to reduce the impact of fragmentation build-up.

I encourage you to try the 4.0-Beta on a test environment and see how that affects the memory usage with your workload.

1 Like