Memory usage over quota on Couchbase-server 3.0.1

Hi,

I’m running a Couchbase cluster, in 3.0.1 community edition, where I see the memory usage growing every day, over the quota defined per server. I tried to find out which part is eating the memory, but any Couchbase graph reports a so high usage. Here is an example of what I’m talking about. In my cluster, the per RAM server quota is set to 71Go.

On a particular node, I see this reports, from Couchbase graphs:

  • Meta data usage: 24 Go
  • User RAM usage: 45 Go
  • Global RAM usage: 51 Go

And from the system, a pmap -d on memcached process report 77 Go used by this processus:

> pmap -d 1412
> ...
> 00000000016de000 77381432 rw--- 0000000000000000 000:00000   [ anon ]
> ...
> mapped: 77799464K    writeable/private: 77710396K    shared: 0K

There is 6 Go difference between the quota and the memory really used, not reported by Couchbase, but only by the system, and it’s still growing. The system is running a Linux 3.16.0-4 under Debian. I disabled transparent huge pages, and set the swapiness to 0.

Have you any idea of the origin of the problem ?
Thanks in advance for the help.

Are you looking at Virtual (VSZ) or Resident (RSS)? Much like many applications the virtual memory footprint of the memcached process may well be higher than the quota, but the RSS should be within the Server Quota.

Examine top, or run something like the following to see the RSS and VSZ of the memcached process:

ps -o rss,vsz -p $(pgrep memcached)

For example on a test machine which is configured for 300MB quota I see (values in KB):

$ ps -o rss,vsz -p $(pgrep memcached)
  RSS    VSZ
178356 506520

i.e. while the virtual memory size is 494MB, the actual resident is 174MB.

Thanks for your reply. I see what you mean, unfortunately, RSS and VSZ are almost the same:

ps -o rss,vsz -p $(pgrep memcached)
RSS    VSZ
77091732 77799472

Can you take a look at the memory statistics?

cbstats HOSTNAME:11210 memory

and

cbstats HOSTNAME:11210 allocator

replacing HOSTNAME with your node name (or just localhost if running cbstats locally)?

Here is the result of the two commands:

For allocation:

MALLOC:    57461808912 (54799.9 MiB) Bytes in use by application
MALLOC: +  11297824768 (10774.4 MiB) Bytes in page heap freelist
MALLOC: +   9862720336 ( 9405.8 MiB) Bytes in central cache freelist
MALLOC: +       524256 (    0.5 MiB) Bytes in transfer cache freelist
MALLOC: +     27055552 (   25.8 MiB) Bytes in thread cache freelists
MALLOC: +    311734432 (  297.3 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  78961668256 (75303.7 MiB) Actual memory used (physical + swap)
MALLOC: +    269885440 (  257.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  79231553696 (75561.1 MiB) Virtual address space used
MALLOC:
MALLOC:        4123541              Spans in use
MALLOC:             38              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Total size of freelists for per-thread caches,
transfer cache, and central cache, by size class
------------------------------------------------
class   1 [        8 bytes ] :    52934 objs;   0.4 MiB;   0.4 cum MiB
class   2 [       16 bytes ] :   119140 objs;   1.8 MiB;   2.2 cum MiB
class   3 [       32 bytes ] :   855193 objs;  26.1 MiB;  28.3 cum MiB
class   4 [       48 bytes ] :   184196 objs;   8.4 MiB;  36.8 cum MiB
class   5 [       64 bytes ] :    11118 objs;   0.7 MiB;  37.4 cum MiB
class   6 [       80 bytes ] :    13394 objs;   1.0 MiB;  38.5 cum MiB
class   7 [       96 bytes ] :  2063127 objs; 188.9 MiB; 227.3 cum MiB
class   8 [      112 bytes ] :   577069 objs;  61.6 MiB; 289.0 cum MiB
class   9 [      128 bytes ] :    45066 objs;   5.5 MiB; 294.5 cum MiB
class  10 [      144 bytes ] :    14008 objs;   1.9 MiB; 296.4 cum MiB
class  11 [      160 bytes ] :    31038 objs;   4.7 MiB; 301.1 cum MiB
class  12 [      176 bytes ] :    14382 objs;   2.4 MiB; 303.5 cum MiB
class  13 [      192 bytes ] :     3461 objs;   0.6 MiB; 304.2 cum MiB
class  14 [      208 bytes ] :    10907 objs;   2.2 MiB; 306.3 cum MiB
class  15 [      224 bytes ] :    11663 objs;   2.5 MiB; 308.8 cum MiB
class  16 [      240 bytes ] :     8463 objs;   1.9 MiB; 310.8 cum MiB
class  17 [      256 bytes ] :     5991 objs;   1.5 MiB; 312.2 cum MiB
class  18 [      288 bytes ] :    14746 objs;   4.1 MiB; 316.3 cum MiB
class  19 [      320 bytes ] :     1480 objs;   0.5 MiB; 316.7 cum MiB
class  20 [      352 bytes ] :     1152 objs;   0.4 MiB; 317.1 cum MiB
class  21 [      384 bytes ] :     1227 objs;   0.4 MiB; 317.6 cum MiB
class  22 [      416 bytes ] :     1087 objs;   0.4 MiB; 318.0 cum MiB
class  23 [      448 bytes ] :     1030 objs;   0.4 MiB; 318.4 cum MiB
class  24 [      480 bytes ] :      973 objs;   0.4 MiB; 318.9 cum MiB
class  25 [      512 bytes ] :     1933 objs;   0.9 MiB; 319.8 cum MiB
class  26 [      576 bytes ] :      575 objs;   0.3 MiB; 320.2 cum MiB
class  27 [      640 bytes ] :      514 objs;   0.3 MiB; 320.5 cum MiB
class  28 [      704 bytes ] :      569 objs;   0.4 MiB; 320.8 cum MiB
class  29 [      768 bytes ] :      484 objs;   0.4 MiB; 321.2 cum MiB
class  30 [      832 bytes ] :      382 objs;   0.3 MiB; 321.5 cum MiB
class  31 [      896 bytes ] :    16082 objs;  13.7 MiB; 335.2 cum MiB
class  32 [      960 bytes ] :   890389 objs; 815.2 MiB; 1150.4 cum MiB
class  33 [     1024 bytes ] :  1069929 objs; 1044.9 MiB; 2195.3 cum MiB
class  34 [     1152 bytes ] :  1633535 objs; 1794.7 MiB; 3989.9 cum MiB
class  35 [     1280 bytes ] :      280 objs;   0.3 MiB; 3990.3 cum MiB
class  36 [     1408 bytes ] :      254 objs;   0.3 MiB; 3990.6 cum MiB
class  37 [     1536 bytes ] :      277 objs;   0.4 MiB; 3991.0 cum MiB
class  38 [     1792 bytes ] :      200 objs;   0.3 MiB; 3991.4 cum MiB
class  39 [     2048 bytes ] :   138693 objs; 270.9 MiB; 4262.2 cum MiB
class  40 [     2304 bytes ] :    64545 objs; 141.8 MiB; 4404.1 cum MiB
class  41 [     2560 bytes ] :      162 objs;   0.4 MiB; 4404.5 cum MiB
class  42 [     2816 bytes ] :    18357 objs;  49.3 MiB; 4453.8 cum MiB
class  43 [     3072 bytes ] :    98603 objs; 288.9 MiB; 4742.6 cum MiB
class  44 [     3328 bytes ] :    60816 objs; 193.0 MiB; 4935.7 cum MiB
class  45 [     4096 bytes ] :   122084 objs; 476.9 MiB; 5412.5 cum MiB
class  46 [     4608 bytes ] :    77812 objs; 341.9 MiB; 5754.5 cum MiB
class  47 [     5120 bytes ] :    84056 objs; 410.4 MiB; 6164.9 cum MiB
class  48 [     6144 bytes ] :   139578 objs; 817.8 MiB; 6982.8 cum MiB
class  49 [     6656 bytes ] :    68770 objs; 436.5 MiB; 7419.3 cum MiB
class  50 [     8192 bytes ] :    70818 objs; 553.3 MiB; 7972.6 cum MiB
class  51 [     9216 bytes ] :    51878 objs; 456.0 MiB; 8428.5 cum MiB
class  52 [    10240 bytes ] :    12598 objs; 123.0 MiB; 8551.5 cum MiB
class  53 [    12288 bytes ] :      503 objs;   5.9 MiB; 8557.4 cum MiB
class  54 [    13312 bytes ] :        3 objs;   0.0 MiB; 8557.5 cum MiB
class  55 [    16384 bytes ] :        2 objs;   0.0 MiB; 8557.5 cum MiB
class  59 [    32768 bytes ] :       19 objs;   0.6 MiB; 8558.1 cum MiB
class  72 [   139264 bytes ] :       10 objs;   1.3 MiB; 8559.4 cum MiB
class  75 [   163840 bytes ] :        2 objs;   0.3 MiB; 8559.7 cum MiB
------------------------------------------------
PageHeap: 119 sizes; 10774.4 MiB free;  257.4 MiB unmapped
------------------------------------------------
     1 pages * 147778 spans ~ 1154.5 MiB; 1154.5 MiB cum; unmapped:    2.5 MiB;    2.5 MiB cum
     2 pages * 118252 spans ~ 1847.7 MiB; 3002.2 MiB cum; unmapped:    0.1 MiB;    2.6 MiB cum
     3 pages *  32226 spans ~  755.3 MiB; 3757.5 MiB cum; unmapped:    0.1 MiB;    2.7 MiB cum
     4 pages *  30645 spans ~  957.7 MiB; 4715.2 MiB cum; unmapped:    0.1 MiB;    2.7 MiB cum
     5 pages *   5889 spans ~  230.0 MiB; 4945.2 MiB cum; unmapped:    0.0 MiB;    2.8 MiB cum
     6 pages *   4618 spans ~  216.5 MiB; 5161.7 MiB cum; unmapped:    0.0 MiB;    2.8 MiB cum
     7 pages *    632 spans ~   34.6 MiB; 5196.2 MiB cum; unmapped:    0.1 MiB;    2.9 MiB cum
     8 pages *    627 spans ~   39.2 MiB; 5235.4 MiB cum; unmapped:    0.1 MiB;    2.9 MiB cum
     9 pages *    107 spans ~    7.5 MiB; 5242.9 MiB cum; unmapped:    0.1 MiB;    3.0 MiB cum
    10 pages *    146 spans ~   11.4 MiB; 5254.3 MiB cum; unmapped:    0.1 MiB;    3.1 MiB cum
    11 pages *     34 spans ~    2.9 MiB; 5257.3 MiB cum; unmapped:    0.0 MiB;    3.1 MiB cum
    12 pages *     23 spans ~    2.2 MiB; 5259.4 MiB cum; unmapped:    0.1 MiB;    3.2 MiB cum
    13 pages *      8 spans ~    0.8 MiB; 5260.2 MiB cum; unmapped:    0.1 MiB;    3.3 MiB cum
    14 pages *      3 spans ~    0.3 MiB; 5260.6 MiB cum; unmapped:    0.0 MiB;    3.3 MiB cum
    15 pages *      1 spans ~    0.1 MiB; 5260.7 MiB cum; unmapped:    0.0 MiB;    3.3 MiB cum
    16 pages *      6 spans ~    0.8 MiB; 5261.4 MiB cum; unmapped:    0.0 MiB;    3.3 MiB cum
    18 pages *   3517 spans ~  494.6 MiB; 5756.0 MiB cum; unmapped:    0.8 MiB;    4.1 MiB cum
    19 pages *    794 spans ~  117.9 MiB; 5873.9 MiB cum; unmapped:    0.7 MiB;    4.9 MiB cum
    20 pages *   3058 spans ~  477.8 MiB; 6351.7 MiB cum; unmapped:    0.9 MiB;    5.8 MiB cum
    21 pages *    646 spans ~  106.0 MiB; 6457.7 MiB cum; unmapped:    0.8 MiB;    6.6 MiB cum
    22 pages *   2288 spans ~  393.2 MiB; 6850.9 MiB cum; unmapped:    1.0 MiB;    7.6 MiB cum
    23 pages *    527 spans ~   94.7 MiB; 6945.6 MiB cum; unmapped:    0.9 MiB;    8.5 MiB cum
    24 pages *   1597 spans ~  299.4 MiB; 7245.0 MiB cum; unmapped:    1.1 MiB;    9.7 MiB cum
    25 pages *    328 spans ~   64.1 MiB; 7309.1 MiB cum; unmapped:    1.0 MiB;   10.6 MiB cum
    26 pages *   1241 spans ~  252.1 MiB; 7561.2 MiB cum; unmapped:    1.2 MiB;   11.9 MiB cum
    27 pages *    239 spans ~   50.4 MiB; 7611.6 MiB cum; unmapped:    1.1 MiB;   12.9 MiB cum
    28 pages *   1001 spans ~  219.0 MiB; 7830.6 MiB cum; unmapped:    1.3 MiB;   14.2 MiB cum
    29 pages *    193 spans ~   43.7 MiB; 7874.3 MiB cum; unmapped:    1.4 MiB;   15.6 MiB cum
    30 pages *    783 spans ~  183.5 MiB; 8057.8 MiB cum; unmapped:    1.4 MiB;   17.0 MiB cum
    31 pages *    133 spans ~   32.2 MiB; 8090.0 MiB cum; unmapped:    1.2 MiB;   18.2 MiB cum
    32 pages *    655 spans ~  163.8 MiB; 8253.8 MiB cum; unmapped:    1.5 MiB;   19.7 MiB cum
    33 pages *    105 spans ~   27.1 MiB; 8280.8 MiB cum; unmapped:    1.3 MiB;   21.0 MiB cum
    34 pages *    526 spans ~  139.7 MiB; 8420.6 MiB cum; unmapped:    1.6 MiB;   22.6 MiB cum
    35 pages *     78 spans ~   21.3 MiB; 8441.9 MiB cum; unmapped:    1.4 MiB;   24.0 MiB cum
    36 pages *    454 spans ~  127.7 MiB; 8569.6 MiB cum; unmapped:    1.7 MiB;   25.6 MiB cum
    37 pages *     77 spans ~   22.3 MiB; 8591.8 MiB cum; unmapped:    1.4 MiB;   27.1 MiB cum
    38 pages *    396 spans ~  117.6 MiB; 8709.4 MiB cum; unmapped:    1.8 MiB;   28.9 MiB cum
    39 pages *     60 spans ~   18.3 MiB; 8727.7 MiB cum; unmapped:    1.5 MiB;   30.4 MiB cum
    40 pages *    364 spans ~  113.8 MiB; 8841.4 MiB cum; unmapped:    1.9 MiB;   32.3 MiB cum
    41 pages *     41 spans ~   13.1 MiB; 8854.6 MiB cum; unmapped:    1.6 MiB;   33.9 MiB cum
    42 pages *    294 spans ~   96.5 MiB; 8951.0 MiB cum; unmapped:    2.0 MiB;   35.8 MiB cum
    43 pages *     36 spans ~   12.1 MiB; 8963.1 MiB cum; unmapped:    1.3 MiB;   37.2 MiB cum
    44 pages *    279 spans ~   95.9 MiB; 9059.0 MiB cum; unmapped:    1.7 MiB;   38.9 MiB cum
    45 pages *     19 spans ~    6.7 MiB; 9065.7 MiB cum; unmapped:    1.8 MiB;   40.7 MiB cum
    46 pages *    211 spans ~   75.8 MiB; 9141.5 MiB cum; unmapped:    2.2 MiB;   42.8 MiB cum
    47 pages *     27 spans ~    9.9 MiB; 9151.5 MiB cum; unmapped:    1.8 MiB;   44.7 MiB cum
    48 pages *    191 spans ~   71.6 MiB; 9223.1 MiB cum; unmapped:    1.9 MiB;   46.5 MiB cum
    49 pages *     28 spans ~   10.7 MiB; 9233.8 MiB cum; unmapped:    1.9 MiB;   48.4 MiB cum
    50 pages *    188 spans ~   73.4 MiB; 9307.2 MiB cum; unmapped:    2.3 MiB;   50.8 MiB cum
    51 pages *     20 spans ~    8.0 MiB; 9315.2 MiB cum; unmapped:    2.4 MiB;   53.2 MiB cum
    52 pages *    179 spans ~   72.7 MiB; 9387.9 MiB cum; unmapped:    2.4 MiB;   55.6 MiB cum
    53 pages *     21 spans ~    8.7 MiB; 9396.6 MiB cum; unmapped:    2.1 MiB;   57.7 MiB cum
    54 pages *    157 spans ~   66.2 MiB; 9462.9 MiB cum; unmapped:    2.5 MiB;   60.2 MiB cum
    55 pages *     12 spans ~    5.2 MiB; 9468.0 MiB cum; unmapped:    1.7 MiB;   61.9 MiB cum
    56 pages *    129 spans ~   56.4 MiB; 9524.4 MiB cum; unmapped:    2.6 MiB;   64.6 MiB cum
    57 pages *     18 spans ~    8.0 MiB; 9532.5 MiB cum; unmapped:    2.7 MiB;   67.2 MiB cum
    58 pages *    125 spans ~   56.6 MiB; 9589.1 MiB cum; unmapped:    2.3 MiB;   69.5 MiB cum
    59 pages *     14 spans ~    6.5 MiB; 9595.6 MiB cum; unmapped:    2.3 MiB;   71.8 MiB cum
    60 pages *    114 spans ~   53.4 MiB; 9649.0 MiB cum; unmapped:    2.3 MiB;   74.1 MiB cum
    61 pages *     12 spans ~    5.7 MiB; 9654.7 MiB cum; unmapped:    2.4 MiB;   76.5 MiB cum
    62 pages *     97 spans ~   47.0 MiB; 9701.7 MiB cum; unmapped:    2.4 MiB;   79.0 MiB cum
    63 pages *     12 spans ~    5.9 MiB; 9707.6 MiB cum; unmapped:    2.5 MiB;   81.4 MiB cum
    64 pages *     81 spans ~   40.5 MiB; 9748.1 MiB cum; unmapped:    2.5 MiB;   83.9 MiB cum
    65 pages *     16 spans ~    8.1 MiB; 9756.2 MiB cum; unmapped:    2.5 MiB;   86.5 MiB cum
    66 pages *     80 spans ~   41.2 MiB; 9797.5 MiB cum; unmapped:    3.1 MiB;   89.5 MiB cum
    67 pages *     13 spans ~    6.8 MiB; 9804.3 MiB cum; unmapped:    3.1 MiB;   92.7 MiB cum
    68 pages *     71 spans ~   37.7 MiB; 9842.0 MiB cum; unmapped:    3.2 MiB;   95.9 MiB cum
    69 pages *      9 spans ~    4.9 MiB; 9846.9 MiB cum; unmapped:    2.7 MiB;   98.6 MiB cum
    70 pages *     63 spans ~   34.5 MiB; 9881.3 MiB cum; unmapped:    2.7 MiB;  101.3 MiB cum
    71 pages *      4 spans ~    2.2 MiB; 9883.5 MiB cum; unmapped:    2.2 MiB;  103.5 MiB cum
    72 pages *     39 spans ~   21.9 MiB; 9905.5 MiB cum; unmapped:    2.8 MiB;  106.3 MiB cum
    73 pages *      4 spans ~    2.3 MiB; 9907.7 MiB cum; unmapped:    2.3 MiB;  108.6 MiB cum
    74 pages *     53 spans ~   30.6 MiB; 9938.4 MiB cum; unmapped:    3.5 MiB;  112.1 MiB cum
    75 pages *     12 spans ~    7.0 MiB; 9945.4 MiB cum; unmapped:    2.9 MiB;  115.0 MiB cum
    76 pages *     45 spans ~   26.7 MiB; 9972.1 MiB cum; unmapped:    3.0 MiB;  118.0 MiB cum
    77 pages *     10 spans ~    6.0 MiB; 9978.1 MiB cum; unmapped:    3.6 MiB;  121.6 MiB cum
    78 pages *     37 spans ~   22.5 MiB; 10000.7 MiB cum; unmapped:    3.0 MiB;  124.6 MiB cum
    79 pages *      4 spans ~    2.5 MiB; 10003.2 MiB cum; unmapped:    1.9 MiB;  126.5 MiB cum
    80 pages *     39 spans ~   24.4 MiB; 10027.5 MiB cum; unmapped:    2.5 MiB;  129.0 MiB cum
    81 pages *      4 spans ~    2.5 MiB; 10030.1 MiB cum; unmapped:    2.5 MiB;  131.5 MiB cum
    82 pages *     35 spans ~   22.4 MiB; 10052.5 MiB cum; unmapped:    3.8 MiB;  135.4 MiB cum
    83 pages *      1 spans ~    0.6 MiB; 10053.1 MiB cum; unmapped:    0.6 MiB;  136.0 MiB cum
    84 pages *     24 spans ~   15.8 MiB; 10068.9 MiB cum; unmapped:    3.3 MiB;  139.3 MiB cum
    85 pages *      2 spans ~    1.3 MiB; 10070.2 MiB cum; unmapped:    0.0 MiB;  139.3 MiB cum
    86 pages *     37 spans ~   24.9 MiB; 10095.1 MiB cum; unmapped:    3.4 MiB;  142.7 MiB cum
    87 pages *      3 spans ~    2.0 MiB; 10097.1 MiB cum; unmapped:    2.0 MiB;  144.7 MiB cum
    88 pages *     30 spans ~   20.6 MiB; 10117.7 MiB cum; unmapped:    3.4 MiB;  148.1 MiB cum
    89 pages *      2 spans ~    1.4 MiB; 10119.1 MiB cum; unmapped:    1.4 MiB;  149.5 MiB cum
    90 pages *     19 spans ~   13.4 MiB; 10132.5 MiB cum; unmapped:    3.5 MiB;  153.0 MiB cum
    91 pages *      4 spans ~    2.8 MiB; 10135.3 MiB cum; unmapped:    2.8 MiB;  155.9 MiB cum
    92 pages *     23 spans ~   16.5 MiB; 10151.9 MiB cum; unmapped:    3.6 MiB;  159.5 MiB cum
    93 pages *      3 spans ~    2.2 MiB; 10154.0 MiB cum; unmapped:    2.2 MiB;  161.7 MiB cum
    94 pages *     17 spans ~   12.5 MiB; 10166.5 MiB cum; unmapped:    3.7 MiB;  165.3 MiB cum
    95 pages *      2 spans ~    1.5 MiB; 10168.0 MiB cum; unmapped:    1.5 MiB;  166.8 MiB cum
    96 pages *     15 spans ~   11.2 MiB; 10179.3 MiB cum; unmapped:    3.8 MiB;  170.6 MiB cum
    97 pages *      2 spans ~    1.5 MiB; 10180.8 MiB cum; unmapped:    1.5 MiB;  172.1 MiB cum
    98 pages *     15 spans ~   11.5 MiB; 10192.3 MiB cum; unmapped:    4.6 MiB;  176.7 MiB cum
    99 pages *      3 spans ~    2.3 MiB; 10194.6 MiB cum; unmapped:    2.3 MiB;  179.0 MiB cum
   100 pages *     20 spans ~   15.6 MiB; 10210.2 MiB cum; unmapped:    4.7 MiB;  183.7 MiB cum
   101 pages *      1 spans ~    0.8 MiB; 10211.0 MiB cum; unmapped:    0.8 MiB;  184.5 MiB cum
   102 pages *     18 spans ~   14.3 MiB; 10225.3 MiB cum; unmapped:    4.0 MiB;  188.5 MiB cum
   104 pages *     10 spans ~    8.1 MiB; 10233.5 MiB cum; unmapped:    3.2 MiB;  191.7 MiB cum
   106 pages *     14 spans ~   11.6 MiB; 10245.1 MiB cum; unmapped:    4.1 MiB;  195.8 MiB cum
   107 pages *      3 spans ~    2.5 MiB; 10247.6 MiB cum; unmapped:    2.5 MiB;  198.4 MiB cum
   108 pages *     11 spans ~    9.3 MiB; 10256.9 MiB cum; unmapped:    4.2 MiB;  202.6 MiB cum
   109 pages *      1 spans ~    0.9 MiB; 10257.7 MiB cum; unmapped:    0.9 MiB;  203.4 MiB cum
   110 pages *      8 spans ~    6.9 MiB; 10264.6 MiB cum; unmapped:    4.3 MiB;  207.7 MiB cum
   111 pages *      2 spans ~    1.7 MiB; 10266.3 MiB cum; unmapped:    1.7 MiB;  209.5 MiB cum
   112 pages *      5 spans ~    4.4 MiB; 10270.7 MiB cum; unmapped:    4.4 MiB;  213.8 MiB cum
   113 pages *      1 spans ~    0.9 MiB; 10271.6 MiB cum; unmapped:    0.9 MiB;  214.7 MiB cum
   114 pages *     12 spans ~   10.7 MiB; 10282.3 MiB cum; unmapped:    4.5 MiB;  219.2 MiB cum
   116 pages *     10 spans ~    9.1 MiB; 10291.3 MiB cum; unmapped:    5.4 MiB;  224.6 MiB cum
   117 pages *      1 spans ~    0.9 MiB; 10292.2 MiB cum; unmapped:    0.9 MiB;  225.5 MiB cum
   118 pages *      6 spans ~    5.5 MiB; 10297.8 MiB cum; unmapped:    4.6 MiB;  230.1 MiB cum
   120 pages *      7 spans ~    6.6 MiB; 10304.3 MiB cum; unmapped:    5.6 MiB;  235.8 MiB cum
   122 pages *      5 spans ~    4.8 MiB; 10309.1 MiB cum; unmapped:    4.8 MiB;  240.5 MiB cum
   124 pages *      6 spans ~    5.8 MiB; 10314.9 MiB cum; unmapped:    3.9 MiB;  244.4 MiB cum
   125 pages *      2 spans ~    2.0 MiB; 10316.9 MiB cum; unmapped:    2.0 MiB;  246.3 MiB cum
   126 pages *      2 spans ~    2.0 MiB; 10318.8 MiB cum; unmapped:    2.0 MiB;  248.3 MiB cum
>255   large *    251 spans ~  713.0 MiB; 11031.8 MiB cum; unmapped:    9.1 MiB;  257.4 MiB cum

For memory:

 bytes:                               57365397968
 ep_blob_num:                         30098210
 ep_blob_overhead:                    1514622591
 ep_item_num:                         11254
 ep_kv_size:                          50892806250
 ep_max_size:                         73400320000
 ep_mem_high_wat:                     62390272000
 ep_mem_low_wat:                      55050240000
 ep_mem_tracker_enabled:              true
 ep_oom_errors:                       0
 ep_overhead:                         2758090214
 ep_storedval_num:                    362772373
 ep_storedval_overhead:               1514622591
 ep_storedval_size:                   29021789840
 ep_tmp_oom_errors:                   0
 ep_value_size:                       25498739888
 mem_used:                            57365397968
 tcmalloc_current_thread_cache_bytes: 27647072
 tcmalloc_max_thread_cache_bytes:     33554432
 total_allocated_bytes:               57452351696
 total_fragmentation_bytes:           9894530864
 total_free_mapped_bytes:             11303051264
 total_free_unmapped_bytes:           269885440
 total_heap_bytes:                    78919819264

With the exact value from the RSS (77091732/1024/1024), I got 73.5 Go, which is a smaller difference, but still above the limit. Maybe there is a small overhead which explain that. The growth is slow, maybe it will finally stops, but I have to wait to know that.

See this section from the TCMalloc allocator stats:

TCMalloc (the memory allocator used by Couchbase) doesn’t immediately return memory to the OS, in case it needs the memory again in the near future (and it would have been a waste of time to return it to have to then re-allocate it).

As you can see from that extract; there’s 19.7GB in TCMalloc’s freelists; should account for the difference you are seeing. That’s quite a large amount, TCMalloc doesn’t normally keep back such a quantity of memory. Has the node been up a long time?

In terms of resolving this you can always swap-rebalance out the node with a new one (and then re-insert the original node after restarting couchbase-server if you wish) - this will clear out the TCMalloc heap.

Another option is an (undocumented, unsupported) method to request TCMalloc to release it’s freelist memory which you could look into. It’s unsupported as it can have an impact on the performance of any future writes to the database as the memory allocator may not have any already free memory available). Additionally it’s not tested by Couchbase QA afiak.

If you decide to give it a try, you’ll need to run the mcctl command on the specific node(s):

/opt/couchbase/bin/mcctl -h localhost:11210 set release_free_memory

Note that this only requests TCMalloc to release memory, it doesn’t always necessarily release all free memory…

OK, thanks the detailled explaination, it explains where the missing RAM space is used. The server is not running for a long time, I restarted the whole cluster on tuesday with consecutive rebalance, to release the memory. I think TCMalloc doesn’t release the memory because of the heavy traffic received by the cluster. I create tens of millions new documents per day, and compaction is running most of the time. Currently only write operations are performed, no read.

Thanks for the tips to force TCMalloc to release memory, it can help, I will test it if the memory usage don’t stop growing.

Memory usage is stable for a while. Memory overhead usage is explained, thanks again for your help.

We seems to have similar VSZ crazyness:

ps -o pid,rss,vsz,args:200 -p $(pgrep beam.smp) PID RSS VSZ COMMAND 17891 2756708 666722476 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e10240 18042 26888 889340 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname 37400 27348 1348780 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -ho

The machine have 32Gb:

free -m total used free shared buffers cached Mem: 32170 31668 501 30 325 15336 -/+ buffers/cache: 16007 16162 Swap: 16143 0 16143

So the first process seems to use 600Gb of virtual memory…this is something really really strange…

This is a node from a 4 node cluster, other nodes have similar behaviour, one of the node I restarted this morning have:

ps -o pid,rss,vsz,args:400 -p $(pgrep beam.smp) PID RSS VSZ COMMAND 40502 25912 1282068 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -ho 40536 2148212 481494160 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e10240 40632 27004 889596 /opt/couchbase/lib/erlang/erts-5.10.4/bin/beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progname