*** glibc detected *** /usr/bin/node: corrupted double-linked list: 0x0000000003374210 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75dee)[0x7fa3ae9c0dee]
/lib64/libc.so.6(+0x7a5a4)[0x7fa3ae9c55a4]
/lib64/libc.so.6(__libc_malloc+0x5c)[0x7fa3ae9c5aac]
/usr/bin/node(_ZN2v88internal13JSArrayBuffer19SetupAllocatingDataENS0_6HandleIS1_EEPNS0_7IsolateEmbNS0_10SharedFlagE+0x63)[0xcf4123]
/usr/bin/node[0x988ff3]
/usr/bin/node[0x99662e]
[0x1f9663c092a7]
======= Memory map: ========
00400000-01aa4000 r-xp 00000000 fc:00 3692482 /usr/bin/node
01ca4000-01cbd000 rw-p 016a4000 fc:00 3692482 /usr/bin/node
01cbd000-01cd1000 rw-p 00000000 00:00 0
0325a000-0523a000 rw-p 00000000 00:00 0 [heap]
b45ae00000-b45af00000 rw-p 00000000 00:00 0
1713f100000-1713f200000 rw-p 00000000 00:00 0
1c3b9a00000-1c3b9b00000 rw-p 00000000 00:00 0
352e8300000-352e8314000 rw-p 00000000 00:00 0
394f9a00000-394f9b00000 rw-p 00000000 00:00 0
47980d00000-47980e00000 rw-p 00000000 00:00 0
4b477300000-4b477400000 rw-p 00000000 00:00 0
4d479000000-4d479100000 rw-p 00000000 00:00 0
542c4600000-542c4700000 rw-p 00000000 00:00 0
59d68400000-59d68500000 rw-p 00000000 00:00 0
5ed9f800000-5ed9f900000 rw-p 00000000 00:00 0
603f3200000-603f3300000 rw-p 00000000 00:00 0
7c7f0200000-7c7f02f5000 rw-p 00000000 00:00 0
8069b100000-8069b200000 rw-p 00000000 00:00 0
82d70500000-82d70600000 rw-p 00000000 00:00 0
83c42b00000-83c42c00000 rw-p 00000000 00:00 0
911eee00000-911eef00000 rw-p 00000000 00:00 0
931a5300000-931a5400000 rw-p 00000000 00:00 0
93baeb00000-93baec00000 rw-p 00000000 00:00 0
a3184e00000-a3184f00000 rw-p 00000000 00:00 0
a535d600000-a535d700000 rw-p 00000000 00:00 0
a5b78e00000-a5b78f00000 rw-p 00000000 00:00 0
aa87b200000-aa87b300000 rw-p 00000000 00:00 0
b882a800000-b882a900000 rw-p 00000000 00:00 0
b8a5f100000-b8a5f200000 rw-p 00000000 00:00 0
c08d1f00000-c08d2000000 rw-p 00000000 00:00 0
c2d36200000-c2d36300000 rw-p 00000000 00:00 0
daf2b600000-daf2b700000 rw-p 00000000 00:00 0
ec3b4f00000-ec3b5000000 rw-p 00000000 00:00 0
ef769500000-ef769600000 rw-p 00000000 00:00 0
efaea000000-efaea100000 rw-p 00000000 00:00 0
fa2a1d00000-fa2a1e00000 rw-p 00000000 00:00 0
fde26b00000-fde26c00000 rw-p 00000000 00:00 0
ff75cf00000-ff75d000000 rw-p 00000000 00:00 0
1036db700000-1036db800000 rw-p 00000000 00:00 0
10b95f500000-10b95f600000 rw-p 00000000 00:00 0
122980100000-122980200000 rw-p 00000000 00:00 0
1255b6500000-1255b6600000 rw-p 00000000 00:00 0
126040400000-126040500000 rw-p 00000000 00:00 0
128a89700000-128a89800000 rw-p 00000000 00:00 0
12f83c400000-12f83c500000 rw-p 00000000 00:00 0
14158a600000-14158a700000 rw-p 00000000 00:00 0
144580400000-144580500000 rw-p 00000000 00:00 0
16fe06e00000-16fe06f00000 rw-p 00000000 00:00 0
17616da00000-17616db00000 rw-p 00000000 00:00 0
17a926da1000-17a9271a1000 rw-p 00000000 00:00 0
182a21900000-182a21a00000 rw-p 00000000 00:00 0
187a74400000-187a74500000 rw-p 00000000 00:00 0
190b13f00000-190b14000000 rw-p 00000000 00:00 0
195a17000000-195a17100000 rw-p 00000000 00:00 0
195cafe00000-195caff00000 rw-p 00000000 00:00 0
19fb07000000-19fb07100000 rw-p 00000000 00:00 0
1a0680200000-1a0680300000 rw-p 00000000 00:00 0
1ae216b00000-1ae216c00000 rw-p 00000000 00:00 0
1b7574c00000-1b7574d00000 rw-p 00000000 00:00 0
1bd14e100000-1bd14e200000 rw-p 00000000 00:00 0
1bed25600000-1bed25700000 rw-p 00000000 00:00 0
1c1812800000-1c1812900000 rw-p 00000000 00:00 0
1c2245100000-1c2245200000 rw-p 00000000 00:00 0
1c6d11800000-1c6d11900000 rw-p 00000000 00:00 0
1cedaef00000-1cedaf000000 rw-p 00000000 00:00 0
1d10d9700000-1d10d9800000 rw-p 00000000 00:00 0
1d4415600000-1d4415700000 rw-p 00000000 00:00 0
1d5836500000-1d5836600000 rw-p 00000000 00:00 0
1dc03ba00000-1dc03bb00000 rw-p 00000000 00:00 0
1e1a2a300000-1e1a2a400000 rw-p 00000000 00:00 0
1e873e800000-1e873e900000 rw-p 00000000 00:00 0
1e92c1400000-1e92c1500000 rw-p 00000000 00:00 0
1ee1ed500000-1ee1ed600000 rw-p 00000000 00:00 0
1ef866100000-1ef866200000 rw-p 00000000 00:00 0
1f5da8800000-1f5da8900000 rw-p 00000000 00:00 0
1f899cd00000-1f899ce00000 rw-p 00000000 00:00 0
1f96638fc000-1f9663900000 ---p 00000000 00:00 0
1f9663900000-1f9663905000 rw-p 00000000 00:00 0
1f9663905000-1f9663906000 ---p 00000000 00:00 0
1f9663906000-1f9663908000 rwxp 00000000 00:00 0
1f9663908000-1f9663a00000 ---p 00000000 00:00 0
1f9663a00000-1f9663a05000 rw-p 00000000 00:00 0
1f9663a05000-1f9663a06000 ---p 00000000 00:00 0
1f9663a06000-1f9663a08000 rwxp 00000000 00:00 0
1f9663a08000-1f9663b00000 ---p 00000000 00:00 0
1f9663b00000-1f9663b05000 rw-p 00000000 00:00 0
1f9663b05000-1f9663b06000 ---p 00000000 00:00 0
1f9663b06000-1f9663b08000 rwxp 00000000 00:00 0
1f9663b08000-1f9663c00000 ---p 00000000 00:00 0
1f9663c00000-1f9663c05000 rw-p 00000000 00:00 0
1f9663c05000-1f9663c06000 ---p 00000000 00:00 0
1f9663c06000-1f9663c4b000 rwxp 00000000 00:00 0
1f9663c4b000-1f9664300000 ---p 00000000 00:00 0
1f9664300000-1f9664305000 rw-p 00000000 00:00 0
1f9664305000-1f9664306000 ---p 00000000 00:00 0
1f9664306000-1f96643ff000 rwxp 00000000 00:00 0
1f96643ff000-1f9664400000 ---p 00000000 00:00 0
1f9664400000-1f9664405000 rw-p 00000000 00:00 0
1f9664405000-1f9664406000 ---p 00000000 00:00 0
1f9664406000-1f96644ff000 rwxp 00000000 00:00 0
1f96644ff000-1f9664500000 ---p 00000000 00:00 0
1f9664500000-1f9664505000 rw-p 00000000 00:00 0
1f9664505000-1f9664506000 ---p 00000000 00:00 0
1f9664506000-1f96645ff000 rwxp 00000000 00:00 0
1f96645ff000-1f9664600000 ---p 00000000 00:00 0
1f9664600000-1f9664605000 rw-p 00000000 00:00 0
1f9664605000-1f9664606000 ---p 00000000 00:00 0
1f9664606000-1f96646ff000 rwxp 00000000 00:00 0
1f96646ff000-1f9664700000 ---p 00000000 00:00 0
1f9664700000-1f9664705000 rw-p 00000000 00:00 0
1f9664705000-1f9664706000 ---p 00000000 00:00 0
1f9664706000-1f96647ff000 rwxp 00000000 00:00 0
1f96647ff000-1f96838fc000 ---p 00000000 00:00 0
1f9e2d700000-1f9e2d800000 rw-p 00000000 00:00 0
21e20d400000-21e20d500000 rw-p 00000000 00:00 0
21e986100000-21e986200000 rw-p 00000000 00:00 0
21f1dfe00000-21f1dff00000 rw-p 00000000 00:00 0
21f9e0700000-21f9e0800000 rw-p 00000000 00:00 0
228adc400000-228adc500000 rw-p 00000000 00:00 0
22a3e5100000-22a3e5200000 rw-p 00000000 00:00 0
22b215b00000-22b215c00000 rw-p 00000000 00:00 0
230b33900000-230b33a00000 rw-p 00000000 00:00 0
230dcc500000-230dcc600000 rw-p 00000000 00:00 0
2319ae500000-2319ae600000 rw-p 00000000 00:00 0
2359e9a00000-2359e9b00000 rw-p 00000000 00:00 0
24fe77e00000-24fe77f00000 rw-p 00000000 00:00 0
250939d00000-250939e00000 rw-p 00000000 00:00 0
263dfdc00000-263dfdd00000 rw-p 00000000 00:00 0
270444500000-270444600000 rw-p 00000000 00:00 0
2729eee00000-2729eef00000 rw-p 00000000 00:00 0
272b1d800000-272b1d900000 rw-p 00000000 00:00 0
274113500000-274113600000 rw-p 00000000 00:00 0
27760e100000-27760e200000 rw-p 00000000 00:00 0
27981cf00000-27981d000000 rw-p 00000000 00:00 0
27eee0200000-27eee0300000 rw-p 00000000 00:00 0
280616700000-280616800000 rw-p 00000000 00:00 0
280ea5900000-280ea5a0c000 rw-p 00000000 00:00 0
284232c00000-284232d00000 rw-p 00000000 00:00 0
284625200000-284625300000 rw-p 00000000 00:00 0
284e22300000-284e22400000 rw-p 00000000 00:00 0
28b839a00000-28b839b00000 rw-p 00000000 00:00 0
28e662d00000-28e662e00000 rw-p 00000000 00:00 0
2a2bd9e00000-2a2bd9f00000 rw-p 00000000 00:00 0
2ad2fc100000-2ad2fc200000 rw-p 00000000 00:00 0
2c2a5c500000-2c2a5c600000 rw-p 00000000 00:00 0
2ca9f8000000-2ca9f8100000 rw-p 00000000 00:00 0
2cba92500000-2cba92600000 rw-p 00000000 00:00 0
2cbf53525000-2cbf53540000 ---p 00000000 00:00 0
2cbf53540000-2cbf53560000 rw-p 00000000 00:00 0
2cbf53560000-2cbf53565000 ---p 00000000 00:00 0
2cf46cd00000-2cf46ce00000 rw-p 00000000 00:00 0
2d9107e00000-2d9107f00000 rw-p 00000000 00:00 0
2e77e2400000-2e77e2500000 rw-p 00000000 00:00 0
2e91f0100000-2e91f0200000 rw-p 00000000 00:00 0
303e04600000-303e04700000 rw-p 00000000 00:00 0
30b22a800000-30b22a900000 rw-p 00000000 00:00 0
31074db00000-31074dc00000 rw-p 00000000 00:00 0
310adba00000-310adbb00000 rw-p 00000000 00:00 0
31cf66800000-31cf66900000 rw-p 00000000 00:00 0
321e12800000-321e12900000 rw-p 00000000 00:00 0
33dc64500000-33dc64600000 rw-p 00000000 00:00 0
34d28f200000-34d28f300000 rw-p 00000000 00:00 0
34ecc7600000-34ecc7700000 rw-p 00000000 00:00 0
35285bc00000-35285bd00000 rw-p 00000000 00:00 0
35e0cf000000-35e0cf100000 rw-p 00000000 00:00 0
360d44300000-360d44400000 rw-p 00000000 00:00 0
368fdf800000-368fdf900000 rw-p 00000000 00:00 0
38ae14300000-38ae14400000 rw-p 00000000 00:00 0
394eb5300000-394eb5400000 rw-p 00000000 00:00 0
3a46db900000-3a46dba00000 rw-p 00000000 00:00 0
3a47b1400000-3a47b1500000 rw-p 00000000 00:00 0
3a4d0a000000-3a4d0a100000 rw-p 00000000 00:00 0
3ad2c6900000-3ad2c6a00000 rw-p 00000000 00:00 0
3ae46c600000-3ae46c700000 rw-p 00000000 00:00 0
3b41a6a00000-3b41a6b00000 rw-p 00000000 00:00 0
3b6719f00000-3b671a000000 rw-p 00000000 00:00 0
3bfd30f00000-3bfd31000000 rw-p 00000000 00:00 0
3c6e83600000-3c6e83700000 rw-p 00000000 00:00 0
3c77eb500000-3c77eb600000 rw-p 00000000 00:00 0
3cf4c6600000-3cf4c6700000 rw-p 00000000 00:00 0
3d4ee7400000-3d4ee7500000 rw-p 00000000 00:00 0
3de6b2d00000-3de6b2e00000 rw-p 00000000 00:00 0
3e625cb00000-3e625cc00000 rw-p 00000000 00:00 0
3e871b400000-3e871b500000 rw-p 00000000 00:00 0
3f03dbf00000-3f03dc000000 rw-p 00000000 00:00 0
3fb02c400000-3fb02c500000 rw-p 00000000 00:00 0
7fa384000000-7fa384021000 rw-p 00000000 00:00 0
7fa384021000-7fa388000000 ---p 00000000 00:00 0
7fa38c000000-7fa38c021000 rw-p 00000000 00:00 0
7fa38c021000-7fa390000000 ---p 00000000 00:00 0
7fa390000000-7fa390021000 rw-p 00000000 00:00 0
7fa390021000-7fa394000000 ---p 00000000 00:00 0
7fa394000000-7fa394021000 rw-p 00000000 00:00 0
7fa394021000-7fa398000000 ---p 00000000 00:00 0
7fa398000000-7fa398237000 rw-p 00000000 00:00 0
7fa398237000-7fa39c000000 ---p 00000000 00:00 0
7fa39df02000-7fa39df18000 r-xp 00000000 fc:00 3817900 /lib64/libresolv-2.12.so
7fa39df18000-7fa39e118000 ---p 00016000 fc:00 3817900 /lib64/libresolv-2.12.so
7fa39e118000-7fa39e119000 r--p 00016000 fc:00 3817900 /lib64/libresolv-2.12.so
7fa39e119000-7fa39e11a000 rw-p 00017000 fc:00 3817900 /lib64/libresolv-2.12.so
7fa39e11a000-7fa39e11c000 rw-p 00000000 00:00 0
7fa39e11c000-7fa39e121000 r-xp 00000000 fc:00 3817888 /lib64/libnss_dns-2.12.so
7fa39e121000-7fa39e320000 ---p 00005000 fc:00 3817888 /lib64/libnss_dns-2.12.so
7fa39e320000-7fa39e321000 r--p 00004000 fc:00 3817888 /lib64/libnss_dns-2.12.so
7fa39e321000-7fa39e322000 rw-p 00005000 fc:00 3817888 /lib64/libnss_dns-2.12.so
7fa39e322000-7fa39e32f000 r-xp 00000000 fc:00 3817890 /lib64/libnss_files-2.12.so
7fa39e32f000-7fa39e52e000 ---p 0000d000 fc:00 3817890 /lib64/libnss_files-2.12.so
7fa39e52e000-7fa39e52f000 r--p 0000c000 fc:00 3817890 /lib64/libnss_files-2.12.so
7fa39e52f000-7fa39e530000 rw-p 0000d000 fc:00 3817890 /lib64/libnss_files-2.12.so
7fa39e530000-7fa39e5f2000 r-xp 00000000 fc:00 3950942 /npm_prebuild/vodContentIngest/node_modules/couchbase/build/Release/couchbase_impl.node
7fa39e5f2000-7fa39e7f2000 ---p 000c2000 fc:00 3950942 /npm_prebuild/vodContentIngest/node_modules/couchbase/build/Release/couchbase_impl.node
7fa39e7f2000-7fa39e7fd000 rw-p 000c2000 fc:00 3950942 /npm_prebuild/vodContentIngest/node_modules/couchbase/build/Release/couchbase_impl.node
7fa39e7fd000-7fa39e7fe000 ---p 00000000 00:00 0
7fa39e7fe000-7fa39effe000 rw-p 00000000 00:00 0
7fa39effe000-7fa39efff000 ---p 00000000 00:00 0
7fa39efff000-7fa39f7ff000 rw-p 00000000 00:00 0
7fa39f7ff000-7fa39f800000 ---p 00000000 00:00 0
7fa39f800000-7fa3a0000000 rw-p 00000000 00:00 0
7fa3a0000000-7fa3a026c000 rw-p 00000000 00:00 0
7fa3a026c000-7fa3a4000000 ---p 00000000 00:00 0
7fa3a4000000-7fa3a41b7000 rw-p 00000000 00:00 0
7fa3a41b7000-7fa3a8000000 ---p 00000000 00:00 0
7fa3a8000000-7fa3a8167000 rw-p 00000000 00:00 0
7fa3a8167000-7fa3ac000000 ---p 00000000 00:00 0
7fa3ac146000-7fa3ac147000 ---p 00000000 00:00 0
7fa3ac147000-7fa3ac947000 rw-p 00000000 00:00 0
7fa3ac947000-7fa3ac948000 ---p 00000000 00:00 0
7fa3ac948000-7fa3ad148000 rw-p 00000000 00:00 0
7fa3ad148000-7fa3ad149000 ---p 00000000 00:00 0
7fa3ad149000-7fa3ad949000 rw-p 00000000 00:00 0
7fa3ad949000-7fa3ad94a000 ---p 00000000 00:00 0
7fa3ad94a000-7fa3ae14a000 rw-p 00000000 00:00 0
7fa3ae14a000-7fa3ae14b000 ---p 00000000 00:00 0
7fa3ae14b000-7fa3ae94b000 rw-p 00000000 00:00 0
7fa3ae94b000-7fa3aead5000 r-xp 00000000 fc:00 3817872 /lib64/libc-2.12.so
7fa3aead5000-7fa3aecd5000 ---p 0018a000 fc:00 3817872 /lib64/libc-2.12.so
7fa3aecd5000-7fa3aecd9000 r--p 0018a000 fc:00 3817872 /lib64/libc-2.12.so
7fa3aecd9000-7fa3aecdb000 rw-p 0018e000 fc:00 3817872 /lib64/libc-2.12.so
7fa3aecdb000-7fa3aecdf000 rw-p 00000000 00:00 0
7fa3aecdf000-7fa3aecf6000 r-xp 00000000 fc:00 3817898 /lib64/libpthread-2.12.so
7fa3aecf6000-7fa3aeef6000 ---p 00017000 fc:00 3817898 /lib64/libpthread-2.12.so
7fa3aeef6000-7fa3aeef7000 r--p 00017000 fc:00 3817898 /lib64/libpthread-2.12.so
7fa3aeef7000-7fa3aeef8000 rw-p 00018000 fc:00 3817898 /lib64/libpthread-2.12.so
7fa3aeef8000-7fa3aeefc000 rw-p 00000000 00:00 0
7fa3aeefc000-7fa3aef12000 r-xp 00000000 fc:00 3688456 /lib64/libgcc_s-4.4.7-20120601.so.1
7fa3aef12000-7fa3af111000 ---p 00016000 fc:00 3688456 /lib64/libgcc_s-4.4.7-20120601.so.1
7fa3af111000-7fa3af112000 rw-p 00015000 fc:00 3688456 /lib64/libgcc_s-4.4.7-20120601.so.1
7fa3af112000-7fa3af195000 r-xp 00000000 fc:00 3817882 /lib64/libm-2.12.so
7fa3af195000-7fa3af394000 ---p 00083000 fc:00 3817882 /lib64/libm-2.12.so
7fa3af394000-7fa3af395000 r--p 00082000 fc:00 3817882 /lib64/libm-2.12.so
7fa3af395000-7fa3af396000 rw-p 00083000 fc:00 3817882 /lib64/libm-2.12.so
7fa3af396000-7fa3af47e000 r-xp 00000000 fc:00 3689556 /usr/lib64/libstdc++.so.6.0.13
7fa3af47e000-7fa3af67e000 ---p 000e8000 fc:00 3689556 /usr/lib64/libstdc++.so.6.0.13
7fa3af67e000-7fa3af685000 r--p 000e8000 fc:00 3689556 /usr/lib64/libstdc++.so.6.0.13
7fa3af685000-7fa3af687000 rw-p 000ef000 fc:00 3689556 /usr/lib64/libstdc++.so.6.0.13
7fa3af687000-7fa3af69c000 rw-p 00000000 00:00 0
7fa3af69c000-7fa3af6a3000 r-xp 00000000 fc:00 3817902 /lib64/librt-2.12.so
7fa3af6a3000-7fa3af8a2000 ---p 00007000 fc:00 3817902 /lib64/librt-2.12.so
7fa3af8a2000-7fa3af8a3000 r--p 00006000 fc:00 3817902 /lib64/librt-2.12.so
7fa3af8a3000-7fa3af8a4000 rw-p 00007000 fc:00 3817902 /lib64/librt-2.12.so
7fa3af8a4000-7fa3af8a6000 r-xp 00000000 fc:00 3817880 /lib64/libdl-2.12.so
7fa3af8a6000-7fa3afaa6000 ---p 00002000 fc:00 3817880 /lib64/libdl-2.12.so
7fa3afaa6000-7fa3afaa7000 r--p 00002000 fc:00 3817880 /lib64/libdl-2.12.so
7fa3afaa7000-7fa3afaa8000 rw-p 00003000 fc:00 3817880 /lib64/libdl-2.12.so
7fa3afaa8000-7fa3afac8000 r-xp 00000000 fc:00 3817861 /lib64/ld-2.12.so
7fa3afcb8000-7fa3afcbf000 rw-p 00000000 00:00 0
7fa3afcc1000-7fa3afcc2000 rw-p 00000000 00:00 0
7fa3afcc2000-7fa3afcc3000 ---p 00000000 00:00 0
7fa3afcc3000-7fa3afcc7000 rw-p 00000000 00:00 0
7fa3afcc7000-7fa3afcc9000 r--p 0001f000 fc:00 3817861 /lib64/ld-2.12.so
7fa3afcc9000-7fa3afcca000 rw-p 00021000 fc:00 3817861 /lib64/ld-2.12.so
7fa3afcca000-7fa3afccb000 rw-p 00000000 00:00 0
7ffc0b565000-7ffc0b587000 rw-p 00000000 00:00 0 [stack]
7ffc0b5d1000-7ffc0b5d3000 r--p 00000000 00:00 0 [vvar]
7ffc0b5d3000-7ffc0b5d5000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]Hey @aterem,
According to this stack trace, the crash did not occur within our SDK. What lead you to believe that it was a flaw in the SDK that cuased your application to crash as it did? Additionally, could you provide more details about which Node.js version, SDK version, Server version, etc… you are using?
Cheers, Brett
The:
7fa39e7f2000-7fa39e7fd000 rw-p 000c2000 fc:00 3950942 /npm_prebuild/vodContentIngest/node_modules/couchbase/build/Release/couchbase_impl.node
Kind of gave it away…
Reproduce 100% of times in our CI. Simply use the function setRemove on stress tests and you will see it.
I manage to work around it by implementing set using normal json object and then just using mutateIn.upsert with the key as the path. This way if I add the same value multiple times it will just be added it once (because keys in object are uniqe), and if I want to delete it, I don’t need to fetch the whole document and reUpsert it. This solve the issue described at:
https://github.com/couchbase/couchnode/blob/master/lib/bucket.js#L2372
So it is not just a performance issue but also more than that