Moxi 2.5.0_1059_rel on Ubuntu 12.04 LTS 64-bit crashing with SIGABRT due to buffer overflow

Hello,

I am struggling updating moxi-server from 1.8.1 to 2.5.0 on Ubuntu 12.04 LTS. I am using the official package. I have tried some debugging to rule out misconfiguration as the source of the problem. I have tried figuring it out from moxi-server sources with no success.

I have not found any similar reports and finally decide to post here. I am not sure, whether this is an actual bug or some unmet dependency. I will appreciate any help. Meanwhile, I am staying with 1.8.1.

Here is what is happening:

I am starting moxi with: $ sudo /opt/moxi/bin/moxi -vvv -X -u moxi -P /var/run/moxi-server/mox-server.pid -s /var/run/moxi-server/moxi-server.sock -a 766 -Z /opt/moxi/etc/moxi.cfg

Then I am connecting to the socket using socat: sudo -u moxi socat - UNIX-CONNECT:/var/run/moxi-server/moxi-server.sock and send B:10.8.64.91:11000 stats. While it works fine with 1.8.1, 2.5.0 is crashing with SIGABRT due to buffer overflow.

Here is what I am getting in stderr:

2015-10-05 06:14:15: (memcached.c.4764) moxi log, mode=1, file=(null)
2015-10-05 06:14:15: (slabs.c.128) slab class   1: chunk size     96 perslab 10922
2015-10-05 06:14:15: (slabs.c.128) slab class   2: chunk size    120 perslab  8738
2015-10-05 06:14:15: (slabs.c.128) slab class   3: chunk size    152 perslab  6898
2015-10-05 06:14:15: (slabs.c.128) slab class   4: chunk size    192 perslab  5461
2015-10-05 06:14:15: (slabs.c.128) slab class   5: chunk size    240 perslab  4369
2015-10-05 06:14:15: (slabs.c.128) slab class   6: chunk size    304 perslab  3449
2015-10-05 06:14:15: (slabs.c.128) slab class   7: chunk size    384 perslab  2730
2015-10-05 06:14:15: (slabs.c.128) slab class   8: chunk size    480 perslab  2184
2015-10-05 06:14:15: (slabs.c.128) slab class   9: chunk size    600 perslab  1747
2015-10-05 06:14:15: (slabs.c.128) slab class  10: chunk size    752 perslab  1394
2015-10-05 06:14:15: (slabs.c.128) slab class  11: chunk size    944 perslab  1110
2015-10-05 06:14:15: (slabs.c.128) slab class  12: chunk size   1184 perslab   885
2015-10-05 06:14:15: (slabs.c.128) slab class  13: chunk size   1480 perslab   708
2015-10-05 06:14:15: (slabs.c.128) slab class  14: chunk size   1856 perslab   564
2015-10-05 06:14:15: (slabs.c.128) slab class  15: chunk size   2320 perslab   451
2015-10-05 06:14:15: (slabs.c.128) slab class  16: chunk size   2904 perslab   361
2015-10-05 06:14:15: (slabs.c.128) slab class  17: chunk size   3632 perslab   288
2015-10-05 06:14:15: (slabs.c.128) slab class  18: chunk size   4544 perslab   230
2015-10-05 06:14:15: (slabs.c.128) slab class  19: chunk size   5680 perslab   184
2015-10-05 06:14:15: (slabs.c.128) slab class  20: chunk size   7104 perslab   147
2015-10-05 06:14:15: (slabs.c.128) slab class  21: chunk size   8880 perslab   118
2015-10-05 06:14:15: (slabs.c.128) slab class  22: chunk size  11104 perslab    94
2015-10-05 06:14:15: (slabs.c.128) slab class  23: chunk size  13880 perslab    75
2015-10-05 06:14:15: (slabs.c.128) slab class  24: chunk size  17352 perslab    60
2015-10-05 06:14:15: (slabs.c.128) slab class  25: chunk size  21696 perslab    48
2015-10-05 06:14:15: (slabs.c.128) slab class  26: chunk size  27120 perslab    38
2015-10-05 06:14:15: (slabs.c.128) slab class  27: chunk size  33904 perslab    30
2015-10-05 06:14:15: (slabs.c.128) slab class  28: chunk size  42384 perslab    24
2015-10-05 06:14:15: (slabs.c.128) slab class  29: chunk size  52984 perslab    19
2015-10-05 06:14:15: (slabs.c.128) slab class  30: chunk size  66232 perslab    15
2015-10-05 06:14:15: (slabs.c.128) slab class  31: chunk size  82792 perslab    12
2015-10-05 06:14:15: (slabs.c.128) slab class  32: chunk size 103496 perslab    10
2015-10-05 06:14:15: (slabs.c.128) slab class  33: chunk size 129376 perslab     8
2015-10-05 06:14:15: (slabs.c.128) slab class  34: chunk size 161720 perslab     6
2015-10-05 06:14:15: (slabs.c.128) slab class  35: chunk size 202152 perslab     5
2015-10-05 06:14:15: (slabs.c.128) slab class  36: chunk size 252696 perslab     4
2015-10-05 06:14:15: (slabs.c.128) slab class  37: chunk size 315872 perslab     3
2015-10-05 06:14:15: (slabs.c.128) slab class  38: chunk size 394840 perslab     2
2015-10-05 06:14:15: (slabs.c.128) slab class  39: chunk size 493552 perslab     2
2015-10-05 06:14:15: (thread.c.264) worker_libevent thread_id 140493983012608
2015-10-05 06:14:15: (thread.c.264) worker_libevent thread_id 140493974619904
2015-10-05 06:14:15: (thread.c.264) worker_libevent thread_id 140493966227200
2015-10-05 06:14:15: (thread.c.264) worker_libevent thread_id 140493957834496
2015-10-05 06:14:17: (cproxy_config.c.912) init_string cycle: 200
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_max: 1024
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_conn_max: 8
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_weight: 0
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_retry: 1
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_protocol: 8
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_timeout: 5000
2015-10-05 06:14:17: (cproxy_config.c.912) init_string downstream_conn_queue_timeout: 200
2015-10-05 06:14:17: (cproxy_config.c.912) init_string wait_queue_timeout: 200
2015-10-05 06:14:17: (cproxy_config.c.912) init_string connect_timeout: 400
2015-10-05 06:14:17: (cproxy_config.c.912) init_string auth_timeout: 100
2015-10-05 06:14:17: (cproxy_config.c.912) init_string time_stats: 0
2015-10-05 06:14:17: (cproxy_config.c.912) init_string mcs_opts: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string connect_max_errors: 5
2015-10-05 06:14:17: (cproxy_config.c.912) init_string connect_retry_interval: 30000
2015-10-05 06:14:17: (cproxy_config.c.912) init_string front_cache_max: 200
2015-10-05 06:14:17: (cproxy_config.c.912) init_string front_cache_lifespan: 0
2015-10-05 06:14:17: (cproxy_config.c.912) init_string front_cache_spec: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string front_cache_unspec: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string key_stats_max: 4000
2015-10-05 06:14:17: (cproxy_config.c.912) init_string key_stats_lifespan: 0
2015-10-05 06:14:17: (cproxy_config.c.912) init_string key_stats_spec: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string key_stats_unspec: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string optimize_set: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string usr: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string host: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string port: 0
2015-10-05 06:14:17: (cproxy_config.c.912) init_string bucket: 
2015-10-05 06:14:17: (cproxy_config.c.912) init_string port_listen: 11211
2015-10-05 06:14:17: (cproxy_config.c.912) init_string default_bucket_name: [  ]
2015-10-05 06:14:17: (cproxy.c.153) cproxy_create on port -2, name default, config mcmux_config
2015-10-05 06:14:17: (cproxy.c.292) cproxy_listen on port -2, downstream mcmux_config
2015-10-05 06:14:17: (memcached.c.398) <43 server listening (auto-negotiate)
2015-10-05 06:14:17: (memcached.c.440) <43 initialized conn_funcs to default
2015-10-05 06:14:17: (cproxy.c.418) <43 cproxy listening on port -2
2015-10-05 06:14:17: (cproxy_config.c.411) moxi listening on -2 with 1 conns
2015-10-05 06:14:22: (memcached.c.3445) 43: drive_machine conn_listening
2015-10-05 06:14:22: (memcached.c.403) <44 new auto-negotiating client connection
2015-10-05 06:14:22: (cproxy.c.524) <44 assigning to first bucket
2015-10-05 06:15:59: (memcached.c.3445) 44: drive_machine conn_new_cmd
2015-10-05 06:15:59: (memcached.c.670) 44: going from conn_new_cmd to conn_waiting
2015-10-05 06:15:59: (memcached.c.3445) 44: drive_machine conn_waiting
2015-10-05 06:15:59: (memcached.c.670) 44: going from conn_waiting to conn_read
2015-10-05 06:15:59: (memcached.c.3445) 44: drive_machine conn_read
2015-10-05 06:15:59: (memcached.c.670) 44: going from conn_read to conn_parse_cmd
2015-10-05 06:15:59: (memcached.c.3445) 44: drive_machine conn_parse_cmd
2015-10-05 06:15:59: (memcached.c.3113) 44: Client using the proxy-upstream-ascii protocol
2015-10-05 06:15:59: (cproxy_protocol_a.c.34) <44 cproxy_process_upstream_ascii B:10.8.64.91:11000 stats
2015-10-05 06:15:59: (cproxy.c.2216) 44: pause_upstream_for_downstream
2015-10-05 06:15:59: (memcached.c.670) 44: going from conn_parse_cmd to conn_pause
2015-10-05 06:15:59: (cproxy.c.2269) wait_queue_timeout started
2015-10-05 06:15:59: (cproxy.c.1767) assign_downstream
2015-10-05 06:15:59: (cproxy.c.871) cproxy_add_downstream 0 1024
2015-10-05 06:15:59: (cproxy.c.1287) cproxy_create_downstream: mcmux_config, 0, 8
2015-10-05 06:15:59: (mcs.c.64) mcs_create using libmemcached
2015-10-05 06:15:59: (cproxy.c.989) -1: release_downstream
2015-10-05 06:15:59: (cproxy.c.1388) check_downstream_config 1
2015-10-05 06:15:59: (cproxy.c.1388) check_downstream_config 1
2015-10-05 06:15:59: (cproxy.c.1865) 44: assign_downstream, matched to upstream
2015-10-05 06:15:59: (cproxy.c.1939) 44: cproxy_forward prot 5 to prot 8
2015-10-05 06:15:59: (cproxy.c.1427) 44: cproxy_connect_downstream server_index -1 in 1
Aborted

Here is what I am getting in stdout:

*** buffer overflow detected ***: /opt/moxi/bin/../bin/moxi.actual terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fc74fa12db7]
/lib/x86_64-linux-gnu/libc.so.6(+0x109cb0)[0x7fc74fa11cb0]
/lib/x86_64-linux-gnu/libc.so.6(+0x108f96)[0x7fc74fa10f96]
/opt/moxi/bin/../bin/moxi.actual[0x41ddd9]
/opt/moxi/bin/../bin/moxi.actual[0x427b52]
/opt/moxi/bin/../bin/moxi.actual[0x41cc26]
/opt/moxi/bin/../bin/moxi.actual[0x422d4e]
/opt/moxi/bin/../bin/moxi.actual[0x40ddc4]
/opt/moxi/bin/../bin/moxi.actual[0x40e6b0]
/opt/moxi/bin/../bin/moxi.actual[0x443378]
/opt/moxi/bin/../bin/moxi.actual[0x444ff0]
/opt/moxi/bin/../bin/moxi.actual[0x413905]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc74fccee9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc74f9fc2ed]
======= Memory map: ========
00400000-0046d000 r-xp 00000000 07:fc 1054817                            /opt/moxi/bin/moxi.actual
0066c000-0066d000 r--p 0006c000 07:fc 1054817                            /opt/moxi/bin/moxi.actual
0066d000-00670000 rw-p 0006d000 07:fc 1054817                            /opt/moxi/bin/moxi.actual
00670000-00677000 rw-p 00000000 00:00 0 
01fc2000-02004000 rw-p 00000000 00:00 0                                  [heap]
7fc744000000-7fc744021000 rw-p 00000000 00:00 0 
7fc744021000-7fc748000000 ---p 00000000 00:00 0 
7fc74b51b000-7fc74b530000 r-xp 00000000 07:fc 262443                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc74b530000-7fc74b72f000 ---p 00015000 07:fc 262443                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc74b72f000-7fc74b730000 r--p 00014000 07:fc 262443                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc74b730000-7fc74b731000 rw-p 00015000 07:fc 262443                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc74b731000-7fc74b732000 ---p 00000000 00:00 0 
7fc74b732000-7fc74bf32000 rw-p 00000000 00:00 0                          [stack:29551]
7fc74bf32000-7fc74bf33000 ---p 00000000 00:00 0 
7fc74bf33000-7fc74c733000 rw-p 00000000 00:00 0                          [stack:29550]
7fc74c733000-7fc74c734000 ---p 00000000 00:00 0 
7fc74c734000-7fc74cf34000 rw-p 00000000 00:00 0                          [stack:29549]
7fc74cf34000-7fc74cf35000 ---p 00000000 00:00 0 
7fc74cf35000-7fc74d735000 rw-p 00000000 00:00 0                          [stack:29548]
7fc74d735000-7fc74d736000 ---p 00000000 00:00 0 
7fc74d736000-7fc74df36000 rw-p 00000000 00:00 0                          [stack:29547]
7fc74df36000-7fc74df42000 r-xp 00000000 07:fc 262457                     /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc74df42000-7fc74e141000 ---p 0000c000 07:fc 262457                     /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc74e141000-7fc74e142000 r--p 0000b000 07:fc 262457                     /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc74e142000-7fc74e143000 rw-p 0000c000 07:fc 262457                     /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc74e143000-7fc74e14d000 r-xp 00000000 07:fc 262459                     /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc74e14d000-7fc74e34d000 ---p 0000a000 07:fc 262459                     /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc74e34d000-7fc74e34e000 r--p 0000a000 07:fc 262459                     /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc74e34e000-7fc74e34f000 rw-p 0000b000 07:fc 262459                     /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc74e34f000-7fc74e366000 r-xp 00000000 07:fc 262454                     /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc74e366000-7fc74e565000 ---p 00017000 07:fc 262454                     /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc74e565000-7fc74e566000 r--p 00016000 07:fc 262454                     /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc74e566000-7fc74e567000 rw-p 00017000 07:fc 262454                     /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc74e567000-7fc74e569000 rw-p 00000000 00:00 0 
7fc74e569000-7fc74e571000 r-xp 00000000 07:fc 262455                     /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc74e571000-7fc74e770000 ---p 00008000 07:fc 262455                     /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc74e770000-7fc74e771000 r--p 00007000 07:fc 262455                     /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc74e771000-7fc74e772000 rw-p 00008000 07:fc 262455                     /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc74e772000-7fc74e774000 r-xp 00000000 07:fc 262439                     /lib/x86_64-linux-gnu/libdl-2.15.so
7fc74e774000-7fc74e974000 ---p 00002000 07:fc 262439                     /lib/x86_64-linux-gnu/libdl-2.15.so
7fc74e974000-7fc74e975000 r--p 00002000 07:fc 262439                     /lib/x86_64-linux-gnu/libdl-2.15.so
7fc74e975000-7fc74e976000 rw-p 00003000 07:fc 262439                     /lib/x86_64-linux-gnu/libdl-2.15.so
7fc74e976000-7fc74e979000 r-xp 00000000 07:fc 269393                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fc74e979000-7fc74eb78000 ---p 00003000 07:fc 269393                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fc74eb78000-7fc74eb79000 r--p 00002000 07:fc 269393                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fc74eb79000-7fc74eb7a000 rw-p 00003000 07:fc 269393                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7fc74eb7a000-7fc74eb8b000 r-xp 00000000 07:fc 140192                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fc74eb8b000-7fc74ed8a000 ---p 00011000 07:fc 140192                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fc74ed8a000-7fc74ed8b000 r--p 00010000 07:fc 140192                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fc74ed8b000-7fc74ed8c000 rw-p 00011000 07:fc 140192                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7fc74ed8c000-7fc74ed9c000 r-xp 00000000 07:fc 140194                     /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12
7fc74ed9c000-7fc74ef9b000 ---p 00010000 07:fc 140194                     /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12
7fc74ef9b000-7fc74ef9c000 r--p 0000f000 07:fc 140194                     /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12
7fc74ef9c000-7fc74ef9d000 rw-p 00010000 07:fc 140194                     /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12
7fc74ef9d000-7fc74f017000 r-xp 00000000 07:fc 269408                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fc74f017000-7fc74f217000 ---p 0007a000 07:fc 269408                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fc74f217000-7fc74f218000 r--p 0007a000 07:fc 269408                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fc74f218000-7fc74f21b000 rw-p 0007b000 07:fc 269408                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7fc74f21b000-7fc74f2cf000 r-xp 00000000 07:fc 140197                     /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8
7fc74f2cf000-7fc74f4cf000 ---p 000b4000 07:fc 140197                     /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8
7fc74f4cf000-7fc74f4d5000 r--p 000b4000 07:fc 140197                     /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8
7fc74f4d5000-7fc74f4d6000 rw-p 000ba000 07:fc 140197                     /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8
7fc74f4d6000-7fc74f4d7000 rw-p 00000000 00:00 0 
7fc74f4d7000-7fc74f4ef000 r-xp 00000000 07:fc 140231                     /usr/lib/x86_64-linux-gnu/librtmp.so.0
7fc74f4ef000-7fc74f6ef000 ---p 00018000 07:fc 140231                     /usr/lib/x86_64-linux-gnu/librtmp.so.0
7fc74f6ef000-7fc74f6f0000 r--p 00018000 07:fc 140231                     /usr/lib/x86_64-linux-gnu/librtmp.so.0
7fc74f6f0000-7fc74f6f1000 rw-p 00019000 07:fc 140231                     /usr/lib/x86_64-linux-gnu/librtmp.so.0
7fc74f6f1000-7fc74f707000 r-xp 00000000 07:fc 262483                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fc74f707000-7fc74f906000 ---p 00016000 07:fc 262483                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fc74f906000-7fc74f907000 r--p 00015000 07:fc 262483                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fc74f907000-7fc74f908000 rw-p 00016000 07:fc 262483                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7fc74f908000-7fc74fabd000 r-xp 00000000 07:fc 262433                     /lib/x86_64-linux-gnu/libc-2.15.so
7fc74fabd000-7fc74fcbc000 ---p 001b5000 07:fc 262433                     /lib/x86_64-linux-gnu/libc-2.15.so
7fc74fcbc000-7fc74fcc0000 r--p 001b4000 07:fc 262433                     /lib/x86_64-linux-gnu/libc-2.15.so
7fc74fcc0000-7fc74fcc2000 rw-p 001b8000 07:fc 262433                     /lib/x86_64-linux-gnu/libc-2.15.so
7fc74fcc2000-7fc74fcc7000 rw-p 00000000 00:00 0 
7fc74fcc7000-7fc74fcdf000 r-xp 00000000 07:fc 262468                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc74fcdf000-7fc74fede000 ---p 00018000 07:fc 262468                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc74fede000-7fc74fedf000 r--p 00017000 07:fc 262468                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc74fedf000-7fc74fee0000 rw-p 00018000 07:fc 262468                     /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc74fee0000-7fc74fee4000 rw-p 00000000 00:00 0 
7fc74fee4000-7fc74ffdf000 r-xp 00000000 07:fc 262447                     /lib/x86_64-linux-gnu/libm-2.15.so
7fc74ffdf000-7fc7501de000 ---p 000fb000 07:fc 262447                     /lib/x86_64-linux-gnu/libm-2.15.so
7fc7501de000-7fc7501df000 r--p 000fa000 07:fc 262447                     /lib/x86_64-linux-gnu/libm-2.15.so
7fc7501df000-7fc7501e0000 rw-p 000fb000 07:fc 262447                     /lib/x86_64-linux-gnu/libm-2.15.so
7fc7501e0000-7fc7501e7000 r-xp 00000000 07:fc 262471                     /lib/x86_64-linux-gnu/librt-2.15.so
7fc7501e7000-7fc7503e6000 ---p 00007000 07:fc 262471                     /lib/x86_64-linux-gnu/librt-2.15.so
7fc7503e6000-7fc7503e7000 r--p 00006000 07:fc 262471                     /lib/x86_64-linux-gnu/librt-2.15.so
7fc7503e7000-7fc7503e8000 rw-p 00007000 07:fc 262471                     /lib/x86_64-linux-gnu/librt-2.15.so
7fc7503e8000-7fc7503fc000 r-xp 00000000 07:fc 1054813                    /opt/moxi/lib/libmemcached.so.6
7fc7503fc000-7fc7505fb000 ---p 00014000 07:fc 1054813                    /opt/moxi/lib/libmemcached.so.6
7fc7505fb000-7fc7505fc000 r--p 00013000 07:fc 1054813                    /opt/moxi/lib/libmemcached.so.6
7fc7505fc000-7fc7505fd000 rw-p 00014000 07:fc 1054813                    /opt/moxi/lib/libmemcached.so.6
7fc7505fd000-7fc750607000 r-xp 00000000 07:fc 1054812                    /opt/moxi/lib/libvbucket.so.1
7fc750607000-7fc750806000 ---p 0000a000 07:fc 1054812                    /opt/moxi/lib/libvbucket.so.1
7fc750806000-7fc750807000 r--p 00009000 07:fc 1054812                    /opt/moxi/lib/libvbucket.so.1
7fc750807000-7fc750808000 rw-p 0000a000 07:fc 1054812                    /opt/moxi/lib/libvbucket.so.1
7fc750808000-7fc750858000 r-xp 00000000 07:fc 1054811                    /opt/moxi/lib/libconflate.so.0
7fc750858000-7fc750a57000 ---p 00050000 07:fc 1054811                    /opt/moxi/lib/libconflate.so.0
7fc750a57000-7fc750a59000 r--p 0004f000 07:fc 1054811                    /opt/moxi/lib/libconflate.so.0
7fc750a59000-7fc750a5a000 rw-p 00051000 07:fc 1054811                    /opt/moxi/lib/libconflate.so.0
7fc750a5a000-7fc750a7c000 r-xp 00000000 07:fc 262424                     /lib/x86_64-linux-gnu/ld-2.15.so
7fc750be3000-7fc750c6c000 rw-p 00000000 00:00 0 
7fc750c78000-7fc750c7c000 rw-p 00000000 00:00 0 
7fc750c7c000-7fc750c7d000 r--p 00022000 07:fc 262424                     /lib/x86_64-linux-gnu/ld-2.15.so
7fc750c7d000-7fc750c7f000 rw-p 00023000 07:fc 262424                     /lib/x86_64-linux-gnu/ld-2.15.so
7fff7f2ad000-7fff7f2ce000 rw-p 00000000 00:00 0                          [stack]
7fff7f384000-7fff7f386000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

and finally, here is my config:

concurrency=1024,
connect_max_errors=5,
connect_retry_interval=30000,
connect_timeout=400,
cycle=200,
downstream_conn_max=8,
downstream_conn_queue_timeout=200,
downstream_timeout=5000,
wait_queue_timeout=200

Thanks,

Michał Roszka

This does appear to be a bug of some sort.

Unfortunately, the symbols are not showing in that backtrace. Because of how this works on Linux, we can’t really extract them from the core necessarily. If you have a core file available, can you open it in gdb (usually, the easiest way to be sure it resolves the symbols is to start gdb from the directory where moxi’s binary is) and get a backtrace of all threads?