Python SDK upsert_multi free(): invalid next size (fast)

Python SDK
1

Hello,
I’m trying to use upsert_multi to upload ~10k json documents onto my couchbase server, and its been failing for a little while, so thought i’d write in here.
Firstly it was a segmentation fault (corrupted double linked list), but after upping my ulimit stack size and running my program with faulthandler enabled gave a fatal python error and printed the exception “free(): invalid next size (fast)”.

Trace:

free(): invalid next size (fast)
Fatal Python error: Aborted

Thread 0x00007f3cb3fff700 (most recent call first):
  File "/home/magnus/.local/lib/python3.6/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f3cbb197700 (most recent call first):
  File "/home/magnus/.local/lib/python3.6/site-packages/pymongo/periodic_executor.py", line 131 in _run
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f3cc0b67740 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/couchbase_core/client.py", line 685 in upsert_multi
  File "iedi_archive_clean.py", line 35 in <module>
Aborted

Code creates a dict of documents to upload and does it in a upsert_multi. This is running in a loop as I found python would crash when trying to upload >20k or so files.
Also, after multiple tests i’ve found that its sorta random when it’ll crash, sometimes after the first loop, sometimes after the fourth.

Best regards,
Magnus

2

Thanks, sounds like a memory allocation/corruption issue.
Are you able to produce a simple script to replicate this?

The client can be built with Address Sanitizer support by doing the following:

export PYCBC_SAN_OPT_address=TRUE
export PYCBC_SAN_OPT_undefined=TRUE
pip uninstall --yes couchbase
pip install couchbase -v

When you run the client it may complain about a missing sanitizer runtime, and give you instructions how to fix this.

Alternatively you can run the entire script with Valgrind Memcheck, although this will be slow.

Thanks,

Ellis

3

As i’m unable to create a local 4.5 server which is what we use in production, i’m sadly not. After running my current script after the envvars you posted, I had to set LD_PRELOAD to find my ASAN, and then my program gave me the output of:

==6945==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8410520 byte(s) in 13220 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5ab684 (/usr/bin/python3.6+0x5ab684)

Direct leak of 62976 byte(s) in 53 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x54485e in PyUnicode_New (/usr/bin/python3.6+0x54485e)

Direct leak of 23256 byte(s) in 41 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5a41ba (/usr/bin/python3.6+0x5a41ba)

Direct leak of 9320 byte(s) in 10 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5518cb in PyType_GenericAlloc (/usr/bin/python3.6+0x5518cb)

Direct leak of 4680 byte(s) in 6 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x59203b in PyBytes_FromStringAndSize (/usr/bin/python3.6+0x59203b)

Direct leak of 1728 byte(s) in 3 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5abd13 (/usr/bin/python3.6+0x5abd13)

Direct leak of 528 byte(s) in 1 object(s) allocated from:
#0 0x7f3cd6dbad28 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded28)
#1 0x582742 in PyList_New (/usr/bin/python3.6+0x582742)

Direct leak of 2 byte(s) in 2 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5aca9a in PyMem_Malloc (/usr/bin/python3.6+0x5aca9a)

Indirect leak of 323408 byte(s) in 346 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5518cb in PyType_GenericAlloc (/usr/bin/python3.6+0x5518cb)

Indirect leak of 28717 byte(s) in 26 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x54485e in PyUnicode_New (/usr/bin/python3.6+0x54485e)

Indirect leak of 28053 byte(s) in 26 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5ac33a in PyObject_Malloc (/usr/bin/python3.6+0x5ac33a)

Indirect leak of 12883 byte(s) in 7 object(s) allocated from:
#0 0x7f3cd6dbaf30 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdef30)
#1 0x5475d4 (/usr/bin/python3.6+0x5475d4)

Indirect leak of 7912 byte(s) in 13 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5a41ba (/usr/bin/python3.6+0x5a41ba)

Indirect leak of 1728 byte(s) in 3 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5ab684 (/usr/bin/python3.6+0x5ab684)

Indirect leak of 528 byte(s) in 1 object(s) allocated from:
#0 0x7f3cd6dbab40 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb40)
#1 0x5abd13 (/usr/bin/python3.6+0x5abd13)

SUMMARY: AddressSanitizer: 8916239 byte(s) leaked in 13758 allocation(s).

4

Hi, please try installing with debug symbols on, as well as the other parameters:

export PYCBC_DEBUG_SYMBOLS=TRUE

Also, running with gdb could be useful:

gdb `which python`
run <python_script_path> <python_script_args>

(this is also scriptable via gdb --batch --command=script.gdb ...)

Are you able to upgrade to a later Python, in case this is an issue with Python itself (3.6 is several years old now)?

Thanks,

Ellis

5

Yeah i can upgrade, will do this monday, have a pleasant weekend!

1 Like
6

I’m somehow unable to recreate the segmentation fault today, and I have absolutely no idea why. I’ll update the thread if it starts acting up again.
Thanks for the help.

7

OK, yes, keep us updated if it recurs. Thanks :slight_smile: