CentOS 8: undefined symbol: EVP_KDF_ctrl in libcrypto

I just installed CB Server 6.5.1 on CentOS 8 and was having immense stability problems. Eventually got to the logs and found a lot of messages like this:

Service 'indexer' exited with status 127. Restarting. Messages:
/opt/couchbase/bin/indexer: symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b

Googling on the internet, I found this topic on the redhat forum which seemed superficially similar, and it was the case that CB Server has a similar “built-in” version of libcrypto. Checking the symbols with nm confirms that the symbol is missing from the version in the couchbase install package but present in the “default” version in /lib64.

# ldd /opt/couchbase/bin/goxdcr 
	linux-vdso.so.1 (0x00007fff073f6000)
	libpcre.so.1 => /opt/couchbase/bin/../lib/libpcre.so.1 (0x00007f6b43f46000)
	libsigar.so => /opt/couchbase/bin/../lib/libsigar.so (0x00007f6b43d37000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b43b17000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6b43755000)
	libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007f6b43522000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6b44164000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6b432d2000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6b42fe2000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6b42dc6000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6b42bc2000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6b429b1000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b427ad000)
	libcrypto.so.1.1 => /opt/couchbase/bin/../lib/../lib/libcrypto.so.1.1 (0x00007f6b422a0000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6b4209c000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6b41e85000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6b41c5a000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f6b419d6000)

# nm -D /lib64/libk5crypto.so.3 | grep EVP_KDF_ctrl
                 U EVP_KDF_ctrl
# nm -D /opt/couchbase/lib/libcrypto.so.1.1  | grep EVP_KDF_ctrl
# nm -D /lib64/libcrypto.so.1.1  | grep EVP_KDF_c
0000000000173820 T EVP_KDF_ctrl
0000000000173910 T EVP_KDF_ctrl_str

So I moved it to one side and that appears to have fixed the problem, although it took me a little while to realize I needed to take the same actions on all the nodes in my cluster :slight_smile:

Hopefully this helps anybody else who encounters similar symptoms. I had not seen this problem on 6.5, but it has to be said that I applied the latest CentOS updates at the same time as updating Couchbase, so it could be more about their world changing.

UPDATE: In the course of writing this up, I had to reboot the machine and it turns out that on restart, couchbase-server will not start in this configuration because of something to do with generating a random seed:

Eshell V9.3.3.9  (abort with ^G)
(babysitter_of_ns_1@cb.local)1> {"init terminating in do_boot",{{badmatch,{error,{bad_return,{{ns_babysitter,start,[normal,[]]},{'EXIT',{undef,[{crypto,strong_rand_bytes," ",[]},{misc,generate_cookie,0,[{file,"src/misc.erl"},{line,268}]},{ns_babysitter,start,2,[{file,"src/ns_babysitter.erl"},{line,55}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{ns_babysitter_bootstrap,start,0,[{file,"src/ns_babysitter_bootstrap.erl"},{line,30}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{ns_babysitter_bootstrap,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump.1596056556.3054216.babysitter...done

So I had to move it back, restart and then move it out again so that the rest of the system could function.

I’m seeing the same thing. I fortunately have some other systems that I’ve not patched that are on:

openssl-devel-1.1.1c-2.el8_1.1.x86_64
openssl-pkcs11-0.4.8-2.el8.x86_64
openssl-libs-1.1.1c-2.el8_1.1.x86_64
openssl-1.1.1c-2.el8_1.1.x86_64

And work fine.

Does anyone know when the fix for this will hit the community edition? I am working on an Ansible Couchbase collection and noticed this while testing. My installs work fine for enterprise edition but rebalance fails due to this error on Community with an updated Centos8 box.
Thanks