CentOS 8: undefined symbol: EVP_KDF_ctrl in libcrypto

I just installed CB Server 6.5.1 on CentOS 8 and was having immense stability problems. Eventually got to the logs and found a lot of messages like this:

Service 'indexer' exited with status 127. Restarting. Messages:
/opt/couchbase/bin/indexer: symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b

Googling on the internet, I found this topic on the redhat forum which seemed superficially similar, and it was the case that CB Server has a similar “built-in” version of libcrypto. Checking the symbols with nm confirms that the symbol is missing from the version in the couchbase install package but present in the “default” version in /lib64.

# ldd /opt/couchbase/bin/goxdcr 
	linux-vdso.so.1 (0x00007fff073f6000)
	libpcre.so.1 => /opt/couchbase/bin/../lib/libpcre.so.1 (0x00007f6b43f46000)
	libsigar.so => /opt/couchbase/bin/../lib/libsigar.so (0x00007f6b43d37000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b43b17000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6b43755000)
	libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007f6b43522000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6b44164000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6b432d2000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6b42fe2000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6b42dc6000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6b42bc2000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6b429b1000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b427ad000)
	libcrypto.so.1.1 => /opt/couchbase/bin/../lib/../lib/libcrypto.so.1.1 (0x00007f6b422a0000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6b4209c000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6b41e85000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6b41c5a000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f6b419d6000)

# nm -D /lib64/libk5crypto.so.3 | grep EVP_KDF_ctrl
                 U EVP_KDF_ctrl
# nm -D /opt/couchbase/lib/libcrypto.so.1.1  | grep EVP_KDF_ctrl
# nm -D /lib64/libcrypto.so.1.1  | grep EVP_KDF_c
0000000000173820 T EVP_KDF_ctrl
0000000000173910 T EVP_KDF_ctrl_str

So I moved it to one side and that appears to have fixed the problem, although it took me a little while to realize I needed to take the same actions on all the nodes in my cluster :slight_smile:

Hopefully this helps anybody else who encounters similar symptoms. I had not seen this problem on 6.5, but it has to be said that I applied the latest CentOS updates at the same time as updating Couchbase, so it could be more about their world changing.

UPDATE: In the course of writing this up, I had to reboot the machine and it turns out that on restart, couchbase-server will not start in this configuration because of something to do with generating a random seed:

Eshell V9.3.3.9  (abort with ^G)
(babysitter_of_ns_1@cb.local)1> {"init terminating in do_boot",{{badmatch,{error,{bad_return,{{ns_babysitter,start,[normal,[]]},{'EXIT',{undef,[{crypto,strong_rand_bytes," ",[]},{misc,generate_cookie,0,[{file,"src/misc.erl"},{line,268}]},{ns_babysitter,start,2,[{file,"src/ns_babysitter.erl"},{line,55}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{ns_babysitter_bootstrap,start,0,[{file,"src/ns_babysitter_bootstrap.erl"},{line,30}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{ns_babysitter_bootstrap,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump.1596056556.3054216.babysitter...done

So I had to move it back, restart and then move it out again so that the rest of the system could function.

I’m seeing the same thing. I fortunately have some other systems that I’ve not patched that are on:

openssl-devel-1.1.1c-2.el8_1.1.x86_64
openssl-pkcs11-0.4.8-2.el8.x86_64
openssl-libs-1.1.1c-2.el8_1.1.x86_64
openssl-1.1.1c-2.el8_1.1.x86_64

And work fine.