I just installed CB Server 6.5.1 on CentOS 8 and was having immense stability problems. Eventually got to the logs and found a lot of messages like this:
Service 'indexer' exited with status 127. Restarting. Messages:
/opt/couchbase/bin/indexer: symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
Googling on the internet, I found this topic on the redhat forum which seemed superficially similar, and it was the case that CB Server has a similar “built-in” version of libcrypto. Checking the symbols with nm
confirms that the symbol is missing from the version in the couchbase install package but present in the “default” version in /lib64
.
# ldd /opt/couchbase/bin/goxdcr
linux-vdso.so.1 (0x00007fff073f6000)
libpcre.so.1 => /opt/couchbase/bin/../lib/libpcre.so.1 (0x00007f6b43f46000)
libsigar.so => /opt/couchbase/bin/../lib/libsigar.so (0x00007f6b43d37000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b43b17000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6b43755000)
libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007f6b43522000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6b44164000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f6b432d2000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f6b42fe2000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f6b42dc6000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f6b42bc2000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f6b429b1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b427ad000)
libcrypto.so.1.1 => /opt/couchbase/bin/../lib/../lib/libcrypto.so.1.1 (0x00007f6b422a0000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6b4209c000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6b41e85000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f6b41c5a000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f6b419d6000)
# nm -D /lib64/libk5crypto.so.3 | grep EVP_KDF_ctrl
U EVP_KDF_ctrl
# nm -D /opt/couchbase/lib/libcrypto.so.1.1 | grep EVP_KDF_ctrl
# nm -D /lib64/libcrypto.so.1.1 | grep EVP_KDF_c
0000000000173820 T EVP_KDF_ctrl
0000000000173910 T EVP_KDF_ctrl_str
So I moved it to one side and that appears to have fixed the problem, although it took me a little while to realize I needed to take the same actions on all the nodes in my cluster
Hopefully this helps anybody else who encounters similar symptoms. I had not seen this problem on 6.5, but it has to be said that I applied the latest CentOS updates at the same time as updating Couchbase, so it could be more about their world changing.
UPDATE: In the course of writing this up, I had to reboot the machine and it turns out that on restart, couchbase-server will not start in this configuration because of something to do with generating a random seed:
Eshell V9.3.3.9 (abort with ^G)
(babysitter_of_ns_1@cb.local)1> {"init terminating in do_boot",{{badmatch,{error,{bad_return,{{ns_babysitter,start,[normal,[]]},{'EXIT',{undef,[{crypto,strong_rand_bytes," ",[]},{misc,generate_cookie,0,[{file,"src/misc.erl"},{line,268}]},{ns_babysitter,start,2,[{file,"src/ns_babysitter.erl"},{line,55}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{ns_babysitter_bootstrap,start,0,[{file,"src/ns_babysitter_bootstrap.erl"},{line,30}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{ns_babysitter_bootstrap,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
Crash dump is being written to: erl_crash.dump.1596056556.3054216.babysitter...done
So I had to move it back, restart and then move it out again so that the rest of the system could function.