Python client with TLS requires SSL verification on Ubuntu but not Mac

Running into a bit of confusion regarding SSL verification when using the Python client. I’m assuming there’s just something I don’t understand about the SSL verification process in general.

The short version is: I need to set ssl=no_verify on my connection string when running from EC2/Ubuntu, but not when running from my Mac.

We’re running Couchbase Enterprise 6.6.2 on a cluster of EC2 instances. We have a self signed cert that the server uses for TLS.

I’m using version 4.0.3 of the python client. I’ve gotten the same results running under Python 3.8, 3.9. and 3.10.

The test I’ve been using is basically just a copy/paste from the getting started guide:

import os
from datetime import timedelta
from couchbase.auth import PasswordAuthenticator
from couchbase.cluster import Cluster
from couchbase.options import (ClusterOptions, ClusterTimeoutOptions,
                               QueryOptions)

bucket_name = "the_bucket"
username = os.environ["CB_USER"]
password = os.environ["CB_PASSWORD"]
cert_path = os.environ["CB_CERT_PATH"]
endpoint = os.environ["CB_URL"]

auth = PasswordAuthenticator(
    username,
    password,
    # From couchbase docs:
    # NOTE: If using SSL/TLS, add the certificate path.
    # We strongly reccomend this for production use.
    cert_path=cert_path
)

cluster = Cluster(endpoint, ClusterOptions(auth))
cluster.wait_until_ready(timedelta(seconds=10))

print("cluster ready")

Now, this all works great from my Mac laptop (MacOS 10.15.7). I am able to connect with the following CB_URL:

  • couchbases://the.endpoint.com

However, if I run this from one of our application servers (EC2/Ubuntu20.04), I need to disable SSL verification by setting the connection string to:

  • couchbases://the.endpoint.com?ssl=no_verify

Otherwise, I just get an UnambigousTimeoutException.

I’ve confirmed that all the settings are identical between environments.

The question is: why does disabling ssl verification make it work on Ubuntu? Why does this not need to be done from my Mac?

I thought the process of SSL verification involved verifying the hostname of the cert as well as checking that it’s signed by a trusted authority. Since we are using a self-signed cert, which can’t be verified, I assumed we would need to turn off verification everywhere, but this is not the case on Mac somehow.

Any help would be greatly appreciated. I’m sure there’s something fundamental I’m missing.

Hi @josh.crean - Could you provide some details you see from logging? Docs are here. DEBUG level logs would be ideal.

Also can you also see what the openssl s_client command returns?

 openssl s_client -connect <endpoint>:11207

Hey @jcasey ,

Thanks for the response. The debug logs seem to mirror what I’m seeing regarding TLS verification.

From my laptop, I’m seeing “normal” connection logs (IP addresses were redacted):

INFO::2022-08-16 09:48:52::created child logger root
INFO::2022-08-16 09:48:52::replace list of bootstrap nodes with addresses from DNS SRV of "our.endpoint.com": ["ip1:11207", "ip2:11207", "ip3:11207"]
DEBUG::2022-08-16 09:48:52::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0]: use TLS verify file: "cb-cert.pem"
DEBUG::2022-08-16 09:48:52::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1:11207> attempt to establish MCBP connection
DEBUG::2022-08-16 09:48:52::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1:11207> connecting to ip1:11207, timeout=2000ms
DEBUG::2022-08-16 09:48:52::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1:11207> connected to ip1:11207
DEBUG::2022-08-16 09:48:52::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1/ip1:11207> user_agent={"a":"cxx/1.0.0/80b3686","i":"2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582"}, requested_features=[tcp_nodelay, mutation_seqno, xattr, xerror, select_bucket, json, duplex, alt_request_support, tracing, sync_replication, vattr, collections, subdoc_create_as_deleted, preserve_ttl, unordered_execution, snappy]
...
DEBUG::2022-08-16 09:48:53::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1/ip1:11207> received new configuration: ...
DEBUG::2022-08-16 09:48:53::[2d6ab619-e64c-4647-6e0e-99eb37d2dcd0/b6322cb9-f7b0-4f5c-54eb-e06417771582/tls/-] <ip1/ip1:11207> detected network is "default"
DEBUG::2022-08-16 09:48:53::PYCBC: create conn callback completed

But from other platforms, the logs show SSL verification errors:

INFO::2022-08-16 13:53:40::created child logger root
INFO::2022-08-16 13:53:40::replace list of bootstrap nodes with addresses from DNS SRV of "our.endpoint.com": ["ip1:11207", "ip2:11207", "ip3:11207"]
DEBUG::2022-08-16 13:53:40::[2a2f9635-8006-43a5-c14c-7415aad6adc7]: use TLS verify file: "cb-cert.pem"
DEBUG::2022-08-16 13:53:40::[2a2f9635-8006-43a5-c14c-7415aad6adc7/31df85a0-15a8-4228-d282-9ab5f4f2be3a/tls/-] <ip1:11207> attempt to establish MCBP connection
DEBUG::2022-08-16 13:53:40::[2a2f9635-8006-43a5-c14c-7415aad6adc7/31df85a0-15a8-4228-d282-9ab5f4f2be3a/tls/-] <ip1:11207> connecting to ip1:11207, timeout=2000ms
WARNING::2022-08-16 13:53:41::[2a2f9635-8006-43a5-c14c-7415aad6adc7/31df85a0-15a8-4228-d282-9ab5f4f2be3a/tls/-] <ip1:11207> unable to connect to ip1:11207: 337047686 (error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed). is_open=true
ERROR::2022-08-16 13:53:41::[2a2f9635-8006-43a5-c14c-7415aad6adc7/31df85a0-15a8-4228-d282-9ab5f4f2be3a/tls/-] <ip1:11207> no more endpoints left to connect, will try another address

So, I’m not sure why it would verify from the laptop, but not from anywhere else.

The s_client output is a little more interesting:

From laptop - working:

CONNECTED(00000003)
depth=1 CN = *.zone.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---

From Ubuntu - not working:

CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = *.zone.com
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=0 CN = Couchbase Server
verify error:num=68:CA signature digest algorithm too weak
verify return:1
depth=1 CN = *.zone.com
verify return:1
depth=0 CN = Couchbase Server
verify return:1
CONNECTED(00000003)
depth=1 CN = *.zone.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---

It seems relevant that the failing system is using openssl 1.1.1. Based on some googling, it sounds like this version of openssl may have restricted the ciphers it allows.

# openssl s_client -connect ip1:11207
New, SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit

So our server cert is using AES128-SHA. Perhaps this is one that is no longer accepted by default?

My mac is using LibreSSL 2.8.3 and not complaining about the cipher.

I was going to ask what version of Ubuntu you are using. If 20.04, running “openssl version -a” should show that DOPENSSL_TLS_SECURITY_LEVEL is set to 2. If that is the case, you are correct, AES128-SHA would not be acceptable.

From OpenSSL docs:

Level 2
Security level set to 112 bits of security. As a result RSA, DSA and DH keys shorter than 2048 bits and ECC keys shorter than 224 bits are prohibited. In addition to the level 1 exclusions any cipher suite using RC4 is also prohibited. SSL version 3 is also not allowed. Compression is disabled.

I believe you can update your openssl.cnf to change the security level. Haven’t looked closely, but a quick Google turned up this. Which seems promising :slight_smile:

Also, do you not have OpenSSL on your mac? I am quite shocked that the SDK is working w/o OpenSSL.

Well on the Mac, I have LibreSSL 2.8.3 installed, which is an older version of their fork of openssl. I’m assuming that version of LibreSSL doesn’t have the concept of a minimum security level like recent versions of OpenSSL have. That’s why I’m able to connect without issue.

I did confirm that if I set the SecurityLevel to 1 on our Ubuntu instance, I am able to connect normally.

This was an interesting journey and now I know about security levels in openssl. Thanks for your help!

Awesome!! Glad things are working. Always (or at least mostly) fun to learn something new. :slight_smile: