"server TLS certificate untrusted" error connecting to sync gateway

Mobile
1

I’m running into an error connecting a Couchbase Lite Android client to sync gateway over wss:// TLS, getting the error “server TLS certificate untrusted”. These are the logcat logs:

2021-01-14 12:27:45.607 8147-9153/? E/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#5} WebSocket failed to connect! (reason=Network error 8)
2021-01-14 12:27:45.611 8147-8250/? E/CouchbaseLite/REPLICATOR: {Repl#6} Got LiteCore error: Network error 8 "server TLS certificate untrusted"
2021-01-14 12:27:45.671 8147-8147/? W/Couchbase: ReplicatorChange{replicator=Replicator{@0xe741373(<*>),Database{@0x1039856, name='911705151-synced'} => URLEndpoint{url=wss://********:4984/db}}, status=Status{activityLevel=CONNECTING, progress=Progress{completed=0, total=0}, error=CouchbaseLiteException{CouchbaseLite,5008,'server TLS certificate untrusted
       (CouchbaseLite Android v2.8.1-1 (EE/release, Commit/a9c0287d1e@a10b5970562a Core/2.8.1 (1) at 2020-10-31T20:34:35.308Z) on Java; Android 11; sdk_gphone_x86_arm)'}}}
2021-01-14 12:27:45.720 8147-8147/? W/Couchbase: ReplicatorChange{replicator=Replicator{@0xe741373(<*>),Database{@0x1039856, name='911705151-synced'} => URLEndpoint{url=wss://********:4984/db}}, status=Status{activityLevel=STOPPED, progress=Progress{completed=0, total=0}, error=CouchbaseLiteException{CouchbaseLite,5008,'server TLS certificate untrusted
       (CouchbaseLite Android v2.8.1-1 (EE/release, Commit/a9c0287d1e@a10b5970562a Core/2.8.1 (1) at 2020-10-31T20:34:35.308Z) on Java; Android 11; sdk_gphone_x86_arm)'}}}

If I change the endpoint to the ws:// scheme, it refuses the connection:

2021-01-14 12:22:22.576 7779-7938/? E/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#1} WebSocket failed to connect! (reason=WebSocket/HTTP status 1000)
2021-01-14 12:22:22.584 7779-7919/? E/CouchbaseLite/REPLICATOR: {Repl#2} Got LiteCore error: WebSocket error 1001 "WebSocket connection closed by peer"
2021-01-14 12:22:23.228 7779-7779/? W/Couchbase: ReplicatorChange{replicator=Replicator{@0x9b119e1(<*>),Database{@0xc2fc818, name='911705151-synced'} => URLEndpoint{url=ws://********:4984/db}}, status=Status{activityLevel=CONNECTING, progress=Progress{completed=0, total=0}, error=CouchbaseLiteException{CouchbaseLite,11001,'WebSocket connection closed by peer
       (CouchbaseLite Android v2.8.1-1 (EE/release, Commit/a9c0287d1e@a10b5970562a Core/2.8.1 (1) at 2020-10-31T20:34:35.308Z) on Java; Android 11; sdk_gphone_x86_arm)'}}}
2021-01-14 12:22:23.235 7779-7779/? W/Couchbase: ReplicatorChange{replicator=Replicator{@0x9b119e1(<*>),Database{@0xc2fc818, name='911705151-synced'} => URLEndpoint{url=ws://********:4984/db}}, status=Status{activityLevel=STOPPED, progress=Progress{completed=0, total=0}, error=CouchbaseLiteException{CouchbaseLite,11001,'WebSocket connection closed by peer
       (CouchbaseLite Android v2.8.1-1 (EE/release, Commit/a9c0287d1e@a10b5970562a Core/2.8.1 (1) at 2020-10-31T20:34:35.308Z) on Java; Android 11; sdk_gphone_x86_arm)'}}}

I can connect to the sync gateway over https in the browser just fine, including in Chrome on the Android device. The root certificate is signed by Amazon.

2

it’s correct to use wss:// for CBLite client to connect sync gateway over TLS.

To investigate your problem, there are several checkpoints: (1) your sync gateway has ssl enabled and have certificate setup properly, (2) include sync gateway certificate in your android application as a resource file (3) when you initialize a ReplicatorConfiguration, call setPinnedServerCertificate(YourSyncGatewayCert) to set the certificate to the replicator.

Hope this helps.

1 Like
3

From reading the docs, certificate pinning should be optional, right? Shouldn’t it work without this step with the default setAcceptOnlySelfSignedServerCertificate(false) and a server certificate signed by a valid certificate authority (Amazon)? The docs say:

“If there are no pinned certificates and setAcceptOnlySelfSignedServerCertificate is false (default), the client validates the server’s certificates against the system CA certificates. The server must supply a chain of certificates whose root is signed by one of the certificates in the system CA bundle.”

The system CA bundle would seem to have the Amazon signed root certificate, since it works to connect to the sync gateway over https in Chrome on the Android device, right?

I just tested with setAcceptOnlySelfSignedServerCertificate(true) and get the same error. So it’s not that it thinks the certificate is self-signed (a different error). For some reason it just isn’t trusting it.

4

Ok, figured this out. I added a regular GET request to the sync gateway endpoint over https within the app code and got this error:

2021-01-14 14:53:54.939 5873-6039/com.salesrabbit.android.sales.universal W/CouchbaseLite/NETWORK: WebSocketListener failed with response null
    javax.net.ssl.SSLHandshakeException: Domain specific configurations require that hostname aware checkServerTrusted(X509Certificate[], String, String) is used
        at com.android.org.conscrypt.SSLUtils.toSSLHandshakeException(SSLUtils.java:362)
        at com.android.org.conscrypt.ConscryptEngine.convertException(ConscryptEngine.java:1134)
        at com.android.org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1089)
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:876)
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:747)
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:712)
        at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:849)
        at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.access$100(ConscryptEngineSocket.java:722)
        at com.android.org.conscrypt.ConscryptEngineSocket.doHandshake(ConscryptEngineSocket.java:238)
        at com.android.org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:217)
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)
     Caused by: java.security.cert.CertificateException: Domain specific configurations require that hostname aware checkServerTrusted(X509Certificate[], String, String) is used
        at android.security.net.config.RootTrustManager.checkServerTrusted(RootTrustManager.java:113)
        at com.couchbase.lite.internal.replicator.CBLTrustManager.doCheckServerTrusted(CBLTrustManager.java:73)
        at com.couchbase.lite.internal.replicator.CBLTrustManager.checkServerTrusted(CBLTrustManager.java:63)
        at com.android.org.conscrypt.Platform.checkServerTrusted(Platform.java:254)
        at com.android.org.conscrypt.ConscryptEngine.verifyCertificateChain(ConscryptEngine.java:1644)
        at com.android.org.conscrypt.NativeCrypto.ENGINE_SSL_read_direct(Native Method)
        at com.android.org.conscrypt.NativeSsl.readDirectByteBuffer(NativeSsl.java:568)
        at com.android.org.conscrypt.ConscryptEngine.readPlaintextDataDirect(ConscryptEngine.java:1095)
        at com.android.org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1079)
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:876) 
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:747) 
        at com.android.org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:712) 
        at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:849) 
        at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.access$100(ConscryptEngineSocket.java:722) 
        at com.android.org.conscrypt.ConscryptEngineSocket.doHandshake(ConscryptEngineSocket.java:238) 
        at com.android.org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:217) 
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379) 
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) 
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) 
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) 
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) 
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) 
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) 
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) 
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) 
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) 
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) 
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) 
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
        at java.lang.Thread.run(Thread.java:923)

Which lead me to believing it had something to do with the <domain-config cleartextTrafficPermitted="true"> I added while testing sync gateway without TLS locally. And sure enough, removing this network config it’s now connecting as it should.

5

We’ve had instances of folks getting stump the other way - where they want cleartext in dev/test and forget to set the flag . We have a note in our docs to that affect

As of Android Pie, version 9, API 28, cleartext support is disabled, by default. Although wss: protocol URLs are not affected, in order to use the ws: protocol, applications must target API 27 or lower, or must configure application network security as described here.

6

I believe that the cleartextTrafficPermitted should affect only the ws: protocol connection (and only by allowing it). Are you saying that with the flag, the wss: protocol connection did not work and that removing it made it work?

If that is the case, we need to update our documentation.

7

@blake.meike yes, apparently there’s a requirement that if you have a <domain-config> defined (“Domain specific configuration”), even if it’s to set cleartextTrafficPermitted="true", it requires implementing checkServerTrusted(X509Certificate[], String, String) in some way. This function is part of the X509TrustManager API. The only time I’ve had to work with this API was getting a local proxy working, where I wanted to bypass certificate checks.

I’m targeting the latest API 30. Not sure which API level this behavior begins in. This is the network_security_config.xml I was using to allow insecure ws: traffic:

<network-security-config>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">localhost</domain>
        <domain includeSubdomains="true">192.168.100.158</domain>
    </domain-config>
</network-security-config>

Maybe the fact that the sync gateway endpoint I was connecting to with wss: wasn’t one of the local hostname or IP in the config had something to do with it? After commenting out the config entirely, it connected without issue.

8

Thanks @jeff.lockhart! This is really good to know. I’m gonna do a couple experiments and then propose updates to our documentation.

… and, btw, bonus points for using 192.168.x.x instead of 10.x.x.x… :stuck_out_tongue_winking_eye:

9

I’ve created a JIRA ticket to track updating the documentation:

If I understand, though, you have a workaround and are good to go. Will mark this discussion closed unless I hear otherwise.

10

@blake.meike thanks. Yes, for now I’m just adding or removing the <domain-config> as needed to either test locally without TLS or connect to remote server with TLS. There may be a way to add an additional <domain-config> that both allows insecure traffic to specific URLs as well as allows normal secure traffic to everything else.

11

As per the comments above, given the fact that there’s a requirement that if you have a <domain-config> defined (“Domain specific configuration”), even if it’s to set cleartextTrafficPermitted="true" , it requires implementing checkServerTrusted(X509Certificate[], String, String) … is that not something that couchbase lite should do?

12

I know you fixed this with https://issues.couchbase.com/browse/CBL-1984

But this needs the Proguard rule:

-keep class com.couchbase.lite.internal.replicator.CBLTrustManager {
    public java.util.List checkServerTrusted(java.security.cert.X509Certificate[], java.lang.String, java.lang.String);
 }

Can you just add all the needed rules inside the library? I know some other libraries do this already.

13

GREAT CATCH @vincentjoshuaet!
I will add this to our Proguard rule example.
I don’t know of a way to keep applications that use our library from “proguarding” the library.

14

@geraldapeoples CBL Android implements checkServerTrusted(X509Certificate[], String, String) since v 3.0
@vincentjoshuaet just pointed out to me, though, that the implementation will be useless if you don’t protect it from proguard…

15

The new proguard rules for 3.1. They would work just as well for 3.0

-keep class com.couchbase.lite.ConnectionStatus { <init>(...); }
-keep class com.couchbase.lite.LiteCoreException { static <methods>; }
-keep class com.couchbase.lite.internal.replicator.CBLTrustManager {
    public java.util.List checkServerTrusted(java.security.cert.X509Certificate[], java.lang.String, java.lang.String);
}
-keep class com.couchbase.lite.internal.ReplicationCollection {
    static <methods>;
    <fields>;
}
-keep class com.couchbase.lite.internal.core.C4* {
    static <methods>;
    <fields>;
    <init>(...);
 }
16

You can do it by adding a consumer-rules.pro file.

java - What is the difference between consumer-rules.pro and proguard-rules.pro in Android? - Stack Overflow

2. What is the difference between consumer-rules.pro and proguard-rules.pro in Android?

proguard-rules.pro is the file where you declare rules related to Proguard for your module and its dependencies.

consumer-rules.pro is the file where you declare rules that can be applied to your module by the consumer (whoever would be using your module/library as a dependency – typically library devs).

17

Excellent! Thanks a lot, @vincentjoshuaet

Maybe I can package it with the maven distribution…