Bootstrap Error on .NET 5 and .NET 6

I know .NET 6 isn’t officially supported yet, and this post is more to bring awareness to the issue so that it can be looked at when support for .NET 6 is added.

I downloaded the first release of .NET 6 (6.0.0) and tried it with SDK 3.2.4. It works in one situation, but not another:

  1. .Net 6 Windows Console Application With SDK 3.2.4 connecting to local 6.6 Couchbase Server ==> Able to Bootstrap + Issue Queries

  2. .Net 6 Alpine (3.14) Docker Container Application Hosted On Kubernetes connecting to cloud hosted 6.6 Couchbase Server ==> Bootstrap error below (1 occurrence each for Couchbase, Memcached, and Ephemeral):

“Cannot bootstrap bucket THEBUCKET as Couchbase. System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|187_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize)
at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte reAuthenticationData, Boolean isApm)
at Couchbase.Core.IO.Connections.ConnectionFactory.CreateAndConnectAsync(IPEndPoint endPoint, HostEndpoint hostEndpoint, CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.ConnectionPoolBase.CreateConnectionAsync(CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.<>c__DisplayClass29_0.<g__StartConnection|0>d.MoveNext()
— End of stack trace from previous location —
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.AddConnectionsAsync(Int32 count, CancellationToken cancellationToken)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.InitializeAsync(CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.InitializeAsync()
at Couchbase.Core.DI.ClusterNodeFactory.CreateAndConnectAsync(HostEndpoint endPoint, BucketType bucketType, NodeAdapter nodeAdapter, CancellationToken cancellationToken)
at Couchbase.Core.ClusterContext.CreateAndBootStrapBucketAsync(String name, HostEndpoint endpoint, BucketType type)
at Couchbase.Core.ClusterContext.GetOrCreateBucketAsync(String name)”

Unsure what differences in the environments/systems are causing the issue. Both setups are working fine with .Net Core 3.1 + SDK 3.2.4

Unsure if related/the same as this issue: .NETCore App 6 compatible

Thanks @obawin for brining this to our notice, @Richard_Ponton something to keep in mind.

@obawin

It appears you’re connecting with SSL to the Couchbase cluster, and the problem is happening somewhere in the SSL stream. .NET Core implements this using openssl on Linux under the hood, I believe. I’m wondering if there’s something about using the Alpine distribution of Linux and the version of openssl it has.

Have you tried another distro as the base for your Docker image? The default .NET 6.0 image is Debian 11 and includes openssl 1.1.1k. The Alpine image doesn’t include the openssl command-line, but appears to include libssl and libcrypto 1.1.1l.

Unfortunately, we are pretty firmly locked in with Alpine (chosen due to its small size, security focused approach), and we probably won’t be able to change unless there is a major reason to. It’s possible other distros have no problems, as Alpine does have some quirks given its minimalistic approach to things.

Hi,

Decided to run another test, and I can confirm that the same error happens on .NET 5 and SDK 3.2.4.

The variations I have tested are:

  1. .NET Core 3.1 + SDK 3.2.4 + Alpine 3.13 ==> OK

  2. .NET 5.0 + SDK 3.2.4 + Alpine 3.13 ==> Bootstrap Error as above

  3. .NET 6.0 + SDK 3.2.4 + Alpine 3.13 or Alpine 3.14 ==> Bootstrap Error as above

Did some digging, and I believe this is the cause of the issue in .NET 5 and .NET 6:

I assume Couchbase currently doesn’t support the most modern/restrictive ciphers yet?

Given that, the 2 options available seem to be:

  • modify the openssl config on the alpine image
    or
  • Explicitly configure some working values for CipherSuitePolicy and SslStream objects in the Couchbase SDK code

Would it be a good idea to consider implementing option #2 in a future release? The worry about option #1 is that many (most?) SDK Linux .Net5+ users will be affected, and it might be too big a hurdle/responsibility for them to have to go into the openssl configuration levels?

Perhaps a mixture of the 2 options (and a good balance of both) is to create a new option that accepts a list of allowed Cipher suites, and if explicit ones are specified, those are used in the SslStream calls. The list of available cipher suites is available via this enum:

This would give users the ability to configure/customize the used cipher suites at the SDK level, without having to alter the Alpine openssl config/defaults.

From my perspective, this would be more secure as well as we use the Alpine image for different types of containerized services. If the fix for this issue happens at the open ssl config level, images for services that don’t use Couchbase would also be affected. Even for images for services that use Couchbase, ideally the Cipher suite restriction wouldn’t be loosened on the entire image. Having the ability to change/fix at the SDK level limits the cipher change to the service that is communicating with CB.

@btburnett3 , @jmorris , what are your thoughts?

1 Like

I agree with your assessment, based on my research. I think we should probably expose configuration options for .NET Core 3.1 and later (the first version they are supported by HttpClient/SocketsHttpHandler and SslStream). We should also consider adjusting the defaults for greater out-of-the-box compatibility, but that has risks and needs careful consideration.

I filed this issue for tracking: [NCBC-3019] Allow SSL cipher configuration - Couchbase database

1 Like

Thanks @btburnett3 ! Looking forward to an upcoming release with this change, so that we can start testing SDK 3.2.4 and .NET6!

1 Like

hi @btburnett3 ,

I did some prototyping/testing and it looks like what’s important for our environment setup is being able to set the Enabled SslProtocols (the out-of-the box Cipher Suites in Net 6 worked fine).

Specifically, the CB SDK has TLS 1.0, 1.1, and 1.2 enabled by default. Looks like in this configuration, the lowest (available) may be used/tried. Our environment has anything lower than 1.2 disabled, which is the reason for the error. When I change the code and configure TLS 1.2 and 1.3 as enabled, the CB SDK works good on NET 6 (KV query success).

The changes I made to the CD SDK are:

  1. ClusterOptions.cs:

     /// <summary>
     /// Enabled SSL Protocols
     /// </summary>
     public SslProtocols EnabledSslProtocols { get; set; } = SslProtocols.Tls | SslProtocols.Tls11 | SslProtocols.Tls12;
    

#if NETCOREAPP3_1_OR_GREATER
///


/// List of supported TLS Cipher Suites. If not set, will use default .NET Cipher Suites
///

public TlsCipherSuite? SupportedTLSCipherSuites { get; set; } = null;
#endif

  1. ConnectionFactory.cs:

             //create the sslstream with appropriate authentication
    

#if !NETCOREAPP3_1_OR_GREATER
await sslStream.AuthenticateAsClientAsync(targetHost, certs,
_clusterOptions.EnabledSslProtocols,
_clusterOptions.EnableCertificateRevocation)
.ConfigureAwait(false);
#else
SslClientAuthenticationOptions sslOptions = new SslClientAuthenticationOptions()
{
TargetHost = targetHost,
ClientCertificates = certs,
EnabledSslProtocols = _clusterOptions.EnabledSslProtocols,
CertificateRevocationCheckMode = _clusterOptions.EnableCertificateRevocation ? X509RevocationMode.Online : X509RevocationMode.NoCheck,
CipherSuitesPolicy = _clusterOptions.SupportedTLSCipherSuites != null && _clusterOptions.SupportedTLSCipherSuites.Length > 0
? new CipherSuitesPolicy(_clusterOptions.SupportedTLSCipherSuites) : null
};
await sslStream.AuthenticateAsClientAsync(sslOptions)
.ConfigureAwait(false);
#endif

  1. CouchbaseHttpClientFactory.cs:
    private HttpMessageHandler CreateClientHandler()
    {
    #if !NETCOREAPP3_1_OR_GREATER

    handler.SslProtocols = _context.ClusterOptions.EnabledSslProtocols;

    #else
    //for x509 cert authentication
    if (_context.ClusterOptions.X509CertificateFactory != null)
    {
    handler.SslOptions.EnabledSslProtocols = _context.ClusterOptions.EnabledSslProtocols;

    }

         ...
     	
         if (_context.ClusterOptions.SupportedTLSCipherSuites != null && _context.ClusterOptions.SupportedTLSCipherSuites.Length > 0)
         {
             handler.SslOptions.CipherSuitesPolicy = new CipherSuitesPolicy(_context.ClusterOptions.SupportedTLSCipherSuites);
         }
    

#endif

As mentioned, for the test, I only had to set options.EnabledSslProtocols = SslProtocols.Tls13 | SslProtocols.Tls12, to have it work. However, I think there is still value to have options.SupportedTLSCipherSuites as configurable too.

Is it possible to have (similar) changes as above for 3.2.5?

PS: One point of possible interest is that when I turn on SDK logging, I see this message many times:

“Error processing new clusterOptions System.ArgumentException: Comparing the same configs is not allowed. at Couchbase.Core.Configuration.Server.BucketConfigExtensions.IsNewerThan(BucketConfig newConfig, BucketConfig oldConfig) in … at Couchbase.CouchbaseBucket.ConfigUpdatedAsync(BucketConfig newConfig)”

Unsure if I added the new cluster options (EnabledSslProtocols and SupportedTLSCipherSuites) correctly/in all places (I followed how “ForceIpAsTargetHost” was implemented) and that may be causing the above?

1 Like

Actually, the error:

“Error processing new clusterOptions System.ArgumentException: Comparing the same configs is not allowed. at Couchbase.Core.Configuration.Server.BucketConfigExtensions.IsNewerThan(BucketConfig newConfig, BucketConfig oldConfig)”

seems like an existing issue. When logging is enabled, I see the error message in the SDK even before making the test SSL/Cipher config changes above.

It seems somewhat random when the error starts (saw it happen when service first started, but also only after first request/processing was performed). Once it starts, it logs the error messsage twice every 2-3s and seems to continue indefinitely.

@obawin

The ArgumentException error is a separate known issue: https://issues.couchbase.com/browse/NCBC-3018

As to getting SslProtocols exposed in 3.2.5, I don’t want to speak for the Couchbase SDK team, I’m just a very active community member. However, I do know that it’s the holidays here in the US right now, they are very busy on several items, and I believe the cutoff for 3.2.5 is in about a week, give or take. And I’m currently focusing my efforts on System.Text.Json support.

Therefore, I don’ think 3.2.5 is very likely. That said, community contributions to the SDK are welcome. If you wanted to polish up your experiment and put in a CR at https://review.couchbase.org/ it could get completed sooner.

1 Like