Dependency Injection related exceptions at IIS app pool startup

Since upgrading to the latest Couchbase .NET SDK, and using the new dependency injection extension, we’ve been running into two intermittent issues that appear to be related to the DI. The application in question is .NET Core 3.1.4 hosted in an IIS website. I’m not completely clear on how this is occurring, but what I do know is that when the app pool first starts (like after a deploy of new code), requests to the app pool start to fail with one of the two exceptions listed below. The application is pretty straight forward, in that it calls IServiceCollection.AddCouchbase(), with config being passed into the AddCouchbase method.

The way that I can resolve this when it happens is to remove the app from receiving HTTP requests, recycle the app pool, then re-allow requests to the app to resume.

System.InvalidOperationException: ConfigHandler has already been started.
at Couchbase.Core.Configuration.Server.ConfigHandler.Start(Boolean withPolling)
at Couchbase.Cluster…ctor(ClusterOptions clusterOptions)
at Couchbase.Cluster.ConnectAsync(ClusterOptions options)
at Couchbase.Extensions.DependencyInjection.Internal.ClusterProvider.GetClusterAsync()
at Couchbase.Extensions.DependencyInjection.Internal.BucketProvider.b__4_0(String name)

or

System.ArgumentException: An item with the same key has already been added. Key: Couchbase.Core.ClusterContext
at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer1 comparer) at Couchbase.Core.DI.CouchbaseServiceProvider..ctor(IEnumerable1 serviceFactories)
at Couchbase.ClusterOptions.BuildServiceProvider()
at Couchbase.Core.ClusterContext…ctor(ICluster cluster, CancellationTokenSource tokenSource, ClusterOptions options)
at Couchbase.Cluster…ctor(ClusterOptions clusterOptions)
at Couchbase.Cluster.ConnectAsync(ClusterOptions options)
at Couchbase.Extensions.DependencyInjection.Internal.ClusterProvider.GetClusterAsync()
at Couchbase.Extensions.DependencyInjection.Internal.BucketProvider.b__4_0(String name)

1 Like

I may be wrong, but looking at the exception alone looks like the bootstrapping is attempted twice for somereason.
It would be worthwhile to paste how you bootstarp your code
@btburnett3 / @jmorris thoughts ?

@justusweber - what version of the sdk are you using? If your not on 3.0.2, I would try upgrading first and see if the issue is resolved.

-Jeff

The latest version I see available on Nuget is 3.0.2, which is the SDK version I’m using. And the latest version of the Couchbase.Extensions.DependencyInjection of 3.0.1.

1 Like

@justusweber -

Sorry I posted the wrong version,3.0.2 is the latest published version. As @AV25242 mentioned, it looks like in this scenario the client bootstraps twice causing the exceptions. The SDK should detect that it’s already bootstrapped and not do it again and/or the client code should cleanup so that when the bootstrapping process starts again from a fresh state.

I’ll try to recreate on my side and see what can be done.

-Jeff

1 Like

Thanks Jeff.

In case this adds clarity, this application makes a call to Couchbase upon each HTTP request it receives. It seems that there is a relationship between HTTP requests making it to the app at initial startup and these exceptions. As I mentioned, it never occurs if I start the application up with the server out of the load balancer (i.e., with the server not receiving any HTTP requests), and it always happens if the server is in the load balancer and actively receiving HTTP requests at the time the app is starting up and bootstrapping Couchbase.

Jeff, this comment of yours, whereby the SDK is supposed to elegantly handle bootstrapping when it’s already bootstrapped, made me reconsider a portion of my startup code. I have a wrapper class for the Couchbase functionality, where the IBucketProvider instance is made available via DI thanks to the DI extension package. I registered this wrapper class of mine in the DI container as a singleton. I’ve adjusted my wrapper class to be registered as transient, and the problem seems to be resolved. I’ll do a few more tests to ensure.

3 Likes

@justusweber awesome!

I spoke too soon. The exception of “System.ArgumentException: An item with the same key has already been added. Key: Couchbase.Core.ClusterContext” happened again today on a prod deploy.

@justusweber - can you share a sample project that is similar to your project that we can use to debug the issue?

@jmorris yes, I’ll put something together.

I’m also getting same exception. For me the application is in .Net 4.6.1 with Autofac implemented. Using DI extension for Couchbase 3.0.1. I have few hangfire jobs which runs parallel. I doubt, I’m getting the error when two job fires in same time and trying to connect to Couchbase . Below is the error and any help in this line would be appreciated.
System.InvalidOperationException: ConfigHandler has already been started.
at Couchbase.Core.Configuration.Server.ConfigHandler.Start(Boolean withPolling)
at Couchbase.Cluster…ctor(ClusterOptions clusterOptions)
at Couchbase.Cluster.d__24.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Couchbase.Extensions.DependencyInjection.Internal.ClusterProvider.d__5.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Couchbase.Extensions.DependencyInjection.Internal.BucketProvider.<b__4_0>d.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

@jerinmaryland Can you try upgrading to SDK 3.0.2. It’s still compatible with DI 3.0.1.

@btburnett3 I’m using SDK 3.0.2 and Dependency Injection Extension 3.0.1. As a workaround, I’ve added a thread sleep to all my hangfire job’s before startup. Now seem’s worked for me.