GetAsync failures due to "Cannot access a disposed object" SDK 3.0.1

No issues with this code until recently. The relevant changes were (1) updated SDK from 3.0.0 to 3.0.1 and (2) changed from serial calls to GetAsync to parallel async tasks calling GetAsync.

Note also, this is extremely low volume in this case, passing less than than 100 keys at time, usually 25 or less. The error (from logs) along with its inner exception is shown below followed by a code sample for how it is being called.

Why/when does this exception occur and what is the best way to mitigate/recover within the app code?

  ERROR - GetAsync failed for document key "Cache_Service_DocType_52.454_-1.748_51.5268474_-0.4614485" with exeption: Cannot access a disposed object.
  Object name: 'DataFlowConnectionPool'.

  ErrorMsg: System.AggregateException: One or more errors occurred. (Value cannot be null. (Parameter 'endPoint'))
    System.ArgumentNullException: Value cannot be null. (Parameter 'endPoint')
     at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationToken token, Nullable`1 timeout)
     at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationToken token, Nullable`1 timeout)
     at Couchbase.KeyValue.CouchbaseCollection.ExecuteLookupIn(String id, IEnumerable`1 specs, LookupInOptions options)
     at Couchbase.KeyValue.CouchbaseCollection.GetAsync(String id, GetOptions options)

Code:

	var getCachedDocsTasks = docKeys.Select(async key =>
	{
		cachedDoc = await _cacheService.GetAsync<CacheDocInfo>(key);
		listOfCachedDocsResult.Add(cachedDoc);
	});
	await Task.WhenAll(getCachedDocsTasks);

	public class CacheService : ICacheService
	{
        	public async Task<T> GetAsync<T>(string key)
        	{
                    try
	            {
        	        var getAsyncTask = _collection.GetAsync(key);
                	var result = await getAsyncTask;

	                return result != null && result.Cas != 0 ? result.ContentAs<T>() : default(T);
        	    }
	            catch (Couchbase.Core.Exceptions.KeyValue.DocumentNotFoundException)
        	    {
                	_logger.LogInformation($"document with Key '{key}' not found");
	                return default(T);
        	    }
		    catch (Exception ex)
	            {
        	        _logger.LogError($"ERROR - GetAsync failed for document key {key} with exeption: {ex.Message}");
                	throw;
            	    }
        	}
	}

@atkinsr - I suspect this will be fixed in 3.0.2 which should be released soon.

I’m having a similar bug in the 3.1.2 version

Executing op NoOp on 3.225.106.106:11210 with key and opaque 3876.

dbug: Couchbase.Core.ClusterNode[0]

Op failed: Couchbase.Core.IO.Operations.Noop

System.ObjectDisposedException: Cannot access a disposed object.

Object name: 'MultiplexingConnection'.

at Couchbase.Core.IO.Connections.MultiplexingConnection.SendAsync(ReadOnlyMemory1 request, IOperation operation, ErrorMap errorMap, CancellationToken cancellationToken)`

at Couchbase.Core.IO.Operations.OperationBase.SendAsync(IConnection connection, CancellationToken cancellationToken)

at Couchbase.Core.ClusterNode.ExecuteOp(Func4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)`

dbug: Couchbase.Views.ViewClient[0]

Sending view request to: http://3.225.106.106:8092/bolt-qa-documents/_design/p/_view/p?stale=false&descending=true&on_error=continue

dbug: Couchbase.Query.QueryClient[0]

Sending query 09ead95c-f20e-4d69-9192-ea4c244e970c to node http://3.225.106.106:8093/query.

dbug: Couchbase.Query.QueryClient[0]

Request 09ead95c-f20e-4d69-9192-ea4c244e970c has succeeded.

dbug: Couchbase.Core.ClusterNode[0]

Executing op NoOp on 3.225.106.106:11210 with key and opaque 3877.

dbug: Couchbase.Core.ClusterNode[0]

Op failed: Couchbase.Core.IO.Operations.Noop

System.ObjectDisposedException: Cannot access a disposed object.

Object name: 'MultiplexingConnection'.

at Couchbase.Core.IO.Connections.MultiplexingConnection.SendAsync(ReadOnlyMemory1 request, IOperation operation, ErrorMap errorMap, CancellationToken cancellationToken)`

at Couchbase.Core.IO.Operations.OperationBase.SendAsync(IConnection connection, CancellationToken cancellationToken)

at Couchbase.Core.ClusterNode.ExecuteOp(Func4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)`

This behavior begins afeters service is launched and there is no way to mekae it work again, every connection attempt ends in a timeout and “ObjectedDisposed” exceptions.

We have tried doing the SDK samples implementation, used all flavours of CouchBase, used the Linq2Couchbase and DependendyInjections libraries, but no combination has worked in order to resolve this issue in the .NET based services we have.

Is there any further info about this topic or some king of workaround to avoid this issue?

Thanks

@sebastian_ortiz

This seems like a bug to me. Any information you can provide about what situation creates it, especially debug logs, would be very helpful. For now, I’ve filed a bug for tracking.

https://issues.couchbase.com/browse/NCBC-2829

However, it’s possible there is a workaround in 3.1.2. Try setting ClusterOptions.Experiments.ChannelConnectionPools to true. This will switch you to the new, experimental, higher-performance connection pool. In addition to improved performance, the way it handles disposing of connections has been rewritten. Therefore, it’s possible the bug was incidentally fixed in the redesign. Please let us know how it goes for you, both in terms of stability and performance.

I am working on Health check endpoint for our WebApi and the code for this check is very simple:

        var report = _bucket.Ping(ServiceType.KeyValue);
        var kvOk = report.Services.First().Value.All(x => x.State == ServiceState.Ok);
        return kvOk;

For testing purpose I stopped CouchbaseServer and I got

Cannot access a disposed object.
Object name: ‘System.Net.Sockets.Socket’.

Then I started CouchbaseServer again. But exception is still here for some period of time.
This all in CouchbaseNetClient version 2.7.16. Hope it will help to reproduce in latest SDK.