.Net Core fails to connect after some time

Hi,
I have a .Net core 2.2 application running on Azure Linux Web App.
It connects to Community Edition 6.0.0 build 1693
CouchbaseNetClient v2.7.16
Couchbase.Extensions.DependencyInjection v2.0.2

After starting the app, it is working perfectly fine for some time. Several hours after that, I start experiencing timeouts from both GetDocumentsAsync and UpsertAsync.
This only happens on Azure Web App. It never happened on my local machine or on the on premise servers.
The exception message is:

The operation has timed out [“s”:“kv”,“i”:“2f36”,“c”:“c38c7d20f03f909d/95c0a42407e8755e”,“b”:“Peleg”,“l”:“169.254.0.4:43375”,“r”:“52.164.125.222:11210”,“t”:15000000]

but I can also see this exception, which makes no sense to me, since up until a few minutes prior to this, everything worked fine. And nobody changed any configurations on the app or on the bucket. (Also, this isn’t the user name I set in the app settings)

Authentication failed for user ‘Peleg’

I can also see logs from the Couchbase SDK:

Handling disconnect for connection 4b280dc4-2492-4c17-83f5-ab02a00ec166: Couchbase.IO.SendTimeoutExpiredException: The operation has timed out. [“s”:“kv”,“i”:“2f23”,“c”:“c38c7d20f03f909d/15c3dc31b5c547c7”,“b”:“Gateways”,“l”:“169.254.0.4:42091”,“r”:“52.138.203.44:11210”,“t”:15000000]

Failed to perform SelectBucket operation for ‘Peleg’. Reason None.

Hi @gilad.teller,

Would you mind posting some of the code you use to connect to a cluster?

And since you are using the DependencyInjection extension, have you written any shutdown code (for example, registering with ApplicationStopped?

In addition to what @matthew.groves has requested. Based on what you had mentioned

It works on my local and on premise
I would also check the “Always On” option. I am not sure how you do the bootstrapping (the code requested my matthew will help). But try the below maybe it helps (courtesy microsoft docs)

To keep Web App alive and available all the time need to enable “ Always On ” option. By default, web apps are unloaded if they are idle for some period of time. This lets the system conserve resources. In Basic or Standard mode, you can enable Always On to keep the app loaded all the time.

1 Like

@AV25242 I am already using the “Always On” option.
@matthew.groves
In Startup.cs:
I have these lines in ConfigureServices method:

services.AddCouchbase(Configuration.GetSection("Couchbase"));
services.AddCouchbaseBucket<IUmbracoBucketProvider>("Umbraco");
services.AddCouchbaseBucket<IPelegBucketProvider>("Peleg");
services.AddCouchbaseBucket<IGatewaysBucketProvider>("Gateways");
//all couchbase calls from the app are made through this service
services.AddSingleton<ICache, CouchbaseManager>(); 

I also have this in Configure method:

applicationLifetime.ApplicationStopped.Register(() =>
{
     app.ApplicationServices.GetRequiredService<ICouchbaseLifetimeService>().Close();
});

The actual usage of Couchbase is in one service that I am adding as a singleton:

public CouchbaseManager(ILogger<CouchbaseManager> logger, IPelegBucketProvider pelegBucketProvider, IUmbracoBucketProvider umbracoBucketProvider, IGatewaysBucketProvider gatewaysBucketProvider)
{
	_logger = logger;
	IBucket gatewaysBucket = gatewaysBucketProvider.GetBucket();
	IBucket pelegBucket = pelegBucketProvider.GetBucket();
	IBucket umbracoBucket = umbracoBucketProvider.GetBucket();

	Buckets = new Dictionary<E_Bucket, IBucket>
	{
		{ E_Bucket.Gateways, gatewaysBucket },
		{ E_Bucket.Peleg, pelegBucket },
		{ E_Bucket.Umbraco, umbracoBucket }
	};

	BucketContexts = new Dictionary<E_Bucket, BucketContext>
	{
		{ E_Bucket.Gateways, new BucketContext(gatewaysBucket) },
		{ E_Bucket.Peleg, new BucketContext(pelegBucket) },
		{ E_Bucket.Umbraco, new BucketContext(umbracoBucket) }
	};
}

public async Task<Dictionary<string, T>> GetAsync<T>(IEnumerable<string> keys, E_Bucket bucket)
{
	Dictionary<string, T> result = new Dictionary<string, T>();
	try
	{
		if (keys != null && keys.Any())
		{
			IDocumentResult<string>[] cbRes = await Buckets[bucket].GetDocumentsAsync<string>(keys);
			result = cbRes.Where(x => x.Success).ToDictionary(y => y.Id, z => Tools.FromJson<T>(z.Content));
		}
	}
	catch (Exception ex)
	{
		_logger.LogError($"CB Multi GetAsync exception message: {ex.Message}", ex);
	}
	return result;
}

public async Task<bool> UpsertAsync<T>(Dictionary<string, T> values, E_Bucket bucket, TimeSpan timeSpan)
{
	if (values != null && values.Count > 0)
	{
		try
		{
			List<IDocument<string>> docs = new List<IDocument<string>>();
			foreach (KeyValuePair<string, T> value in values)
			{
				Document<string> doc = new Document<string>()
				{
					Id = value.Key,
					Content = Tools.ToJson(value.Value),
					Expiry = (uint)timeSpan.TotalMilliseconds
				};
				docs.Add(doc);
			}
			IDocumentResult<string>[] results = await Buckets[bucket].UpsertAsync(docs);
			if (results.All(x => !x.Success))
			{
				Exception ex = results.FirstOrDefault(x => x.Exception != null).Exception;
				if (ex != null)
				{
					throw new Exception("CB Multi UpsertAsync all failed", ex);
				}
				else
				{
					throw new Exception("CB Multi UpsertAsync all failed");
				}
			}
			return true;
		}
		catch (Exception ex)
		{
			_logger.LogError($"CB Multi UpsertAsync exception message: {ex.Message}", ex);
		}
	}
	return false;
}

Another thing that might be a problem: As you can see in my code above, the CouchbaseManager service is configured as a singleton. Is this correct? Maybe it should be Scoped?

I see you’re using BucketContext, does this mean you are using Linq2Couchbase as well? I don’t think having CouchbaseManager as a singleton would be a problem, but I’m tagging @btburnett3 just in case.

Yes, I skipped adding the query method to my code. I don’t have data about it’s performance, since it currently has only one usage.

public async Task<IEnumerable<T>> QueryAsync<T>(Expression<Func<T, bool>> predicate, E_Bucket bucket)
{
	try
	{
		IEnumerable<T> qRes = await BucketContexts[bucket].Query<T>().Where(predicate).ExecuteAsync();
		IEnumerable<T> res = qRes.Select(x => x);
		return res;
	}
	catch (Exception ex)
	{
		_logger.LogError($"CB QueryAsync exception message: {ex.Message}", ex);
		return null;
	}
}

I’m calling it something like this:
IEnumerable<MyClass> lst = await _couch.QueryAsync<MyClass>(x => x.Param1 == a1, E_Bucket.Peleg);

@gilad.teller

I don’t see any issue with CouchbaseManager being a singleton. That said, BucketContext isn’t really designed to be a singleton. Much like a DbContext in EntityFramework, it’s generally transient or perhaps scoped to the request. I think it will work okay for read-only operations, but any kind of change tracking and save back would bleed between requests. It also isn’t thread safe, so you could get race conditions or synchronization exceptions.

Just to check, I removed the BucketContext and all Linq2Couchbase usage from my code. But nothing changed.
The same logs, and the same timeouts :frowning:
Is it possible that it is actually a networking problem? I doubt it, because after a restart the app is working flawlessly. Is there something I can check to make sure?

1 Like

If you suspect a network issue, I recommend trying sdk-doctor.

1 Like

Few things I would like to comment here based on my observation and comments from @btburnett3 on BucketContext and Thread safety.

               You have declared the Couchbase Manager as Singleton and any implementations inside it i.e the Bucket Context will also be singleton just a FYI. I read that you have removed  any  reference to the bucket context. I dont have access to your source code but based on all the conversations it appears to me that either your connection is stale and has timed-out or  the call is not making it to the CB server. 

Can you try these may be it help
- Leave your implementation as is and change the Couchbase Manager instance to Scoped (Instance / Request).
- Create a simple app to begin with
Marketplace has a simple asp.net core couchbase application VISX that I had created sometime back it uses simple DI maybe you can use it as reference to begin with.

Some example reference can also be found here