For the past few weeks I’ve encountered memory leak issue when using Couchbase .NET SDK. The app I’m using is a worker which receive a few hundreds messages per minute from Azure Service Bus and it throw
OutOfMemoryException after 2-3 days of running.
Azure.Messaging.ServiceBus package to receive messages and insert the record into Couchbase.
I’m using Couchbase Client latest version 3.2.3, Couchbase server Enterprise Edition 6.0.3 build 2895
The reason I think this is related to Couchbase client is by comment out all the code related to Couchbase, the problem go away. Here all 3 lines of code that related to Couchbase in that worker
var bucket = await _namedBucketProvider.GetBucketAsync(); var collection = bucket.DefaultCollection(); await collection.UpsertAsync("some_document_key", dataAsJsonString, _defaultUpsertOptions.CancellationToken(cancellationToken));
Comment out just the third or both second or third line also have the leak. Interestingly, I can’t reproduce the leak locally (on Windows 10 machine) this only happened on our k8s pod.
I’ve taken a few memory dumps for this issue
- Dump after restart a few minutes (T0): dump-1635754392.zip - Google Drive
- Dump after T0 5 minutes: dump-1635754699.zip - Google Drive
- Dump after T0 30 minutes: dump-1635756534.zip - Google Drive
It seems like there are CancellationTokenSource that aren’t cleaned up/unregistered:
And here is the memory chart (on test instance, I ramp up the messages so it occurred faster):