We have a constellation of two hundred ASP.NET apps (C# SDK 1.3.10) speaking all to the same Couchbase cluster (3.0.3). Each .NET app opens connections to 2 different Buckets, with minPoolSize=2 and maxPoolSize=5. When we were running on a Couchbase cluster of version 3.0.1, we had between 800 and 2000 connections per Couchbase Node as expected. After changing to 3.0.3, we have turned on all apps in the same instant to the Cluster, while the apps stood under high live traffic. The connections have rapidly climbed to 10000 per Couchbase Node, after which the Couchbase has stopped to respond. When we then turned off our feature related on couchbase and restarted all apps, the connections went down to maybe 9000 (at least in the web console) and then stayed forever.
We’re using a singleton pattern to ensure that we have only one instance of CouchbaseClient class per .NET app (and it works well with the 3.0.1 server). How could it happen that Couchbase suddenly received 5x more connections?
I can’t think of any changes between 3.0.1 -> 3.0.3 which would have such an affect on the number of connections used. I would suspect this is some kind of environmental issue, perhaps the client or firewall related?
Can you check the breakdown in connections across the different ports (using netstat). Are you seeing a large number of ports in TIME_WAIT or similar?
Unfortunately I can’t use netstat, now the servers are already restarted.
But what we did is the following (in a test app):
- opened nearly 10000 connections per Node
- edited web.config of this ASP.NET app
- what we’ve seen is that some connections remained open after the app has restarted.
We suppose that this problem appeared when editing a master configuration file that has triggered a restart of all 200 apps.
We’re testing now if the problem persists with .NET SDK 2.1.1. It is a stable version?