What does "curr_connections" actually report?


#1

In the JSON from the stats API, the “curr_connections” reports something that looks like the total number of connections to the whole cluster, numerous times. In my case, an 8 node cluster with 16,000 connections will report 58 items having 16,000 each. The real number of connections to each node is closer to 2,000. I don’t know what the significance of 58 items might be, but it could be the total number of client hosts.

I see something similar reported via DogStatsD. The metric “couchbase.by_bucket.curr_connections” reports something similar, which makes the monitoring graph fairly misleading, especially taken as a sum.

Is this a bug or is there some logical rationale behind this metric report? What I am expecting is the value from the couchbase console under “Server Resources / connections” which in my case does report 2,000 connections (the number of connections for the node that the console hits.)

This is on Community 2.2.


#2

@ingenthr hey could you help here please ?


#3

I’ll have a look, but @pvarley may know of the top of his head.


#4

I assume the REST end point you are using is:

http://localhost:8091/pools/default/buckets/[bucket_name]/stats

curr_connections is the total number of connections on port 11210 from all nodes. It is worth noting that this stat is for the whole cluster and not per bucket.

As detailed in the manual when the stat endpoint is queried it can take a zoom parameter, if one is not set it will default to the minute zoom level. As a result the endpoint will return 60 items for each stat. each one representing a second. The stats can be matched up to the time they’re captured by using the timestamp array that is also returned.

There is a known issue where the stats REST endpoint does not always return 60 items this could explain why there are only 58 in this case. This is fixed in Couchbase Server 4.0, which is currently in beta.

This is on Community 2.2.

Couchbase Server 2.2 is nearly two years old, I would suggest upgrading there have been tons of improvements! :smiley:


#5

If only I could upgrade the server from 2.2. What’s stopping me is an enormous amount of code using the .NET client. While I could probably work around the API changes easily enough, moving to a later version of .NET just so that we could upgrade a client library is not really an option. I’ve started using cbstats instead of the stats REST API now.

What I have is approximately 200 app servers connecting to an 8 node cluster with 3 buckets. After a huge traffic spike, the connection counts go way up and stay up (approaching the 9000 connection limit) and it looks like the client lib takes it time for releasing the connections. This is making me nervous because I don’t know how to predict the behavior or how to tell the client lib (.NET 1.3.11) to be more aggressive at releasing connections. That’s probably a question for the .NET client forum, if there is one.

Thanks for your responses

James


#6

If only I could upgrade the server from 2.2. What’s stopping me is an enormous amount of code using the .NET client.

The Cluster can be upgraded independently of the .Net SDK client see the compatibility table in the SDK documentation.

What I have is approximately 200 app servers connecting to an 8 node cluster with 3 buckets. After a huge traffic spike, the connection counts go way up and stay up (approaching the 9000 connection limit) and it looks like the client lib takes it time for releasing the connections. This is making me nervous because I don’t know how to predict the behavior or how to tell the client lib (.NET 1.3.11) to be more aggressive at releasing connections.

In the 1.X .NET SDK, the number of connections is control by minPoolSize and maxPoolSize settings. The defaults are 10 and 20 respectively. The following formula can be used to work out the maximum number of connections:

Number of Buckets * Number of application server * maxPoolSize = total number of connections

In theory the total number of connections in this case could be 12,000 assuming the defaults are being used. This is above the 9,000 I suggest looking at tuning maxPoolSize down. In Couchbase Server 3 the limits have been increase to 30,000.

I believe the 2.X version of the .NET SDK is much more efficient with connection however that will require code changes.