Possible bug with admin ui? # of connections does not get closed


#1

Hi,

I’ve just noticed that # of connections do not get closed.

Steps to re-produce

  1. Run a custom script to seed some test data (just doing insert operations)
  2. Wait a few minutes.
  3. Restart couchbase server
  4. Disconnect lan cable just in case
  5. Open up admin ui then click a bucket
  6. Even at this point, I can see # of connections are over 70.

Is this a known bug?


#2

I am not aware of a bug similar to the behaviour that you have just described.

I do have a few questions, though:

  1. What version of Couchbase Server are you running?
  2. What were the number of connections before running the test script, perhaps you could share said test script?
  3. Where exactly are you seeing these connections? Is this the total connections graph in the UI?
  4. Could you run a netstat -anp on the node when you see this increased number of connections? This will allow you identify which ports are in use and from which IPs, as well as the program using the socket.

#3

Hello Matt,

  1. I’m on 4.0 CM
  2. It’s hard to say at this point. It’s been hours and I’m still seeing 50+ connection on my local.
  3. On Admin UI, I clicked a bucket then there is “xx connections” box under the “SERVER RESOURCES”
  4. Not exactly sure if I’m seeing right data, but I have a lot of localhost.52305, localhost.11209, and localhost.11210, and similar addresses.

#4

Bump…

Still having the same issue.


#5

bump again…

I see only 200 reads / second, but have 3700+ connections.


#6

Hey Moono,

Sorry for not getting back to you about this, were you able to collect the netstat? It would be interesting to see what state the connections are in.

If possible could you upload the netstat output that you collect after reproducing the issue?
Obviously feel free to sanitise any IPs if you think that they are identifying, although try to keep the same IP sanitised to the same name so it’s easy to match up connections (ideally you could mark the client IP).


#7

Hi Matt,

Here’s output (I replaced ips to a fake number).

I did not know where to run the cmd. I ran it on an index server.


#8

Well it looks as though the connections displayed in the UI are legitimate!

It seems as though a large number of clients (from different IPs assuming that you didn’t randomise the last 3 bits of the foreign address) are connecting to the node, specifically via port 11210 which is the data access port.

The issue of the number of connections constantly increasing might not be a server-side issue (your original value of 70 could be normal considering all of the inter-node connections), but rather a client-side one.

What SDK do you use for your clients?
Would it be possible for you to share your connection + data access code for Couchbase, it may be that you are unexpectedly recreating connections.
Perhaps a little more info about your application setup would be useful here too!


#9

Hi Matt,

We have PHP application with PHP-FPM + nginx (typical setup for PHP) + Couchbase PHP SDK (latest).

Um…I don’t do anything special with my application.

We simply have

$this->cluster = new CouchbaseCluster($host, $username, $password); $bucket = $this->cluster->openBucket("bucketName");

That’s really about it.

Is there anything I can provide more?


#10

Do you use a singleton for your Bucket instance? If not then this could be creating many bucket objects, which each open up more connections.


#11

I use singleton for the bucket object.

Even if I did not, I thought couchbase sdk tries to re-use an existing connection.

Although I still don’t understand how this uses an existing connection, here is a thread.


#12

How did you count # of connections from the netstat output? The output has 837 lines, whereas I have over 3000+ connections.