Sudden High CPU Usage For Couchbase Server

#1

I’m having an issue with very high CPU usage for a Couchbase Linux server (reporting as ~650% for memcached on a machine with 8 cores). The CPU usage went way up just in the last few hours, which I don’ t think it’s ever done before. We have 2 couchbase servers, and normally they both sit at very low CPU usage (around 10%), and have been going on happily for the last 2.5 years or so. The problem is that I know almost nothing about Couchbase or Linux, and the person who set this up has left the company, so it’s very hard for me to diagnose.

A summary of the most recent log entries is…
21:26 June 30th 2015, mb_master000 ns_1@172.24.126.133 Haven’t heard from a higher priority node or a master, so I’m taking over.
21:21 June 30th 2015, menelaus_web102 ns_1@172.24.126.132 Client-side error-report for user undefined on node ‘ns_1@172.24.126.132’
04:42 Nov 3rd 2014, mb_master000 ns_1@172.24.126.132 Haven’t heard from a higher priority node or a master, so I’m taking over.

Notice that last 2 entires happened very recently, and the entry before that was last year. The server with the high CPU is 172.24.126.133.

See the image below for the resources on 172.24.126.133. Please let me know what else would be useful to include.

2015-07-01_11-21-26.jpg988×577 76.5 KB

Any help is much appreciated.

#2

What version/platform is this?

#3

Version is 2.2.0 Enterprise Edition (build-821). The servers still appear to be running ok (I can still log into the application they’re used for). All other resources (memory usage/disk usage are very low and haven’t changed with the CPU usage increase).

#4

Do you think I should just try a manual fail over and re-balance for the server with the high CPU usage? I’m reading a little bit about it, but would that be a relatively safe thing to do?

#5

Can you rebalance upgrade? 2.2 is quite old by now.

#6

I’ll just give the rebalance a go first and see how that goes. I realise it would be a good idea to upgrade, but I’ll leave that as a last resort if the re-balance doesn’t fix anything. Cheers.

#7

Hi,

I’ve decided to go ahead and upgrade Couchbase to the latest version (v3.0.3). The servers are CentOS by the way, and I’ll be installing the package “couchbase-server-enterprise-3.0.3-centos6.x86_64.rpm”. I’m going to go down the online upgrade path with a swap rebalance.

Before I do this however I want to check that the .NET library we’re using to talk to Couchbase won’t have any issues. We’re using the NuGet package CouchbaseNetClient.1.3.4, which I realise isn’t the latest (latest on NuGet is 1.3.12), but I’d rather avoid upgrading to the latest version right now, because it will mean re-compiling and deploying all our projects, which isn’t something we have the time for at the moment.

Is anyone aware of any public API changes from Couchbase Server 2.2 to 3.0.3 that might have changed, or can you direct me to where I might find this information?

Thanks again :smile: