Server idle yet memcached.exe and erl.exe CPU use is extremly high


#1

Dev server is idle yet memcached.exe is using ~70% CPU and erl.exe is using ~25%.

Windows 2008R2 on Azure.

Version: 3.0.1 Community Edition (build-1444)

Cluster State ID: 013-040-44

Restart doesn’t help :hushed:


#2

Hey Itay,

What was happening before the server was idle? I mean, have you been loading data into Couchbase or otherwise working with it?

It looks like it could be this bug:

https://issues.couchbase.com/browse/MB-12775

So far, that bug hasn’t been fixed but you might find something useful in the bug comments.

Thanks,

Matthew.


#3

Nothing interesting happened before the server was idle. it was idle even before being idle.
I read the bug comments but there is no solution there other the basically throw the server away and start a new one :sleepy:

If it is about me installed N1QL some time ago, is there a way to uninstall it ?


#4

10 hours and several restarts later, still 100% CPU on an idle server


#5

20 hours later, still 100% CPU

@cihangirb , Can it be related to a view I created ?
Database is small. Views are usually created in minutes


#6

I agree with @matthew - looks like MB-12755. We have an engineer looking at it.
thanks
-cihan


#7

Still 100%.

This bug is at least 3 months old and without a cure. A part from that, it doesn’t really help me to know that it is documented.

Is there anything I can do to restore the server or should I kill it and lose all the test data, including views ?


#8

is this your only node or only replica of the data or do you have more nodes? one option is to remove the node and bring it back in again with full recovery for short term relief. in the meantime we’ll take a look at the bug but hard to promise a timeline.
thanks
-cihan


#9

just saw the note from Dave on the bug - could you get us this from the machine having trouble?
cbcollect_info could be useful; but if it’s a spin or something then a minidump (Task Manager -> Processes -> right click “memcached.exe” -> Create Dump File) is probably even more helpful.
thanks
-cihan


#10

Still 100%
It is the only node, no replica, for dev
Once I’ll be able to download memcached.exe dump from a stuck machine, I’ll send it to you by email


#11

I am not a Couchbase Server user (don’t even know what that is), but I am using RabbitMQ in a project and RabitMQ uses Erlang.

This morning, when I arrived at work, I found erl.exe running at 25% of the CPU even though there has been absolutely no use been made of RabbitMQ for many days (possibly weeks) now.

Sounds like a Erlang bug.


#12

@cihangirb,

It just started again.memcached.exe is >90% continuously.
I downgraded the node from Azure A2 to A1.

Where you able to learn anything from the dump I sent you by email ?

I followed your suggestion and gracefully failed it. It takes ages to fail a node with 100% CPU. When finished, erl.exe CPU usage was ~15%. memcached.exe practically disappeared.
Node recovered. This should not occur :tired_face:

What should I do the next time ?
What should I do so that there will not be a next time ?

Itay


#13

Hi @Itay, Don’t have a fix for the issue. We need access to the box to do a little more digging. See dave’s note here: https://issues.couchbase.com/browse/MB-12775

Would you be able to give access to the box?
thanks
-cihan


#14

The box is busy doing other stuff :smirk:


#15

Was there a resolution to this issue? I’m facing the same thing on my Windows 2012 R2 Azure VM. The 2 processes, memcached and erl.exe. eat up 99% CPU with memcache spiking between 75% and 90% and erl.exe eating the remainder.


#16

This was reported as https://issues.couchbase.com/browse/MB-14036, which is marked as fixed in 4.0.0. I believe 4.0.0 should be out in the near future.


#17

Does 4.0.0 replaces 3.x.x ?
Or more specifically, should I upgrade from 3.0.1 CE to 4.0.0 or should I continue with the 3.x.x family ?


#18

Hi @itay, 4.0 replaces 3.0 so you should upgrade when 4.0.0 is released. you will get improvements in performance by upgrading even if you are not using the new query language or indexing on the core data operations.
thanks
-cihan


#19

Will 4.0.0 new release be Community Edition ?


#20

yes RC0 that has been published this past monday already has the release candidate for community edition and enterprise edition - couchbase.com/downloads
-cihan