Timed out waiting for operation - failing node

Java SDK
1

Hello,
I would be happy to get your advise on some issue that came up lately.
We are using CouchBase version: 2.1.1 community edition (build-764) on Google compute Engine VM, replicated as 3 total instance. We are using couchbase-client version 1.4.6. Our pom.xml is defined with:

com.couchbase.client
couchbase-client
1.4.6

The problem we’re facing is that after some time, from our client (running also on GCE from using tomcat) we are getting the following error:
Timed out waiting for operation - failing node: …
net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: …
…
…
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: ourcouchbasedns/ip:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167) ~[spymemcached-2.11.5.jar:2.11.5]
at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) ~[spymemcached-2.11.5.jar:2.11.5]
…
In the catalina.out of tomcat we have the following exceptions:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-bio-80-exec-1581"
java.lang.OutOfMemoryError: Java heap spaceException in thread “http-bio-80-exec-1759” Exception in thread “Memcached IO over {MemcachedConnection to ourcouchbasedns-1/ip:11210 ourcouchbasedns-2/ip:11210 ourcouchbasedns-3/ip:11210}” Exception in thread “http-bio-80-exec-1446” Exception in thread “http-bio-80-exec-1767” Exception in thread “http-bio-80-exec-1763” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1758” Exception in thread “http-bio-80-Acceptor-0” Exception in thread “http-bio-80-exec-1591” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1743” java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1741” Exception in thread “http-bio-80-exec-1774” java.lang.OutOfMemoryError: Java heap space

Exception in thread “Couchbase View Thread” Exception in thread “http-bio-80-exec-1656” Exception in thread “http-bio-80-exec-1658” Exception in thread “I/O dispatcher 3” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1324” Exception in thread “Couchbase View Thread” at java.net.SocketInputStream.socketRead0(Native Method)
Exception in thread “http-bio-80-exec-1647” Exception in thread “http-bio-80-exec-1”
…

Since the couchbase do not serve the client and the node filed, after some time our tomcat server also stuck since too many threads are probably stuck with no response.
We are running about 20-30 requests per second, but it also fail after few days with lower traffic.

Can you please suggest what can we do in such matter?

1 Like
2

You most likely have a memory leak in your application or a too small heap space configured. There are currently no known issues that the java client leaks, this is most probably just coincidence that the view thread was trying to allocate some data right before it failed.

This is also the cause of the timeouts, because before you OOM, the JVM deliberately tries to free memory, leading to long GC pauses and this causes timeouts.

I recommend you to enable heap dumping on OOM and then investigate what’s filling it up. When you see OOMs in your logs, it’s too late already :wink:

1 Like
3

@ronic : Did daschl tip help you get past the issue?

We have been facing this too with couchbase server 3.0.3

4

@aksbhatia can you shed more light on your issue too?