Hello,
I would be happy to get your advise on some issue that came up lately.
We are using CouchBase version: 2.1.1 community edition (build-764) on Google compute Engine VM, replicated as 3 total instance. We are using couchbase-client version 1.4.6. Our pom.xml is defined with:
com.couchbase.client
couchbase-client
1.4.6
The problem we’re facing is that after some time, from our client (running also on GCE from using tomcat) we are getting the following error:
Timed out waiting for operation - failing node: …
net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: …
…
…
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: ourcouchbasedns/ip:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167) ~[spymemcached-2.11.5.jar:2.11.5]
at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) ~[spymemcached-2.11.5.jar:2.11.5]
…
In the catalina.out of tomcat we have the following exceptions:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-bio-80-exec-1581"
java.lang.OutOfMemoryError: Java heap spaceException in thread “http-bio-80-exec-1759” Exception in thread “Memcached IO over {MemcachedConnection to ourcouchbasedns-1/ip:11210 ourcouchbasedns-2/ip:11210 ourcouchbasedns-3/ip:11210}” Exception in thread “http-bio-80-exec-1446” Exception in thread “http-bio-80-exec-1767” Exception in thread “http-bio-80-exec-1763” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1758” Exception in thread “http-bio-80-Acceptor-0” Exception in thread “http-bio-80-exec-1591” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1743” java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1741” Exception in thread “http-bio-80-exec-1774” java.lang.OutOfMemoryError: Java heap space
Exception in thread “Couchbase View Thread” Exception in thread “http-bio-80-exec-1656” Exception in thread “http-bio-80-exec-1658” Exception in thread “I/O dispatcher 3” java.lang.OutOfMemoryError: Java heap space
Exception in thread “http-bio-80-exec-1324” Exception in thread “Couchbase View Thread” at java.net.SocketInputStream.socketRead0(Native Method)
Exception in thread “http-bio-80-exec-1647” Exception in thread “http-bio-80-exec-1”
…
Since the couchbase do not serve the client and the node filed, after some time our tomcat server also stuck since too many threads are probably stuck with no response.
We are running about 20-30 requests per second, but it also fail after few days with lower traffic.
Can you please suggest what can we do in such matter?