Timeouts Happening more frequently across java clients


#1

We are seeing a lot of timeouts at application startup. I have checked our file handle limits (they’re set at 100k and we have nowhere near that many opened (about 8k on the system). We also have 13GB available of RAM.

This occurs at startup, as I mentioned, and the application needs to be restarted to make it work. It’s hit or miss when it works successfully or not. It does not recover.

We saw this problem before but it largely went away - pretty sure I posted here and we never really resolved it. We are running Couchbase Server 4.5.1 and a variety of Couchbase Java client versions, but this most recently occurred with 2.4.6.

Thanks - H

Below is a stack trace:
java.util.concurrent.TimeoutException
at com.couchbase.client.core.utils.Blocking.blockForSingle(Blocking.java:74)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:343)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:327)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:316)
at com.company.couch2.CouchbaseClientManager.lambda$initializeBuckets$3(CouchbaseClientManager.java:160)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.company.couch2.CouchbaseClientManager.initializeBuckets(CouchbaseClientManager.java:154)
at com.company.couch2.CouchbaseClientManager.lambda$start$1(CouchbaseClientManager.java:87)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.company.couch2.CouchbaseClientManager.start(CouchbaseClientManager.java:71)
at io.dropwizard.lifecycle.JettyManaged.doStart(JettyManaged.java:27)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.server.Server.start(Server.java:342)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.Server.doStart(Server.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:43)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:76)
at io.dropwizard.cli.Cli.run(Cli.java:70)
at io.dropwizard.Application.run(Application.java:72)
at com.company.application.service.main(Service.java:292)


#2

@unhuman

Did u get a way to resolve above ?

I am facing similar problem.


#3

No. There are some theories here that the (virtual) host (client side) is/was underprovisioned. I haven’t heard many complaints about this lately, but we’ve been building things out more too… So, either people have given up or have found ways around this w/o telling me.


#4

Maybe some clues here: Couchbase client randomly timeouts during document upsert