Couchbase server randomly disconnects KV endpoints/KeepAlive not working

maybe more useful information from the trace log from when this quick disconnect happens:

2016-08-18 14:34:22,203 [cb-io-1-1] (Slf4JLogger.java) INFO  - Connected to Node localhost
2016-08-18 14:34:22,203 [cb-io-1-1] (Slf4JLogger.java) DEBUG - Connected (DISCONNECTED) to Node localhost/127.0.0.1
2016-08-18 14:34:22,203 [cb-io-1-1] (Slf4JLogger.java) TRACE - State (CouchbaseNode) DISCONNECTED -> CONNECTED
2016-08-18 14:34:22,203 [cb-io-1-1] (Slf4JLogger.java) TRACE - State (KeyValueService) CONNECTED -> DEGRADED
2016-08-18 14:34:22,204 [cb-io-1-1] (Slf4JLogger.java) TRACE - State (ServiceStateZipper) CONNECTED -> DISCONNECTED
2016-08-18 14:34:22,204 [cb-io-1-1] (Slf4JLogger.java) INFO  - Disconnected from Node localhost
2016-08-18 14:34:22,204 [cb-io-1-1] (Slf4JLogger.java) DEBUG - Disconnected (CONNECTED) from Node localhost/127.0.0.1
2016-08-18 14:34:22,204 [cb-io-1-1] (Slf4JLogger.java) TRACE - State (CouchbaseNode) CONNECTED -> DISCONNECTED
2016-08-18 14:34:22,205 [cb-io-1-1] (Slf4JLogger.java) DEBUG - [localhost/127.0.0.1:11210][KeyValueEndpoint]: Channel Active.
2016-08-18 14:34:22,205 [cb-computations-4] (Slf4JLogger.java) DEBUG - Successfully enabled Service BINARY on Node localhost/127.0.0.1
2016-08-18 14:34:22,205 [cb-computations-4] (Slf4JLogger.java) DEBUG - Starting to discover config through Carrier Bootstrap

and after that you just get these:

2016-08-18 14:34:22,206 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,207 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,208 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,209 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,210 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,211 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,212 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,213 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100 MICROSECONDS
2016-08-18 14:34:22,214 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 128 MICROSECONDS
2016-08-18 14:34:22,215 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 256 MICROSECONDS
2016-08-18 14:34:22,216 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 512 MICROSECONDS
2016-08-18 14:34:22,217 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 1024 MICROSECONDS
2016-08-18 14:34:22,219 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 2048 MICROSECONDS
2016-08-18 14:34:22,222 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 4096 MICROSECONDS
2016-08-18 14:34:22,227 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 8192 MICROSECONDS
2016-08-18 14:34:22,236 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 16384 MICROSECONDS
2016-08-18 14:34:22,253 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 32768 MICROSECONDS
2016-08-18 14:34:22,287 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 65536 MICROSECONDS
2016-08-18 14:34:22,354 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100000 MICROSECONDS
2016-08-18 14:34:22,461 [cb-core-3-1] (Slf4JLogger.java) TRACE - Retrying GetBucketConfigRequest{observable=rx.subjects.AsyncSubject@76d95e68, bucket='genbook'} with a delay of 100000 MICROSECONDS

until it times out:

Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException
        at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:71)
        at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:290)
        at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:265)
        at com.genbook.utilities.actions.IndexCouchbaseAction.connectCouchbase(IndexCouchbaseAction.java:331)
        at com.genbook.utilities.actions.IndexCouchbaseAction.isRunnable(IndexCouchbaseAction.java:209)
        at com.genbook.utilities.UtilityBase.main(UtilityBase.java:147)
Caused by: java.util.concurrent.TimeoutException
        ... 6 more

Now, I’ve seen many posts already about TimeoutExceptions and that Blocking.blockForSingle, it seems to be called from a lot of different places. For example, this might be the most extensive one on that subject: Java Client 2.3.1 : N1QL queries throwing TimeoutException [upgraded to Couchbase 4.5]