Client intermittently fails to open bucket or connect to a node during bootstrap


#1

Environment:
Couchbase Server: Version: 4.1.0-5005 Enterprise Edition (build-5005)
Couchbase Client: couchbase-java-client-2.2.8.jar, couchbase-core-io-1.2.9.jar, rxjava-1.0.17.jar
3-node cluster (node155, node158, node159)

When I bring up my client on a 3-node cluster I will intermittently see connection issues to one node, or to the bootstrap node, on one or more of my applications on the same box bootstrapped at the same time. Sometimes everything works fine, this leads me to believe there may be a race condition.

2016.07.15 18:03:30:179 EDT | Info       | CommonPersistency | localhost-startStop-1

	[Couchbase] ioPoolSize is less than 3 (2), setting to: 3

2016.07.15 18:03:30:179 EDT | Info       | CommonPersistency | localhost-startStop-1

	[Couchbase] computationPoolSize is less than 3 (2), setting to: 3

2016.07.15 18:03:30:449 EDT | Info       | CommonPersistency | localhost-startStop-1

	[Couchbase] CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=3, computationPoolSize=3, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=2, viewServiceEndpoints=2, queryServiceEndpoints=2, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=true, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}

2016.07.15 18:03:30:451 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Diagnostics {
  gc.ps marksweep.collectionCount=2,
  gc.ps marksweep.collectionTime=174,
  gc.ps scavenge.collectionCount=8,
  gc.ps scavenge.collectionTime=130,
  heap.pendingFinalize=0,
  heap.used=init = 134217728(131072K) used = 38855704(37945K) committed = 238026752(232448K) max = 477626368(466432K),
  mem.physical.free=302321664,
  mem.physical.total=1929281536,
  mem.swap.free=1658286080,
  mem.swap.total=2147479552,
  mem.virtual.comitted=3188330496,
  offHeap.used=init = 2555904(2496K) used = 49304184(48148K) committed = 50683904(49496K) max = -1(-1K),
  proc.cpu.time=5570000000,
  runtime.name=5329@jberglund-PS,
  runtime.spec=Oracle Corporation/Java Virtual Machine Specification: 1.8,
  runtime.startTime=1468620197606,
  runtime.sysProperties={awt.toolkit=sun.awt.X11.XToolkit, file.encoding.pkg=sun.io, java.specification.version=1.8, com.couchbase.client.deps.io.netty.packagePrefix=com.couchbase.client.deps., sun.cpu.isalist=, sun.jnu.encoding=ISO-8859-1, bw.nameservice.denyTimeBoundedDuplicateLookups=true, java.class.path=/usr/local/tomcat/tomcat_base/bin/bootstrap.jar:/usr/local/tomcat/tomcat_base/bin/tomcat-juli.jar, java.vm.vendor=Oracle Corporation, bw.nameservice.cachePolicy=Configured, sun.arch.data.model=64, catalina.useNaming=true, java.vendor.url=http://java.oracle.com/, sun.net.spi.nameservice.provider.1=dns,BroadSoft, user.timezone=America/New_York, os.name=Linux, java.vm.specification.version=1.8, oracle.jdbc.timezoneAsRegion=false, sun.java.launcher=SUN_STANDARD, user.country=US, java.net.preferIPv6Addresses=false, sun.boot.library.path=/usr/local/java/jdk1.8.0_72/jre/lib/amd64, sun.java.command=org.apache.catalina.startup.Bootstrap start, bw.nameservice.nCachePolicy=Configured, sun.cpu.endian=little, user.home=/var/empty/bworks, user.language=en, java.specification.vendor=Oracle Corporation, java.naming.factory.url.pkgs=org.apache.naming, java.home=/usr/local/java/jdk1.8.0_72/jre, file.separator=/, line.separator=
, java.vm.specification.vendor=Oracle Corporation, java.specification.name=Java Platform API Specification, bw.nameservice.timeBoundedPoolSize=2, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, entityExpansionLimit=1024, bw.nameservice.useAdditionalSrvRrs=true, maxNotificationTransmitterInternalQueueSize=1000, package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.tomcat., package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.naming.,org.apache.tomcat., bw.config.container.name=tomcat, sun.boot.class.path=/usr/local/java/jdk1.8.0_72/jre/lib/resources.jar:/usr/local/java/jdk1.8.0_72/jre/lib/rt.jar:/usr/local/java/jdk1.8.0_72/jre/lib/sunrsasign.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jsse.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jce.jar:/usr/local/java/jdk1.8.0_72/jre/lib/charsets.jar:/usr/local/java/jdk1.8.0_72/jre/lib/jfr.jar:/usr/local/java/jdk1.8.0_72/jre/classes, server.loader=, java.util.logging.config.file=/usr/local/tomcat/tomcat_base/conf/logging.properties, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, bw.nameservice.nCacheTtlSecs=600, java.runtime.version=1.8.0_72-b15, java.naming.factory.initial=org.apache.naming.java.javaURLContextFactory, user.name=bworks, java.net.preferIPv4Stack=true, path.separator=:, common.loader="${catalina.base}/lib","${catalina.base}/lib/*.jar","${catalina.home}/lib","${catalina.home}/lib/*.jar", os.version=3.10.0-327.el7.x86_64, java.endorsed.dirs=/usr/local/tomcat/tomcat_base/endorsed, java.runtime.name=Java(TM) SE Runtime Environment, com.broadsoft.webcontainer.jvm=true, file.encoding=ISO-8859-1, sun.nio.ch.bugLevel=, bw.nameservice.unreachableServerLingerSecs=60, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, org.apache.catalina.STRICT_SERVLET_COMPLIANCE=true, java.vendor.url.bug=http://bugreport.sun.com/bugreport/, java.io.tmpdir=/usr/local/tomcat/tomcat_base/temp, tomcat.util.scan.StandardJarScanFilter.jarsToScan=log4j-core*.jar,log4j-taglib*.jar, catalina.home=/usr/local/tomcat/8.0.22, java.version=1.8.0_72, tomcat.util.scan.StandardJarScanFilter.jarsToSkip=bootstrap.jar,commons-daemon.jar,tomcat-juli.jar,annotations-api.jar,el-api.jar,jsp-api.jar,servlet-api.jar,websocket-api.jar,catalina.jar,catalina-ant.jar,catalina-ha.jar,catalina-storeconfig.jar,catalina-tribes.jar,jasper.jar,jasper-el.jar,ecj-*.jar,tomcat-api.jar,tomcat-util.jar,tomcat-util-scan.jar,tomcat-coyote.jar,tomcat-dbcp.jar,tomcat-jni.jar,tomcat-websocket.jar,tomcat-i18n-en.jar,tomcat-i18n-es.jar,tomcat-i18n-fr.jar,tomcat-i18n-ja.jar,tomcat-juli-adapters.jar,catalina-jmx-remote.jar,catalina-ws.jar,tomcat-jdbc.jar,tools.jar,commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,commons-math*.jar,commons-pool*.jar,jstl.jar,taglibs-standard-spec-*.jar,geronimo-spec-jaxrpc*.jar,wsdl4j*.jar,ant.jar,ant-junit*.jar,aspectj*.jar,jmx.jar,h2*.jar,hibernate*.jar,httpclient*.jar,jmx-tools.jar,jta*.jar,log4j*.jar,mail*.jar,slf4j*.jar,xercesImpl.jar,xmlParserAPIs.jar,xml-apis.jar,junit.jar,junit-*.jar,ant-launcher.jar,cobertura-*.jar,asm-*.jar,dom4j-*.jar,icu4j-*.jar,jaxen-*.jar,jdom-*.jar,jetty-*.jar,oro-*.jar,servlet-api-*.jar,tagsoup-*.jar,xmlParserAPIs-*.jar,xom-*.jar, sun.net.inetaddr.ttl=0, user.dir=/usr/local/broadworks/PS_Rel_22.0_1.916/bin, os.arch=amd64, bw.nameservice.cacheTtlSecs=86400, java.vm.specification.name=Java Virtual Machine Specification, sun.security.pkcs11.enable-solaris=false, java.awt.printerjob=sun.print.PSPrinterJob, sun.os.patch.level=unknown, sun.net.inetaddr.negative.ttl=0, catalina.base=/usr/local/tomcat/8.0.22, shared.loader=, p=tomcat, java.util.logging.manager=org.apache.juli.ClassLoaderLogManager, java.library.path=/usr/local/broadworks/bw_base/lib:/usr/local/TimesTen/tt_base/lib, java.vendor=Oracle Corporation, java.vm.info=mixed mode, java.vm.version=25.72-b15, sun.io.unicode.encoding=UnicodeLittle, java.ext.dirs=/usr/local/java/jdk1.8.0_72/jre/lib/ext:/usr/java/packages/lib/ext, tomcat.util.buf.StringCache.byte.enabled=true, java.class.version=52.0},
  runtime.uptime=12881,
  runtime.vm=Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM: 25.72-b15,
  sys.cpu.loadAvg=5.03,
  sys.cpu.num=2,
  sys.os.arch=amd64,
  sys.os.name=Linux,
  sys.os.version=3.10.0-327.el7.x86_64,
  thread.count=71,
  thread.peakCount=71,
  thread.startedCount=72
}

2016.07.15 18:03:30:496 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Setting seed hosts to [/10.17.4.155]

2016.07.15 18:03:30:509 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Got instructed to open bucket bwecl

2016.07.15 18:03:30:511 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Loading Config for bucket bwecl

2016.07.15 18:03:30:512 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Loading Config for bucket bwecl

2016.07.15 18:03:30:536 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Got instructed to add Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:536 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Connecting Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:537 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] [couchbase155.rtx.broadsoft.com.]: Got instructed to connect.

2016.07.15 18:03:30:538 EDT | FieldDebug | CommonPersistency | localhost-startStop-1

	[Couchbase] Connect finished, registering for use.

2016.07.15 18:03:30:540 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] Successfully added Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:542 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] Got instructed to add Service BINARY, to Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:542 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] [couchbase155.rtx.broadsoft.com.]: Adding Service BINARY

2016.07.15 18:03:30:545 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] [couchbase155.rtx.broadsoft.com.]: Adding Service BINARY to registry and connecting it.

2016.07.15 18:03:30:545 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Got instructed to connect.

2016.07.15 18:03:30:578 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Initializing connect on Endpoint.

2016.07.15 18:03:30:578 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (KeyValueEndpoint) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:578 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (EndpointStateZipper) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:578 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (KeyValueService) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:579 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (ServiceStateZipper) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:589 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (CouchbaseNode) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:616 EDT | FieldDebug | CommonPersistency | cb-computations-3

	[Couchbase] [couchbase155.rtx.broadsoft.com.][KeyValueService]: Initializing connect on Endpoint.

2016.07.15 18:03:30:616 EDT | Debug      | CommonPersistency | cb-computations-3

	[Couchbase] State (KeyValueEndpoint) DISCONNECTED -> CONNECTING

2016.07.15 18:03:30:738 EDT | FieldDebug | CommonPersistency | cb-io-1-2

	[Couchbase] Requesting supported features: [MUTATION_SEQNO, TCPNODELAY]

2016.07.15 18:03:30:740 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] Requesting supported features: [MUTATION_SEQNO, TCPNODELAY]

2016.07.15 18:03:30:742 EDT | FieldDebug | CommonPersistency | cb-io-1-2

	[Couchbase] Negotiated supported features: [MUTATION_SEQNO, TCPNODELAY]

2016.07.15 18:03:30:743 EDT | FieldDebug | CommonPersistency | cb-io-1-2

	[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Connected Endpoint.

2016.07.15 18:03:30:743 EDT | Debug      | CommonPersistency | cb-io-1-2

	[Couchbase] State (KeyValueEndpoint) CONNECTING -> CONNECTED

2016.07.15 18:03:30:744 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] Negotiated supported features: [MUTATION_SEQNO, TCPNODELAY]

2016.07.15 18:03:30:744 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Connected Endpoint.

2016.07.15 18:03:30:744 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (KeyValueEndpoint) CONNECTING -> CONNECTED

2016.07.15 18:03:30:745 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (EndpointStateZipper) CONNECTING -> CONNECTED

2016.07.15 18:03:30:745 EDT | Debug      | CommonPersistency | cb-io-1-2

	[Couchbase] State (EndpointStateZipper) CONNECTING -> DEGRADED

2016.07.15 18:03:30:745 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (KeyValueService) CONNECTING -> CONNECTED

2016.07.15 18:03:30:746 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (ServiceStateZipper) CONNECTING -> CONNECTED

2016.07.15 18:03:30:746 EDT | Info       | CommonPersistency | cb-io-1-1

	[Couchbase] Connected to Node couchbase155.rtx.broadsoft.com.

2016.07.15 18:03:30:748 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] Connected (CONNECTING) to Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:748 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (CouchbaseNode) CONNECTING -> CONNECTED

2016.07.15 18:03:30:748 EDT | FieldDebug | CommonPersistency | RxCachedThreadScheduler-1

	CouchbaseEvent: NodeConnectedEvent{host=couchbase155.rtx.broadsoft.com./10.17.4.155}

2016.07.15 18:03:30:749 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (KeyValueService) CONNECTED -> DEGRADED

2016.07.15 18:03:30:749 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (ServiceStateZipper) CONNECTED -> DISCONNECTED

2016.07.15 18:03:30:749 EDT | Info       | CommonPersistency | cb-io-1-1

	[Couchbase] Disconnected from Node couchbase155.rtx.broadsoft.com.

2016.07.15 18:03:30:757 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] Disconnected (CONNECTED) from Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:757 EDT | Debug      | CommonPersistency | cb-io-1-1

	[Couchbase] State (CouchbaseNode) CONNECTED -> DISCONNECTED

2016.07.15 18:03:30:757 EDT | FieldDebug | CommonPersistency | cb-io-1-2

	[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Channel Active.

2016.07.15 18:03:30:829 EDT | FieldDebug | CommonPersistency | cb-computations-2

	[Couchbase] Successfully enabled Service BINARY on Node couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:829 EDT | FieldDebug | CommonPersistency | cb-computations-2

	[Couchbase] Starting to discover config through Carrier Bootstrap

2016.07.15 18:03:30:830 EDT | FieldDebug | CommonPersistency | cb-io-1-1

	[Couchbase] [couchbase155.rtx.broadsoft.com./10.17.4.155:11210][KeyValueEndpoint]: Channel Active.

2016.07.15 18:03:30:905 EDT | Info       | CommonPersistency | RxCachedThreadScheduler-1

	SystemEvent: Node Connected Event for host couchbase155.rtx.broadsoft.com./10.17.4.155

2016.07.15 18:03:30:905 EDT | FieldDebug | CommonPersistency | RxCachedThreadScheduler-1

	CouchbaseEvent: NodeDisconnectedEvent{host=couchbase155.rtx.broadsoft.com./10.17.4.155}

2016.07.15 18:03:30:905 EDT | Info       | CommonPersistency | RxCachedThreadScheduler-1

	SystemEvent: Node Disconnected Event for host couchbase155.rtx.broadsoft.com./10.17.4.155

#2

Note that this issue is occuring frequently in our labs/testing, and the issue is easy to reproduce by putting the VM under some CPU load while the application executes the openBucket request. The issue does not appear 100% of the time under CPU load, but it does occur greater than 50% of the time. When the issue does occur, 75 seconds later a log appears showing the bucket was successfully opened (e.g. some timer popped). Some of the initial logs (right after the openBucket request) and the last few logs showing the “open bucket” response are shown below. If you need additional logs to help resolve this issue, please contact myself or jberglund.

2016.07.19 17:08:37:744 EDT | Debug | CommonPersistency | cb-io-1-3
[Couchbase] State (EndpointStateZipper) DEGRADED -> CONNECTED

2016.07.19 17:08:37:744 EDT | Debug | CommonPersistency | cb-io-1-3
[Couchbase] State (QueryService) DEGRADED -> CONNECTED

2016.07.19 17:08:37:744 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] [id: 0x2e967224, L:/10.16.128.27:40320 - R:couchbase158.rtx.broadsoft.com./10.17.4.158:8093] ACTIVE

2016.07.19 17:08:37:744 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] [couchbase158.rtx.broadsoft.com./10.17.4.158:8093][QueryEndpoint]: Channel Active.

2016.07.19 17:08:37:745 EDT | FieldDebug | CommonPersistency | cb-io-1-2
[Couchbase] [couchbase159.rtx.broadsoft.com./10.17.4.159:8092][ViewEndpoint]: Connected Endpoint.

2016.07.19 17:08:37:745 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] State (ViewEndpoint) CONNECTING -> CONNECTED

2016.07.19 17:08:37:745 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] State (EndpointStateZipper) CONNECTING -> DEGRADED

2016.07.19 17:08:37:745 EDT | Debug | CommonPersistency | cb-io-1-2
[Couchbase] State (ViewService) CONNECTING -> DEGRADED

… many more logs

2016.07.19 17:08:37:850 EDT | FieldDebug | CommonPersistency | cb-io-1-3
[Couchbase] Connected (DEGRADED) to Node couchbase159.rtx.broadsoft.com./10.17.4.159

2016.07.19 17:08:37:850 EDT | Debug | CommonPersistency | cb-io-1-3
[Couchbase] State (CouchbaseNode) DEGRADED -> CONNECTED

2016.07.19 17:08:37:851 EDT | FieldDebug | CommonPersistency | RxCachedThreadScheduler-1
CouchbaseEvent: NodeConnectedEvent{host=couchbase159.rtx.broadsoft.com./10.17.4.159}

2016.07.19 17:08:37:851 EDT | Info | CommonPersistency | RxCachedThreadScheduler-1
SystemEvent: Node Connected Event for host couchbase159.rtx.broadsoft.com./10.17.4.159

2016.07.19 17:08:37:852 EDT | FieldDebug | CommonPersistency | cb-io-1-3
[Couchbase] [couchbase159.rtx.broadsoft.com./10.17.4.159:11210][KeyValueEndpoint]: Channel Active.

2016.07.19 17:08:43:429 EDT | FieldDebug | CommonPersistency | cb-computations-1
[Couchbase] Received signal for outdated configuration.

… many more logs (and approx. 75 seconds after the openBucket request)

2016.07.19 17:09:52:922 EDT | FieldDebug | CommonPersistency | cb-computations-1
[Couchbase] [couchbase159.rtx.broadsoft.com.]: Service QUERY already added, skipping.

2016.07.19 17:09:52:922 EDT | Info | CommonPersistency | cb-computations-1
[Couchbase] Opened bucket bwsub

The previous post listed the version of the couchbase jars we are using (shown below as well), but I verified the issue exists in older jar files (from 5 months ago) as well.
couchbase-core-io-1.2.9.jar
couchbase-java-client-2.2.8.jar
rxjava-1.0.17.jar

Thanks


openBucket fails with password mismatch, next openBucket request (for a different/valid bucket) times out