Java Client Connection keep closing and reopening, Auth failure

Java Client: 1.1.9
Couchbase Server: 2.1.1 Enterprise
OS: CentOS in VirtualBox

int numOfCouchbaseConnection = 1 List hosts = Arrays.asList(new URI("http://127.0.0.1:8091/pools"));
CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
builder.setProtocol(ConnectionFactoryBuilder.Protocol.BINARY);
builder.setShouldOptimize(true);
builder.setOpTimeout(32);

couchbaseClients = new ArrayList<>();
for (int i = 0; i < numOfCouchbaseConnection; i++) {
  couchbaseClients.add(new CouchbaseClient(builder.buildCouchbaseConnection(hosts, "products", "Administrator", "")));
}

If I switch the Couchbase Server to 2.0.1 Community, there will be another error

2013-09-05 12:38:56.932 WARN net.spy.memcached.auth.AuthThread: Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:56.938 ERROR net.spy.memcached.protocol.binary.SASLStepOperationImpl: Error: Auth failure
2013-09-05 12:38:56.939 INFO com.couchbase.client.CouchbaseConnection: Reconnection due to exception handling a memcached operation on {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL steps operation, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure.
OperationException: SERVER: Auth failure
at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:164)
at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:176)
at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.finishedPayload(SASLBaseOperationImpl.java:93)
at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:162)
at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.readFromBuffer(SASLBaseOperationImpl.java:41)
at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:563)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:480)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:261)
at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
2013-09-05 12:38:56.940 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL steps operation, topWop=null, toWrite=0, interested=1}, attempt 0.
2013-09-05 12:38:56.940 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: SASL steps operation
2013-09-05 12:38:57.039 WARN net.spy.memcached.auth.AuthThread: Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:58.941 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=0}
2013-09-05 12:38:58.942 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@7fe431c2
2013-09-05 12:38:58.943 WARN net.spy.memcached.auth.AuthThreadMonitor: Incomplete authentication interrupted for node {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=8}
2013-09-05 12:38:58.944 WARN net.spy.memcached.auth.AuthThread: Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:58.948 INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.100.20.250/10.100.20.250:11210

I have the same issue. I spend stress testing and at the same time kill one of nodes. There are no operations I have “WARN net.spy.memcached.auth.AuthThread: Authentication failed to …” What mechanism will allow to continue the work of the application after the restoration of failing node.

1 Like

Hello,

When you connect to Couchbase cluster you should not use the Administrator user. The Administrator should only be used to do some administration operation such as create/edit bucket using the ClusterManager class

When you connect to the cluster to do “normal operations” you have 2 options

  • connect without any security/password
  • connect with an SASL password (that you set at the bucket level).

So for example in your case you will set a password to the products bucket, and connect using:

CouchbaseClient cb = new CouchbaseClient(builder.buildCouchbaseConnection(hosts, “products”, “mypassword”));

Also I see in the code snippet you put in the question that you are creating a list of CouchbaseClient. This is not a a good idea. You just need one client in your application. The “pool” is manage internally in the client.

Regards
Tug
@tgrall

Hey tgrall! I did a performance test for my pool

#threads #connections #queries
1 1 21869
2 1 22191
4 1 21212
2 2 40548
4 4 103663
1 2 23193
1 4 20889

My Testing Code

//Couchbase.java
…
System.out.println(System.currentTimeMillis());
couchbaseClients = new ArrayList<>();
for (int i = 0; i < numOfCouchbaseConnection; i++) {
CouchbaseClient couchbaseClient = new CouchbaseClient(hosts, “products”, “”);
couchbaseClients.add(new AbstractMap.SimpleEntry<>(couchbaseClient, couchbaseClient.getView(“products”, “default”)));
}

	for (int i = 0; i < numOfCurrency; i++)
	{
		Thread thread = new QueryThread(categories.get(random.nextInt(categories.size())));
		threads.add(thread);
	}

for (int i = 0; i < numOfCurrency; i++)
	{
		threads.get(i).start();
	}

public synchronized static Map.Entry<CouchbaseClient, View> getCouchbaseClient() {
couchbaseIndex = couchbaseIndex % couchbaseClients.size();
return couchbaseClients.get(couchbaseIndex++);
}
…
//

//QueryThread.java
…
public void run()
{
System.out.println(System.currentTimeMillis());
long start = System.currentTimeMillis();
long end = start + Couchbase.seconds;
while(System.currentTimeMillis() < end)
{
Map.Entry<CouchbaseClient, View> a = Couchbase.getCouchbaseClient();
Query query = new Query();
query.setStale(Stale.OK);
query.setRange(ComplexKey.of(“waejk;jkgrad”, “3789560789”, m_category, null), ComplexKey.of(“waejk;jkgrad”, “3789560789”, m_category, “”));
ViewResponse viewResponse = a.getKey().query(a.getValue(), query);
Couchbase.atomicInteger.getAndIncrement();
}
System.out.println(System.currentTimeMillis());
System.out.println(Couchbase.atomicInteger.getAndIncrement());
}

No matter how many threads share the same connection, the number of queries of different threads are nearly same. if you have multiple connections for different threads, there will be a huge difference in performance

No matter if you use auth or you don’t, authentication failure after one of servers restart/crash.
If only one thread is making requests through the client (single instance in my case, not a list of) then occasionally auth failure disappears (after ~5-10 minutes), if many threads try to do operations on the client (many means 50-100), reconnection time increases or even no reconnection occurs.

Any thoughts?