Client throws TemporaryFailureException when insert in large numbers


#1

Greetings ,

We are in the processing of extending YCSB benchmark tool to use new Couchbase 2 SDK.
Below is the legacy code used to insert a record:

 @Override
    public int insert(Map<Object, Object> context, String table, String key, Map<String, ByteIterator> values) {
        try {
            OperationFuture<Boolean> future = client.add(key, 0, JsonByteIterator.fromMap(values), transcoder);
            if (future.getStatus().isSuccess())
                return observe(context, future, false);
            return getReturnCode(future);
        } catch (Exception e) {
            if (log.isErrorEnabled()) {
                log.error("Error inserting value", e);
            }
            return ERROR;
        }
    }

New code with Couchbase 2 SDK :

 @Override
	public int insert(Map<Object, Object> context, String table, String key,
			Map<String, ByteIterator> values) {
		try {
			JsonNode jsonNode = JsonByteIterator.fromMap(values);
			RawJsonDocument rawJsonDoc = RawJsonDocument.create(key, OM.writeValueAsString(jsonNode));
			bucket.insert(rawJsonDoc);
			return OK;
		} catch (Exception e) {
			if (log.isErrorEnabled()) {
				log.error("Error inserting value",e);
			}
			
			return ERROR;
		}

	}

But when we run this with 20k record inserts , new code throws following exception after about 15K records ,while the previous code runs successfully even with millions of insertions.

What could be causing this error ?

[2015-03-27 14:22:33,582] ERROR com.rakuten.gep.ycsb.couchbase.v3.Couchbase3DB Error inserting value
com.couchbase.client.java.error.TemporaryFailureException
at com.couchbase.client.java.CouchbaseAsyncBucket$12.call(CouchbaseAsyncBucket.java:434)
at com.couchbase.client.java.CouchbaseAsyncBucket$12.call(CouchbaseAsyncBucket.java:415)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:55)
at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:224)
at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:101)
at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:198)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: com.couchbase.client.core.message.kv.InsertResponse.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:98)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:58)
… 11 more


#2

the TemporaryFailureException indicates that the server indicated being too busy or in “temporary failure”. This kind of exception usually means that the operation can be retried with backoff.

you are saying that you’re seeing these errors consistently while using SDK 2.x and not while using 1.4.x under similar load?


#3

Hi simon,
Apparently thats the case , I did the test with same factors/environment .SDK 2 based implementation throws this exception while the 1.4.x version inserted all the documents without any issues.


#4

@simonbasle

Sorry , seems both of the clients failed to insert the total number of records .ITs just YCSM is configured to ignore the error when using 1.4.x client.

I did the test for 20K documents using above two codes .Following are the stats .Seems still SDK2 is slower compared to 1.4.x

1) With Client 1.4.x : inserting 20,000 documents

Successfully inserted document: 19967

[OVERALL], RunTime(ms), 11880.0
[OVERALL], Throughput(ops/sec), 1683.5016835016836
[INSERT], Operations, 20000
[INSERT], AverageLatency(us), 1569.19335
[INSERT], MinLatency(us), 399
[INSERT], MaxLatency(us), 135299
[INSERT], 95thPercentileLatency(us), 3442.6279069767443
[INSERT], 99thPercentileLatency(us), 17627.8
[INSERT], Return=0, 20000

2) With SDK 2 : inserting 20,000 documents

Successfully inserted document : 19942

[OVERALL], RunTime(ms), 14374.0
[OVERALL], Throughput(ops/sec), 1391.4011409489356
[INSERT], Operations, 20000
[INSERT], AverageLatency(us), 1838.27915
[INSERT], MinLatency(us), 336
[INSERT], MaxLatency(us), 184376
[INSERT], 95thPercentileLatency(us), 3444.387573964497
[INSERT], 99thPercentileLatency(us), 22257.333333333332
[INSERT], Return=0, 20000