Couch base Transaction SDK 3.0 - REACTIVE insert

Java SDK
1

Hi All,
we are evaluating couchbase trasaction api for our project, we are doing a POC for the same. we are trying to insert 10000 records inside a couchbase transaction context(via a reactive api aprroach). Here is the code sample,
HashMap<String, String> data1 = DataUtil.createDummyData(1, 10000); //holds the data
ReactiveBucket buc1 = reactivecluster.bucket(“poc”);
ReactiveCollection collection1 = buc1.defaultCollection();
buc1.waitUntilReady(Duration.ofMinutes(2));
Transactions transactions = Transactions.create(cluster, TransactionConfigBuilder.create()
.build());
transactions.reactive().run((ctx) -> {
return Flux.fromIterable(data1.entrySet())
.flatMap(entry -> ctx.insert(collection1, entry.getKey(), JsonObject.fromJson(entry.getValue())))
.then(Mono.defer(() -> {
System.out.println(“commit the data”);
return ctx.commit();
}));
}).block();
This code takes 120 seconds (i understand it is based on payload and your nodes) but all i wanted to know is, is this correct approach? or we are do it in better manner to improve the performace?

Thanks.

2

Hey @nagarajan_subbaraj

There’s some requirements you need to handle to make sure operations are rolled back nicely on failure if there’s any problems, when doing this kind of parallelism. Please see https://docs.couchbase.com/java-sdk/3.0/howtos/distributed-acid-transactions-from-the-sdk.html#concurrent-operations-with-the-async-api for the details.

There’s a few things that can impact performance:

  • Key-value latency. Your code wants to be running as close to the cluster as possible, IMO. Though you can stage the inserts in parallel, as you’re doing there, they will be committed in serial. (We may consider adding a parallel-commit feature in future). If your code has a 150 milli roundtrip latency to the cluster, that’ll be 10,000 docs * 150 millis = 25 secs minimum to commit them. And the durability level you’ve specified, and the performance of your cluster hardware, will also impact key-value latency.
  • Any write-write conflict on the same document with another transaction, and some transient failures (e.g. if the server reports that it is currently overloaded, things along those lines), will cause the transaction to rollback everything so far and retry. So this could be happening, you’d need to check either TransactionResult.logs or TransactionResult.attempts to know.

If you do want to drill into performance, the TransactionResults.logs are a good starting point. They’ll tell you how long each underlying key-value operation is taking, and whether the transaction is retrying, Hopefully they’re somewhat human readable. You might want to read https://docs.couchbase.com/java-sdk/3.0/howtos/distributed-acid-transactions-from-the-sdk.html#mechanics first (this section was added today) to get a grounding in what’s going on under the hood.

But basically you are doing the right thing in that code for performance. (It does need some tweaks as mentioned in the docs, but this is for rollback safety not performance).