Couchbase do not store all documents

#1

Hello,

I’m trying to generate and store in couchbase several million documents but most of the documents are not add in the database. For example, of one million documents only 80-100k thousand will be saved in the database after the first iteration. I use JavaSDK, and method through which I store docs is add with parameter PersistTo.ZERO.

What explains this behavior?

#2

Can you show some code?

How do you do the set ? do you get/check the status of your operation

cb.set(K,V).get();

#3

My code is pretty simple. For the hashmap containing one million items

for (Entry<String, String> en : hashmap.entrySet()) {
couchbaseClient.set(en.getKey(), en.getValue(), PersistTo.ZERO);
}

After these for-each the number of elements in the database not equals 1 Million.

#4

A small addition. When I do

set(key, value).get()

all docs completly puts to database, but the number of operations per seconds is reduced in 10 times. I am very interested in what’s happening

#5

Hello,

So now that it is clear what the source of the problem is let me explain it.

As you probably know the set operation is asynchronous, so the client push the key,value on the network without blocking or checking anything. (it will only failed at this point if the socket/connection is not valid anymore).

So what you have to do in your application is to “check” the status of the operation, the way it is done with Couchbase client API is using a java Future, the OperationFuture object that allows you to check to get the status.

When I asked you to do a client.set(K,V).get() it is a way to do a “synchronous” call since it will wait to have the response to go to the next operation.

The proper way to work if you want to “inject” many many things is to:

  • inject document in Couchbase
  • keep a reference to the OperationFuture
  • check the status of the operation, and in case of failure redo the operation and/or log the error.

I have created a gist with the code:

Some part of it:

…
for (int i = 0 ; i <= 500000 ; i++) {
allOps.add(cb.set(“key::” + i, UUID.randomUUID().toString()));
}
…
for(OperationFuture op : allOps) {
if (! op.getStatus().isSuccess()) {
System.out.println(“Key: key:”+ op.getKey() +" not saved due to ‘"+ op.getStatus().getMessage() +"’. Please retry" );
}
}
…

So if you look in the stats of your bucket using the Web console or the cbstats command you will see the error too.

Regards
Tug
@tgrall