Rapid multiple inserts in Couchbase


#1

Couchbase newbie question (clubbing 5 related questions together):

I am trying to insert 1 million records in couchbase in quick succession, but I see that nearly about 0.5 million records get inserted (Admin console shows 517365 as Item Count). Also, from the admin GUI, I can only see 1000 records (10 pages of 100 records each)

Am wondering where rest of the records are vanishing !

I suspect when Couchbases’s internal buffer is full, further requests are dropped. Can somebody please help me with the following ?

  1. How can I configure the couchbase internal buffer size or atleast check its default size?

  2. I used client.set(n + “”, 0, n + “”).get() to make sure that the data is persisted. But still 0.7 million records were set. Also, I don’t like it as I am doing a unnecessary get operation here.

  3. When I close the couchbase client, are the set operations queued in couchbase buffer aborted ?

  4. Which log file I should be looking at to find insertion failure errors ? (There are tad too many files in Couchbase\Server\var\lib\couchbase\logs folder)

  5. Does Couchbase GUI not paginate all the records ?

public class Test { public static void main(String[] args) { ArrayList nodes = new ArrayList(); String cbUrl = "http://127.0.0.1:8091/pools"; String dbName = "default"; CouchbaseClient client = null; try { nodes.add(URI.create(cbUrl));
        client = new CouchbaseClient(nodes, dbName, "");

        insertRecords(client);

        System.out.println("Test Over");

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        // client.shutdown();
    }
}

public static void insertRecords(CouchbaseClient client) throws Exception {
    int num = 1000000;

    for (int n = 1; n <= num; n++) {
        System.out.println("Adding: " + n);
        client.set(n + "", 0, n + "");
    }
}

}