Parallel read from a bucket

Couchbase Server
1

I’m new to Couchbase and wondering if there is any manner to implement a parallel read from bucket. Given that, a bucket contains 1024 vbuckets by default. So could it be possible to split a N1QL query select * from bucket1 into several queries? What I mean is that one of those queries just reads data from a range of vbucket, e.g, from vbucket1 to vbucket100. Because the partition key is used to decide which node the value should be persisted. I think it could be possible to read a part of data from bucket according to a range of partition key. Could someone help me out of this? Thanks a lot.

2

N1QL already does fetch from KV by using pool of parallel threads (2.5* Number of Cores) by all fetch requests. For CE the Number of cores limited is 4.
If really need much parallel get document keys using following N1QL and use streaming SDK asynchronous calls multiple threads to get documents.

SELECT RAW META().id FROM bucket1

As you know need any processing of data ,this avoids 2-hops of data (Data Node --> N1QL --> Client) , uses client machine resources and you can control the threads in the client

3

Thanks for the response!

4

Thanks for the update! What do you mean by 2-hops of data?

5

In fact, I have another question. Would it be possible to do batch-wise read by splitting a bucket into severals? The idea is to divide the bucket into small batches of given size and then to read these small batches one by one. Thanks in advance for the reply.

6

You can use OFFSET/LIMIT repeat statement next offset.
SELECT RAW META().id FROM bucket1 OFFSET 0 LIMIT 10000;

You can use range scan using LIKE change value every time based on your keys “a%”, “b%”, …“A%”…
SELECT RAW META().id FROM bucket1 META().id LIKE “a%”;

7

Thanks a lot! And I’m wondering if I could firstly divide the bucket by batch size in bytes and then read them with “LIMIT”. Could you help me out of this?

8

There are limit on 10 buckets. If you want divide you can divide by number of documents. Why can’t use LIMIT OFFSET run multiple queries with different offsets

9

Thanks for the response!