I’m encountering an issue that I’m not sure how to resolve, seen on both Couchbase 4.1 and 4.5 DP1 enterprise using java SDK 2.2.3 and 2.2.5 (tested on both)
I’ve got a bucket (single server cluster) with about 1.7 million docs, no mapreduce views, 3 GSI indexes
RAM usage 2.52GB /7.4GB
Data usage 1.4GB /1.98GB
Data RAM quota: 8192mb
Index RAM quota 5120mb
I’m running an async N1QL query that, when 100% successful, returns about 170k docs.
When I run this query under any sort of load (such as running the same query on another instance of the client on a different VM) the subscribe.onComplete function triggers before all 170k are retrieved, typically after about 270-300 seconds having processed about 100-110k documents. No exceptions occur, onError is not triggered on the observable and onComplete is invoked.
Here’s the query with the GSI indexed parts bolded (where some names have been changed to protect the innocent).
SELECT * FROM
thelog WHERE order_a = $order_a AND order_b = $order_b AND time BETWEEN $startTime AND $endTime AND
type = $type ORDER BY
On a successful run when all rows are retrieved it takes about 5-6 minutes (which is OK, we’re running this async on a lower priority thread…).
I’ve set the serverTimeout to 30 minutes for the query parms and the environment vars have the queryEndpoints set to 4 (the number of threads I expect to process requests concurrently)
The observable grabs the AsyncN1qlQueryResult and flatMaps to AsyncN1qlQueryRow, then maps to the JsonObject inside, then subscribe takes the JsonObject and writes to a file on disk after some light formatting. the subscribe.onComplete counts down the latch waiting on the main thread.