@k_reid shared the logs with me, and looking them over, I think @daschl’s suggestion of being out of connections may be the most likely. Connections look healthy. The thing is, he’s on client 2.5.1, so there should be pooling by default on queryEndpoints, right @daschl?
Note the queries are with request_plus and many of them run for ~1.5s.
A future enhancement, it might be nice to log at the “could not dispatch request” line that we have consumed X out of Y endpoints, or something like that.
@k_reid: to diagnose, add this to your CouchbaseEnvironment:
.queryServiceConfig(QueryServiceConfig.create(10,200))
You’ll want to calculate a realistic number, but if that helps, you just need a wider highway to the query service given the number of concurrent requests.