First, I wanted to confirm that we’re now testing with the 2.3.3 GA version. I don’t see any behavior differences after moving away from the test version we had before.
Second, I re-ran my test with queryEndpoints=20 and with no queryEndpoints setting. With 20 endpoints, I see the same pattern as I did in the most recent screen capture - the first query node in the cluster averages about 3000 queries/sec and the others are either 1k or 1.5k/sec. These rates stay consistent and I can’t account for why they would vary to such a degree, even between the 1k and 1.5k nodes. The exact same query against an empty bucket returning an empty result set is used for every call.
The test with no queryEndpoint override had some interesting results. The overall query rate was much slower. It seemed that about half of the query nodes received about 300 queries/sec and the other half was 500-600 queries/second. The first node in the cluster was in the slow group this time.
I can run a tcpdump, but wanted to confirm what you wanted me to capture. Would you like to see what happens during a run with queryEndpoints=20 or something else?