I don’t know why there was error in retrieving list of bucket. I don’t see any logging in query.log regarding this. Attached indexer.log does not have indexer blocked messages. May be the problem got resolved due to index deletion and rebuild. I can see a lot of “Index scan timed out” errors in query logs.
Looking at the index stats, it looks like a lot of documents are queued on the indexer side, but are not yet indexed. Average drain rate is zero. A very few (less than 20) items are getting indexed (flushed to index storage) every minute. At this time, there are around 168 million docs already indexed.
On this node, do you see CPU or disk i/o saturation? If yes, then that can be the bottleneck. Please check which process is consuming most of the CPU and disk i/o. Also, if CPU is under pressure and indexer process is taking most of the CPU, you can collect cpu profile of indexer using following command. Also let me know couchbase server version (including minor version).
curl -X GET -u <couchbase-admin-username>:<password> http://<host-ip>:9102/debug/pprof/profile
Meanwhile, I will check if I can find any numbers around scalability of the indexes in community edition.