I have a 2 node cluster on XLarge instances in Amazon.
There’s a one big bucket of 18M documets - 2K each doc and 60 bytes for the key, 1 replica configured (+replica for indexes).
There are about 3 disk creates per sec, 3 CAS operations per sec, 10 gets per sec.
Each node has 9GB RAM allocated for this bucket, and 1TB disk (regular EBS, not PIOPS)
There are 2 design documents, each one has 4 views, each view has 40 emits (there are checks on each property to verify that it exists before emitting).
We have 4 problems:
- When publishing a new design document, it takes 3-4 days until indexing is finished.
- We added a third node 2 weeks ago, and the rebalancing hasn’t finished yet (now it’s stuck on 70%).
- We can’t use the views, probably because of the reindexing (which I don’t understand why it suddenly started without publishing a new view)
- The @indexes dir on the second node reached 920GB, which is not normal. On the first node it takes only 120GB.
Can anyone help?