Actually, I have another critical problem.
I force-stopped 3 of 2.2.0 CE nodes and off-line upgrade to 3.0.1 CE and tried rebalancing the cluster.
The 3 of old nodes (previously 2.2.0 CE) have 50GB disk. (whose disk usage was 91% before rebalancing)
The 3 of new nodes (3.0.1 CE) have 200GB disk.
I marked the 3 old nodes as ‘Remove’ and performed ‘Rebalance’.
I expected the data from 3 old nodes goes into the new nodes.
But after a few hours, 2 of the 3 old nodes (50GB), went down because of disk full.
And I couldn’t understand the situation. How could disk usage for old nodes (marked as ‘Remove’) go up when rebalancing?
2 dead nodes didn’t go back up so I had to fail them over. (loosing data within them)
Now I have 1 more 50GB node to remove, and I am afraid that the same thing would happend.
Am I missing something here?
How could I properly rebalance the cluster and safely remove 50GB node from the cluster?