Re-indexing takes very long

We have 4 node cluster (AWS m1.xlarge).
We have 6 design docs and 26 views.
2 design doc have 11 views each.

We have 1 billion events / docs in the bucket.
Now when we modify views, it starts re-indexing (expected).
But what is problematic is , it took 2 days to complete.
We are more worried about when move to production, this will transform into downtime.
Question is:

  1. What are different ways that this downtime can be reduced to 2-4 hours ?
    we are open to using second backup CB cluster, but not any other third party system/tool.
    Events will be continuously flowing into system.
  2. We want all data (old and new indexes) from old and new docs will be immediately available during re-indexing.

Those are pretty small instance types, and a small number of nodes for that many views!

For a start I’d strongly recommend migrating to some beefier instances if you want to use that many views - m1.xlarge is pretty old now - EC2 is up to m4 instance types in the current generation. I believe you can get faster instances, with more RAM and disk for the same or less money.

View (re)building speed is essentially a function of the amount of data, number of views, and the amount of CPU and disk you make available to them. I’d suggest one (or more) of:

  • Simplify / unify your views - each one needs to be recalculated when a document changes.
  • Measure your resource utilisation on the nodes, and increase whatever the bottleneck is. AWS disks (particularly on old instance types) can be very constrained, so you might want to consider increasing your IOPS, or moving to local SSD.

Thanks @drigby !

This is our POC / Test environment.
We will be definitely using better configurations than this in production.
Given, we can have optimal hardware configurations (for this setup),
Is it possible to bring down re-building views to 2-4 Hours (for ~2 billion docs) ?

1 billion docs on 4 nodes WOW!!! What your active resident % (% of docs in memory)?