Best practices for programming for node failure


I understand that a cluster is not down just because one node is down. BUT, the documents whose active vBucket are on the down node are partially unavailable. You cannot update those docs, and you can only read them if you are willing to accept a replica copy that might be stale.

So I am wondering how other programmers handle this. If a doc is on a down vBucket, do you wait for failover, then try the update again? If you are reading, do you take the replica copy? I know the answers will varying depending on the application, but I want to gather some input.

(This is sort of a continuation of this thread…
On node failure, how long are insert/upserts blocked?)