I’ve got a Couchbase(2.0.1) cluster with 6 nodes(3 replicas). On the client side I use a Moxi server(1.8.1) and memcached clients. I’ve encountered an issue with this setup. When a cluster node goes down, the Moxi server continues considering this node up and tries to write to it. As I read in the Moxi server documentation, a Moxi server communicates with a cluster through HTTP/REST and gets an actual cluster configuration and, when the configuration changes, the cluster notifies connected Moxi servers about reconfiguration. So in my case, it seems that the Moxi server is not getting reconfiguration messages.
For the sake of curiosity, I tried to connect to the cluster directly through libcouchbase(2.0.7) and went to the same problem. When a cluster node goes down, the library still tries to write to this node as the Moxi server did.
The Moxi server configuration: