Nodes are only removed from the client if they are failed over, meaning that they have been either manually removed from the cluster, or they have been removed via auto-failover (an optional feature allowing the cluster to automatically remove servers it thinks are dead). Otherwise, the client library must assume that any sort of socket or connectivity error is temporary. Note that Failover is not just a colloquial term to indicate a dead server, but corresponds to an actual Couchbase Cluster Management API call (via REST) to indicate that a specific server has been “Failed over”, as failover is a potentially destructive operation with respect to data.
If a socket connection has been broken (
ECONNRESET for example), the client will try to reconnect. It will not try to reconnect on a timeout error because a timeout does not indicate a broken connection – it does indicate that the connection is still open, but for some reason the remote host is not responding - often due to resource exhaustion. In such a scenario, it is certainly not a good idea to add more TCP connections to an already-exhausted server.
It is possible to pre-empt and determine if a given document ID (key) will be routed to a specific server, and thus you can write code which can check for each given key if it would be routed to a server which is known to be “dead” (where “dead” is something defined in your application, and exceeds Couchbase’s definition of dead, which is failed-over). I am not sure if such an API is exposed in the node.js library, but is certainly available in the underlying C library