Client connection become unstable when time goes by


#1

I have one service which has million online customers, it is a node.js app with Couchbase cluster, it started very well, but when time goes by, 1% requests will return 500 timeout error from the connection to Couchbase server. It goes back to no error when I restart the node.js server.
My confuse is: If I have 3 workers link to the Couchbase cluster which has 3 nodes, only 1 can get this timeout error, others are just fine, but meanwhile all the 3 Couchbase nodes are stable. After restart node.js server, all 3 workers are fine. Anyone can give me some advise about it?
Detail info:

  • the libcouchbase version is: 2.4.6
  • the error message is: {Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout",“code”:23}