We have a cluster of 6 nodes with one replica set running Membase 1.7. I noticed when one of the node goes down (it crashes - different issue), our app stops responding correctly. It’s not until we remove and rebalance the cluster that it starts working again. In our PHP logs, I’m seeing the following errors. At the time of impact, I did log onto the server and tried to telnet to the port AND it worked fine.
We do have auto-failover enabled, what I dont recall is if this issue occurs when more than 1 of the node goes down. I read the couchbase manual and it states that it will failover the first node properly but it won’t do an autofailover on the second node. My question is, does it do a rebalance after the first node is ejected?
No servers added to memcache connection in /<PATH_TO_FILE> on line 72
[24-Sep-2013 12:22:16 Etc/GMT+4] PHP Warning: Memcache::set() [memcache.set]: No servers added to memcache connection in /<PATH_TO_FILE> on line 87
[24-Sep-2013 20:38:13 Etc/GMT+4] PHP Notice: Memcache::get() [memcache.get]: Server localhost (tcp 11211) failed with: Failed reading line from stream (0) in /<PATH_TO_FILE> on line 72
[24-Sep-2013 20:38:14 Etc/GMT+4] PHP Warning: Memcache::pconnect() [memcache.pconnect]: Can’t connect to localhost:11211, Unknown error (0) in /<PATH_TO_FILE> on line 66