I’m doing test on SDK V2.0.3.
There’s reconfigration problem after failover.
Exception
[com.couchbase.client.core.endpoint.Endpoint] - [search-easter-cache10/10.41.117.140:11210][KeyValueEndpoint]: Could not connect to endpoint, retrying with delay 4096ms:
java.net.ConnectException: Connection refused: search-easter-cache10/10.41.117.140:11210
Test Results
Load Server : 15
Client Server : 6 (L7)
Data Seed
request per 40 keys
Normal Performance
concurrence(per load server) : 1
- QPS : 3045 (4ms)
- Response time : 4 (no shaking)
- OPS : 90K (shaking :100~120K)
concurrence(per load server) : 4
- QPS : 9705 (5ms)
- Response time : 4 (5~12, shaking)
- OPS : 90K (shaking : 0~400K)
Node Down
concurrence(per load server) : 4
- QPS : 1095 (53 ms)
- Response time : 53 ms (no shaking)
- OPS : 36K (no shaking)
Failover => IF do not reboot client server, throughput IS NOT recovered .
- QPS : 1350 (45ms) => recovered to 9500(ops) after client reboot
- Response time: 45ms (no shaking) => recovered to 5~6ms after client reboot
- OPS: 48K => recovered to 9650(ops) after client reboot
Code
public class HammerGather implements Func1<String, Observable<StringDocument>> {
private final Logger logger = LoggerFactory.getLogger(HammerGather.class);
private final Bucket bucket;
private final long masterTimeout;
private final long replicaTimeout;
public HammerGather(Bucket bucket, long master_timeout, long replica_timeout) {
this.bucket = bucket;
this.masterTimeout = master_timeout;
this.replicaTimeout = replica_timeout;
}
@Override
public Observable<StringDocument> call(final String imageHash) {
return bucket.async().get(imageHash, StringDocument.class)
.timeout(masterTimeout, TimeUnit.MILLISECONDS)
.onErrorResumeNext(new Resumer(imageHash));
}
class Resumer implements Func1<Throwable, Observable<? extends StringDocument>> {
final private String imageHash;
public Resumer(final String imageHash) {
this.imageHash = imageHash;
}
@Override
public Observable<? extends StringDocument> call(Throwable throwable) {
logger.warn(String.format("master error %s, timeout(%s), %s", imageHash, masterTimeout, throwable.toString()));
return Observable.<StringDocument>empty();
}