Unable to deserialize JsonObject and getting TimeoutException when extreme load test on Couchbase?

java

#1

I can’t able to share my code, logs, etc. directly as I am working on Virtual Desktop and Couchbase community forum is blocked from inside that network. So pardon me if there is any mistake.

I am running a n1ql query through java as below.

Statement statement = Select.select("*").from(“my_url”).where(Expression.x(“user”).eq("$user").and(Expression.x(“url”).eq("$url"))).limit(1);

JsonObject placeholderValues = JsonObject.create().put(“user”,myRequest.getUsername()).put(“url”,myRequest.getUrl());

N1qlQuery query = N1qlQuery.parameterized(statement,placeholderValues);
N1qlQueryResult results = couchbaseBucket.query(query); //TimeoutException
final Iterator result = results.iterator();

if(result.hasNext()) {

ObjectMapper objectMapper = new ObjectMapper();
MyURL myURL = null;

try {

myURL = objectMapper.readValue(result.next().value().get(“my_url”).toString(),MyURL.class); //TranscodingException

} catch (IOException e) {

} catch(TranscodingException e) {
e.printStackTrace();
}

}

}

If url does not exist( against the user) in bucket my_url then I am inserting an object containing the url as below

EntityDocument doc = EntityDocument.create(myURL);
couchbaseBucket.repository().insert(doc);

My Couchbase configuration in spring is as follow

@Configuration
@EnableCouchbaseRepositories(basePackages = {“com.my.model”})
public class MyCouchbaseConfig extends AbstarctCouchbaseConfiguration {

@Override
protected List getBootStrapHosts(){
return Collections.singletonList(“xxxx,yyyy”);
}

@Override
protected String getBucketName(){
return “my_url”;
}

@Override
protected String getBucketPassword(){
return “my_pass”;
}

@Override
protected CouchbaseEnvironment getEnvironment(){
return DefaultCouchbaseEnvironment.builder().connectTimeout(10000).computationPoolSize(15).build();
}

}

Now when I am executing load test using Jmeter, initially for around 2 min everything works fine. But then it started throughing exception with message like TranscodingException: Error deserilizing row value from bytes to JsonObject (check above comment for line details). Then again it started working fine for a while and at the end of load test it started throughing RuntimeException: java.util.concurrent.TimeoutException (check above comment for line details).
Load test with 200 concurrent user for 500 rounds i.e. 1Lakh sample, getting 30% to 50% failure.

Relevent indexing details below

CREATE INDEX ‘url_user_index’ ON ‘my_url’(‘url’,‘user’) USING GSI

Tested with spring-data-couchbase-2.2.0.RELEASE.jar + java-client-2.2.7.jar and with spring-data-couchbase-2.2.0.RELEASE.jar + COUCHBASE SDK 2.2.8/2.2.7. But nothing works for me.

Couchbase server 4.1.1 - 5914.


#2

As no one is responding…I am proposing my solution as below…Although I didn’t found the exact solution but did some work around which make my application work fine…

Initially I tried to achieve the above using View instated of writing n1ql query i.e.

select * from my_url where user='xxx' and url='http://www.couchbase.com'

replaced by a view function as below

function (doc,meta){
if(doc.user && doc.url){
emit([doc.user,doc.url],doc);
}
}

and in my java class I replaced the code

Statement statement = Select.select("*").from(“my_url”).where(Expression.x(“user”).eq("$user").and(Expression.x(“url”).eq("$url"))).limit(1);

JsonObject placeholderValues = JsonObject.create().put("user",myRequest.getUsername()).put("url",myRequest.getUrl());

N1qlQuery query = N1qlQuery.parameterized(statement,placeholderValues);
N1qlQueryResult results = couchbaseBucket.query(query); //TimeoutException
final Iterator result = results.iterator();

if(result.hasNext()) {

ObjectMapper objectMapper = new ObjectMapper();
MyURL myURL = null;

try {

myURL = objectMapper.readValue(result.next().value().get(“my_url”).toString(),MyURL.class);
} catch (IOException e) {

} catch(TranscodingException e) {
e.printStackTrace();
}

}

with below code

ViewResult viewResult = couchbaseBucket.query(ViewQuery.from(“view_url”,“my_url”).reduce(false).development(false).limit(1).key(JsonArray.from(myRequest.getUsername(),myRequest.getUrl())));

for(ViewRow row : viewResult) {
try {

myURL = objectMapper.readValue(row.value().toString(),MyURL.class);
} catch (IOException e) {

} catch(TranscodingException e) {
e.printStackTrace();
}
break;
}

The above does give solution to my problem but the performance is not up to the mark. it’s the half of my expectation. So I change the configuration a little bit as below

@Override
protected CouchbaseEnvironment getEnvironment(){
return DefaultCouchbaseEnvironment.builder().queryTimeout(2000).connectTimeout(30000).computationPoolSize(15).build();
}

By default the query timeout is about 70 sec but I configured it to 2 sec and the connection timeout to 30 sec. Then I again revert my code to n1ql and handle those exception i.e. TranscodingException and TimeoutException in one catch block i.e. RuntimeException. Now inside this catch block I wrote the code for View and it is started performing with throughput nearly 580 tps per 1000 concurrent user.

Summary:- if someone facing the same problem as I do and performance is not the priority then he should replace his n1ql query with view blindly.


#3

Now I am facing a new problem with the above configuration. Although I manage to handle both the exception but after a long idle run of my application, about 1 hrs, it is repeatedly throughing TimeoutException against n1ql query when I execute load test. This is directly impacting on my application performance.


#4

Adding @daschl and @ingenthr


#5

Thank you @geraldss for ur response.

Hello @daschl/@ingenthr,

It seems to me the connections which held by my java client has been expired at the Couchbase server. I am just assuming this, please ignore if it is not relevant.

The problem comes when I tried to get a doc or select any data using N1QL query. But it normal execution happen when tried to insert or select using view.

I guess someone is facing the same problem:-https://forums.couchbase.com/t/couchbase-server-randomly-disconnects-kv-endpoints-keepalive-not-working/9607/13

Thanks,


#6

Hi @Chi,

I’ll note that there have been a number of improvements in both connection pooling and keepalive handling. The version you indicate you’re using in the first posting is 2.2.7, which is from May 2016. With that vintage, you should expect to need to tune queryEndpoints if you expect one client instance to have a very high throughput.

However, we’ve made this more automatic. Can you update to the latest client, 2.4.1, and let us know if you still see something you believe is related to connection problems?

Also, I’d highly recommend watching the logs. You should see INFO/WARN level logs for any connections being dropped unexpectedly.


#7

I updated to 2.4.1 and got java.lang.noclassdeffounderror:com.couchbase.client/core/utils/ConnectionString exception. And earlier I run it through 2.4.2 and I guess I got the same error(don’t remember). In maven spring-data-couchbase-2.2.0.RELEASE.jar is compiled with java-client-2.2.8. I have no plan to move spring 5.

What is SearchEndpoint? How is it differ from QueryEndpoint?

About log details, I can’t see anything suspicious in Couchbase Web-Console. Is there any other log files I am missing?


#8

To use the updated JAR, update both the couchbase-client maven artifact and the core-io maven artifact to the dependency described in the maven POM.

You’re safe updating to the latest SDK since we follow semantic versioning and it’s the right place to diagnose things. Can you give it a shot?

Regarding searchEndpoint and queryEndpoint, the former is for Couchbase’s FTS service. The latter is for the N1QL Query service.

Finally, regarding the logs, I was referring to the logging of your Spring app. Those won’t be in the Couchbase Web Console. You may need to configure logging per Spring’s docs. The SDK will log an INFO level message at startup with environment details. If you see that, then there should be further logging if there are any dropped connections, etc.


#9

1>Okay, it’s my mistake. I update core-io and rxjava jar to 1.4.2 and 1.2.3 respectively and everything went cool. But even if doing so I am still getting the java.util.concurrent.TimeoutException.

2>Thanks for the clarification. Is there any endpoint I need to tune to increase get(doc) or insert(doc) like query? Cuz tuning the queryEndpoint really increase the performance drastically for which I am really glad.

3> No, there is no such log info in Spring.

4>Yesterday, I again load test on the application with 10Milion samples and 1000 concurrent user. This time it gave me above >1000 throughput which is excellent but when I try to execute again with the same load the throughput drop to 60 which is a huge difference. Then I again restart the application and got the exact same behaviour.

    @Override
    protected CouchbaseEnvironment getEnvironment() {
    	return DefaultCouchbaseEnvironment.builder()
    			.queryTimeout(1)
    			.queryEndpoints(10)
    			.build();
    }

Am I making any mistake?


#10

It seems I might able to resolve the issue. Thanks to this post N1QL queries timing out when run in java sdk. Still I’ll wait for at least two to three days just to making sure everything is ok.

Thank you @ingenthr you are a big help.

I am still looking ans for the below

2>Thanks for the clarification. Is there any endpoint I need to tune to increase get(doc) or insert(doc) like query? Cuz tuning the queryEndpoint really increase the performance drastically for which I am really glad.


#11

Okay, even after 3 days my application working smoothly. The mistake I was doing earlier is even if I downgarde my SDK jar to couchbase-java-client-2.2.7 I forgot to downgrade my couchbase-core-io and rxjava jar. Doing so resolved the issue.

@CouchbaseTeam :- there is definitely something wrong with the new version of CB SDK.


#12

Thanks for the update @Chi. I have to admit though, I’m confused on where things stand. Based on my read of the above, you upgraded the client version, made some code changes and saw some better results. However, this last message indicates you downgraded.

What’s a simple summary of the issue?

@daschl and I would definitely like to look into things further if there is a problem here.


#13

There is a kvEndpoints tuneable which is logically similar to the queryEndpoints tuneable. However, you shouldn’t typically need to adjust this, as KV operations are interleaved between client threads and pipelined. The primary situation where you’ll need to adjust the kvEndpoints is when you have a lower amount of data in your working set. If most of the data is in memory, it usually isn’t necessary.


#14

Summary:-

1)If performance is not an issue then blindly go for View instead N1QL.

2)For N1QL, you can set the queryTimeout to a very low value say 3 sec. This will make retry faster instead waiting for a long queryTimeout i.e 70 sec by default.

3)The connections still might drop due to unknown reason :wink: Switch to SDK 2.2.7 and the problem might resolve.