org.apache.camel.RuntimeCamelException after certain days..gets solved after restart

Hi All,
My project has routes written in camel which sends data from activeMq server to Couchbase server.
Suppose my activemq server is at address “X”. Couchbase is at address “Y”. My webapp is deployed inside tomcat present at address “Z”. This functionality works fine on average for 2 days. Sometime in logs of activeMq I see it is unable to connect to tomcat server. This network glitch persist only for a minute. Now after then when my application is trying to put data in Couchbase using that Camel route we get TimeOut exception. To of surprise this exception now appears for all further request until I restart webapp. One of my guess is when we get timeOut exception It may corrupt camel route. But in logs I can see message is coming from “From endpoint of amq” and fails in “to endpoint of couchbase”.

Any suggestion would be helpful . Open for discussion.

I’d recommend looking at the logs from your Camel deployment. A TimeoutException is always an effect from some other cause. For instance, maybe the connection is being destroyed somewhere in the network infrastructure? This would lead to two entries in the logs. One being the TimeoutException from trying to carry out the operation and another being from the connection being disrupted.

I should also note that in some failure modes, a TCP connection might end up being ‘half open’, meaning the client thinks it’s open but the server or something in between has killed the TCP connection. In that case, our client usually detects the down connection with a regular keepalive request or will automatically try to drop/rebuild the connection after a number of continuous TimeoutExceptions.

Thanks ingenthr for such a quick reply. My major concern is when we get timeout exception for first time because of network glitch between jms and tomcat. After that timeoutException persist for all further request until we restart webapp.

your 2nd para is quite interesting. Yeah while verifying in browser console I can see request is stalled for all time. Is there any tool out there to verify whether TCP connection is half open ?

I am using couchbase-java-client-2.1.5.jar. Request is getting failed in

Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
** at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:93) ~[couchbase-java- client-2.1.5.jar:2.1.5]**
** at com.couchbase.client.java.CouchbaseBucket.insert(CouchbaseBucket.java:236) ~[couchbase-java-client-2.1.5.jar:2.1.5]**

Is there any catch is this api because I am not able to completely understand this Observable API.

To see if a TCP connection is half-open, you can check the state of the connection with netstat on both the server and the client. If the server doesn’t have the connection or it’s in a CLOSE_WAIT state or similar, but the client shows ESTABLISHED, then you’d be in this situation.

About the exception, like I said looking at the TimeoutException alone probably isn’t enough. Is there anything earlier in the logs? You might want to check with the Apache Camel folks to be sure the error handling there is correct.

Hi ingenthr,

Thanks for reply. Thanks for giving this cool suggestion on netstat command.

Always in logs of tomcat I see this pattern…
04-Feb-2017 01:42:23.300 | WARN | [serverName:61616@57029] | FailoverTransport:273 - Transport (tcp://serverName:61616) failed, attempting to automatically reconnect
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0_111]
at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0_111]
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594) ~[activemq-client-5.12.1.jar:5.12.1]
at java.io.DataInputStream.readInt(Unknown Source) ~[na:1.8.0_111]
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:267) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213) ~[activemq-client-5.12.1.jar:5.12.1]
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) ~[activemq-client-5.12.1.jar:5.12.1]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]

…around 10 times followed by

04-Feb-2017 17:46:53.507 | WARN | [ cb-io-1-5] | Endpoint:151 - [serverName:8093][QueryEndpoint]: Socket connect took longer than specified timeout.
04-Feb-2017 17:46:53.507 | WARN | [ cb-io-1-4] | Endpoint:151 - [serverName:11210][KeyValueEndpoint]: Socket connect took longer than specified timeout.
04-Feb-2017 17:47:01.198 | WARN | [ cb-io-1-6] | Endpoint:151 - [serverName:8092][ViewEndpoint]: Socket connect took longer than specified timeout.

Hence I have increased the socketConnectionTimeOut parameter from 5000ms to 30000ms.

In the logs of couchbase I have not find anything helpful.

This defect takes around 3 to 4 days for getting replicated. now fingers crossed.

Hi ingenthr,

Thanks for help. This issue is fixed now. Changes I have done are

  1. Introduced viewTimeout parameter.
  2. Increased socketConnectionTimeout to 30 seconds from default timeout of 5 seconds.

My understanding from above issue is due to network glitch camel route used to loose connection from activeMq server and Coucbase server. After a minute when network was fine Camel was unable to connect to Couchbase server as default timeout is 5s which is very less.

My concern is in logs of activeMq I can see it has lost connection with tomcat port but In logs of coucbase there was no mention about it.

Do we have to set logging at coucbase server to Debug mode or any other setting is available ?

@saurabhprataps and @ingenthr,

Hi guys,

Any idea about camel interaction with couchbase in async way. Is it possible/supported ?

The reason I am asking is because camel is working on worker threads in NIO way and couchbase internally using thread local variables.

We tried using it through simple camel route tests and trying to make parallel calls to couchbase through split/parallel processing is not working and getting random Unambiguious timeout issues. And when we run the same code in normal java test without the use of camel route, it all works fine.

couchbase code snippet

getBucket().defaultCollection().reactive().exists(documentId)
.subscribe(new Consumer() {

Your inputs will really help us.

Thanks a lot .