Couchbase NodeJS Error : Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout

So i upgraded the couchbase server from version 3.0.x CE to version 4.5 CE

Now after doing this, im facing an issue in my NodeJS project for all CRUD operations

Error received when trying to perform a GET
CouchbaseError: Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout
** message: ‘Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout’**

I tried increasing the bucket operation timeout to 1 minute, but that did not work.

I deliberately changed the bucket password to a wrong one, and it throws an authentication exception, on using the correct credentials it connects, but after that i received a timeout error on all CRUD operations.

And surprisingly, im only facing this issue in NodeJS, i have another project in .NET that connects to server version 4.5 CE and everything works seamlessly over there.

Any help on this would be highly appreciated!!!

Hey,

Can you provide some details on your server configuration, and how you are connecting to the server with the Node.js SDK (how many nodes do you have, is there a firewall, are your nodes on the same subnet, what is you’re connection string in the SDK, etc…)

P.S. You could also try to use our sdk-doctor to diagnose any network issues between your application and Couchbase Server: https://github.com/couchbaselabs/sdk-doctor/releases

Cheers, Brett

Hey @brett19 i am answering on behalf of neville ,

We are Connecting this way through Node sdk :-

const cluster = new couchbase.Cluster(url);
const bucket = cluster.openBucket(bucket_name, bucket_password);
Connection string - “couchbase://172.xx.xx.xx”

There is only one node in server .

Hey @chirag_b,

You guys should try to run the doctor diagnostics against your cluster (from your application server). It should either direct you to the cause of your problems, or drop a log with sufficient details for us to discover where the problem lies.

Cheers, Brett

Hey @brett19 ,please help me running sdk-doctor , i am on ubuntu 16.04 ,i’ve downloaded sdk-doctor-linux but its an exe file i am not able to run it on ubuntu platform

Hey @chirag_b,

There are three binaries available on the releases page for sdk-doctor. You should select the linux one rather than the exe. You should then be able to run it with no parameters and it will advise you on how to properly run it.

Something like sdk-doctor diagnose couchbase://172.x.y.z. is likely what you will want to run.

Cheers, Brett

Hey @brett19

I got following output after running sdk-doctor :

:arrow_forward: Parsing connection string couchbase://172.x.y.z:8091/
14:47:07.868 INFO :arrow_forward: Connection string identifies the following CCCP endpoints:
14:47:07.868 INFO :arrow_forward: 1. 172.x.y.z:8091
14:47:07.868 INFO :arrow_forward: Connection string identifies the following HTTP endpoints:
14:47:07.868 INFO :arrow_forward: Connection string specifies bucket default
14:47:07.868 WARN :arrow_forward: Your connection string specifies only a single host. You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
14:47:07.868 INFO :arrow_forward: Performing DNS lookup for host 172.x.y.z
14:47:07.868 INFO :arrow_forward: Attempting to connect to cluster via CCCP
14:47:07.868 INFO :arrow_forward: Attempting to fetch config via cccp from 172.x.y.z:8091

Hey @chirag_b,

You should be using the connection string from your application rather than the example I placed above.

Cheers, Brett

Hey @brett19

I apologize ,for security reasons i can’t show full string ,but this output is of actual connection string through which i am trying to connect to Couchbase server . So basically only url is edited in Logs above ,apart from that everything is actual log i got

Thanks
Chirag

Hey @chirag_b,

I see now. Sorry for the confusion. Please drop the 8091 port number from your connection string and your application should behave as expected again. Explicitly providing a port number with a couchbase:// scheme connection string is highly discouraged (in this case, your actually passing an http port for a memcached service).

Cheers, Brett

Hey @brett19

I tried without specifying port ,still same error comesin nodejs program-

{ CouchbaseError: Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout
message: ‘Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout’,
code: 23 }

Also this time sdk-doctor has given full logs :

command - ./sdk-doctor-linux diagnose couchbase://172.x.y.z

Output:
15:15:11.230 INFO :arrow_forward: Parsing connection string couchbase://172.x.y.z
15:15:11.231 INFO :arrow_forward: Connection string was parsed as a potential DNS SRV record
15:15:11.232 INFO :arrow_forward: Connection string identifies the following CCCP endpoints:
15:15:11.232 INFO :arrow_forward: 1. 172.x.y.z:11210
15:15:11.232 INFO :arrow_forward: Connection string identifies the following HTTP endpoints:
15:15:11.232 INFO :arrow_forward: 1. 172.x.y.z:8091
15:15:11.232 INFO :arrow_forward: Connection string specifies bucket default
15:15:11.233 WARN :arrow_forward: Your connection string specifies only a single host. You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
15:15:11.233 INFO :arrow_forward: Performing DNS lookup for host 172.x.y.z
15:15:11.233 INFO :arrow_forward: Attempting to connect to cluster via CCCP
15:15:11.233 INFO :arrow_forward: Attempting to fetch config via cccp from 172.x.y.z:11210
15:15:11.238 ERRO :arrow_forward: Failed to fetch configuration via cccp from 172.x.y.z:11210 (error: invalid bucket name/password)
15:15:11.238 INFO :arrow_forward: Attempting to connect to cluster via HTTP (Terse)
15:15:11.238 INFO :arrow_forward: Attempting to fetch terse config via http from 172.x.y.z:8091
15:15:11.240 ERRO :arrow_forward: Failed to fetch terse configuration via http from 172.x.y.z:8091 (error: incorrect bucket/password)
15:15:11.240 INFO :arrow_forward: Attempting to connect to cluster via HTTP (Full)
15:15:11.240 INFO :arrow_forward: Failed to connect via HTTP (Full), as it is not yet supported by the doctor
15:15:11.240 ERRO :arrow_forward: All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible
15:15:11.240 INFO :arrow_forward: Diagnostics completed

Summary:
[WARN] Your connection string specifies only a single host. You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
[ERRO] Failed to fetch configuration via cccp from 172.x.y.z:11210 (error: invalid bucket name/password)
[ERRO] Failed to fetch terse configuration via http from 172.x.y.z:8091 (error: incorrect bucket/password)
[ERRO] All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible

Found multiple issues, see listing above.

Hey,

Please specify your password to the sdk-doctor when you invoke it.

Cheers, Brett

Hi @brett19,

I get a similar problem but only intermittently and only on our production and staging servers. We connect to a CB cluster using the couchbase://xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy string (ip’s obfuscated) and every so often we get the client timeout issue listed above. I have run sdk-doctor several times and sometimes get:

 Summary:
←[31m[ERRO]←[0m Failed to connect to CAPI service at `xxx.xxx.xxx.xxx:8092` (error: Get http://xxx.xxx.xxx.xxx:8092/: net/http: request canceled (Client.Timeout exceeded while awaiting headers))

I have set the operationTimeout to 60 seconds but the timeout error happens well before that time. I am connecting to the bucket like so:

const environment = process.env.NODE_ENV

const cluster = new couchbase.Cluster(config.environments[environment].couchbase.server)

const getBucket = function(bucketName, password, callback) {
    return cluster.openBucket(
        bucketName,
        password,
        function(err) {
            if (err) {
                log.fatal('Couchbase getBucket. Got error getting bucket: ' + bucketName + ': %j', err)
            } else {
                log.info('Couchbase bucket: ' + bucketName + ': connected')
                if (typeof callback === "function") {
                    callback()
                }
            }
            this.operationTimeout = config.environments[environment].couchbase.operationTimeout;
        })
}

I guess we may have a network issue somewhere but how to find out? The connection works ~95% of the time but never fails on my dev or QA boxes (node and CB are on the same machine so no network latency)

Any help would be greatly appreciated.

Cheers,

Fabrice

Hey Fabrice,

It does seem likely that there is some sort of network issue between your application and Couchbase Server. This could take the form of a miss-configured firewall, bad router, bad nic, bad drivers, etc… There are a substantial number of things that could cause your packets to get dropped as they are.

Out of curiosity, do the connection issues you get periodically always happen for only a single service, or is it random.

Cheers, Brett

Hey @brett19 thanks for mentioning about sdk-doctor ,Issue is resolved now .
The issue was:
My Couchbase server IP is 172.x.y.z and my cochbase server Node is pointing to another subnet IP - 192.x.y.z

So i fixed server node’s IP to 172.x.y.z ,Now its working fine .
Thanks to sdk-doctor ,due to which i am able to identify my couchbase node’s IP is pointing to different subnet .

Thanks to u @brett19 .

Hi Brett,

I suspected that there is some network interference but I can’t get the client timeout to work so may be missing something there. I’m setting operationTimeout on the bucket but to no avail. As for the services it’s totally random.

Cheers,

Fabrice

We have had this issue in the .net client. Is dropping the port still recommended since I don’t see that in any of the code examples from the documentation?

I try to use sdk-docktor-linux on GCP ubuntu 14.04
but I got error message below.
any idea?

/sdk-doctor-linux: line 1: syntax error near unexpected token `<’
./sdk-doctor-linux: line 1:

<html><body>You are being <a href="https://github-production-release-asset-2e65be.s3.amazonaws.com/7456015
0/5ac5783a-eec4-11e6-9623-f13596642247?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171222%2Fus-east-
1%2Fs3%2Faws4_request&amp;X-Amz-Date=20171222T153650Z&amp;X-Amz-Expires=300&amp;X-Amz-Signature=0c2d9bf85303460f8af96ad521a2305a915311c
3467199d1f6a5748baf1983e8&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;response-content-disposition=attachment%3B%20filename%3Dsdk-d
octor-linux&amp;response-content-type=application%2Foctet-stream">redirected</a>.</body></html>

Hey @willyou,

Can you show us the command line invocation you used for the doctor? It looks like you’ve specified a strange node list which points to a non-Couchbase server?

Cheers, Brett

first, curl -O from github
then

sudo chmod a+x sdk-doctor-linux
./sdk-doctor-linux  diagnose --help