Getting couchbase error: Client-Side timeout exceeded for operation

Hi,

We have been struggling with this Error:
{ “errorMessage”: “Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout”,
“errorType”: “CouchbaseError”,
“stackTrace”:
}

And after this we get error on queries:
“Cannot perform operation on a shutdown bucket”

Now, we are using AWS lambda for this operation but I don’t think that matters as we have a regular couchbase server 4.5 and couchbase node.js 2.2 SDK with Ottoman as ORM. I have gone through the previously posted issues on this forum as well as other. Following is the one which we looked into.

We have tried all the possible options we could to overcome this, like, by increasing the operationTimeout and n1qlTimeout to a sufficiently high value but we still get this. Also we have tried to narrow down the issue which happens mostly whenever we try to query user. We get this error on the bucket as well connection string.


Below is our sample code:
exports.handler = function(event, context, callback) {
function sendResponse(error, data) {
ottoman.bucket.disconnect();
console.log("error : ", error);
console.log("data : ", data);
callback(error, data);
context.succeed({
success: true
});
}
console.log(event);
if(event.id != undefined) {
user.find({ _id: event.id }, function(error, result) { > //this line generates the error
if (error) {
sendResponse(error, null);
}
console.log("22 : ", result);
});
}
});


And this is database connection code:
var couchbase=require(‘couchbase’);
var ottoman=require(‘ottoman’);
var config = require(“./config”);

var myCluster = new couchbase.Cluster(config.couchbase.server);
module.exports.bucket = myCluster.openBucket(config.couchbase.bucket,function (error) {
if(error) {
console.log(error);
}
module.exports.bucket.operationTimeout=20000;
module.exports.bucket.n1qlTimeout=100000;
console.log(“Successfully opened igt bucket”);
ottoman.bucket = module.exports.bucket;
});


Ottoman npm package has other methods or functions to query user but all of them give the same error on query. Issue is we get the timeout error most of the times in the database connection when we execute the lambda function.

Below are few logs from our function and error:


2016-09-01T16:22:40.011Z 4d72d039-7060-11e6-9c97-f338272e12c8 Error: cannot perform operations on a shutdown bucket
at Bucket._maybeInvoke (/var/task/node_modules/couchbase/lib/bucket.js:1063:11)
at Bucket._n1ql (/var/task/node_modules/couchbase/lib/bucket.js:716:8)
at Bucket.query (/var/task/node_modules/couchbase/lib/bucket.js:948:17)
at CbStoreAdapter.find (/var/task/node_modules/ottoman/lib/cbstoreadapter.js:839:15)
at Ottoman._findModels (/var/task/node_modules/ottoman/lib/ottoman.js:593:16)
at Function.ModelInstance.find (/var/task/node_modules/ottoman/lib/modelinstance.js:659:23)
at exports.handler (/var/task/userForgotPasswordDev.js:21:14)


Also let me know from where can I get or generate more verbose logs to view and analyse the error.

Thanks.

I’ve experienced similar issues when trying to execute scripts with the SDK installed on my local machine (laptop) to an AWS hosted couchbase cluster. Increasing the timeout didn’t fix the issue - only suggestion I can make is to execute through the SDK as close to the couchbase cluster/network as possible - ie: a development/staging instance within the same network as the couchbase cluster.

Hey @sumedh908,

Can you confirm that your cluster is fully configured to use publicly available IP addresses, and all Couchbase ports are open to the world? In the case of AWS lambda connecting to your cluster on AWS, the connection is made through the public network. Check your cluster node list to verify that the cluster is interconnected via the public network (so that it advertises its network address as being the public address rather than private one).

See here for further details on how to deploy Couchbase on AWS:
http://developer.couchbase.com/documentation/server/current/install/cloud-deployment.html

Cheers, Brett

Thanks for the suggestion @jernster. I will try to connect couchbase on the same network and will check if it is successfully connected.

@brett19

Yes we have fully configured our cluster using public DNS’s of Amazon EC2 instances and more to that we also public IP’s attached to them.
I have referred the documentation on http://docs.couchbase.com/couchbase-manual-2.5/cb-install/#network-ports and all the ports are publicly open.
Now, we have multiple environments setup for the application. Development environment has a single node and Production environment has multiple nodes each for data, index, query. Both the environments face the same issue which happens randomly. I am using the public IP to connect to couchbase cluster from AWS Lambda.

If you look at the below logs:
2016-09-01T15:27:01.936Z 87cab3a0-7058-11e6-9999-d55c25977b16 Error: cannot perform operations on a shutdown bucket
at Bucket._maybeInvoke (/var/task/node_modules/couchbase/lib/bucket.js:1063:11)
at Bucket._n1ql (/var/task/node_modules/couchbase/lib/bucket.js:716:8)
at Bucket.query (/var/task/node_modules/couchbase/lib/bucket.js:948:17)
at CbStoreAdapter.find (/var/task/node_modules/ottoman/lib/cbstoreadapter.js:839:15)
at Ottoman._findModels (/var/task/node_modules/ottoman/lib/ottoman.js:593:16)
at Function.ModelInstance.find (/var/task/node_modules/ottoman/lib/modelinstance.js:659:23)
at exports.handler (/var/task/userForgotPasswordDev.js:106:24)

REPORT RequestId: 87cab3a0-7058-11e6-9999-d55c25977b16 Duration: 43.54 ms Memory Size: 128 MB Max Memory Used: 36 MB


This function ran for less than a tenth of a second before returning this error.

Hey @sumedh908,

Would you be able to attach a listener to the buckte’s error event? This should provide you with some more information on the reason for the failed connection. I’ve provided an example below.

var cluster = ...;
var bucket = cluster.openBucket('default');
bucket.on('error', function(err) {
  console.log(err);
});

Cheres, Brett

@brett19

We have attached the listener on bucket. Below is our latest code:


module.exports.bucket.on(‘error’, function(err) {
console.log(‘CONNECT ERROR:’, err);
});
module.exports.bucket.operationTimeout=20000;
module.exports.bucket.n1qlTimeout=100000;
console.log(“Successfully opened bucket”);
ottoman.bucket = module.exports.bucket;

Following are the logs in CloudWatch of Lambda:
2016-09-02T07:43:01.588Z dc71ceeb-70e0-11e6-9b8e-659d2dcaffd8 Successfully opened bucket

2016-09-02T07:43:02.068Z dc71ceeb-70e0-11e6-9b8e-659d2dcaffd8 error : { [CouchbaseError: Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout] message: ‘Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout’, code: 23 }

2016-09-02T07:43:02.070Z dc71ceeb-70e0-11e6-9b8e-659d2dcaffd8
{
“errorMessage”: “Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout”,
“errorType”: “CouchbaseError”,
“stackTrace”: []
}

It doesn’t seem to console the listener event.

Hey,
Sorry, I am getting confused between the two errors. At one point you were mentioning the shutdown bucket error, which the bucket error event should detect. On the other hand, receiving timeout errors usually will indicate that there is trouble with the network between your cluster and the application server. Have you been able to reproduce this issue outside of lambda (which is known to have suspicious network behaviour)?
Cheers, Brett

@brett19

Even we are not sure why these errors are generated. Because when we a triggering the lambda function we get the timeout error. And after testing it consequent requests, we have observed the shutdown bucket error. This is happening when we query the users collection. Some logs show timeout and some print the shutdown bucket errors.
When we run the lambda function locally, we are getting timeout errors but the frequency is 80% less than actually running it on lambda.

We’re seeing something that feels similar using the latest Ruby SDK: Ruby SDK Times Out with Back-to-Back Queries

In our case, this is two Ubuntu instances in the same region and same security group, but two different availability zones (us-west-2a test instance hitting us-west-2c database box). We’re using single server 4.1.1-5914-enterprise with no other client connections but the test code, and we’re using the 1.3.14 ruby gem with libcouchbase 2.6.2.

I know the cross post is annoying, but hopefully not too confusing and potentially helpful if this could be libcouchbase related instead of an issue with couchbase itself.

Hello Brett,

I get this error occasionally as well and it does make sense the throughput is being exceeded. I am using only a single instance at this time as I try things out. Would you say adding an additional instance or two alleviate this issues? Or, is this network throughput related?

Thanks

Hey @jgcoding,

Typically timeout issues encountered with the Node.js SDK can come from a number of causes. Identifying the exact source usually requires a bit of diagnostics. It is worth viewing your systems overall network throughput as well as looking at the Couchbase UI for more information on the number of operations being performed as well as the number of disk fetches that need to be performed (disk fetches are significantly slower than memory fetches on the server, and can easily cause your client to timeout if lots of operations are dispatched at once).

Cheers, Brett

1 Like

Understood. Thanks for your reply, I appreciate it.

JG