Error 16 on view query

Basically I have a route that looks up a view. If I call this route concurrently with multiple connections, even 50 at a time, couchbase starts failing with Error code 16. Logs:

15:04:45 event-api.1 | 41238ms [I0] {513} [INFO] (connection - L:450) <127.0.0.1:8092> (SOCK=0x103417340) Starting. Timeout=75000000us
15:04:45 event-api.1 | 2015-07-24T18:04:45.216Z - debug: [pt-node-models] find personas by profile. profile: 52700b8e-e22f-4c48-838b-d8a57e69017d
15:04:45 event-api.1 | 41238ms [I0] {513} [DEBUG] (lcbio_mgr - L:416) <127.0.0.1:8092> (HE=0x105804800) Creating new connection because none are available in the pool
15:04:45 event-api.1 | 41238ms [I0] {513} [DEBUG] (lcbio_mgr - L:321) <127.0.0.1:8092> (HE=0x105804800) Starting connection on I=0x103415d90
15:04:45 event-api.1 | 41238ms [I0] {513} [INFO] (connection - L:450) <127.0.0.1:8092> (SOCK=0x103415ec0) Starting. Timeout=75000000us
15:04:45 event-api.1 | 2015-07-24T18:04:45.216Z - debug: [pt-node-models] find personas by profile. profile: 702e8e50-e564-432e-adb8-1225d4753464
15:04:45 event-api.1 | 41239ms [I0] {513} [DEBUG] (lcbio_mgr - L:416) <127.0.0.1:8092> (HE=0x105804800) Creating new connection because none are available in the pool
15:04:45 event-api.1 | 41239ms [I0] {513} [DEBUG] (lcbio_mgr - L:321) <127.0.0.1:8092> (HE=0x105804800) Starting connection on I=0x1030022d0
15:04:45 event-api.1 | 41239ms [I0] {513} [INFO] (connection - L:450) <127.0.0.1:8092> (SOCK=0x103002460) Starting. Timeout=75000000us
15:04:45 event-api.1 | 2015-07-24T18:04:45.217Z - debug: [pt-node-models] find personas by profile. profile: 77a0b4af-195c-4a85-ab8c-dedf31fab929
15:04:45 event-api.1 | 41239ms [I0] {513} [DEBUG] (lcbio_mgr - L:416) <127.0.0.1:8092> (HE=0x105804800) Creating new connection because none are available in the pool
15:04:45 event-api.1 | 41239ms [I0] {513} [DEBUG] (lcbio_mgr - L:321) <127.0.0.1:8092> (HE=0x105804800) Starting connection on I=0x10302b0c0
15:04:45 event-api.1 | 41239ms [I0] {513} [INFO] (connection - L:450) <127.0.0.1:8092> (SOCK=0x103006e90) Starting. Timeout=75000000us
15:04:45 event-api.1 | 2015-07-24T18:04:45.217Z - debug: [pt-node-models] find personas by profile. profile: d1afbf7a-2a3a-4309-8dba-60689962c712
15:04:45 event-api.1 | 41240ms [I0] {513} [DEBUG] (lcbio_mgr - L:416) <127.0.0.1:8092> (HE=0x105804800) Creating new connection because none are available in the pool
15:04:45 event-api.1 | 41240ms [I0] {513} [DEBUG] (lcbio_mgr - L:321) <127.0.0.1:8092> (HE=0x105804800) Starting connection on I=0x103000640
15:04:45 event-api.1 | 41240ms [I0] {513} [INFO] (connection - L:450) <127.0.0.1:8092> (SOCK=0x10300c550) Starting. Timeout=75000000us
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (connection - L:127) <127.0.0.1:8092> (SOCK=0x10300c550) Failed: lcb_err=0x10, os_errno=24
15:04:45 event-api.1 | 41240ms [I0] {513} [DEBUG] (lcbio_mgr - L:271) <127.0.0.1:8092> (HE=0x105804800) Received result for I=0x103000640,C=0x0; E=0x10
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (http-io - L:226) Connection to failed with Err=0x10
15:04:45 event-api.1 | !!!
15:04:45 event-api.1 | 16
15:04:45 event-api.1 | null
15:04:45 event-api.1 | 2015-07-24T18:04:45.218Z - debug: [cvent-pt-ticket-client] Searching for existing profile for this ticket
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:420) <127.0.0.1:8092> (HE=0x105804800) Not creating a new connection. There are still pending ones
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (ioctx - L:151) <127.0.0.1:8092> (CTX=0x103309300,mgmt/capi) Destroying. PND=0,ENT=1,SORC=1
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (lcbio_mgr - L:486) <127.0.0.1:8092> (HE=0x105804800) Closing idle connection. Too many in quota
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (ioctx - L:151) <127.0.0.1:8092> (CTX=0x103311710,mgmt/capi) Destroying. PND=0,ENT=1,SORC=1
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (lcbio_mgr - L:486) <127.0.0.1:8092> (HE=0x105804800) Closing idle connection. Too many in quota
15:04:45 event-api.1 | 41241ms [I0] {513} [TRACE] (connection - L:344) <127.0.0.1:8092> (SOCK=0x10340bcb0) Received completion handler. Status=0. errno=0
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:116) <127.0.0.1:8092> (SOCK=0x10340bcb0) Connected
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:121) <127.0.0.1:8092> (SOCK=0x10340bcb0) Couldn’t set TCP_NODELAY
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:271) <127.0.0.1:8092> (HE=0x105804800) Received result for I=0x103417d60,C=0x10340bcb0; E=0x0
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:223) <127.0.0.1:8092> (HE=0x105804800) Assigning R=0x10340a3f0 SOCKET=0x10340bcb0
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (ioctx - L:101) <127.0.0.1:8092> (CTX=0x100d1cfd0,unknown) Pairing with SOCK=0x10340bcb0
15:04:45 event-api.1 | 41241ms [I0] {513} [TRACE] (connection - L:344) <127.0.0.1:8092> (SOCK=0x103417340) Received completion handler. Status=0. errno=0
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:116) <127.0.0.1:8092> (SOCK=0x103417340) Connected
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:121) <127.0.0.1:8092> (SOCK=0x103417340) Couldn’t set TCP_NODELAY
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:271) <127.0.0.1:8092> (HE=0x105804800) Received result for I=0x10341ed40,C=0x103417340; E=0x0
15:04:45 event-api.1 | 41241ms [I0] {513} [TRACE] (connection - L:344) <127.0.0.1:8092> (SOCK=0x103415ec0) Received completion handler. Status=0. errno=0
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:116) <127.0.0.1:8092> (SOCK=0x103415ec0) Connected
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:121) <127.0.0.1:8092> (SOCK=0x103415ec0) Couldn’t set TCP_NODELAY
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:271) <127.0.0.1:8092> (HE=0x105804800) Received result for I=0x103415d90,C=0x103415ec0; E=0x0
15:04:45 event-api.1 | 41241ms [I0] {513} [TRACE] (connection - L:344) <127.0.0.1:8092> (SOCK=0x103002460) Received completion handler. Status=0. errno=0
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:116) <127.0.0.1:8092> (SOCK=0x103002460) Connected
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:121) <127.0.0.1:8092> (SOCK=0x103002460) Couldn’t set TCP_NODELAY
15:04:45 event-api.1 | 41241ms [I0] {513} [DEBUG] (lcbio_mgr - L:271) <127.0.0.1:8092> (HE=0x105804800) Received result for I=0x1030022d0,C=0x103002460; E=0x0
15:04:45 event-api.1 | 41241ms [I0] {513} [TRACE] (connection - L:344) <127.0.0.1:8092> (SOCK=0x103006e90) Received completion handler. Status=0. errno=0
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:116) <127.0.0.1:8092> (SOCK=0x103006e90) Connected
15:04:45 event-api.1 | 41241ms [I0] {513} [INFO] (connection - L:121) <127.0.0.1:8092> (SOCK=0x103006e90) Couldn’t set TCP_NODELAY

The !!! stuff is me inserting this code:

console.log(’!!!’);
console.log(errCode)
console.log(val)

in the “else” case in Bucket.prototype._viewReq

I have also tried turning off http poolsize using

export LCB_OPTIONS=http_poolsize=0

Is query view connection pooling the problem? Is there a way to turn off connection pooling for view quering within couchnode lib?

But that did not seem to work.

Any help is appreciated.

Thanks,

Bojan

I think @brett19 might be able to help you on this.

Hey bojand,

Error 16 indicates that the network connection failed for some reason. This seems likely to be caused by some kind of firewall or other operating system problem which is restricting the number of connections that are possible (and thus the number of concurrent view requests).

Cheers, Brett

The logs state 15:04:45 event-api.1 | 41240ms [I0] {513} [ERROR] (connection - L:127) <127.0.0.1:8092> (SOCK=0x10300c550) Failed: lcb_err=0x10, os_errno=24. 24 means EMFILE which means you have too many open file descriptors. You should enable detailed error codes in the connection string and it should give you a more accurate error message.

It seems to have been EMFILE issue, increasing max file limit solved the issue.

Thanks,