Syncgateway TimeOut with 2 Data Node Cluster

[wanda_a]

Greetings!
I am in a rather peculiar position where my syncgateway keeps timing out at launch each time i have more than one data node (works perfectly when other nodes are not data nodes). I first reported about it here a few days ago but barely understood what was happening so my question wasn’t very clear either. The error message is:

@wanda_a

Timeout 1m15s exceeded -- auth.(*Authenticator).rebuildChannels() at auth.go:176
RebuildChannels returned error: [1080] Timeout 1m15s exceeded --auth(*Authenticator).getPrincipal.func1() at auth.go:119
[ERR] Error opening database db_name: [1080] Timeout 1m15s exceeded

and adjusting the view_query_timeout_secs value to 5 minutes (from default 75 seconds) doesn’t help.

I’ve tried to simply resync the gateway after the rebalance (ensuring it is offline before the nodes are added) but the resync always goes on through the night and past the maintenance window yet it usually takes 10 minutes max. I’ve followed this thread about multi-node clusters and gateways. I still have this trouble whether all data nodes or one data node is listed in the config file.

Besides adding the extra IP addresses to the config file, are there any other fields to adjust to get the gateway responding smoothy to multiple data nodes? Am using Couchbase 6.0.3 and Gateway 2.6. This error is always cleared when i remove the added data nodes. Any insight would be highly appreciated.

Here are my gateway settings, besides the logging and sync functions:

@wanda_a

“databases”: {
“db_name”: {
“server”: “###:8091”,
“bucket”: “##”,
“bucket_op_timeout_ms”: 300,
“username”: “##”,
“password”: “##”,
“num_index_replicas”: 0,
“revs_limit”:20,
“enable_shared_bucket_access”: true,
“import_docs”: true,
“users”: {
“#”: {“password”:“#”},
“#”: {“password”:“#”}
},
“roles”:{
“#”: {},
“#”: {}
},

[sridevi.saragadam]

hi @wanda_a, when you say you used more than one data node and based on the link you provide, I assume that you are using 3 couchbase server nodes in the cluster. Could you let me know how many sync gateways you are using?

Btw, were you able to add all 3 couchbase server nodes in sync gateway config file which you referenced the link

in your sync gateway config file, add all 3 server ips of sserver cluster
"server":“http://11.111.11.111,22.222.22.222,33.333.33.333:8091”,

[sridevi.saragadam]

hi @wanda_a, can you make sure all the servers which you added in the cluster are up and running? Make sure rebalance is not failed. You see this error when one of the server goes down.
Also data bucket username and password you entered on SGW is the right credentials of bucket, right?
Other than that you do not need to enter anything on sgw config file. I do not see anything required.
Also, can u make sure sync gateway node is reachable to new nodes that you added in server cluster?

[wanda_a]

yes, i am using 3 nodes. 1 data and eventing specific, 1 index, query, search and analytics specific and 1 with all services including data. the one with all services was the sole server before, but we trying to branch out now as seen with the other two. we are using one syncgateway, hosted on the same server as the sole server (there’s plenty of hardware resources in that server though).
All 3 nodes were successfully added, and the rebalance was successful. and i added all 3 as in the 123 pattern example.

[wanda_a]

all servers running, re balance successful. credentials are correct am sure (the same set work when i narrow it back to one data node). could you please expound on the “reachable from servers” bit?

also, does it make sense to include the IP of a server that’s not a data server?

[sridevi.saragadam]

@wanda_a, "reachable to server from sync gateway meaning " is sync gateway can communicate to server. Based on your post, sync gateway works fine with one couchbase server node, which means sync gateway is able to talk to couchbase server . When you add new nodes, is sync gateway is able to communicate to new nodes. If new node is not reachable by sync gateway, then it will fail

When you say “does it make sense to include the IP of a server that’s not a data server?” adding data node server ip on sync gateway config? If you have not add that ip, can you please add it and restart the sync gateway?