New user creation takes very long time

Mobile Sync Gateway
1

Hi guys,
currently I have cluster with 3 nodes (8 cores cpu 64gb ram) CB 4.0 CE
also I have 2 same configuration servers running only SG 1.2.1 CE
this 5 machines are working for syncing data in my mobile app
currently there is more than 200 millions docs synchronised
average load is about 100 000 daily users,
and it’s about 30 000 new users daily
I’m using session based auth for all my users
I have no problems with getting session for existing users
However creating new user with Admin API takes very-very long time (about 20 minutes)
I’m creating new user with POST request to Admin API
When I create new user I pass only name, password and admin_channels
Any suggestions why it takes so long? What steps can I do to reduce this time
maybe it’s possible to authenticate with basic auth but not creating user? or smth like that
Thank you in advance

2

Are you saying it’s taking 20 minutes to create a single user, or 20 minutes for your daily 30K new users?

20 minutes for a single user would obviously be very unexpected. User creation does result in a stale=false view query to be executed against the server (to identify whether the new user has already been granted channel access by documents), so I assume that’s related to the delay, but 20 minutes seems excessive given the number of docs and the size of your cluster, unless your server cluster is already under very high load.

3

it takes 20 minutes for every single user
I’m not sure if my cluster is under high load
it has next parameters, see screenshots




first bucket is just test bucket, problems is with second bucket
also every user has it’s own channel, so it can access only his data

4

I’m not an expert on Couchbase Server sizing - you might need to direct that part of the question to the server forums - but from the Sync Gateway side…are you seeing anything in the Sync Gateway logs during the user creation operation?

5

i’ve checked logs, the most common message is

and

6

log level in config file is ‘info’

7

In general that log output suggests that your SG cluster might be overloaded. To test whether the slow user creation is the result of load on the SG or Couchbase Server side, you could spin up a test SG node that’s not servicing requests, and attempt user creation via that node.

It’s still surprising to me that you’d see slow response time only on user creation - particularly if session creation is happening quickly.

8

@yuzz,

Could you log into a sync gateway node and run below command as sync_gateway user.

#ulimit -a

and share output.

9

sure:

10

@yuzz Another thing to look for would be slow execution logging for the view request related to user creation. I’d expect to see entries like

go-couchbase: call to ViewCustom("sync_gateway", "access") in

in your Sync Gateway logs. Do you see any output like that?

11

yes! many of them

12

@yuzz,

looks like you have the linux default number of file descriptors.

Here is the recommend for SG: http://developer.couchbase.com/documentation/mobile/current/develop/guides/sync-gateway/os-level-tuning/max-file-descriptors/index.html

Could you run

#ps aux | grep sync_gateway
output the PID of SG

#lsof -p {pid of sync gateway} | wc -l
number count of file descriptors open

This will give you how many currently used file descriptors that Sync Gateway is using.
source: http://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html

13

Hi guys,
I’ve fixed issue with file descritptors, as recomended, also I’ve found that I had full eviction settings for my bucket, it fixed issue with big number of reads/wrtites to disk.
However problem wasn’t solved.
Now I see next logs in SG log, I think it’s too much time

14

Agreed - basic Get operations against the bucket shouldn’t be taking that long. We’re back to the same question I raised above, though: whether the slow operations are due to an overloaded Sync Gateway, or an slow Couchbase Server response.

If you’re able to try the above suggestion - spin up another Sync Gateway that’s not under load and check response time while the server is under load from the existing Sync Gateway - that will help narrow things down.

Another option would be to check CPU usage on the existing Sync Gateway machine - slow operation time can be the result of CPU starvation when too many concurrent operations are running.

15

so, I’ve started sync gateway server on fresh environment, not accepting requests
same picture, very long time creating user. it seems like issue with CB server perfomance
do I need to move to neighbour forum about CB server?

16

I’d probably recommend that - the server team should be able to give you better advice on how to dig into the problem using the server diagnostic tools. If it comes back to something Sync Gateway can do differently, I can jump back in.