SyncGW 1.2 - More than 1GB Memory and 10% CPU

Mobile Sync Gateway
1

Following is the setup details

CB Couchbase Server 4.0.0-4051 (CE) - 1 Node
SyncGW 1.2 CE - 1 Node

CBS Disk usage is 1.67GB. Traffic tool is pumping 8ops. syncGW config file has webhook configured. On running the traffic tool it is seen that syngw process private memory is constantly increasing. On running traffic tool for 6 hours it is seen syngw process private memory is going above 1.3 GB and the CPU taken by syncgw also is increasing to 10%.

Following is the syncgw log snippet of processing one document.

2016-04-07T17:08:31.846+05:30 HTTP: #089: PUT /ptxdata/s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc (as 918553899955)
2016-04-07T17:08:31.847+05:30 CRUD+: Invoking sync on doc “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” rev 1-e72fb706a2b23a3d359671bb5a22bbdc
2016-04-07T17:08:31.870+05:30 Cache: SAVING #1368598
2016-04-07T17:08:31.870+05:30 CRUD: Stored doc “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” / "1-e72fb706a2b23a3d359671bb5a22bbdc"
2016-04-07T17:08:31.870+05:30 HTTP+: #089: --> 201 (424.9 ms)
2016-04-07T17:08:31.870+05:30 Events+: Event queue worker sending event Document change event for doc id: s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc to: Webhook handler [http://XXXXX]
2016-04-07T17:08:31.871+05:30 Events+: Webhook handler ran for event. Payload posted to URL http://XXXXXXX, got status 200 OK
2016-04-07T17:08:31.878+05:30 HTTP: #098: PUT /ptxdata/s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc (ADMIN)
2016-04-07T17:08:31.878+05:30 CRUD+: Invoking sync on doc “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” rev 2-11e59717832379d3a6ad4e4582858c2d
2016-04-07T17:08:31.879+05:30 CRUD+: Saving old revision “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” / “1-e72fb706a2b23a3d359671bb5a22bbdc” (110 bytes)
2016-04-07T17:08:31.879+05:30 CRUD+: Backed up obsolete rev “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc”/"1-e72fb706a2b23a3d359671bb5a22bbdc"
2016-04-07T17:08:31.879+05:30 CRUD: Doc “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” in channels "{s918553899955_ptx1460029083919490744-content-CH}"
2016-04-07T17:08:31.879+05:30 Cache: SAVING #1368599
2016-04-07T17:08:31.879+05:30 CRUD: Stored doc “s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc” / "2-11e59717832379d3a6ad4e4582858c2d"
2016-04-07T17:08:31.879+05:30 HTTP+: #098: --> 201 (1.3 ms)
2016-04-07T17:08:31.879+05:30 Events+: Event queue worker sending event Document change event for doc id: s918553899955_ptx1460029083919490744_A1460029083_Attachment.doc to: Webhook handler [http://XXXX]

–Gracelin

2

Sync Gateway maintains an in-memory channel cache, which retains at minimum the 50 most recently accessed documents for each channel. What’s the channel model for the documents you’re pushing with the traffic tool? If you have very few docs/channel, you might see memory increasing in the manner described.

3

Adam,

Thanks for the response.

Is there a provision in syncgw to view how many channels are there totally, how many channel are in use presently and how many documents are assigned to each of the channels currently in use?

Gracelin

4

We don’t currently have any channel index diagnostics exposed, but it’s on the roadmap, hopefully for 1.3.

See https://github.com/couchbase/sync_gateway/issues/1149 - I’ve just added a comment to ensure the diagnostics are also exposed for the channel index.

5

Adam,

Channel to document map available in memory would definitely a good diagnostics required.

All the documents pushing with the traffic tool are assigned to the same channel only Memory usage is increasing steadily with the document. Is there a way to diagnose what is causing the memory increase in sync gateway node?

6

@sgracelin Is the traffic tool generating inserts, updates, or a mix? If you’re doing a lot of updates, it’s possible the revision cache is slowly growing, as the size of the revision trees in the cache increases.

If it’s mainly inserts, my guess is that there’s something not getting released in the webhook handling.

If you can share the details on the insert/update/delete breakdown, I’ll file a ticket to try to reproduce and investigate further.

7

Thanks a lot @adamf for the quick and helpful response.

Traffic tool is generating mixed traffic. New document creation, GET of those documents and Updates are also happening. Can you please elaborate on what all will be stored in the memory by the sync gateway??. If we want to arrive on the memory requirement for syncgw node what is the requirement?

As I understand from the discussion with you, if number of channels are more and less documents are assigned to each channel then the memory requirement will be more compared to assigning good amount of documents to a single channel. If X is the number of channels what is the recommended Y ie number of documents that can be assigned to each channel? When X number of channels and Y number of documents are assigned to each of the channel, then what is the expected memory usage?. If there are N number of sync GW nodes how does it get distributed among them?

Please help to understand better in the memory utilisation by a SyncGW node.

Regards,
Gracelin