How to monitor replication "checking" phase

richard.perfect · September 17, 2018, 10:57pm

Our application is successfully synchronizing two Couchbase Server database using the Sync Gateway (not using XDCR). I am able to create/delete replication tasks and also monitor the tasks themselves using the “/_active_tasks” endpoint all ok.

However when a replication task is created there seems to be a fairly long period (10’s of mins in our case) of time where the Sync Gateway is checking to see which documents need to be copied before actually doing anything. This means that the Active Tasks results doesn’t show any documents being read/written for quite some period of time after a replication task has been created.

Is there any way to monitor what the Sync Gateway is doing during this “checking” phase so that we can provide feedback to the user that their replication request is actually being worked on, and they just need to wait?

At this stage my best option seems to be to watch the file size of the “sync_gateway_error.log” and just say that the Sync Gateway is working when it sees the file size increasing. Is there a better way?

traun · September 18, 2018, 10:08pm

Great to hear that! The only thing I can think of right now is to look at some of the stats in the _expvars.

Unfortunately we don’t have a lot of docs for the expvars, the best I know of are here:

If you poke around the mobile-testkit repo, there is some example usage of pulling data from expvars:

github.com

couchbaselabs/mobile-testkit/blob/4742fa288dba8d4a205c4a29a559fff9dd60467f/libraries/utilities/monitor_sync_gateway.py#L31-L43


expvars = [
    "mem:memstats.Alloc",
    "mem:memstats.Sys",
    "mem:memstats.HeapAlloc",
    "mem:memstats.HeapInuse",
    "memstats.NumGC",
    "memstats.PauseTotalNs",
    "memstats.PauseNs",
    "syncGateway_db.channelChangesFeeds",
    "syncGateway_db.document_gets",
    "syncGateway_db.revisionCache_adds",
    "syncGateway_db.revs_added"
]

richard.perfect · September 22, 2018, 9:53pm

Thanks for the suggestion. I had a look at the expvars but nothing seemed to change in any useful way when the system is doing the sync checking phase so we’ve gone with monitoring the file size of the log file.

At the moment it looks like a CPU monitor you just see a graph showing the delta difference in the file size for each polling period. If nothing is happening then the line graph is low when something is happening then the line graph bounces up to a much higher level.