Sync gateway taking 13 - 20s to get changes_view


#1

We’ve been having a lot of performance problems lately and while investigating I saw in the sync gateway logs that it is taking a very long time to query the changes_view:
14:32:41.911325 2016-06-14T14:32:41.911Z changes_view: Query took 17.2230945s to return 10 rows, options = db.Body{“endkey”:[]interface {}{“channel1”, 0x6e42d4}, “stale”:false, “startkey”:[]interface {}{“channel1”, 0x1}}

Normally in our logs it shows this kind of call taking about 200 - 800ms. We aren’t using many resources on the VM sync gateway is running on either, less than 20% of RAM and less than 25% CPU. The last time we had issues like this it was from calling access() too often in the sync gateway config, but we don’t do that anymore so what else could be causing sync gateway to run so slowly? We are using sync gateway v1.1.1, couchbase server 3.1


#2

The slow query is made when Sync Gateway doesn’t have the full history of the channel in memory, and needs to execute a Couchbase Server view to retrieve older entries. If Sync Gateway is not under significant load, the slow query response time is likely on the Couchbase Server side.

If you’re seeing the query being executed repeatedly in the logs for the same channel (based on the startkey), then you might be able to mitigate by changing Sync Gateway’s channel cache size, to avoid repeated queries.

Since this sounds like a relatively new problem, though, I’d probably look first for potential bottlenecks on the server side.


#3

Would the number of allowed revisions affect this? I just found out recently that we can set a max number of revisions. Currently we have no limit so some documents have hundreds or thousands of revisions. The actual old revision docs get deleted fairly quickly but the document’s metadata is huge because of all the revisions listed in it. Just wondering if setting that value might help us going forward.


#4

I’m not sure what significant load would be but we don’t have a huge user base so I don’t think it is. In ~1.5hrs I saw 40 changes_view requests. 9 of them took longer the 5 seconds and 3 of them took longer than 20 seconds. Would that be considered a heavy load?

We are aware of very bad performance for couchbase server when retrieving documents from disk and we’re working on migrating to a new architecture with a better disk setup. If that could cause the issue then it is most likely that, though this problem just started happening for us a few weeks ago and our load has not increased significantly in the past couple months.


#5

The frequency of the channels view query wouldn’t be impacted by the number of allowed revisions - the channel cache only retains one revision per doc id, so the revision limit shouldn’t affect how frequently the view call is required.

It’s possible that larger document size (a consequence of the larger number of revisions) could be increasing your indexing/view response time, but I wouldn’t expect a sudden jump of the type you’ve seen - it would be more gradual performance degradation. It’s probably worth lowering your max revision count. Note that it’s important that your max revision count is still quite a bit larger than your update frequency, to avoid redundant conflicts.

It does sound like this is more likely to be related to your other Couchbase Server performance issues, although I don’t know conclusively how much disk retrieval impacts view indexing/query performance.