Push many documents using couchbase lite and sync gateway


#1

Hello,
I’m implementing a solution with multiple .Net Mobile clients that runs couchbase lite, and sync documents through sync gateway.
I’m testing couchbase lite against a big number of documents.
Let’s say we have 100k document per Client, the Client is authenticated on the Gateway using the user “stressuser1”.
The pull request ends in a few seconds. It logs a hundred of lines like this:

16:15:48.482681 HTTP: #057: PUT /syncdatasampledb/_local/cdcec8cd4951410c5a046bca413f704095a0bd2c (as stressuser1)

When i launch the pull request, Sync gateway takes a lot, it seems it process every document for the channel interested. It logs many lines (one per document?) that looks like this one:

2015/03/12 16:06:51 go-couchbase: call to Do("_sync:user:stressuser1") in github.com/couchbaselabs/go-couchbase.(*Bucket).casNext took 1m2.031121565s
16:06:51.061736 HTTP: #1386: POST /syncdatasampledb/_revs_diff (as stressuser1)

If the push works this way (one like per documents check), i can not use sync gateway for many documents.
Does anyone have tested the sync gateway and mobile lite over many documents, or has any kind of suggestion?

At present I’m thinking about writing those documents to the Server using the CouchBase SDK, then I’ve to exclude those documents from any update, event from the shadow bucket gateway update, if possible.

Thank you for any suggestion.


#2

PUT /syncdatasampledb/_local/cdcec8cd4951410c5a046bca413f704095a0bd2c

This is an internal bookkeeping call that Couchbase Lite makes, and is used only by the client to keep track of how far it has gotten through the sync process. It doesn’t represent your user documents. With 100k documents, it will chunk pulls and pushes. Is your app’s normal usage pattern really sync’ing 100k documents per session? This seems like a very synthetic test, without knowing more about your application directly.

At present I’m thinking about writing those documents to the Server using the CouchBase SDK

We do not currently support making writes to the bucket used by Sync Gateway. Doing so nearly guarantees that SG will not see that document, as the rules for correctly writing the metadata are very difficult to implement outside of Sync Gateway.

If you tell us what you goal you are trying to reach, we can very likely help you find a better approach to getting there.


#3

First: thanks for your interest.

The solution I’m trying to build is syntetized here: Architectural information for a Server/Mobile app
Additional note:
I have two bucket “bucketsync” (used by the sync gateway) and “bucket” (shadow bucket).
“At present I’m thinking about writing those documents to the Server using the CouchBase SDK”… i mean writing in the shadowed bucket (“bucket”). I noticed that if i write documents to the shadowed bucket (“bucket”) they where updated by the sync gateway to his bucket (“bucketsync”).
Also consider that the logged documents are really important for this application, they could not be lost, I’m afraid because I do not know how to stop sync gateway to delete those “logged” documents from the shadowed bucket. I would like to mantain those documents once they where loaded.

Another alternative could be using a NodeJS (or other) to implement a RESTful service, just to write document I’ve to log in another bucket (not the one shadowed by the sync gateway, nor the sync gateway bucket, let’s say “bucketlog”), this way it would be fully external to the sync gateway world.

I hope I explained my question in the right way. If not, please ask me more.


#4

I’m not sure from your post what the actual problem you’re having is. Is it performance? Reliability? Just too much logging?

2015/03/12 16:06:51 go-couchbase: call to Do("_sync:user:stressuser1") in github.com/couchbaselabs/go-couchbase.(*Bucket).casNext took 1m2.031121565s

This looks like a performance problem in the database server — a single Set operation took a minute to complete (when it should take milliseconds). So if you’re seeing poor performance, it looks like something to do with how your Couchbase Server is configured, or loaded, or the connection between it and Sync Gateway.


#5

Thank you for help.
The problem I have is that a push with 100k documents takes too much to perform. This is probably due to poor connection with the sync gateway. The connection between my clients is a 3G, and may fails. Also, the couchbase server, at present is a virtual linux machine running on a i7, with just 1Gb ram for this machine.
The solution I’m thinking about is:
Use sync gateway (liked to a bucket and his shadow bucket) to sync just the 1000 (more or less) documents that’s need to be synched.
Use a NodeJS RESTful service to write down logged documents (almost 5000 per day per clients) to another bucket.
What is your opinion with respect to this solution?