Syncing selected documents between different servers


#1

I know that Sync Gateway is designed to sync selected documents between a Couchbase server and multiple client devices. But can it also be used to sync documents between two separate Couchbase server installations?

I’m working on an application that would let companies collaborate with one another on common projects. Each company would have its own Couchbase server that would store both private-scoped documents (objects) related to its projects, and public-scoped documents that it needs to share with other companies working on the same projects. Those other companies would have their own separate Couchbase servers running the same application.

So can Sync Gateway be used to link different Couchbase servers in this way, so that they can share selected documents?


#2

This is possible, but there also at least a handful of ways you could accomplish your basic goal of some shared data, some private data, so maybe a quick run through will help.

The big question is: are you talking about integrating a bunch of apps that already use Couchbase Server, or are you writing from scratch? If you are writing from scratch, then it’s worth considering ways to use the Couchbase Mobile data model all the way at the application layer. In short, Couchbase Mobile documents keep a concurrency control identifier on the document (the _rev field) for deeper control over conflict resolution. Couchbase Server has a different concurrency control model, so if you are talking about a bunch of code that already interfaces with CBS, then there are patterns to make that across locations, but they’ll be different from the patterns you get if you use Couchbase Mobile APIs full stack.

Lemme know a little more about your requirements (HTML5 UI? Native desktop apps? existing backend processes? green field? etc) and I can give you more detail.


#3

Sync Gateway acts as a “passive partner” in replications — it implements its side of the API, but it’s Couchbase Lite that does the work of figuring out which revisions need to be copied. That’s just because it was easier to implement that way.

Traun Leyden (@traun) implemented a proof-of-concept program in Go that acts as the active replicator: point it at the URLs of two Sync Gateways and it will replicate between them. We’ve talked about integrating this functionality into a future version of the Gateway itself, but it’s not a confirmed feature yet.


#4

Thanks for your replies.

This is a new application that’s still in the conceptual phase (although I’ve been developing the concept for a number of years). It will be an offline first app - mainly for desktop but also for mobile (but with limited functionality). And it would be native in both cases.

I’ve considered many different approaches over the years, including CouchDB and Couchbase. Lately I’ve been leaning towards a Cloudant-based solution, using Cloudant Sync (Java) on desktop and mobile devices. But I’ve just realised that trying to do filtered replication with many users probably isn’t viable in terms of performance, which is why I’m taking a closer look at Sync Gateway and Couchbase.

An advantage of using a Cloudant-based approach is that I can have separate databases for private and public data, then use filtered replication to copy the necessary data between them, and restrict outside access to only the public database. But I don’t know if the same thing can be achieved using Couchbase buckets as I thought they were isolated?

I have some other technical issues I’d also like to discuss, but I don’t want to give too much detail in a public forum.

Thanks again,
Denis


#5

Yes, this is why @jchrisa and I moved away from the purely CouchDB-like approach and he designed the channel system in SG.

But I don’t know if the same thing can be achieved using Couchbase buckets as I thought they were isolated?

If you want to replicate the entire database, both private and public data, you’d run the replication over the admin port, so all docs would be accessible.


#6

Here’s the github repo for sg-replicate:


#7

With my application, I want to be able to share documents between companies working on the same projects. For example, the company I work for is a system integrator with 20 employees, and it probably does work for up to 100 different customers.

Now I’d like to isolate the data that I share with those companies, so that Company X doesn’t know what we’re doing with Company Y. With Cloudant I can do that by using separate databases with unique access rights. But with Couchbase Server it looks like I’d have to use a separate bucket per company to achieve isolation.

Now there seem to be a couple of problems with that approach. One is that Couchbase Server is limited to 128 buckets (although the recommended maximum is 10). That’s sufficient for our company, but it may not be for larger companies.

But the bigger problem seems to be that Sync Gateway only works with one Couchbase bucket, as its -bucket configuration option specifies the name of the bucket to use. If that’s correct, it means I have to put all the shared company data into a single bucket, which I think is too much of a security risk.


#8

The channels in Sync Gateway allow you to apply read access privileges to documents based on their contents. So you can keep both companies’ docs in one bucket/database and still have privacy. (As well as being able to have shared docs of course.)


#9

Also, no, you can configure the gateway to serve any number of databases/buckets.


#10

Thanks for clarifying that, Jens.

Can sg-replicate be configured to connect to multiple Sync Gateways?


#11

Yes; as I understand it, that’s it’s primary purpose.