We have 2 repositories with one cluster for each. One is where we will keep app related data. It will be having data changed by apps. And second repository is for other business requirements. Both repositories will have different granularity. The app repository will generally have documents broken into different parts so that less amount of data will be synced. We will be keeping static and volatile data in different documents and technologies as diffgrams are not implemented yet and complete documents are synced. Also App repository will not have complete data, it will have data only required by the apps.
But we need to keep both repository in sync.
To do this I think there are 2 options which I can think of right now
Using web hooks :- We will create a web service on which Sync gateway will post documents using web hooks and this service will apply business logic to merge multi documents coming from app repository into one and post it on sync gateway of other repository. But what if the services are down, Then the queue can fill out quickly and the changes will not sync. Or if there is an error on the service and we are setting very less time for timeout in web hooks. In this case we will need to write a fallback method. What do you suggest as the best way to write fallback. Which will make sure all the data is processed and nothing is missing even if some documents missed with web hook method.One method is to create a daemon service which will ask sync gateway about the data changed from last sync periodically. But I need to know if there are better methods then this.
creating the service as a client by embedding CBL in the service. This will work like a mobile device. And when the service will be saving document in its local DB we will set TTL of the document very short so that the service local DB wont have much documents as in reality it doesnt need. But then I am skeptical about the performance of this approach. Theoretically the sync will always happen. But will the CBL be able to cater the load. Will speed be an issue. Please suggest. I also want to know at what number of documents CBL will start having issue.
do you need bi-directional sync between app repository and the other repository?
do you mean that different apps with different user in Sync Gateway, so what the relationship between app repository and the other repository about Sync Gateway user?
may be you can use websocket to listen the changed documents with last_seq value stored somewhere, so that if your daemon service is down, you can read the changed data from last_seq, so you will not missing any changed data.
How about store the data to local, so that this data will not sync.
Yes we will need it if possible easily and have less failure.
The same app repository will be used for multiple apps in a single framework. lot of data is shared between apps. You can assume it as one app for employees and one for guests. All data in the app repository is not required in other repository because multiple values in app repository creates a single new value in other repository. And lot of other data is not required to be store in other repository and is only app specific. Apart from this the other repository will be synced with help of SG replicate once in a day (whenever the network is good) to some other repository on the cloud. So in summary the other repository will have data which will go to cloud , with less noise and very specific with different granularity.
As far as users are concerned we will have a customised authentication, With an in house authentication and authorisation service. but all users will be linked to Couch base mobile with different channels subscribed.
It means you suggest that we don’t use web hooks, normal pull methods or having CBL embedded in service. And create a daemon service which will work as a client to sync gateway without using CBL it looks a good idea and las_seq is the number till where the data is synced.
I would like to know more about “las_seq”. As far as storing this las_seq is concerned we can save it in file or cache, persisted to disk in a finite interval.
Do you have some link of a sample where this kind of functionality is implemented. It can be in java/C# or any language.
how many local documents can be created ? What is there life ? Are they kept in local RAM or persisted ?
Sorry for loads of question I am not from NoSQL back ground.
the typical use case of web hooks is
Sending notifications when specific documents change, for example, by email or SMS
Customized auditing or logging
you can send post/get request to _changes with since={last_seq} to get the change info since last change. FYI
local documents will be stored persisted. there should not hard limit except storage limit.
I am not sure if Document expiration (TTL) works on local documents.
notice that if you use _changes on public REST API ,and you disable guest users, you need send user’s Certificate to get the specific user’s change info.