Use custom filters for changes feed of sync gateway


#1

Hello,

is there a way to use custom filters for the changes feed of sync gateway? As I’ll explain below, I would need a filter that works exactly like sync_gateway/by_channel, but the other way round: I want to sync all documents except those belonging to a single channel.

I need this because I am writing an app that synchronizes using PouchDB and the sync gateway to a couchbase server, where I have two kinds of documents:

  1. Documents that should be synced (about 100). These documents are in many different channels that change often.
  2. Documents that should not be synced (a few 10000). These documents should be accessible on-demand, however. They all belong to a single channel ‘doNotSync’.

My first approach to solve this was to use the sync_gateway/by_channel function to sync only documents that belong to channels I’m interested in. The documents of type 2) were fetched on-demand using HTTP GET requests to http://sync_gateway/bucket/documentID. However, this way the application misses some document updates because the channels change often, as you can see in this example:

  1. The Application has access to the channels ‘user1’ and ‘user2’, but listens only to _changes filtered by channel ‘user1’
  2. ‘user2’ publishes something in channel ‘user2’
  3. the application now listens also to channel ‘user2’
  4. the document of 2) does not reach the application

Therefore my idea was to sync all documents except those belonging to the channel ‘doNotSync’. In the documentation for the _changes feed I found this option:

filter: “Reference to a filter function from a design document that will filter whole stream emitting only filtered events.”

How can I create a design document with a filter function that I can call? I tried to create a ddoc _design/bar with the following content
{"filters":{"foo":"function() { ... }"}}
but accessing http://localhost:4985/data/_changes?filter=bla/foo returns “unknown filter”.

Thank you in advance!


#2

The only changes filters currently supported by Sync Gateway are sync_gateway/bychannel and _doc_ids (as documented here: http://developer.couchbase.com/documentation/mobile/1.2/develop/references/sync-gateway/rest-api/database-public/get-changes/index.html). I think you may have been looking at the Couchbase Lite REST API for the filter function case.

There isn’t currently support for an ‘exclude channel X’ filter, but I think that would be a reasonable enhancement of the `sync_gateway/bychannel’ filter (so that you could specify something like channels=["*","- doNotSync"] as your set of channels. That approach shouldn’t have significant performance overhead. I’ve filed an ER for that here: https://github.com/couchbase/sync_gateway/issues/1960 - please comment with additional specifics of your use case if you like.


#3

Hello,

thank you for your response and for filing an ER! Having an exclude filter would be great (if it is possible to implement without performance degradation). Meanwhile I solved it by creating two couchbase sync gateway users for each real user, one for syncing, one for downloading stuff.