How do changes REST API feed options work?

I am trying to write an NodeJS agent for Sync Gateway that listens for changes, but I don’t understand how to use the feed options: normal, continuous, longpoll, websocket.

I get how I could use a timer and the normal option and do a polling, but don’t see how the other options work. Does anyone have a good example or even just a more complete description of how continuous, longpoll, or websocket works?

hey @sweetiewill can you help out here please

Hey @chrismbeckett,

The timer with normal option works and can look into examples on the topic but regarding a more complete description of the feed options:

normal: Immediately the ‘_changes’ endpoint will return with the results
longpoll: Server will have the client HTTP request stay open for a duration and have new results returned to the client.
continuous: This is similar to the ‘longpoll’ feed option but in that the socket is remained opened and never closed.

Curious what are you using the NodeJS agent for other than listening for changes.

I am implementing dynamic permissions into my application. I have a group document that needs to become an actual sync gateway role. I added a provisioning property to my group document, and I need a server side agent to watch for group documents, and then create the role on the admin REST API when a new group document is created.

I found this explanation of longpoll and continuous this morning (http://guide.couchdb.org/draft/notifications.html). It basically says don’t use longpoll, but I was thinking that for a server-side agent continuous or websocket might be the best choice, but am concerned about issues with the connection closing etc.

Is normal with a timer and polling just the best practical/recommended option? Seems the most straight-forward and least likely to break in production.

It sounds like a good use case for using a webhook (released in Sync Gateway 1.1).
You can find examples of how to use webhooks in this blog post and roles in this guide.

James

1 Like

Hey @jamiltz, thanks for this suggestion and the links. Web hooks look great and seem a much better solution than polling.

1 Like

I tried to use the _changes endpoint with the feed option set to ‘websocket’ and with a real WebSocket. But it didn’t seem to work very well.

I managed to get a successful Upgrade handshake. (See the log below)

2021-02-03T15:22:05.126Z [INF] HTTP: #065: GET /health/_changes?filter=sync_gateway/bychannel&channels=someone@example.com&feed=websocket&since=28 (as ADMIN)
2021-02-03T15:22:05.126Z [INF] HTTP+: #065: → 101 Upgraded to WebSocket protocol (0.0 ms)

But after that, the web socket did not seem to return anything and stayed idle, even after I had updated a document Do I miss something here ? Do I need to set up a sub protocol ?

Incorrect post - see below for detail.

I would try adding &continuous=true - which tells the connection to stay open, even after “catching up” with the latest changes.

Edit: Further investigation reveals query parameters do not work for websocket changes feeds - see my next comment in this thread.

I’ve just tried the continuous flag you suggested. But it didn’t work either.

But thanks for your help anyway. I really appreciated it.

Ah, I remembered why you’re seeing this behaviour.

The Websocket API takes in its options as an initial message from the client, rather than via the query parameters, so actually what you need to do is open the connection, and then send an options object to start the feed. In the simplest case this can just be {}, but for channels, since values, etc. you can follow this struct:

var input struct {
		Feed           string        `json:"feed"`
		Since          db.SequenceID `json:"since"`
		Limit          int           `json:"limit"`
		Style          string        `json:"style"`
		IncludeDocs    bool          `json:"include_docs"`
		Filter         string        `json:"filter"`
		Channels       string        `json:"channels"` // a filter query param, so it has to be a string
		DocIds         []string      `json:"doc_ids"`
		HeartbeatMs    *uint64       `json:"heartbeat"`
		TimeoutMs      *uint64       `json:"timeout"`
		AcceptEncoding string        `json:"accept_encoding"`
		ActiveOnly     bool          `json:"active_only"` // Return active revisions only
	}

Working example using wsd (like curl for ws) to drive the websocket connection where I connect, create doc1, and doc2, then send {} to start the feed, and then create doc3 and doc4.

$ wsd -url 'ws://127.0.0.1:4985/db1/_changes?feed=websocket'
connecting to ws://127.0.0.1:4985/db1/_changes?feed=websocket from http://localhost/...
successfully connected to ws://127.0.0.1:4985/db1/_changes?feed=websocket

> {}
< []"seq":2,"id":"doc1","changes":[{"rev":"1-cd809becc169215072fd567eebd8b8de"}]},{"seq":3,"id":"doc2","changes":[{"rev":"1-cd809becc169215072fd567eebd8b8de"}]}]
< []
< [{"seq":4,"id":"doc3","changes":[{"rev":"1-cd809becc169215072fd567eebd8b8de"}]}]
< [{"seq":5,"id":"doc4","changes":[{"rev":"1-cd809becc169215072fd567eebd8b8de"}]}]
> ^C
$ curl -X PUT http://localhost:4985/db1/doc1 -H 'Content-Type: application/json' -d '{"foo":"bar"}'
{"id":"doc1","ok":true,"rev":"1-cd809becc169215072fd567eebd8b8de"}
$ curl -X PUT http://localhost:4985/db1/doc2 -H 'Content-Type: application/json' -d '{"foo":"bar"}'
{"id":"doc2","ok":true,"rev":"1-cd809becc169215072fd567eebd8b8de"}

# sending {} via wsd here

$ curl -X PUT http://localhost:4985/db1/doc3 -H 'Content-Type: application/json' -d '{"foo":"bar"}'
{"id":"doc3","ok":true,"rev":"1-cd809becc169215072fd567eebd8b8de"}
$ curl -X PUT http://localhost:4985/db1/doc4 -H 'Content-Type: application/json' -d '{"foo":"bar"}'
{"id":"doc4","ok":true,"rev":"1-cd809becc169215072fd567eebd8b8de"}
2021-02-04T18:43:10.870Z [INF] HTTP:  #001: GET /db1/_changes?feed=websocket (as ADMIN)
2021-02-04T18:43:10.870Z [INF] HTTP+: #001:     --> 101 Upgraded to WebSocket protocol  (0.0 ms)
2021-02-04T18:43:15.372Z [INF] HTTP:  #002: PUT /db1/doc1 (as ADMIN)
2021-02-04T18:43:15.373Z [INF] HTTP+: #002:     --> 201   (0.4 ms)
2021-02-04T18:43:18.760Z [INF] HTTP:  #003: PUT /db1/doc2 (as ADMIN)
2021-02-04T18:43:18.760Z [INF] HTTP+: #003:     --> 201   (0.2 ms)
2021-02-04T18:43:22.054Z [INF] Changes: c:#001 MultiChangesFeed(channels: {*}, options: {Since: 0, Limit: 0, Conflicts: false, IncludeDocs: false, Wait: true, Continuous: true, HeartbeatMs: 0, TimeoutMs: 300000, ActiveOnly: false}) ...
2021-02-04T18:43:22.055Z [INF] Cache:   Querying 'channels' for "*" (start=#1, end=#4, limit=5000)
2021/02/04 18:43:22 SG-Bucket: 	... view returned 2 rows
2021-02-04T18:43:22.068Z [INF] Cache:     Got 2 rows from query for "*": #2 ... #3
2021-02-04T18:43:22.068Z [INF] Cache:   Initialized cache of "*" with 2 entries from query (#2--#3)
2021-02-04T18:43:22.068Z [INF] Cache: GetChangesInChannel("*") --> 2 rows
2021-02-04T18:43:26.985Z [INF] HTTP:  #004: PUT /db1/doc3 (as ADMIN)
2021-02-04T18:43:26.985Z [INF] HTTP+: #004:     --> 201   (0.2 ms)
2021-02-04T18:43:26.985Z [INF] Cache: GetCachedChanges("*", 3) --> 1 changes valid from #4
2021-02-04T18:43:29.164Z [INF] HTTP:  #005: PUT /db1/doc4 (as ADMIN)
2021-02-04T18:43:29.164Z [INF] HTTP+: #005:     --> 201   (0.2 ms)
2021-02-04T18:43:29.164Z [INF] Cache: GetCachedChanges("*", 4) --> 1 changes valid from #5

After the parameters were sent as a “stringified” JSON object via the WebSocket, everything seemed to be working great !.

Thank YOU very much for your help!