Strange sequence number and changes-feed issue

Mobile Sync Gateway
1

We are using Pouchdb to replicate against the sync gateway. I noticed that we have strange sequence numbers in our changes-feed that look like this "9941:9901". So its neither an integer nor a compond sequence number. Is this still a valid sequence number?

My pouchdb replication has a default batch_size of 100 and therefore calls the _changes endpoint several times.
Here are the requests made by pouchdb:

https:///_changes?style=all_docs&since=0&limit=100
https:///_changes?style=all_docs&since=9094&limit=100
https:///_changes?style=all_docs&since=9941%3A7767&limit=100
https:///_changes?style=all_docs&since=9941%3A8951&limit=100
https:///_changes?style=all_docs&since=9941%3A9051&limit=100
https:///_changes?style=all_docs&since=9941%3A9339&limit=100
https:///_changes?style=all_docs&since=9941%3A9632&limit=100
https:///_changes?style=all_docs&since=9941%3A9741&limit=100
https:///_changes?style=all_docs&since=9967&limit=100

After the replication is done there is one missing document that does not appear in any of the fetched changes. But the user has permission to see this document and I can confirm that the document appears in the _changes feed if I don’t set a limit.

Afterward I increased the batch_size to 1000 and suddenly the document was replicated by pouchdb.
Here are the requests made by pouchdb:

https:///_changes?style=all_docs&since=0&limit=1000
https:///_changes?style=all_docs&since=9967&limit=1000

Now all since-querie parameters are integers and the strange sequence numbers are skipped.
Any ideas whats going wrong?

Thanks

2

Sequences in the replication protocols (CouchDB or Couchbase Mobile) are arbitrary JSON values, not necessarily numbers. Sync Gateway sometimes uses strings to represent edge cases; my SG knowlege is way out of date, but IIRC it’s to cover a sequence when your access privileges changed due to an update of a document. (@adamf?)

1 Like
3

@Sebastian_Single That is in fact a compound sequence number, and in that instance looks like it’s related to an access grant, as Jens has indicated. In this specific case, it looks like the document w/ sequence 9941 granted new access to a channel, and the contents of that channel were backfilled to the user. There were more than 100 docs in that channel, so required more than one changes request with limit=100 to complete the backfill. Once the backfill is complete, SG can return to using non-compound sequences.

The limit value shouldn’t result in any missing documents, however. To work out what’s going on, I’d start by looking at the sequence of the document that’s missing, and issue a changes request w/ limit that you’d expect to return that sequence. If you’re able to identify two changes requests that should contain that sequence, that show the document being retrieved with limit=1000 and not with limit=100, that would be a bug.

1 Like
4

@adamf Thanks for your response.

There are 562 changes that belong to the 9941-channel. If I call the changes-route with a sequence number before 9941 I get this:

curl -X GET "http://localhost:4984/default/_changes?limit=564&since=9921" -H  "accept: application/json" -u <my-test-user:password>

{"results":[
{"seq":9922,"id":...}
,{"seq":"9941:7755","id":...}

....

,{"seq":"9941:9926","id":...}
,{"seq":9941,"id":...}
],
"last_seq":"9941"}

The change with “9941” is the document that doesn’t get replicated. My changes feed has 562 “9941:<>” changes and then the “9941”-change itself.

If I use the first compound sequence number “9941:7755” as since-parameter I get this:

curl -X GET "http://localhost:4984/default/_changes?limit=564&since=9941:7755" -H  "accept: application/json" -u <my-test-user:password>

{"results":[
{"seq":"9941:7756","id":...}
,{"seq":"9941:7757","id":...}

.... 

,{"seq":"9941:9926","id":...}
,{"seq":9967,"id":...}
,{"seq":9980,"id":...}
,{"seq":9989,"id":...}
],
"last_seq":"9989"}

The change “9941” is missing and should be between “9941:9926” and “9967”. It looks like the changes-endpoint has a bug If I use a compound sequence number as since-parameter. The workaround with batch-size 1000 gets all changes with a compound sequence number at once and therefore only calls the changes endpoint wit “normal” sequence numbers.

5

I have created a bug ticket on github - https://github.com/couchbase/sync_gateway/issues/4825