Need to get a timestamp from a TAP feed event to pass in TAP BackFill argument

#1

In Sync Gateway we use TAP feed to receive events from the underlying Couchbase Server bucket.

If the Sync Gateway connection to the TAP feed is lost, when the connection is re-established it is possible that some events will have been lost.

When the connection is lost we want to start a new TAP feed that includes all events from the point where the connection was lost.

In the arguments passed when requesting a new TAP feed, there is a ‘Backfill’ parameter

BACKFILL ( =0x01=) contains a single 64-bit body that represents the oldest entry (from epoch) you're interested in. Specifying a time in the future (for the server you are connecting to), will cause it to start streaming only current changes.

What we need is a way to get a timestamp (CBS clock) from a TAP feed event to use when requesting a new TAP feed after a lost connection.

We are thinking that the most appropriate event to use would be the most recently received ‘TAP_CHECKPOINT_END’.

Is this currently possible?

#2

Unfortunately, I don’t think you can get what you’re looking for @andy. Note that while this was built in to the TAP protocol, part of that protocol is that it is allowed to send you duplicate items. In the current (and only) implementation, if you send any timestamp in the past, it will send you all of the data.

The original plan was to later extend it.

Plans change however, and that was taken a different direction in Database Change Protocol (DCP). As you probably already know, DCP gives you the ability to pick up from a given checkpoint and that’s the best way to handle this.