I am trying out the new Java DCP client. When I want to stream data from 0 upto K where K is a sequence number that is less than current max sequence number M, the stream does not end at the sequence number K but continues beyond that. Is it that the stream always terminates at the end of a snapshot?
Hi Athalye. Your intuition is correct. I asked some experts and here’s how they responded:
If you specify an end seqno for a DCP stream you will receive at least up to that seqno. That means that the server may send you more items past that end seqno. This happens because the server must send a full snapshot and it may not have a full snapshot ending at exactly the end seqno you requested.
Also, at least in the ephemeral case, the backfill may have to continue past the end seqno to ensure you aren't missing items. For example, if items were written in the order (number being the seqno assigned) A1 B2 C3 B4 and you then requested up to seqno 3, we can't send A1 C3 then stop, because that state never really existed at any point in time (B having been updated, B2 is no longer available to send, so the range would be extended to seqno 4 to give you the new copy of B)