Hey, thank you for your help.
Our use case is as follows:
We have a server that generates a sequence of events at a high rate.
The events are sent to others servers and asynchronously stored to DB.
We have to avoid under any circumstances that we ever emit different events with the same sequence number (or skip a sequence number).
To achieve this we use a lockfile that is generated on startup, and deleted on graceful server shutdown (it waits for all the in-flight events to finish being written to DB).
If the lockfile does not already exist on startup, we have to be 100% sure that the DB will either be able to deliver all and any of the previous events or give us an error (so we can fix the whole system through manual intervention).
If the lockfile does exist already on startup, we know that the DB might not contain all entries and we need manual intervention on the whole system.
On startup we retrieve the last event with REQUEST_PLUS to be sure the index picked it up as well.
This “100% or die” requirement is what I am trying to solve.
If couchbase’s automatic-failover is deactivated then storing the events on shutdown with persist_to=1 and waiting for them to all return with “success” before deleting the lockfile would be enough to fulfill the requirement, right?
What if automatic-failover is activated? Does persist_to=4 then fulfill the requirement (if the replica count is set to 3)?
And finally, if it is possible to fulfill the requirement, we could replace the lockfile with a “performed clean shutdown” entry in the databases instead, which is removed during startup before operation is resumed, so no system state information is stored on the servers’ local hdd.
Is there a way to guarantee that a XDCR setup has all the events persisted with the same “100% or die” requirement before the “performed clean shutdown” entry is replicated to the other datacenter?