I’m finding it hard to find good information about Couchbase’s guarantees. Specifically tuning them. I’m trying to find out:
• Are writes acknowledged after written to disk or only after written to memory?
• Can writes be configured to be acknowledged only after safely committed to disk?
• Does sync_gateway acknowledge writes only after safely committed to disk?
I know there is some tradeoffs on performance but we have a low-write high-read workload and also sensitive data that we would highly prefer to not be acknowledged until safely committed to disk.
in general couchbase acknowledges write operations as successful once they have been written to RAM on the primary node (partition). Persistence and replication is handled asynchronously which means that with this default setting you could be subject to data loss if a node fails before the data is replicated and/or persisted depending on the setup and actions afterwards.
Of course we have a solution for that, called durability requirements. From a server SDK you can define replicate and persist durability requirements that, once the initial operation succeeded, will make the client poll the affected nodes and ask for the state of the mutation. If your requirements can be fulfilled you’ll get success, otherwise a failure and then its up to you to take appropriate action (since we don’t know what you need to do in this case).
How sync gateway exposes/handles that I don’t know to be honest, but I’m sure someone from the mobile team like @jens or @sweetiewill do.
It’s much too expensive (an order of magnitude slower, IIRC) to commit each individual write to disk. There should be a facility to flush writes when a replication saves a checkpoint; I know CouchDB’s replicator does this, but I can’t recall if we support that in SG. @adamf, do you know?
(FYI, William aka sweetiewill isn’t at Couchbase any more. Our current mobile dev advocate is @hod.greeley.)
No, Sync Gateway doesn’t currently have a way to force persistence on a write (primarily for the performance reason @jens stated), and there’s no flush-writes-on-checkpoint.
I don’t know of a bucket-level write persist operation (that SG could use at checkpoint time) - particularly since when the checkpoint is sent, SG doesn’t know the set of documents pushed during that replication.
It would probably be a relatively straightforward enhancement to set the write options globally via the Sync Gateway config - a bit more complex to handle on individual write and bulk write operations.
Didn’t SG originally use a wait-until-persisted flag when doing a document update? And we took that out because it was so slow/expensive for the server?
FYI, the CouchDB facility is something like POST /_ensure_full_commit. I think I even hooked up a no-op handler for it in SG long ago, because the CouchDB replicator sends it after it saves the checkpoint, and if it gets a 404 status back the replicator treats that as an error.
SG used to do something similar for user document updates - it set a wait-until-indexed flag, to ensure the user document changes were visible to a subsequent stale=false view query.
However, this is no longer needed in Couchbase Server 3.0 and higher, due to a change in how stale=false works.
I don’t recall a time when we set the wait-until-persist flag was used by Sync Gateway, though.
So I guess what I’m trying to sort out is if we used sync_gateways REST API for creating documents it will acknowledge the write before it has been fully and safely committed to disk?
CouchDB never acknowledges a write until it’s saved to disk (unless you configure it otherwise), while the replication protocol’s checkpoints are saved after that (so some replication may replay in the event of a failure).
What we’re really trying to evaluate is the pitfalls of using Couchbase + Sync Gateway and it sounds like unless you’re using the replication protocol you can lose writes due to the delay between write ack’s and disk flushes.
As a not-a-server guy I’m somewhat out of my depth here, but let me point out that Couchbase Server is clustered, with redundant writes to multiple nodes and transparent failover, so writing to disk isn’t as urgent as it is with a single-node system like CouchDB. (Unless the power to your entire data center goes down, of course.)
@robertjpayne I want to clarify further here. I’m not sure, but it seems like you might be confusing Sync Gateway persisting to disk (which it is not intended to do) with Couchbase Server.
Sync Gateway is a stateless glue piece that manages replications. Some of the discussion in the thread is about persisting SG checkpoints. These are used to make replications more efficient, but I believe there’s no loss if they don’t happen (@adamf or @jens can you confirm?).
So Couchbase Server has a mechanism for ensuring you know you’ve written to disk, but it sounds like it’s not currently controllable from Sync Gateway. Jens’ last point was that this isn’t typically as much of an issue as it sounds because not only do you have writes to disk, you have redundancy in having a write also getting pushed to your backup nodes in a cluster.
If you’re sure you need a guarantee of a write to disk after you create a document via Sync Gateway, it sounds like this would be an enhancement request we’d have to look at (again, @adamf or @jens if you can confirm). Of course there are legitimate situations where that’s needed, but most don’t. (Meaning I’m not intending to be rude in questioning whether you absolutely need the guarantee, just that sometimes people realize it’s not a real requirement.)