Read What I Write

I am using couchbase strictly as a key/value store. When I perform a couchbaseBucket.get(key) soon after (~15-50ms) performing an insert couchbaseBucket.insert(doc, ReplicateTo.ONE) it returns null. If I perform the same query a few seconds later it successfully returns the item. How quickly should documents be available for retrieval by key?

My Couchbase cluster is running on AWS and was setup using terraform.

Statistics:

  • Nodes: 10 m5.xlarge (16gb mem, 4 vCPU).
  • 2,500 writes/second
  • 500 reads/second
  • Cluster CPU usage: ~30%
  • Cluster Mem usage: ~40%

Any time after the insert() is handled, the get() should return the item assuming we’re talking about a single cluster, and not one using XDCR, which would actually be multiple clusters.

Double check that you’re doing all of the error handling of the couchbaseBucket.insert(doc, ReplicateTo.ONE) and that you’re not accidentally using any async API that would let it happen out of order.

If you have a small code sample that reproduces this, you may want to crank the log level up to DEBUG to see the two operations flowing, ensure they’re hitting the same node, etc.

Thanks for the input. When you say the “insert() is handled” does handled mean the insert() call returns? Or does it get put on a queue and is “handled” at some later point?

I am using a single cluster (no XDCR), and as far as I know am using synchronous calls. The insert() and get() are happening on different threads, but the get() thread will not spawn until after the insert() call returns.

// Cluster creation
CouchbaseEnvironment couchbaseEnvironment = DefaultCouchbaseEnvironment.builder()
				.kvTimeout(30)
				.keyValueServiceConfig(KeyValueServiceConfig.create(1))
				.build();
couchbaseCluster = CouchbaseCluster.fromConnectionString(couchbaseEnvironment, url).authenticate(username, password);

// Bucket creation 
this.couchbaseBucket = couchbaseCluster.openBucket(couchbaseBucketName);

// Thread 1 - Insert
BinaryDocument doc = BinaryDocument.create(key, expirationSeconds, Unpooled.wrappedBuffer(dataBytes));
couchbaseBucket.insert(doc, ReplicateTo.ONE);

// Thread 2 (created after the above lines complete) - Retrieval
BinaryDocument doc = couchbaseBucket.get(key, BinaryDocument.class);

Errors should be handled correctly, but I will double check that now and attempt to get a smaller test working to see the DEBUG output

Yes, your code seems reasonable. As long as the get() is executed after the insert(), it should be there. My only suspicion is that there’s something happening with how your threads are executing. If there’s any chance they’ll execute in parallel, it’s possible for the get() to be processed first. In fact, it’s likely because it’s less work. The other thread has to go through an encoding step. before putting bits on the wire.

The DEBUG should show the order pretty clearly, so will look forward to the result.

You were correct! We are sending the keys back to an external partner, and they sending us request #2 (Thread 2 above) before request #1 (Thread 1 above).

Thanks for the update! I suspected there wasn’t a Couchbase problem here. This issue (and a missing TTL issue) is seemingly reported once every two years or so and it hasn’t yet been a defect. :slight_smile:

I totally understand how these things happen though, so I’m glad you were able to find it.