I’m trying to check for the existence of a document only on the master node (ReplicateTo.Zero), and without concern if it has been persisted to disk or not (PersistTo.Zero). Using Bucket.Exists works fine after an insert operation; however, fails after a remove operation (acting as though the document still exists). Source to repro -
using System;
using System.Collections.Generic;
using System.Threading;
using Couchbase;
using Couchbase.Configuration.Client;
namespace CouchbaseManagementReset
{
class Program
{
static void Main(string[] args)
{
var config = new ClientConfiguration
{
Servers = new List<Uri> { new Uri("http://localhost:8091/") },
BucketConfigs = new Dictionary<string, BucketConfiguration>
{
{ "test_1", new BucketConfiguration { BucketName = "test_1" } },
},
};
using (var cluster = new Cluster(config))
{
using (var bucket = cluster.OpenBucket("test_1"))
{
var key = Guid.NewGuid().ToString();
bucket.Insert(key, "{\"who\":\"Tommy\", \"what\":\"fails\"}", ReplicateTo.Zero);
if (!bucket.Exists(key))
{
Console.WriteLine($"1 - After Insert, expected {key} to exist.");
}
bucket.Remove(key);
if (bucket.Exists(key))
{
Console.WriteLine($"2 - After Remove, expected {key} to not exist.");
}
Thread.Sleep(1000); // Let's wait until the Remove operation persists to disk (PersistTo)
if (bucket.Exists(key))
{
Console.WriteLine($"3 - After Remove + Sleep, expected {key} to not exist.");
}
}
}
}
}
}
produces the output
2 - After Remove, expected d744c9f4-1ae4-4d3d-8f4b-cc663d8c9e5a to not exist.
Press any key to continue . . .
First off, is the behavior of Exists returning true even after a remove operation when the operation has not yet reached disk expected? Secondly, if so, is there an alternative way of checking for existence only on the master without concern for it having persisted to disk? I know Exists wraps Observe, but without knowing if it’s an insert or remove operation, and knowing the CAS if it was an insert operation, I wasn’t able to get my desired result using Observe.
@mhederi -
I need to dig a bit deeper into this to be sure, but I am guessing that Exist is finding an non-evicted replica and returning true. You could probably add a thread sleep after the call to remove for a few ms to confirm.
-Jeff
Hey Jeff, thanks for taking a look. I did a little deeper digging into the docs for Couchbase Server’s Observe operation and under the Response section, found the snippet
In the response, the keystate field is intended to be a bitmask for various states. For example:
0x00 = found, not persisted
0x01 = found, persisted
0x80 = not found
0x81 = logically deleted
…
It is important for client developers to understand the difference between not found and logically deleted. If a client does a set then not found and logically deleted mean mean the same thing from the clients perspective. For a set the first thing the client needs to know is whether or not an item has made it to the appropriate servers and in this case both not found and logically deleted mean that this hasn’t happened yet. In the case of a delete not found and logically deleted mean two different things. If you are returned not found for a delete then you can assume that the delete has been persisted on that node, but if you are returned logically deleted then it means that the delete has made it to the server, but has not yet been persisted.
Sure enough, putting a breakpoint inside of CouchbaseBucket.Exists and hitting it right after a Remove operation, I see that the KeyState is LogicalDeleted, not NotFound as asserted inside of Exists.
So, currently Exists means two different things depending on the context of checking after an Insert or a Remove operation. After an Insert, Exists simply means that it exists on the primary in ram, not necessarily that it’s been persisted to disk yet; however, after a Remove operation, Exists means that it neither exists in ram nor disk. I think to make the two consistant, Exists really should be updated to check for KeyState LogicalDeleted, i.e.
public bool Exists(string key)
{
var observe = new Observe(key, null, _transcoder, _operationLifespanTimeout);
var result = _requestExecuter.SendWithRetry(observe);
return result.Success && result.Value.KeyState != KeyState.LogicalDeleted;
}
1 Like
@mhederi,
Great research!
However, I would propose that exists should return false if KeyState == LogicalDeleted or KeyState == NotFound, rather than just LogicalDeleted.
Brant
1 Like
@mhederi -
Awesome job sleuthing! I created a [Jira ticket][1] to track this bug. If you feel inclined, go ahead and fix it (with integration tests) and push a pull request, otherwise I’ll pick it up later this week or early next for [v2.2.2][2].
@btburnett3 -
Yes, I think you are correct here: NotFound or LogicalDeleted means the key does not exist.
-Jeff
[1]: https://issues.couchbase.com/browse/NCBC-1020
[2]: https://issues.couchbase.com/projects/NCBC/versions/13003
@btburnett3 - You are correct! I had it in my head that KeyState was a flag enum for some reason.
@jmorris - Thanks for creating the ticket. I’ll be keeping my eye on github, I’m really interested in seeing how you write integration tests for this one!
Thanks both of you.
-Miller
@mhederi
I went ahead and made a fix with tests myself, in review here if you’re curious:
http://review.couchbase.org/#/c/56309/
-Brant
1 Like
@mhederi, @btburnett3 -
The patch has been merged with master and it’s available on github. It will be officially released as part of 2.2.2.
Thanks for the excellent work guys!
-Jeff
