PLEASE HELP! Error in .NET client in atomic int ops

.net

#1

I am trying to Get<ulong> item from Couchbase server like this:

IOperationResult result = this._client.Get<ulong>(this.Key);

But it fails on BitConverter.ToUInt64().

Test Name: ValueTest
Test FullName: FistCore.UnitTest.MembaseBucketTest.ValueTest
Test Source: E:\SVN\FistCore\FistCore.Testing.UnitTest\Membase\MembaseAtomicIntTest.cs : line 163
Test Outcome: Failed
Test Duration: 0:00:00,0343377

Result StackTrace:
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.BitConverter.ToUInt64(Byte[] value, Int32 startIndex)
at Couchbase.IO.Converters.DefaultConverter.ToUInt64(Byte[] buffer, Int32 offset, Boolean useNbo)
at Couchbase.Core.Transcoders.DefaultTranscoder.Decode[T](Byte[] buffer, Int32 offset, Int32 length, OperationCode opcode)
at Couchbase.Core.Transcoders.DefaultTranscoder.Decode[T](Byte[] buffer, Int32 offset, Int32 length, Flags flags, OperationCode opcode)
at Couchbase.IO.Operations.OperationBase`1.GetValue()
— End of inner exception stack trace —
at FistCore.Extend.Membase.Object.Atomic.MembaseAtomicInt.Refresh(Boolean withTouch) in E:\SVN\FistCore\FistCore.Extend.Membase\Object\Atomic\MembaseAtomicInt.cs:line 294
at FistCore.UnitTest.MembaseBucketTest.ValueTest() in E:\SVN\FistCore\FistCore.Testing.UnitTest\Membase\MembaseAtomicIntTest.cs:line 172
Result Message:
Test method FistCore.UnitTest.MembaseBucketTest.ValueTest threw exception:
FistCore.Extend.Membase.MembaseException: Error occured in membase. Refresh method failed./r/nKey affected: testInt/r/nStore mode: None —> System.ArgumentException: Destination array is not long enough to copy all the items in the collection. Check array index and length.

Item was created by Increment operation and set to 222. I can see it as actual 222 number if I look into it from Couchbase Server UI. Is this a known bug in SDK?


#2

@d_fist

We you do a Get<ulong> operation, it is expecting to find an 8-byte integer stored in raw binary form. If you see it in the Couchbase UI, then it sounds like it was stored in string form. I’m not personally familiar with how the Increment operation stores in Couchbase, but it sounds like it may be as a string.

I would try using _client.Get and then parsing with ulong.Parse(). See if that works.

Alternatively, if you’re doing an Increment operation normally you would just get the value directly from that method. It returns the value to you as part of the atomic Increment operation. If you go back and read it again, it won’t be atomic. Another client may have updated it between the increment and the get.

Brant


#3

Ok I tried with string and it works. But now I have a problem with storing strings. They are automatically converted to their base64 representation?!?!?

result = this._client.Upsert<string>(“testString”, “test”);

With this I get this in Couchbase UI:

KEY: testString
VALUE: dGVzdA==

When I try to retrieve this document by _client.Get<string>(“testString”) it returns EMPTY value.

Am I the only one in entire .NET Couchbase community that actually uses key-value store instead of JSON dynamic values???


#4

EDIT:

Well I finally found out that Append and Prepend on strings don’t have the same behavior as Increment and Decrement in integers in version 2 of .NET SDK. Specifically Append and Prepend don’t return any values while Increment and Decrement DO return values.

After FORCED usage of serialization and deserialization (without the possibility to switch it OFF) in version 2+ my final thought is:

  • whoever designed this .NET SDK 2 (after much better 1+ version) should rethink their purpose in life :stuck_out_tongue: .

#5

This is incorrect. They are not converted to base64 by the client. They are converted to base64 by the Couchbase Web UI. There are some couchdb related legacy reasons for this and yes, it’s not the best behavior for strings. I’ve filed myself a few enhancements for the web UI.


#6

Isn’t that a little harsh? And it turns out it’s not based on facts.


#7

It is stored as a string. That mostly has to do with backward compatibility to the original memcached text protocol. @trond probably remembers the details. In the binary protocol, we were able to tighten up a lot of the details around incr/decr.


#8

You are correct, I am harsh. And I am sorry but do you know why? Let me tell you a little story how I started using Membase (yes it was called Membase back then when my ex company was one of it’s first clients):

I think it was around 2010 when we started searching for key-value store for our company AdServer. After a lot of debate about (not so many) available key-value stores, we had gone with Membase because of persistency feature. We were even featured as use case for you guys on your website. Anyway long story short, at some point in (I think it was 2012) one of our servers broke down for the first time in full production mode (we had 3 servers). You would think fortunately we have replication and Couchbase (then Membase) is fully fault tolerant database that will work normally. Well guess what, it didn’t work. We had about 1/3 data loss, well data was there (in replication) but the system was not getting to it. So after a lot of work and testing and so on and on we finally concluded that Couchbase (then Membase) doesn’t simply do what it is supposed to do. After another year or so digging we found out the problem lies somewhere in .NET client. And then this started:
https://issues.couchbase.com/browse/NCBC-84

The famous task of creating 1 simple function in that is about 90% of what Couchbase (then Membase) is/was…
You would be amazed how long it takes to create this GetFromReplica function. Apparently 3 years and major version later.

Well when I finally found out that it is finally fixed in version 2.1. we delayed the port of our framework untill now and after starting the port we found out:

  1. you can’t easily disable serialization/deserialization - was it so hard to leave simple Get ?
  2. Undocumented features (GetFromReplica says "Gets data from a " or something like that, its just missing half of the text)
  3. Inconsistent behaviour - for instance append and prepend vs increment and decrement
  4. Fix to broken connection pooling when recycling processes was to “flag” buckets with “disposed” (really???)
  5. But the main thing is the mentality of the devs - “I need to remove serialization/deserialization” - “Use Get<string>, we are pretty sure it will not do serialization/deserialization but you know it might…” and then off to github to debug their code.
  6. There needs to be low level access to Operaton objects that leaves the dev possible to bypass JSON features for speed

#9

Hello again @ingenthr,

We finally did the port from 1.3.4 to 2.4.2 and UnitTest is green across the board. I would like to ask just a couple more things - will this code ensure that item will be retrieved in case of node failure and/or timeouts (please notice the comments):

        IOperationResult<string> result = this._client.Get<string>(this.Key);

        bool isFromReplica = false;

        if (!result.Success)
        {
            if (result.ShouldRetry())
            {
                Thread.Sleep(10);  // SHOULD WE FORCE WAIT HERE? AND FOR HOW LONG?
                result = this._client.GetFromReplica<string>(this.Key); // SHOULD WE DO Get OR GetFromReplica HERE?
            }

            if (result.Status == Couchbase.IO.ResponseStatus.NodeUnavailable)
            {
                result = this._client.GetFromReplica<string>(this.Key); // IS THIS THE RIGHT PLACE TO GetFromReplica?
                isFromReplica = true;
            }

            if (!result.Success)
            {
                if (result.Exception != null && MembaseHelper.IsErrorSevere(result.Status))
                    throw new MembaseException(this.Key, StoreMode.None, "Error occured in membase. Refresh method failed.", result.Exception);

                return false;
            }
        }

Thank you for your help. And sorry about my outburst but I still think you should not force users to use serialization/deserialization - because, for instance, we use “late” serialization/deserialization when the object properties are actually accessed and/or changed to speed things up and fully use async pattern and now we need to use Get<string> which is weird (to say at least) and we are still not sure what happens in this case since there are some unknowns happening here https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Core/Transcoders/DefaultTranscoder.cs:

  1. How does GetFormat work on line 43?
  2. What does weird IF doing on line 167?
  3. Also does IF on line 223 basically answer my question on Get<string> ?
  4. If 3 is yes then what is the difference between DataFormat and TypeCode flags? When are they set?

Thanks again, and sorry again for my outburst.


#10

@d_fist

  • You can avoid serialization/deserialization by pushing byte arrays: bucket.Set<byte[]>(key, theBytes). Also, objects are stored as JSON, strings as strings, byte arrays (blobs) pass through serialization/deserialization

  • You should avoid using result.ShouldRetry() since it’s used internally by the SDK to determine if an operation should be retried. By the time you can call the method, its already done (or tried to do) its thing.

  • Checking for ResponseStatus.NodeUnavailable and then trying a replica read should work; you may also want to try on TransportFailure or ClientFailure, which may come before a NodeUnavailable.

  • GetFormat checks the flags (metadata about the value stored) to help in how to unpack the body. You can see that certain .NET types are stored as JSON, byte arrays as binary, and strings as strings. This allows the SDK’s to share buckets without corrupting data. See [here]
    (https://docs.google.com/document/d/1V653a6FF6DOqdT4d-fKIjGkHabDaNGZsvbtsUKJyeLc/edit) for details.

  • “Weird IF” ensures that we treat data stored using decr/incr with the correct byte order - which happens to be different that regular unsigned longs.

  • Its a mapping between .NET Types and Common Flag Types.

Note, if the transcoding or serialization do not fit your needs, you can always write your own implementation of each respective interface.

-Jeff


#11

Hi again,

Could you please elaborate this:

You should avoid using result.ShouldRetry() since it’s used internally by the SDK to determine if an operation should be retried. By the time you can call the method, its already done (or tried to do) its thing.

Does that mean that Couchbase client by default does “retries” after “busy” timeouts? Can you point me to the place where that is in client code on github? I am asking this because the old client did not have this behavior and Couchbase strongly suggested that developer should handle retries after “server busy” timeouts on operations.

Next:

“Weird IF” ensures that we treat data stored using decr/incr with the correct byte order - which happens to be different that regular unsigned longs.

Also when does this “regular unsigned long” happen? For instance if I have this code:

client.Upsert<ulong>(“someKey”, 2);
client.Increment(“someKey”,2,1);
client.Get<ulong>(“someKey”);

What will happen after the code is executed in this order?